kserve by kserve

A programmable Mixture-of-Models router for heterogeneous LLM inference. Maintained by vllm-project on GitHub, where it has earned 5,116 stars from the community.

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton. Maintained by ome-projects on GitHub, where it has earned 485 stars from the community.

AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text. Maintained by kubeai-project on GitHub, where it has earned 1,238 stars from the community.