llmariner by llmariner

Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed. Maintained by defilantech on GitHub, where it has earned 185 stars from the community.

Arks is a cloud-native inference framework running on Kubernetes. Maintained by scitix on GitHub, where it has earned 50 stars from the community.

Easy, advanced inference platform for large language models on Kubernetes. Star to support our work!. Maintained by InftyAI on GitHub, where it has earned 308 stars from the community.