$ Discover the best projects built with vLLM

LvLLM is a special NUMA extension of vllm that makes full use of CPU and memory resources, reduces GPU memory requirements, and features an efficient GPU parallel and NUMA parallel architecture, supporting hybrid inference for MOE large models. Maintained by guqiong96 on GitHub, where it has earned 401 stars from the community.

Python package for LLM compression. Maintained by FujitsuResearch on GitHub, where it has earned 398 stars from the community.

Low latency JSON generation using LLMs. Maintained by varunshenoy on GitHub, where it has earned 396 stars from the community.

Setup and run a local LLM and Chatbot using consumer grade hardware. Maintained by jasonacox on GitHub, where it has earned 346 stars from the community.

Versatile Almost Local, Eventually Reasonable Assistant. Maintained by vakovalskii on GitHub, where it has earned 331 stars from the community.

Persist and reuse KV Cache to speedup your LLM. Maintained by ModelEngine-Group on GitHub, where it has earned 312 stars from the community.

Easy, advanced inference platform for large language models on Kubernetes. Star to support our work!. Maintained by InftyAI on GitHub, where it has earned 308 stars from the community.

Implementation for FP8/INT8 Rollout for RL training without performence drop. Maintained by yaof20 on GitHub, where it has earned 308 stars from the community.

Qwen3.5-122B-A10B on DGX Spark: 28.3 → 51 tok/s (+80%). Maintained by albond on GitHub, where it has earned 306 stars from the community.

AI-powered offensive security testing using autonomous agents, directly in your terminal. Maintained by pensarai on GitHub, where it has earned 299 stars from the community.

Blazing-fast LLM inference in pure Rust. No PyTorch and Python runtime. Maintained by guoqingbao on GitHub, where it has earned 298 stars from the community.

ML software (llama.cpp, ComfyUI, vLLM) builds for AMD gfx906 GPUs, e.g. Radeon VII / MI50 / MI60. Maintained by mixa3607 on GitHub, where it has earned 296 stars from the community.

The production framework for Predictive and Generative AI. Serve any model as an API in one line, with OpenAI/Anthropic/Ollama-compatible endpoints, a built-in chat UI, and native MCP. Maintained by vortico on GitHub, where it has earned 296 stars from the community.

A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization, speculative decoding, benchmarking. Maintained by patchy631 on GitHub, where it has earned 296 stars from the community.

CacheRoute is an innovative LLM scheduling scheme dedicated to enabling flexible KV cache reuse across LLM systems, improving task performance and system efficiency. Maintained by AstraNetLab on GitHub, where it has earned 293 stars from the community.

High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends. Maintained by thushan on GitHub, where it has earned 273 stars from the community.

LLM ，。. Maintained by lework on GitHub, where it has earned 271 stars from the community.

[ICML-2026] Official implementation of "SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience". Maintained by SunzeY on GitHub, where it has earned 259 stars from the community.

An Open-source, self-hosted AI model hub with Hugging Face compatibility, accelerating vLLM/SGLang performance. Maintained by matrixhub-ai on GitHub, where it has earned 258 stars from the community.

A CPU Realtime VLM in 500M. Surpassed Moondream2 and SmolVLM. Training from scratch with ease. Maintained by lucasjinreal on GitHub, where it has earned 256 stars from the community.

Home Assistant LLM integration for local OpenAI-compatible services (llamacpp, vllm, etc). Maintained by skye-harris on GitHub, where it has earned 256 stars from the community.

gpt_serverLLMs、Embedding、Reranker、ASR、TTS、、。. Maintained by shell-nlp on GitHub, where it has earned 253 stars from the community.

One-click Qwen3.6-27B inference on Windows. 158 tok/s on RTX 5090, 72 tok/s on RTX 3090. Native, no WSL, no Docker, no telemetry. Maintained by devnen on GitHub, where it has earned 227 stars from the community.

A PyTorch native library for training speculative decoding models. Maintained by lightseekorg on GitHub, where it has earned 219 stars from the community.

a fun and educational take on vLLM. Maintained by ovshake on GitHub, where it has earned 215 stars from the community.

Simple LLM service identification - translate IP:Port to Ollama, vLLM, LiteLLM, or 60+ other AI services in seconds. Maintained by praetorian-inc on GitHub, where it has earned 204 stars from the community.

A general-purpose API load testing platform that supports LLM services and business HTTP interfaces, enabling one-click performance testing, result comparison, and AI-powered intelligent analysis and summarization. API ， HTTP ，、 AI. Maintained by MigoXLab on GitHub, where it has earned 201 stars from the community.

vLLM Documentation in Chinese Simplified / vLLM. Maintained by hyperai on GitHub, where it has earned 190 stars from the community.

Open-source tools for training and evaluating Vision Language Models for OCR. Maintained by Roots-Automation on GitHub, where it has earned 190 stars from the community.

Fully-featured, beautiful web interface for vLLM - built with NextJS. Maintained by yoziru on GitHub, where it has earned 188 stars from the community.

Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed. Maintained by defilantech on GitHub, where it has earned 185 stars from the community.

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang. Maintained by novitalabs on GitHub, where it has earned 184 stars from the community.

[ICCV2025] Referring any person or objects given a natural language description. Code base for RexSeek and HumanRef Benchmark. Maintained by IDEA-Research on GitHub, where it has earned 184 stars from the community.

Booster - open accelerator for LLM models. Better inference and debugging for AI hackers. Maintained by gotzmann on GitHub, where it has earned 169 stars from the community.

Deep Learning Deployment Framework: Supports tf/torch/trt/trtllm/vllm and other NN frameworks. Support dynamic batching, and streaming modes. It is dual-language compatible with Python and C++, offering scalability, extensibility, and high performance. It helps users quickly deploy models and provide services through HTTP/RPC interfaces. Maintained by NetEase-Media on GitHub, where it has earned 169 stars from the community.

Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows. Maintained by InternScience on GitHub, where it has earned 167 stars from the community.

Automated system for LLM evaluation via agents. Doc as below:. Maintained by OpenDCAI on GitHub, where it has earned 160 stars from the community.

Cognithor · Agent OS: Local-first autonomous agent operating system. 19 LLM providers, 18 channels, 145 MCP tools, 6-tier memory, Agent Packs marketplace, zero telemetry. Python 3.12+, Apache 2.0. Maintained by Alex8791-cyber on GitHub, where it has earned 153 stars from the community.

GLM-5.2-NVFP4-REAP-469B serving on SM120 (4× RTX PRO 6000 Blackwell) — one-command vLLM launch recipe, 250K context, DeepSeek Sparse Attention + MTP speculative decode. Maintained by 0xSero on GitHub, where it has earned 152 stars from the community.

Never give AI companies your secrets! A local LLM-based privacy filter for LLM users. Seamless integration with your existing AI tools as a Python library / OpenAI SDK replacement / API Gatetway / Web Server. Maintained by cxumol on GitHub, where it has earned 148 stars from the community.

DocMind AI is a powerful, open-source Streamlit application leveraging LlamaIndex, LangGraph, and local Large Language Models (LLMs) via Ollama, LMStudio, llama.cpp, or vLLM for advanced document analysis. Analyze, summarize, and extract insights from a wide array of file formats, securely and privately, all offline. Maintained by BjornMelin on GitHub, where it has earned 147 stars from the community.

vLLM-5090: Docker Container for RTX 5090 + OpenCode. Maintained by BoltzmannEntropy on GitHub, where it has earned 142 stars from the community.

Unified management and routing for llama.cpp, MLX and vLLM models with web dashboard. Maintained by lordmathis on GitHub, where it has earned 139 stars from the community.

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI. Maintained by Sandermage on GitHub, where it has earned 131 stars from the community.

Privacy-first LLM proxy and AI gateway - load balancing, multi-provider routing, API key management, usage tracking, rate limiting. Self-hosted. Zero knowledge of your prompts. Maintained by voidmind-io on GitHub, where it has earned 125 stars from the community.

A project worth exploring. Maintained by GaeaRuiW on GitHub, where it has earned 123 stars from the community.

Your AI intranet: network the computers you already own for inference and training. Maintained by autonomous-ai on GitHub, where it has earned 121 stars from the community.

A High-Performance LLM Inference Engine with vLLM-Style Continuous Batching. Maintained by lumia431 on GitHub, where it has earned 120 stars from the community.

KEDA External gRPC Scaler for GPU workloads - native NVML metrics via DaemonSet, no Prometheus required. Maintained by pmady on GitHub, where it has earned 113 stars from the community.

Convert PowerPoint files into semantically rich text using vision language models. Maintained by ALucek on GitHub, where it has earned 113 stars from the community.

Data Center and Client workload and software optimizations for Intel hardware. Maintained by intel on GitHub, where it has earned 107 stars from the community.

Efficient LLM inference on Slurm clusters. Maintained by VectorInstitute on GitHub, where it has earned 106 stars from the community.

One-command vLLM installation for NVIDIA DGX Spark with Blackwell GB10 GPUs (sm_121 architecture). Maintained by eelbaz on GitHub, where it has earned 105 stars from the community.

Local diagnostic CLI for NVIDIA DGX Spark (GB10). Detects power caps, UMA pressure, thermal risk, CUDA 13/SM_121 wheel mismatches, Docker issues, and validates vLLM/Ollama/llama.cpp/SGLang recipes. Maintained by joeynyc on GitHub, where it has earned 101 stars from the community.

Kubernetes-native platform for deploying and managing AI inference across multiple providers. Maintained by ai-runway on GitHub, where it has earned 96 stars from the community.

Practical local LLM recipes and benchmarks for RTX 5060 Ti setups. Maintained by 5p00kyy on GitHub, where it has earned 96 stars from the community.

Nano vLLM with vLLM v1's request scheduling strategy and chunked prefill. Maintained by slwang-ustc on GitHub, where it has earned 95 stars from the community.

llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deployment, such as UI, RESTful API, auto-scaling, computing resource management, monitoring, and more. Maintained by OpenCSGs on GitHub, where it has earned 95 stars from the community.

FlowSteer: agents designing agentic workflows via reinforced progressive canvas editing. Maintained by beita6969 on GitHub, where it has earned 94 stars from the community.

Extensible generative AI platform on Kubernetes with OpenAI-compatible APIs. Maintained by llmariner on GitHub, where it has earned 94 stars from the community.