$ Discover the best projects built with vLLM

Native Windows vLLM 0.26.0: CPython 3.13, CUDA 12.8, SM 7.5/8.6/8.9/12.0 for RTX 20/30/40/50, OpenAI-compatible serving, Triton/FlashAttention, 10 KV-cache formats, Multi-TurboQuant, and experimental CPU/RAM/NVMe prompt-KV offload. No WSL or Docker. Maintained by aivrar on GitHub, where it has earned 36 stars from the community.

GLM-5.2 (744B/40B MoE) on a 4× DGX Spark / GB10 (sm_121) cluster: portable Triton sparse-MLA kernels, a data-free expert prune, MTP draft, and a one-script bootstrap. Maintained by CosmicRaisins on GitHub, where it has earned 36 stars from the community.

Two-node DGX Spark/ASUS GX10 DeepSeek V4 Flash DSpark NVFP4 port for vLLM 0.25, with live dashboard and reproducible deployment. Maintained by Anemll on GitHub, where it has earned 36 stars from the community.

From-scratch C++/CUDA inference engine for the NVIDIA RTX 5090 (sm_120a) — the best single-GPU backend for agentic AI: tool calling, long-context loops, reasoning and concurrent sub-agents on top of the fastest single-stream decode on the 5090 (beats llama.cpp, at-or-ahead of vLLM on NVFP4). 100% written by Claude Code. Maintained by kekzl on GitHub, where it has earned 35 stars from the community.

An 18 notebook course that isolates and measures each component of agentic loop engineering on real, industry standard software datasets. Maintained by FareedKhan-dev on GitHub, where it has earned 34 stars from the community.

Benchmarking Open-Ended Inference Optimization by AI Agents. Maintained by aisa-group on GitHub, where it has earned 34 stars from the community.

AIfred-Intelligence — self-hosted Multi-Agent Assistant with Debate Modes (Symposion/Tribunal), Voice (STT + Streaming-TTS), RAG with Long-Term Memory, Web Research and Tool-Calling. Reachable via Web-UI or Telegram/Discord/Email/EPIM. Multi-Backend: llama-swap, Ollama, vLLM, TabbyAPI. Maintained by Peuqui on GitHub, where it has earned 33 stars from the community.

Lightweight proxy for LLM. Maintained by yatesdr on GitHub, where it has earned 33 stars from the community.

vllm. Maintained by gameofdimension on GitHub, where it has earned 31 stars from the community.

Serve Llama 3.3 70B (with AWQ quantization) using vLLM and deploy it on BentoCloud. Maintained by kingabzpro on GitHub, where it has earned 31 stars from the community.

Powerful no-code LLM fine-tuner: upload data → train → deploy in minutes. Unsloth 2-5× acceleration · QLoRA/DPO/RLHF/PPO/ORPO · Reward Model training · GGUF export · vLLM inference · BLEU/ROUGE/BERTScore · full CLI · Heretic Mode to unlock full model potential. Maintained by Yog-Sotho on GitHub, where it has earned 30 stars from the community.

AI Coding Assistant CLI for offline enterprise environments - Local LLM platform with Plan & Execute architecture, Supervised Mode, and auto-update system. Maintained by HanSyngha on GitHub, where it has earned 30 stars from the community.

A REST API for vLLM, production ready. Maintained by France-Travail on GitHub, where it has earned 29 stars from the community.

Experimental RAG playground for exploring retrieval quality, corpus construction, and filter-chain design. Features configurable ranking and filtering pipelines, visual document grounding, chat interfaces, web search, Open WebUI integration, and rich debugging insights. Ollama and vLLM, running in Dev Containers or natively on Linux and Windows. Maintained by HarinezumIgel on GitHub, where it has earned 29 stars from the community.

[CVPR2025] Official Repository for IMMUNE: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment. Maintained by itsvaibhav01 on GitHub, where it has earned 29 stars from the community.

Policy-driven seamless lazy loading. Maintained by cloudpilot-ai on GitHub, where it has earned 29 stars from the community.

a simple lightweight large language model pipeline framework. Maintained by sherlockchou86 on GitHub, where it has earned 27 stars from the community.

A web-based memory usage and performance calculator for Huggingface GGUF models. Maintained by gdevenyi on GitHub, where it has earned 27 stars from the community.

Dockerized LLM inference server with constrained output (JSON mode), built on top of vLLM and outlines. Faster, cheaper and without rate limits. Compare the quality and latency to your current LLM API provider. Maintained by phospho-app on GitHub, where it has earned 27 stars from the community.

An open-source, AI-driven Social Media Digital Twin powered by LLMs (Ollama, vLLM) for Computational Social Science simulations, network analysis, and human-agent interaction. Maintained by YSocialTwin on GitHub, where it has earned 27 stars from the community.

A hybrid router that uses Spot GPU instances to reduce costs and Serverless GPUs for making Cold Starts faster. Maintained by Tandemn-Labs on GitHub, where it has earned 27 stars from the community.

Serve GLM-5.2 469B (REAP-pruned, NVFP4) across 3× NVIDIA DGX Spark with vLLM pipeline parallelism — 256K context, production-ready config and patches. Maintained by bird on GitHub, where it has earned 27 stars from the community.

Official implementation for Text Generation Beyond Discrete Token Sampling. Maintained by EvanZhuang on GitHub, where it has earned 26 stars from the community.

Self-hosted meeting transcription portal — speech-to-text, speaker diarization, LLM-corrected transcripts, structured summaries and Word minutes, on your own GPUs. Flask + PostgreSQL, GDPR audit trail, distributed GPU topologies, docker. Maintained by Martossien on GitHub, where it has earned 26 stars from the community.

Orpheus TTS Server with streaming support (TTFB ~160ms). Maintained by taresh18 on GitHub, where it has earned 26 stars from the community.

A "standard library" of Triton kernels. Maintained by stackav-oss on GitHub, where it has earned 26 stars from the community.

Dual-engine (llama.cpp + vLLM) LLM benchmarking pipeline for GGUF & safetensors on NVIDIA GPUs — speed, quality, live dashboard, publishable cards. Maintained by notwitcheer on GitHub, where it has earned 26 stars from the community.

Patched native-Windows build of vLLM. Three Windows-specific fixes (CPU-relay for Gloo, Qwen3 reasoning parser, wildcard model name) on top of SystemPanic 0.19.0. Engine for devnen/qwen3.6-windows-server. Maintained by devnen on GitHub, where it has earned 25 stars from the community.

Python SDK for Agent Vector Protocol – transfer KV-cache between LLM agents instead of text. Maintained by VectorArc on GitHub, where it has earned 25 stars from the community.

Unified KV-cache compression for LLM inference: 12 Python-native methods, Debian-tested isolated add-ons, Godzilla KVarN/TriAttention, exact Godzilla/Gigatoken profiles, CUDA weight sharing, and multi-GPU planning. Maintained by aivrar on GitHub, where it has earned 25 stars from the community.

Training-free KV cache compression via E8 lattice VQ. 2-bit KV that preserves retrieval (30/30 NIAH vs TurboQuant 0/30). Calibration-free, 9 architectures validated. Maintained by jagmarques on GitHub, where it has earned 24 stars from the community.

Rust SDK for building AI agents with local OpenAI-compatible servers (LMStudio, Ollama, llama.cpp, vLLM). Features streaming, tools, hooks, retry logic, and comprehensive examples. Maintained by slb350 on GitHub, where it has earned 24 stars from the community.

Run the LLM Swarm Router on machines to distribute Local Ai to the Swarm - More Machines - MORE SPEED. Maintained by matthewdcage on GitHub, where it has earned 24 stars from the community.

， LLM. Maintained by zRzRzRzRzRzRzR on GitHub, where it has earned 24 stars from the community.

Probing the limitations of multimodal language models for chemistry and materials research. Maintained by lamalab-org on GitHub, where it has earned 24 stars from the community.

This project is the backend engine for a fully autonomous AI-powered call center. It integrates a large language model (LLM), speech recognition, and text-to-speech to manage real-time phone conversations via Asterisk. Maintained by klikz-dev on GitHub, where it has earned 24 stars from the community.

Local AI workstation — discover, run, chat, benchmark, and generate images from open-weight models. DFlash/DDTree speculative decoding, TurboQuant & TriAttention cache compression strategies, MLX + llama.cpp + vLLM + MTPLX backends. Maintained by cryptopoly on GitHub, where it has earned 24 stars from the community.

Local-first desktop assistant for the whole job hunt — scrape listings, match with a local LLM, generate applications in your own voice, and track everything on a Kanban board. Maintained by Keljian on GitHub, where it has earned 23 stars from the community.

An imperative command-line-interface for AI workload orchestration. Maintained by theoddden on GitHub, where it has earned 23 stars from the community.

Say 'dreamier' and your ComfyUI workflow shifts — instantly, reversibly. An AI co-pilot for VFX artists: zero-LLM recipes (dreamier/sharper/faster), 133 MCP tools, EXR-aware vision, workflow.lock provenance, full undo, native sidebar, 5 swappable brains (Claude, GPT, Gemini, Ollama, Nemotron). Maintained by JosephOIbrahim on GitHub, where it has earned 23 stars from the community.

Enterprise-grade platform to generate and execute Cypress, Playwright, WebdriverIO, and Appium end-to-end tests from natural language requirements. (Appium is experimental and requires external mobile infrastructure.). Maintained by aiqualitylab on GitHub, where it has earned 22 stars from the community.

Open-source, self-hosted AI workspace for local and open-weight LLMs with vLLM, LiteLLM, autonomous agents, MCP tools, deep research, artifacts, and BYOK. Maintained by openmake on GitHub, where it has earned 22 stars from the community.

DeepSeek-V3, R1 671B on 8xH100 Throughput Benchmarks. Maintained by dzhsurf on GitHub, where it has earned 22 stars from the community.

NVFP4 inference on Blackwell GeForce (RTX 5090/5080/5070 Ti/RTX PRO 6000) — SM120 patches for vLLM + FlashInfer + CUTLASS. 175 tok/s on Qwen3.6-35B MoE. Maintained by lna-lab on GitHub, where it has earned 22 stars from the community.

Run and manage both vLLM and llama.cpp model servers from one terminal UI + CLI — shared Docker Compose profiles, live tok/s, and cross-backend port/GPU conflict checks. NVIDIA + Linux. Maintained by Changroro on GitHub, where it has earned 22 stars from the community.

Run the AEON Bench suite on your own hardware: verified HuggingFace pull → serve → benchmark (text · agentic ×3 harnesses · vision · audio · arena · perf) → ed25519-signed attested submit. Maintained by AEON-7 on GitHub, where it has earned 21 stars from the community.

[ACL 2026] CoCoA: Collaborative Chain-of-Agents for Parametric-Retrieved Knowledge Synergy. Maintained by liunian-Jay on GitHub, where it has earned 21 stars from the community.

OpenAI-compatible, vLLM-served OCR API for the Surya-OCR-2 model — multilingual document OCR (layout + text recognition) with request batching, a local CLI, and Docker packaging. Maintained by wuxuedaifu on GitHub, where it has earned 21 stars from the community.

Low-Cost Cross-Domain Web Structured Information Extraction using specialized LoRA adapters. Maintained by abdo-Mansour on GitHub, where it has earned 20 stars from the community.

Layered prefill changes the scheduling axis from tokens to layers and removes redundant MoE weight reloads while keeping decode stall free. The result is lower TTFT, lower end-to-end latency, and lower energy per token without hurting TBT stability. Maintained by scale-snu on GitHub, where it has earned 20 stars from the community.

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → parallelism → vLLM/SGLang, with GPU labs on open models. Maintained by jiahongsigma on GitHub, where it has earned 20 stars from the community.

RAG using LlamaIndex：Computer Network Q&A System powered by LlamaIndex | LlamaIndex - HyDE+ + vLLM +Ragas. Maintained by userHanlh on GitHub, where it has earned 20 stars from the community.

[AAAI 2025 oral] Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit. Maintained by qizhou000 on GitHub, where it has earned 19 stars from the community.

Comprehensive, scalable ML inference architecture using Amazon EKS, leveraging Graviton processors for cost-effective CPU-based inference and GPU instances for accelerated inference. Guidance provides a complete end-to-end platform for deploying LLMs with agentic AI capabilities, including RAG and MCP. Maintained by aws-solutions-library-samples on GitHub, where it has earned 19 stars from the community.

A Scheduler for Batched LLM Inference. Maintained by iPieter on GitHub, where it has earned 19 stars from the community.

ICE-PIXIU：A Cross-Language Financial Megamodeling Framework. Maintained by YY0649 on GitHub, where it has earned 19 stars from the community.

Manalyzer: End-to-end Automated Meta-analysis with Multi-agent System. Maintained by black-yt on GitHub, where it has earned 19 stars from the community.

Intelligent load balancer for distributed vLLM server clusters vLLM. Maintained by xerrors on GitHub, where it has earned 19 stars from the community.

A lightweight post-training framework for LLMs and VLMs. 51 algorithms, 38 verified models. Scales with DeepSpeed, vLLM, and Ray. Maintained by warlockee on GitHub, where it has earned 19 stars from the community.

Drop-in OIDC & Google A2A auth + Weaviate memory for Ollama, vLLM and any local LLM server. Maintained by attach-dev on GitHub, where it has earned 18 stars from the community.