$ Discover the best projects built with vLLM

A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: AI Gateway (LiteLLM) LLM Serving (vLLM, SGLang, Ollama) Vector Databases, Embedding Models (TEI) Observability (Langfuse, Phoenix) etc. Fast-track your GenAI deployment with Kubernetes. Maintained by aws-samples on GitHub, where it has earned 91 stars from the community.

This repository contains hands-on projects, code examples, and deployment workflows. Explore multi-agent systems, LangChain, LangGraph, AutoGen, CrewAI, RAG, MCP, automation with n8n, and scalable agent deployment using Docker, AWS, and BentoML. Maintained by MDalamin5 on GitHub, where it has earned 90 stars from the community.

Granite Switch — Build AI models like you build software. Maintained by generative-computing on GitHub, where it has earned 90 stars from the community.

Qwen2+SFT+DPO， SFTTrainer/DPOTrainer/TRPOTrainer，，（neo4j, milvus, LDA, ）。， vllm, vllm API embedder + Reranker RAG 。 MDAgents ， vllm api 。. Maintained by NJUxlj on GitHub, where it has earned 89 stars from the community.

Enterprise-grade LLM automated deployment tool that makes AI servers truly "plug-and-play". Maintained by EM-GeekLab on GitHub, where it has earned 88 stars from the community.

Real-time hardware and LLM inference monitoring — GPU, CPU, memory, and vLLM metrics streamed to a dashboard. Maintained by niklasfrick on GitHub, where it has earned 86 stars from the community.

Zero trust LLM gateway. OpenAI-compatible proxy with semantic routing and load balancing across OpenAI, Anthropic, Ollama, vLLM, and any compatible backend. Identity-based access, virtual API keys, and end-to-end encryption via OpenZiti. Maintained by openziti on GitHub, where it has earned 86 stars from the community.

The open-core AI workbench — notebooks, agents, RAG, voice, and images across any model: OpenAI, Anthropic, Google, xAI, or local via Ollama/vLLM. BSL 1.1, auto-converting to Apache-2.0 on a two-year clock. Your AI keeps running when theirs doesn't. Maintained by Bike4Mind on GitHub, where it has earned 85 stars from the community.

A unified evaluation toolkit and leaderboard for rigorously assessing the scientific intelligence of large language and vision–language models across the full research workflow. Maintained by InternScience on GitHub, where it has earned 85 stars from the community.

Agent and LLM API performance monitoring via network packet probe. Measures performance of OpenClaw, Claude, Codex, DeepAgents and more — deployed on the provider side, no SDK changes required. Maintained by Netis on GitHub, where it has earned 80 stars from the community.

A High-Efficiency System of Large Language Model Based Search Agents. Maintained by tiannuo-yang on GitHub, where it has earned 79 stars from the community.

A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving. Maintained by asprenger on GitHub, where it has earned 79 stars from the community.

First open-source implementation of Google TurboQuant (ICLR 2026) -- near-optimal KV cache compression for LLM inference. 5x compression with near-zero quality loss. Maintained by OnlyTerp on GitHub, where it has earned 78 stars from the community.

Production inference for encoder models - ColBERT, GLiNER, ColPali, embeddings etc. - as vLLM plugins for online and in-process deployment. Maintained by latenceainew on GitHub, where it has earned 76 stars from the community.

Reliable and Efficient Semantic Prompt Caching with vCache. Maintained by vcache-project on GitHub, where it has earned 76 stars from the community.

[ICML 2026] Decoding Tree Sketching (DTS): a training-free & model agonistic & plug-in framework for LLM parallel reasoning. Maintained by ZichengXu on GitHub, where it has earned 71 stars from the community.

This project aims to provide a high effective KV cache manage framework for llm inference and improve memory utilization and inference speed. Maintained by TheToughCrane on GitHub, where it has earned 71 stars from the community.

The htop for LLM inference see exactly where every GB of VRAM goes and get measured quantization savings. Maintained by helasaoudi on GitHub, where it has earned 68 stars from the community.

EmbodiedAgents is a fully-loaded ROS2 based framework for creating interactive physical agents that can understand, remember, and act upon contextual information from their environment. Maintained by automatika-robotics on GitHub, where it has earned 66 stars from the community.

PDF Parsing Tool: GOT's vLLM acceleration implementation, MinerU for layout recognition, and GOT for table formula parsing. Maintained by liunian-Jay on GitHub, where it has earned 66 stars from the community.

Automated Deep Research with LLMs, web search, paper parsing, and didactic summarization. Maintained by protonspy on GitHub, where it has earned 66 stars from the community.

An intelligent tuner for vLLM that automatically monitors GPU metrics, uses Bayesian optimization to tune parameters. Maintained by jranaraki on GitHub, where it has earned 66 stars from the community.

Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant kernel paths, CUDA toolchains, KV allocation, eval harnesses, versioning. Symptom-first, with the check that catches each. Maintained by Blackwellboy on GitHub, where it has earned 64 stars from the community.

LLM-Inference-Bench. Maintained by argonne-lcf on GitHub, where it has earned 63 stars from the community.

Real-time speech-to-text WebSocket server with pluggable ASR backends, energy-based VAD, streaming partial results, and Prometheus observability. Maintained by victoryangzhijie on GitHub, where it has earned 63 stars from the community.

A tool for benchmarking LLMs on Modal. Maintained by modal-labs on GitHub, where it has earned 62 stars from the community.

Turn documents into structured JSON with local-first document AI. Run 100% locally by default, with API, CLI, and Web UI. Maintained by parsehawk on GitHub, where it has earned 61 stars from the community.

Sardeenz is a proof-of-concept application that allows you to load more than one model on a given GPU. It allows you to add more and more models onto a GPU, until it is fully utilized. Maintained by rh-aiservices-bu on GitHub, where it has earned 61 stars from the community.

LLM-powered security log analyzer: detect threats & anomalies with zero regex — just declare a Pydantic schema. Real-time Telegram alerts, SIEM-ready with Elasticsearch/Kibana. Supports OpenAI, Ollama, vLLM. Maintained by call518 on GitHub, where it has earned 60 stars from the community.

Modern AI chatbot supporting multiple LLMs. Switch between Gemini, Mistral, Llama, Claude and ChatGPT. Maintained by intelligentnode on GitHub, where it has earned 59 stars from the community.

An endpoint server for efficiently serving quantized open-source LLMs for code. Maintained by wangcx18 on GitHub, where it has earned 59 stars from the community.

A physics-grounded, cost-aware optimization loop for vLLM. Maintained by jungledesh on GitHub, where it has earned 59 stars from the community.

Open-source, self-hostable social listening. Maintained by obris-dev on GitHub, where it has earned 58 stars from the community.

High-Performance KV Cache Storage Engine on CXL Shared Memory for LLM Inference. Maintained by xcena-dev on GitHub, where it has earned 56 stars from the community.

OpenAI-compatible multilingual TTS server — Chatterbox on vLLM with real-time PCM audio streaming, low time-to-first-byte (~0.7 s), voice cloning, and 23 languages. Maintained by wuxuedaifu on GitHub, where it has earned 56 stars from the community.

vLLM Router. Maintained by llm-semantic-router on GitHub, where it has earned 56 stars from the community.

Real-time streaming TTS server for XTTS-v2 on vLLM — OpenAI-compatible API, ~0.5s TTFB, Docker. Maintained by wuxuedaifu on GitHub, where it has earned 55 stars from the community.

DFlash vLLM for DGX Spark — Plug & Play Block-Diffusion Speculative Decoding. Maintained by AEON-7 on GitHub, where it has earned 54 stars from the community.

The open-source AI platform for enterprises that can't send data to the cloud. OpenAI-compatible API, full management dashboard, zero data egress. Maintained by xinity-ai on GitHub, where it has earned 54 stars from the community.

DGX Spark / GB10 vLLM Docker stack for large-model serving, presets, patches, and validation notes. Maintained by bjk110 on GitHub, where it has earned 54 stars from the community.

[ICPP'25] TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference. Maintained by MLSysU on GitHub, where it has earned 52 stars from the community.

Arks is a cloud-native inference framework running on Kubernetes. Maintained by scitix on GitHub, where it has earned 50 stars from the community.

vLLM Qwen 3.6-27B (AWQ-INT4) + DFlash speculative decoding on AMD Strix Halo (gfx1151 iGPU, 128 GB UMA, ROCm 7.13). 24.8 t/s single-stream, vision, tool calling, 256K context, OpenAI-compatible, Docker. Matches DGX Spark FP8+DFlash+MTP at a third of the cost. No CUDA. Maintained by hec-ovi on GitHub, where it has earned 50 stars from the community.

Open source Enterprise AI Gateway for API, MCP and agent, and AI model traffic. One Apache-2.0 binary: 72 native providers behind an OpenAI-compatible API, or serve vLLM and llama.cpp on your own GPUs. Keys, budgets, guardrails, semantic cache, WAF. Maintained by soapbucket on GitHub, where it has earned 49 stars from the community.

Batch Deployment for Document Parsing with AWS Batch & Qwen-2.5-VL. Maintained by jeremyarancio on GitHub, where it has earned 48 stars from the community.

TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs. Maintained by Alberto-Codes on GitHub, where it has earned 48 stars from the community.

A high-performance RDMA distributed file system for fast LLM Inference and GPU Training. Maintained by blackbird-io on GitHub, where it has earned 47 stars from the community.

Legacy: self-hosted LLM/RAG stack installer for DGX Spark (arm64). The work continued as AGmind Systems Lab — measured local-LLM performance evidence (Strix Halo, DGX Spark, llama.cpp, vLLM): https://agmind.ai. Maintained by botAGI on GitHub, where it has earned 47 stars from the community.

Native macOS menu bar app for realtime dictation with optional LLM polishing. Connects to any OpenAI Realtime-compatible backend — fully local on Apple Silicon with voxmlx + mlx-swift-lm. Maintained by T0mSIlver on GitHub, where it has earned 45 stars from the community.

Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop. Maintained by outsourc-e on GitHub, where it has earned 44 stars from the community.

ElasticMM: Elastic and Efficient MLLM Serving System. Maintained by hpdps-group on GitHub, where it has earned 44 stars from the community.

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features. Maintained by mudler on GitHub, where it has earned 44 stars from the community.

Android keyboard with local AI (Ollama, Whisper, MCP) or cloud (Gemini, Groq, OpenAI). Maintained by SvReenen on GitHub, where it has earned 43 stars from the community.

Bleeding edge vLLM Docker image for the NVIDIA DGX Spark (GB10 / sm_121a). Maintained by timothystewart6 on GitHub, where it has earned 42 stars from the community.

useful Gentoo overlay Curated ebuilds, AI, tools & science. Maintained by istitov on GitHub, where it has earned 42 stars from the community.

Synthetic medical VQA pipeline: 119K images annotated by frontier VLMs, cross-validated at 93% agreement, fine-tuned on 3 model families (2-3B params). Maintained by openmed-labs on GitHub, where it has earned 39 stars from the community.

Code for building self-expanding knowledge graphs with Outlines, vLLM, neo4j, and Modal. Maintained by just-cameron on GitHub, where it has earned 38 stars from the community.

Carbon Limiting Auto Tuning for Kubernetes. Maintained by Climatik-Project on GitHub, where it has earned 38 stars from the community.

Hit your limit? Need privacy? Just swap the model, everything else stays. Maintained by luongnv89 on GitHub, where it has earned 37 stars from the community.

Self-hosted, OpenAI-compatible inference for the agentic era: reasoning LLMs, universal tool calling, and the Responses API alongside embeddings, speech, and image models — many models sharing your GPUs, one gateway. Powered by Ray Serve. Maintained by modelship-ai on GitHub, where it has earned 37 stars from the community.