booster by gotzmann

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features. Maintained by mudler on GitHub, where it has earned 44 stars from the community.

gpt_serverLLMs、Embedding、Reranker、ASR、TTS、、。. Maintained by shell-nlp on GitHub, where it has earned 253 stars from the community.

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM. Maintained by jmaczan on GitHub, where it has earned 994 stars from the community.