vllm.cpp by mudler

Official implementation for Text Generation Beyond Discrete Token Sampling. Maintained by EvanZhuang on GitHub, where it has earned 26 stars from the community.

LLM inference engine built from scratch in C++. No PyTorch, no frameworks. Maintained by Anirudh171202 on GitHub, where it has earned 12 stars from the community.

The official repo for the paper "Optimizing LLM Inference Throughput via Memory-aware and SLA-constrained Dynamic Batching". Maintained by KevinLee1110 on GitHub, where it has earned 18 stars from the community.