photon_infer by lumia431

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM. Maintained by jmaczan on GitHub, where it has earned 994 stars from the community.

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features. Maintained by mudler on GitHub, where it has earned 44 stars from the community.

A high-performance RDMA distributed file system for fast LLM Inference and GPU Training. Maintained by blackbird-io on GitHub, where it has earned 47 stars from the community.