FastDeploy by PaddlePaddle

Layered prefill changes the scheduling axis from tokens to layers and removes redundant MoE weight reloads while keeping decode stall free. The result is lower TTFT, lower end-to-end latency, and lower energy per token without hurting TBT stability. Maintained by scale-snu on GitHub, where it has earned 20 stars from the community.

Accurate, large-scale, and extensible simulator for LLM inference Systems. Maintained by microsoft on GitHub, where it has earned 654 stars from the community.

A course of learning LLM inference serving on Apple Silicon for systems engineers: build a tiny vLLM + Qwen. Maintained by skyzh on GitHub, where it has earned 4,441 stars from the community.