< cd ../gallery
Layered prefill changes the scheduling axis from tokens to layers and removes redundant MoE weight reloads while keeping decode stall free. The result is lower TTFT, lower end-to-end latency, and lower energy per token without hurting TBT stability. Maintained by scale-snu on GitHub, where it has earned 20 stars from the community.
# install
pip install layered-prefillmore vLLM repos
AI & ML
nano-vllmmoeSpeculative Decoding. Maintained by banfeb on GitHub, where it has earned 11 stars from the community.
4mo ago
AI & ML
Official implementation for Text Generation Beyond Discrete Token Sampling. Maintained by EvanZhuang on GitHub, where it has earned 26 stars from the community.
1y ago
AI & ML
A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving. Maintained by asprenger on GitHub, where it has earned 79 stars from the community.
2y ago
