< cd ../gallery
[AI & ML]
layered-prefill
Layered prefill changes the scheduling axis from tokens to layers and removes redundant MoE weight reloads while keeping decode stall free. The result is lower TTFT, lower end-to-end latency, and lower energy per token without hurting TBT stability.
@scale-snu
maintainer
★ 18 stars
# README
Layered prefill changes the scheduling axis from tokens to layers and removes redundant MoE weight reloads while keeping decode stall free. The result is lower TTFT, lower end-to-end latency, and lower energy per token without hurting TBT stability. Maintained by scale-snu on GitHub, where it has earned 18 stars from the community.
It's actively developed around inference, llm, llm-infernece, and is a solid reference for anyone building with these tools.
# tags
# install
npm install layered-prefillmore vLLM repos
