vector-inference by VectorInstitute

Layered prefill changes the scheduling axis from tokens to layers and removes redundant MoE weight reloads while keeping decode stall free. The result is lower TTFT, lower end-to-end latency, and lower energy per token without hurting TBT stability. Maintained by scale-snu on GitHub, where it has earned 20 stars from the community.

Nano vLLM with vLLM v1's request scheduling strategy and chunked prefill. Maintained by slwang-ustc on GitHub, where it has earned 95 stars from the community.

ElasticMM: Elastic and Efficient MLLM Serving System. Maintained by hpdps-group on GitHub, where it has earned 44 stars from the community.