< cd ../gallery
[ICPP'25] TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference. Maintained by MLSysU on GitHub, where it has earned 52 stars from the community.
# install
pip install td-pipemore vLLM repos
AI & ML
An endpoint server for efficiently serving quantized open-source LLMs for code. Maintained by wangcx18 on GitHub, where it has earned 59 stars from the community.
2y ago
AI & ML
A Scheduler for Batched LLM Inference. Maintained by iPieter on GitHub, where it has earned 19 stars from the community.
1y ago
AI & ML
This project aims to provide a high effective KV cache manage framework for llm inference and improve memory utilization and inference speed. Maintained by TheToughCrane on GitHub, where it has earned 71 stars from the community.
5mo ago
