granite-switch by generative-computing

An endpoint server for efficiently serving quantized open-source LLMs for code. Maintained by wangcx18 on GitHub, where it has earned 59 stars from the community.

[ICPP'25] TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference. Maintained by MLSysU on GitHub, where it has earned 52 stars from the community.

This project aims to provide a high effective KV cache manage framework for llm inference and improve memory utilization and inference speed. Maintained by TheToughCrane on GitHub, where it has earned 71 stars from the community.