time-to-first-token by patchy631

Source-code level analysis of LLM RL training infra: async RL, weight sync, FP8, MoE routing | LLM RL. Maintained by zpqiu on GitHub, where it has earned 14 stars from the community.

An endpoint server for efficiently serving quantized open-source LLMs for code. Maintained by wangcx18 on GitHub, where it has earned 59 stars from the community.

This project aims to provide a high effective KV cache manage framework for llm inference and improve memory utilization and inference speed. Maintained by TheToughCrane on GitHub, where it has earned 71 stars from the community.