sparkrun by spark-arena

Accurate, large-scale, and extensible simulator for LLM inference Systems. Maintained by microsoft on GitHub, where it has earned 654 stars from the community.

A tool for benchmarking LLMs on Modal. Maintained by modal-labs on GitHub, where it has earned 62 stars from the community.

llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deployment, such as UI, RESTful API, auto-scaling, computing resource management, monitoring, and more. Maintained by OpenCSGs on GitHub, where it has earned 95 stars from the community.