llm-inference

llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deployment, such as UI, RESTful API, auto-scaling, computing resource management, monitoring, and more. Maintained by OpenCSGs on GitHub, where it has earned 95 stars from the community.

It's actively developed around deepspeed, llama-cpp, llm-inference, and is a solid reference for anyone building with these tools.