< cd ../gallery
llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deployment, such as UI, RESTful API, auto-scaling, computing resource management, monitoring, and more. Maintained by OpenCSGs on GitHub, where it has earned 95 stars from the community.
# install
pip install llm-inferencemore vLLM repos
AI & ML
Nano vLLM with vLLM v1's request scheduling strategy and chunked prefill. Maintained by slwang-ustc on GitHub, where it has earned 95 stars from the community.
8mo ago
AI & ML
An endpoint server for efficiently serving quantized open-source LLMs for code. Maintained by wangcx18 on GitHub, where it has earned 59 stars from the community.
2y ago
AI & ML
This project aims to provide a high effective KV cache manage framework for llm inference and improve memory utilization and inference speed. Maintained by TheToughCrane on GitHub, where it has earned 71 stars from the community.
5mo ago
