grps

Deep Learning Deployment Framework: Supports tf/torch/trt/trtllm/vllm and other NN frameworks. Support dynamic batching, and streaming modes. It is dual-language compatible with Python and C++, offering scalability, extensibility, and high performance. It helps users quickly deploy models and provide services through HTTP/RPC interfaces. Maintained by NetEase-Media on GitHub, where it has earned 168 stars from the community.

It's actively developed around dynamic-batching, serving, tensorflow, and is a solid reference for anyone building with these tools.