llmq

A Scheduler for Batched LLM Inference. Maintained by iPieter on GitHub, where it has earned 19 stars from the community.

It's actively developed around llm, llm-inference, rabbitmq, and is a solid reference for anyone building with these tools.