wingman

Inference Hub for AI at Scale. Maintained by adrianliechti on GitHub, where it has earned 77 stars from the community.

It's actively developed around llama, llamacpp, openai, and is a solid reference for anyone building with these tools.