olla

High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends. Maintained by thushan on GitHub, where it has earned 256 stars from the community.

It's actively developed around ai, amd, golang, and is a solid reference for anyone building with these tools.