< cd ../gallery
[AI & ML]
guidance-for-scalable-model-inference-and-agentic-ai-on-amazon-eks by aws-solutions-library-samples
Comprehensive, scalable ML inference architecture using Amazon EKS, leveraging Graviton processors for cost-effective CPU-based inference and GPU instances for accelerated inference. Guidance provides a complete end-to-end platform for deploying LLMs with agentic AI capabilities, including RAG and MCP. Maintained by aws-solutions-library-samples on GitHub, where it has earned 19 stars from the community.
# install
pip install guidance-for-scalable-model-inference-and-agentic-ai-on-amazon-eksmore vLLM repos
AI & ML
Intelligent load balancer for distributed vLLM server clusters vLLM. Maintained by xerrors on GitHub, where it has earned 19 stars from the community.
11mo ago
AI & ML
Layered prefill changes the scheduling axis from tokens to layers and removes redundant MoE weight reloads while keeping decode stall free. The result is lower TTFT, lower end-to-end latency, and lower energy per token without hurting TBT stability. Maintained by scale-snu on GitHub, where it has earned 20 stars from the community.
7mo ago
AI & ML
Benchmark OpenAI-compatible AI endpoints and AI Accelerators in a reproducible structured way. Maintained by flexaihq on GitHub, where it has earned 12 stars from the community.
7mo ago
