Sturnus

Sturnus is an automatic latency-based routing sidecar for LLM providers that exposes an OpenAI-compatible API.

Sturnus is a lightweight sidecar that sits beside your app and automatically shifts traffic to whichever LLM provider is fastest and available right now. It features latency- and error-aware routing, session affinity, transparent passthrough, and memory-bounded request buffers. It supports providers such as OpenAI, Azure OpenAI, Google AI Studio, Anthropic, Groq, and local OpenAI-compatible options, with specific support for Vertex AI. It is designed for developers who want automatic traffic steering across LLM providers without setting up a hosted SaaS or a complex control plane. The proxy serves chat completions and embeddings endpoints while exposing Prometheus metrics and structured logging.

Sturnus at a glance

Best for: Software engineer
Runs on: Linux, macOS, Docker, self-hosted
Pricing: Open source under the MIT license
Released: June 2026
Category: AI coding tool
Website: github.com/sturnus-dev/sturnus

Key features

Latency- and error-aware routing
Session affinity with stateless header support
Transparent passthrough for request bodies and SSE streams
Memory-bounded request buffering
Vertex AI support with GKE Workload Identity auth
Prometheus metrics and structured logging endpoints
Zero infrastructure sidecar deployment

Use it for

Routing LLM requests to the fastest available provider
Pinning multi-turn chat conversations to the same provider
Managing LLM traffic within Kubernetes sidecars
Collecting Prometheus metrics on upstream LLM latency and errors

Alternatives to Sturnus

OrcaRouter — Route requests to the best models without paying token markups.
llm-d — Deploys production-grade LLM serving on existing hardware clusters.
FreeLLMAPI — Accesses hundreds of free LLM models and billions of tokens through one API key.
GoRoute — Consumes 5.1MB of RAM with sub-10ms latency for coding agents.
AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
OpenLLM — Routes developer tools through multiple providers and local models with included inference.
LightLLM — It delivers high-speed inference and serving performance for large language models.
Semantic Router — Replaces slow LLM calls with fast deterministic routing.
OmniRoute — Routes requests across 352 providers with automatic fallback and cost-based selection.
New Relic — Unified visibility across applications, infrastructure, and AI models.
Weave Router — Routes prompts to cost-efficient models while preserving speed and quality.
Mesh LLM — Run large local models across multiple machines through one OpenAI-compatible endpoint.

All Sturnus alternatives