whereismyai
LiteLLM
LiteLLM is an open-source AI gateway that calls 100+ language models through a single OpenAI-compatible interface.
LiteLLM is an open-source Python SDK and a self-hosted proxy that send chat, embeddings, images, audio, batches and rerank requests to more than 100 providers, including OpenAI, Anthropic, Gemini, Bedrock and Azure, in the OpenAI request format. The gateway adds virtual keys, spend tracking, guardrails, load balancing and an admin dashboard. It can also invoke A2A agents and attach MCP servers so tools show up on the same /chat/completions path. Netflix is listed among open-source adopters. Latency is quoted at 8 ms P95 at 1,000 requests per second in the project's own benchmarks.
LiteLLM at a glance
- Best for: Software engineer
- Runs on: open source, self-hosted, API
- Pricing: Free and open source; a hosted proxy and an Enterprise tier are also offered
- Released: July 2023
- Category: AI coding tool
- Website: github.com/BerriAI/litellm
Key features
- Python SDK and self-hosted proxy for 100+ LLMs
- OpenAI-compatible chat, embeddings, images, audio and batches
- Virtual keys, spend tracking and an admin dashboard
- Load balancing and guardrails on the gateway
- A2A agent calls and an MCP bridge
- Works as an MCP server for editors such as Cursor
Use it for
- Swap model providers without rewriting application code
- Give a team one gateway with per-key spend limits
- Route editor tool calls through MCP to many models
Companion to
Tools whose articles name LiteLLM as a companion, each with its job for them.
Alternatives to LiteLLM
- OptiLLM — Increases LLM reasoning accuracy by up to 10x without model training.
- WebLLM — Runs high-performance language models directly inside web browsers.
- LitGPT — Readable single-file LLM code with tested finetuning recipes
- LLM CLI — It enables command-line interaction with remote APIs and local machine models.
- MLC LLM — Enables native optimization and deployment of AI models across platforms.
- Mesh LLM — Run large local models across multiple machines through one OpenAI-compatible endpoint.
- OpenLIT — It enables comprehensive monitoring, evaluation, and management of AI agents via OpenTelemetry.
- OpenLLM — It simplifies infrastructure management and optimizes inference speed and cost.
- llm-d — Deploys production-grade LLM serving on existing hardware clusters.
- AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
- LightLLM — It delivers high-speed inference and serving performance for large language models.
- StrictLLM — It keeps your AI workflows secure and reviewable with sandboxed MCP servers and strict tool permissions.
All LiteLLM alternatives