LiteLLM Explained: The Open-Source Model Gateway for Agents

TL;DR — LiteLLM is an open-source Python proxy that gives you a single OpenAI-compatible endpoint for 100+ model providers. You call one API, it routes to OpenAI, Anthropic, Google, Bedrock, local vLLM, or whatever else you’ve configured — with automatic failover, cost tracking, rate limiting, and budget caps. If your agents talk to multiple models (they should), LiteLLM is the layer that keeps you from writing provider-specific code.

The Problem LiteLLM Solves

You’re building an agent. It needs a strong model for planning (Claude), a fast model for simple tool calls (GPT-4o-mini), and a cheap model for summarization (a local Llama via vLLM). That’s three different APIs, three different auth patterns, three different response formats, three different error codes, and three different billing dashboards.

Now multiply that by every model update that changes pricing, every rate limit you hit at 2 AM, every time a provider has an outage and your agents crash instead of gracefully falling back.

LiteLLM collapses this into one endpoint. Your agent code calls POST /chat/completions with a model name. LiteLLM handles everything underneath.

What LiteLLM Actually Is

LiteLLM has two faces:

1. Python SDK — A completion() function that wraps 100+ providers into an OpenAI-compatible interface. Import it and use it in your code directly.
2. Proxy Server (AI Gateway) — A standalone service you deploy. It exposes OpenAI-compatible endpoints and routes requests to configured providers based on rules you define.

The proxy is where the real value lives for production agent stacks. It’s the centralized control plane for all model access.

Key numbers:

Metric	Value
GitHub stars	20K+
OpenRank (May 2026)	156.06
Active contributors	249
Docker pulls	240M+
Supported providers	100+
Production requests served	1B+
License	MIT

Architecture: Where the Gateway Sits

┌──────────────────────────────────────────────────┐
│  Agent Runtime                                    │
│  (calls POST /v1/chat/completions)               │
├──────────────────────────────────────────────────┤
│  LiteLLM Proxy  ←── the gateway                  │
│  ┌────────────────────────────────────────────┐  │
│  │ Router: model → provider mapping           │  │
│  │ Load Balancer: round-robin / least-busy    │  │
│  │ Fallback: provider A fails → try B         │  │
│  │ Cost Tracker: per-token, per-key, per-team │  │
│  │ Rate Limiter: TPM / RPM caps              │  │
│  │ Budget: hard spend limits per virtual key  │  │
│  │ Cache: semantic or exact-match             │  │
│  │ Guardrails: content filtering              │  │
│  └────────────────────────────────────────────┘  │
├──────────────────────────────────────────────────┤
│  Providers                                        │
│  OpenAI | Anthropic | Bedrock | vLLM | SGLang    │
│  Google | Mistral | Cohere | Ollama | ...        │
└──────────────────────────────────────────────────┘

Your agent code doesn’t know (or care) which provider serves which model. That’s the gateway’s job.

Features That Matter for Agent Stacks

Unified API — One endpoint, one format. Switch from Claude to GPT-4o by changing a model name string, not rewriting your tool-calling logic.

Automatic failover — If Anthropic returns a 529, LiteLLM retries on a backup deployment or falls back to a different provider. Your agent session doesn’t die because one API had a hiccup.

Cost tracking — Every request is logged with input/output tokens and cost. You can see per-agent, per-team, per-model breakdowns. When your coding agent burns $40 in a debugging loop, you know exactly which session did it.

Budget caps — Set hard limits per virtual API key. Give each agent instance a $5/day budget. When it’s used up, requests get rejected instead of burning through your credit card.

Load balancing — Multiple API keys for the same provider? Multiple vLLM instances? LiteLLM distributes requests across them. If one is hitting rate limits, traffic shifts to another.

Virtual keys — Create API keys for teams or agents without sharing your real provider keys. Each virtual key has its own budget, rate limits, and model access permissions.

Caching — Exact-match or semantic caching of responses. If two agent sessions ask the same question with the same context, serve the cached response. Saves tokens and latency.

The Cost Routing Story

This is where LiteLLM intersects with a growing trend in agent infrastructure. Research from ICLR 2025 (RouteLLM) and EMNLP 2025 (IPR) showed that routing requests to different model tiers based on difficulty can cut costs by 40-50% while maintaining quality.

In practice, LiteLLM’s GitHub issues tell the same story. Search results show frequent requests for cost-based routing (23 issues), lowest-cost routing (18), budget routing (37), and spend tracking (76). Teams are asking: given quality requirements, how do I route to the cheapest sufficient model?

A basic cost-routing config looks like:

model_list:
  - model_name: agent-planner
    litellm_params:
      model: anthropic/claude-sonnet-4
      api_key: sk-ant-...
    model_info:
      cost_per_token: 0.000003

  - model_name: agent-executor
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: sk-...
    model_info:
      cost_per_token: 0.00000015

  - model_name: agent-summarizer
    litellm_params:
      model: openai/hosted_vllm/llama-3.3-70b
      api_base: http://vllm-internal:8000/v1

Your agent calls different model names for different tasks. The gateway routes each to the right backend. No provider SDK imports in your agent code.

Running LiteLLM Proxy

# Docker (quickest)
docker run -d \
  -p 4000:4000 \
  -v ./litellm_config.yaml:/app/config.yaml \
  -e DATABASE_URL=postgresql://... \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml

# Test
curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-your-virtual-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "agent-planner",
    "messages": [{"role": "user", "content": "Plan the refactor"}],
    "max_tokens": 500
  }'

That request hits LiteLLM, which resolves agent-planner to Claude Sonnet 4, adds auth, forwards the request, logs the cost, and returns the response in OpenAI format.

Failover That Actually Works

The single most valuable LiteLLM feature for production agents is failover — and it’s also the one people configure wrong. The naive setup looks like this:

model_list:
  - model_name: agent-planner
    litellm_params:
      model: anthropic/claude-sonnet-4
      api_key: os.environ/ANTHROPIC_KEY_1
  - model_name: agent-planner          # same name = a fallback group
    litellm_params:
      model: anthropic/claude-sonnet-4
      api_key: os.environ/ANTHROPIC_KEY_2  # different key

router_settings:
  num_retries: 2
  retry_after: 5
  allowed_fails: 3
  cooldown_time: 30
  fallbacks: [{"agent-planner": ["agent-planner-backup"]}]

The subtle part most people miss: retries within the same provider don’t help when the provider itself is down. If Anthropic returns 529 (overloaded), retrying the same Anthropic endpoint 2 seconds later usually returns another 529. Real resilience needs a cross-provider fallback:

  - model_name: agent-planner-backup
    litellm_params:
      model: openai/gpt-4o           # DIFFERENT provider
      api_key: os.environ/OPENAI_KEY

Now when Anthropic is overloaded, the request falls through to GPT-4o. Your agent gets a slightly different model but stays alive. For a planning step, that trade-off is almost always worth it.

The other gotcha: allowed_fails and cooldown_time. When a deployment fails allowed_fails times, LiteLLM puts it in cooldown for cooldown_time seconds — routing traffic elsewhere. Set cooldown_time too low (e.g., 5s) and you’ll keep hammering a struggling provider. Too high (e.g., 300s) and you lose capacity for 5 minutes over a transient blip. 30-60s is the sweet spot for most agent workloads.

The Streaming + Fallback Trap

Here’s a bug that bites agent builders specifically. When you stream responses (which you should, for UX) and the primary provider fails mid-stream — after sending 50 tokens then erroring — what happens?

LiteLLM can fall back to a secondary provider, but the client has already received those 50 tokens. The fallback response starts from scratch, so the user sees 50 tokens, then the stream restarts with different content. For a chat UI this is jarring; for an agent parsing structured output, it’s a parse failure.

The mitigation: for agent tool-call requests where output integrity matters more than latency, disable streaming and use the non-streaming path with fallback. Reserve streaming for user-facing chat where a restart is tolerable. LiteLLM doesn’t solve this for you — you have to decide per-endpoint whether streaming or reliable-fallback matters more.

LiteLLM vs Alternatives

Feature	LiteLLM	OpenRouter	SandBase	Custom Proxy
Self-hosted	✅	❌ (SaaS)	✅	✅
Open source	✅ MIT	❌	Partial	Varies
Provider count	100+	200+	Curated	What you build
Cost tracking	Built-in	Built-in	Built-in	DIY
Budget caps	✅	❌	✅	DIY
Rate limiting	✅	Platform-level	✅	DIY
Failover	✅	Platform-level	✅	DIY
Setup effort	Low	Zero	Low	High

LiteLLM is the open-source option you self-host. OpenRouter is the managed marketplace. SandBase provides a managed gateway with sandbox/execution features. They solve overlapping but different problems — and you can actually use LiteLLM as the routing layer under a higher-level platform.

When You Need a Gateway (and When You Don’t)

You need a gateway when:

• Your agents use more than one model provider
• You need cost visibility and budget controls
• You want automatic failover when a provider goes down
• Multiple teams or agents share API keys and need isolation
• You’re routing between self-hosted models and cloud APIs

You probably don’t need one when:

• Single model, single provider, low volume
• You’re prototyping and don’t care about cost yet
• The provider’s native SDK already handles your retries

For most production agent systems, a gateway becomes essential at the point where you have more than one model or more than one team. The cost of not having one is invisible until you get a surprise $2,000 bill or a 3-hour outage because you hardcoded a single provider endpoint.

FAQ

Can LiteLLM route based on content/difficulty?

Not natively with a built-in classifier, but you can implement it. Set up different model names for different complexity tiers in your agent code, and LiteLLM routes each to the appropriate provider. Some teams put a lightweight classifier in front that picks the model name.

Does LiteLLM add latency?

Minimal — typically 5-15ms per request. It’s a thin proxy, not doing heavy processing. The latency cost is negligible compared to model generation time.

Can I use LiteLLM with streaming?

Yes. Full support for SSE streaming, including with fallover (if the primary stream fails mid-response, it can retry on a fallback).

How does it handle different response formats?

LiteLLM normalizes everything into OpenAI format. Anthropic’s Messages API, Google’s Gemini format, Bedrock’s structure — all come back as standard OpenAI chat completions to your agent code.

Is PostgreSQL required?

Only for persistent cost tracking and budget management. Without a database, you still get routing and failover, but spend data doesn’t survive restarts.

Key Takeaways

• LiteLLM gives agents a single OpenAI-compatible endpoint for 100+ providers, with automatic failover, cost tracking, and budget caps.
• The gateway pattern is essential when agents use multiple models — which they should for cost optimization (strong model for planning, cheap model for simple tasks).
• Cost routing is the emerging trend: route to the cheapest model that meets quality requirements. LiteLLM provides the plumbing; you provide the routing logic.
• Deploy it as a Docker proxy between your agents and model providers. Your agent code never imports a provider SDK again.
• Pair it with inference engines like vLLM or SGLang for self-hosted models, or use it to unify cloud APIs.