Agent Router Python SDK

Call AI models from your agent with LiteLLM's API.
Every call goes through Agent Router, an open-source AI gateway, which handles fallbacks, retries and request limits.

Quick start

pip install theagentrouter
export ANTHROPIC_API_KEY=...

import theagentrouter

resp = theagentrouter.completion(
    model="anthropic/claude-sonnet-5-5",
    messages=[{"role": "user", "content": "Summarise yesterday's refunds"}],
)
print(resp.choices[0].message.content)

OpenAI's Responses API, Anthropic's Messages API and embeddings work the same way, as in LiteLLM:

resp = theagentrouter.responses(model="openai/gpt-6-luna", input="Summarise yesterday's refunds")
print(resp.output_text)

resp = theagentrouter.anthropic.messages.create(
    model="anthropic/claude-sonnet-5-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Summarise yesterday's refunds"}],
)
print(next(block["text"] for block in resp["content"] if block["type"] == "text"))  # thinking may come first

vectors = theagentrouter.embedding(model="openai/text-embedding-3-small", input=["refunds", "returns"])

The first call starts a local gateway on your computer, and the gateway calls the provider with your key. On Windows, Intel Macs and older Linux there is no local gateway; use a remote one instead (see Install).

Why use it

• Already using LiteLLM? Change one import. Functions, arguments, responses and errors stay the same.
• Fallbacks and retries run in a gateway, not in your process. Define a route once, and every program on the computer uses it.
• Google ADK and the OpenAI Agents SDK work without LiteLLM installed.
• Point it at your company's gateway with two environment variables. Provider keys then stay on that gateway, not on laptops.

Examples

You want to	Example
Make one call	first_calls/one_call.py
Move a LiteLLM script over	from_litellm/change_the_import.py
Fall back to another model when one fails	routes/fallback.py
Retry on rate limits and server errors	routes/retry.py
Stream, use tools, get JSON back	first_calls/stream.py, first_calls/tools.py, first_calls/json_schema.py
Use OpenAI's Responses API or Anthropic's Messages API	provider_apis/openai_responses.py, provider_apis/anthropic_messages.py
Run a Google ADK or OpenAI Agents SDK agent	agent_frameworks/google_adk.py, agent_frameworks/openai_agents.py
Send calls to your company's gateway	gateways/your_gateway.py

All twenty-three, with the keys each one needs: examples/.

Coming from LiteLLM

from theagentrouter.compat import litellm   # was: import litellm

Change this one import; the rest of your code stays the same. Some LiteLLM functions are not in the SDK, and some arguments are ignored with a warning. The full list: docs/coming-from-litellm.md.

Fallbacks and retries

A route is a model name you choose, with the models behind it in order. The local gateway calls the first model, retries it if it fails, then moves on to the next one:

import theagentrouter as ar

ar.route("chat", primary="openai/gpt-5.4", fallbacks=["anthropic/claude-sonnet-5-5"], retry=ar.Retry(attempts=2))
ar.completion(model="chat", messages=[{"role": "user", "content": "hi"}])

Here a call to chat goes to openai/gpt-5.4. If it cannot be reached or answers with 429, 500, 502, 503 or 504, the gateway retries it up to 2 more times, then tries anthropic/claude-sonnet-5-5 the same way. Other errors, such as 401, are not retried.

• A route can also limit requests: limit=ar.Limit(60, per="minute") lets 60 requests a minute through and answers the rest with a RateLimitError.
• Routes are saved in ~/.agentrouter/routes.json, so every program on this computer can use them.
• Routes defined in code set up the local gateway. With AGENTROUTER_BASE_URL set, the same code runs, and that gateway's route of the same name applies; local_gateway.export_config() gives it this computer's routes.
• Router from theagentrouter.compat.litellm, with LiteLLM's fallbacks and num_retries, is turned into routes.

Details: Routes.

Agent frameworks

Framework	How
Google ADK	install_as_litellm(), then ADK's LiteLlm model
OpenAI Agents SDK	install_as_litellm(), then the Agents SDK's LitellmModel

install_as_litellm() is in theagentrouter.compat. It makes import litellm load this SDK, so neither framework needs LiteLLM installed. Details and examples: docs/frameworks.md.

Install

pip install theagentrouter

This installs the SDK and the local gateway. The local gateway is Agent Router's aigw program, about 80 MB, and it runs on:

• macOS 13 or newer on Apple chips
• Linux with glibc 2.36 or newer, on x86-64 or Arm

On any other computer (Windows, Intel Macs, macOS 12 or older, Alpine Linux, Linux with glibc older than 2.36) there is no local gateway, so the same command installs the SDK alone. Calls then need a remote gateway (below).

If your code will use a remote gateway, for example on a server, in a container, in a serverless function or in CI, install the SDK alone and skip the 80 MB:

pip install theagentrouter-core

Both packages are imported the same way: import theagentrouter.

Using a remote gateway. To send calls to an Agent Router gateway that your team or company runs, set its address and key:

export AGENTROUTER_BASE_URL=https://gateway.acme.example/v1
export AGENTROUTER_API_KEY=...

With these set, calls go to that gateway, the local gateway never starts, and the provider keys stay on the remote gateway. This works on every computer.

The local gateway keeps running after your program ends, and every program on this computer that uses the SDK shares it. Its files are in ~/.agentrouter/local-gateway/ (or AGENTROUTER_HOME/local-gateway/): config.yaml, state.json and aigw.log. To stop it (the next call starts it again):

python -c "from theagentrouter import local_gateway; local_gateway.stop()"

The local gateway is for development. A new route, model or provider restarts it, which interrupts other programs, and a rotated provider key needs a stop. All known limits: How the local gateway works.

Installing from source, checking the install, upgrading and removing: docs/install.md.

Documentation

Page	What it covers
Install	The SDK, the local gateway, checking it works, upgrading, removing
Usage	Models and keys, parameters, streaming, async, tools, the Responses API, the Messages API, embeddings, errors, cost, gateways, settings
Coming from LiteLLM	What works the same, what is missing, what is ignored, a checklist
Agent frameworks	Google ADK and the OpenAI Agents SDK
How the local gateway works	Known limits; for contributors: packages, config, routing, tests

For coding agents

• skills/: agent skills (SKILL.md). migrate-from-litellm moves a project off LiteLLM; theagentrouter writes new code with the SDK. Copy a folder into your agent's skills folder (for Claude Code, .claude/skills/), then ask it to "migrate this project from LiteLLM to theagentrouter".

More

• DEVELOPMENT.md: what is where in this repository, setting up, tests, the framework tests, real-provider checks, releases.
• CONTRIBUTING.md, SECURITY.md, CHANGELOG.md.

Design

1. Only what agent frameworks use. The SDK has a LiteLLM function or setting only if an agent framework calls it. So there is no image generation, speech, video, files or batches.
2. Agent Router handles the calls. The gateway does fallbacks, retries, request limits, provider keys and logging. The SDK only does what a gateway can't, like building LiteLLM's response objects or running your callbacks.
3. Same API as LiteLLM. Same function names, arguments, responses and errors.

Contact

Join the Agent Router Discord to ask questions. To report a security problem, see SECURITY.md.

License

Apache License 2.0. Documentation in docs/ is under CC BY 4.0.

---

Agent Router is an Agentic AI Foundation project, powered by Envoy. Copyright © Agent Router a Series of LF Projects, LLC. For web site terms of use, trademark policy and other project policies please see https://lfprojects.org.