Usage

Calls: A first call · Models and keys · Parameters · Streaming · Async · Tools · The Responses API · The Messages API · Embeddings
What comes back: What a call returns · Errors · Cost and model details (adding models) · Streams put back together · Callbacks · Counting tokens · Tests
Gateways: Gateways · A gateway you set · The local gateway · Routes · Running the same gateway for a team · Settings

A first call

import theagentrouter

resp = theagentrouter.completion(
    model="anthropic/claude-sonnet-5-5",
    messages=[{"role": "user", "content": "Summarise yesterday's refunds"}],
)
print(resp.choices[0].message.content)

completion takes the same arguments as litellm.completion and returns the same response shape.

The SDK has four kinds of call, each as in LiteLLM:

Call	For	Section
`completion`, `acompletion`	Chat with any model: OpenAI Chat Completions, and Anthropic Messages for Claude	Streaming, Tools
`responses`, `aresponses`	OpenAI's Responses API, for OpenAI models	The Responses API
`anthropic.messages.create`, `acreate`	Anthropic's Messages API, for Claude	The Messages API
`embedding`, `aembedding`	Vectors	Embeddings

Models and keys

A model name is provider/model, such as openai/gpt-5.4-mini. A name with no provider works where it works in LiteLLM: OpenAI and Anthropic names (gpt-5.4-mini, claude-haiku-5-5). Other names need the provider, as in LiteLLM (mistral/mistral-large-latest, groq/llama-3.1-8b-instant). A bare gemini-… name is refused with BadRequestError: LiteLLM sends it to Google Cloud's Vertex AI, which the local gateway does not serve yet, so write gemini/gemini-2.5-flash for the Gemini API. Every gateway gets the name as you wrote it, with or without the provider, and a gateway you set can have names of its own, such as claude-sonnet-5.

Every call goes through a gateway: the local gateway (Agent Router on this computer), or one you set. The local gateway reads each provider's key from the environment when it starts:

Model name starts with	Key
`openai/`	`OPENAI_API_KEY`
`anthropic/`	`ANTHROPIC_API_KEY`
`gemini/`	`GEMINI_API_KEY`
`groq/`	`GROQ_API_KEY`
`mistral/`	`MISTRAL_API_KEY`
`deepseek/`	`DEEPSEEK_API_KEY`
`together/`	`TOGETHER_API_KEY`
`bedrock/`	`AWS_PROFILE`, or `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`; and `AWS_REGION`
`azure/`	`AZURE_API_KEY` and `AZURE_API_BASE` (your resource's endpoint); `AZURE_API_VERSION` if you need a version other than `2024-10-21`

For Azure OpenAI, the name after azure/ is your deployment's name: azure/gpt-5-mini calls the deployment gpt-5-mini. As with Bedrock, the bundled model list has none of these names, so each new one restarts a running gateway with a rule for it. To avoid that, add them with register_model first (see Adding models).

api_key= on a call is used in two cases: with api_base=, as the key for that endpoint, and with a gateway you set, as that gateway's key. With the local gateway it has no effect and warns once, because the local gateway reads provider keys from the environment when it starts.

To send one call to an endpoint you name, in the provider's own format, use api_base= (LiteLLM's argument), for example a local test server. The call goes there in place of the gateway, as in LiteLLM, so routes, fallbacks and limits apply only if that address is an Agent Router gateway that has them; the SDK warns once for each address.

Parameters

For OpenAI Chat Completions, these go to the provider as they are:

temperature, top_p, n, stop, max_tokens, max_completion_tokens, presence_penalty, frequency_penalty, logit_bias, logprobs, top_logprobs, seed, response_format, tool_choice, parallel_tool_calls, reasoning_effort, user, modalities, prediction, audio, service_tier.

For Claude, which goes to the gateway in Anthropic's Messages format (so prompt caching and thinking keep working), the SDK translates temperature, top_p, top_k, stop, max_tokens, max_completion_tokens, tool_choice, parallel_tool_calls and user. parallel_tool_calls=False allows at most one tool call. As in LiteLLM, user is left out when it is an email address or a phone number, which Anthropic refuses, tools also takes Anthropic's own shape and its hosted tools, and n=1 is fine. A JSON schema in response_format maps to the Messages output format, changed as in LiteLLM: every object closes with additionalProperties: false, oneOf becomes anyOf, and what Anthropic refuses, such as maximum, moves into the description. response_format also takes a pydantic class, for every provider. Claude has no JSON mode without a schema, so a plain json_object is not applied: the call goes ahead without it, and the SDK warns once.

To turn on thinking, use reasoning_effort="low" (or another level), or LiteLLM's thinking={"type": "enabled", "budget_tokens": 1024}. What reaches Claude depends on the model, by LiteLLM's rules:

You pass	Before Opus 4.6 and Sonnet 4.6	Opus 4.6 and Sonnet 4.6	Opus 4.7 and newer
`reasoning_effort`	A token budget: 1024 for `minimal` and `low`, 2048 for `medium`, 4096 for `high`, 8192 for `xhigh`, 16384 for `max`	Adaptive thinking at that level. `minimal` is `low`, and `xhigh` raises `BadRequestError`	Adaptive thinking at that level. `minimal` is `low`
`thinking` with `budget_tokens`	As written	As written	Adaptive thinking, at the level that fits the budget
`thinking={"type": "adaptive"}`	A budget of 2048, below `max_tokens`. With `max_tokens` of 1024 or less, no thinking	As written	As written
`thinking={"type": "disabled"}`	As written	As written	As written, but left out on Sonnet 5.5, Opus 5.5, Fable 5 and Fable 5.1, which always think

In adaptive thinking, the model decides how much to think. thinking wins over reasoning_effort, which then only sets the level on a model that has levels. thinking=True is the medium setting. For a Claude name the SDK does not know, the version in the name decides which column applies, and Anthropic answers for the rest.

With reasoning_effort and a max_tokens at or below the token budget, LiteLLM sends the call and Anthropic refuses it. The SDK uses a smaller budget instead, and raises BadRequestError if max_tokens is 1024 or less.

Without max_tokens, Claude gets the model's own limit from the model list, or 4096 for a name not on the list. With a token budget, it gets the budget plus 4096. If you pass both max_tokens and max_completion_tokens, max_completion_tokens wins.

Some options raise UnsupportedParamsError; with drop_params=True (or theagentrouter.drop_params = True, which holds whatever a call says) they are dropped instead. additional_drop_params=["seed"] drops the options you name:

OpenAI-only options such as seed, on every Claude model.
temperature other than 1, top_p and top_k, on Opus 4.7 and every newer Claude model (Opus 4.8, Sonnet 5, Sonnet 5.5, Haiku 5.5, Opus 5, Opus 5.5, Fable 5, Fable 5.1).
A forced tool call (tool_choice="required" or a named tool), on Sonnet 5.5, Opus 5.5 and Fable 5.1. With drop_params=True the call uses tool_choice="auto", and the SDK warns once.

A gateway you set decides for itself: without drop_params, the last two go to it, because its model names are its own.

Claude on Bedrock follows the same rules, by the name after bedrock/; an inference profile's ARN counts as a model before adaptive thinking unless it names a newer Claude. Signed thinking goes back to Bedrock with a tool result, as it does to Anthropic. On Bedrock models, tool_choice="none" raises UnsupportedParamsError, as do seed, n above 1, presence_penalty, frequency_penalty, logit_bias, logprobs and top_logprobs; with drop_params=True they are dropped. Cohere's models take all of these except logit_bias, logprobs and top_logprobs, and a gateway you set gets them all except tool_choice="none". The gateway has no field for four things, so they do not reach Bedrock, and the SDK warns once for each: an effort level (Claude uses its default), top_k, parallel_tool_calls=False and response_format.

Use extra_body for other native Anthropic options. Prompt caching takes LiteLLM's cache_control markers: on a content part, on a whole message, or on a tool; cache reads show in usage.prompt_tokens_details.cached_tokens. With thinking on, the reply carries reasoning_content and thinking_blocks (streamed: delta.reasoning_content); send the assistant message back as it is with the tool result, since Anthropic needs its signed thinking blocks. Model capability lookups describe the bundled model catalog; they do not promise that this SDK translates every native option.

These control the call itself:

Parameter	What it does
`api_key`	The key for `api_base`, or for a gateway you set. Ignored, with a warning, for the local gateway
`api_base`	Send this call to this address instead of the gateway, in the provider's own format. Warns once per address
`timeout`	Seconds before giving up; the default is 600, as in LiteLLM
`extra_headers`	More HTTP headers on the request
`extra_body`	More fields in the request body, for provider-specific options
`mock_response`	Answer on this computer without calling anything, for tests (Tests)

num_retries (or max_retries) sets bounded request retries for connection failures, timeouts, 408, 429 and 5xx. It defaults to zero; authentication and budget errors are not retried. Streams, completion and native Messages alike, retry only before yielding their first chunk. A route's Retry is separate and runs in the gateway, so enabling both can multiply attempts. Other unknown completion arguments warn and are not sent; see Coming from LiteLLM.

Streaming

for chunk in theagentrouter.completion(model="openai/gpt-5.4-mini", messages=messages, stream=True):
    print(chunk.choices[0].delta.content or "", end="")

Every chunk has a choice, as in LiteLLM. The last chunk, the one with the finish reason, carries usage with the token counts; with stream_options={"include_usage": True} the usage comes instead on one more chunk with an empty choice, as LiteLLM sends it. Provider error events raise an SDK exception, including errors that arrive after partial output.

Async

resp = await theagentrouter.acompletion(model="openai/gpt-5.4-mini", messages=messages)

stream = await theagentrouter.acompletion(model="openai/gpt-5.4-mini", messages=messages, stream=True)
async for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Tools

Tools use the OpenAI format for every provider, Anthropic included:

import json

tools = [{
    "type": "function",
    "function": {
        "name": "search_refunds",
        "description": "Find refunds for a day",
        "parameters": {"type": "object", "properties": {"day": {"type": "string"}}, "required": ["day"]},
    },
}]

messages = [{"role": "user", "content": "How much was refunded yesterday?"}]
resp = theagentrouter.completion(model="anthropic/claude-sonnet-5-5", messages=messages, tools=tools)
call = resp.choices[0].message.tool_calls[0]
args = json.loads(call.function.arguments)

messages.append(resp.choices[0].message.to_dict())
messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps({"total_usd": 42})})
final = theagentrouter.completion(model="anthropic/claude-sonnet-5-5", messages=messages, tools=tools)
print(final.choices[0].message.content)

The Responses API

resp = theagentrouter.responses(input="Summarise yesterday's refunds", model="openai/gpt-6-luna", max_output_tokens=200)
resp.output_text                       # the text of the output messages
resp.output[-1].content[0].text        # the reply as OpenAI sends it, by attribute or key; reasoning may come first
resp.usage.input_tokens
resp._hidden_params["response_cost"]

LiteLLM's responses and aresponses, through the gateway's /v1/responses. Agent Router serves the Responses API for OpenAI models, so other providers are refused with BadRequestError; use completion for them. instructions, tools, tool_choice, max_output_tokens, temperature, reasoning, previous_response_id and the other Responses arguments are passed on. Streaming is not supported yet.

The Messages API

resp = theagentrouter.anthropic.messages.create(
    model="anthropic/claude-sonnet-5-5",
    max_tokens=1024,
    system=[{"type": "text", "text": policy, "cache_control": {"type": "ephemeral"}}],
    thinking={"type": "adaptive"},
    messages=[{"role": "user", "content": "Can I get a refund after 40 days?"}],
)
resp["content"]                        # thinking and text blocks, as Anthropic sends them
resp["usage"]["cache_read_input_tokens"]
resp._hidden_params["response_cost"]

for event in theagentrouter.anthropic.messages.create(..., stream=True):
    print(event)                       # b'event: content_block_delta\ndata: {...}\n\n'

LiteLLM's anthropic.messages.create and acreate (also anthropic_messages, the async one), through the gateway's /anthropic/v1/messages. The request and the reply are Anthropic's own format, so every Messages feature works: prompt caching, thinking, tools, citations and anthropic-beta headers (in extra_headers). As in LiteLLM, what a model refuses is changed first. thinking and reasoning_effort follow the table in Parameters, and output_config.effort becomes a token budget before Opus 4.6 and Sonnet 4.6. Opus 4.5 keeps the effort level and gets no adaptive thinking. Sampling options raise UnsupportedParamsError from Opus 4.7 on, unless drop_params=True. Before 4.6, a temperature other than 1 is dropped while thinking is on. metadata keeps only user_id. speed, compaction, context_management, output_format, output_config.format and safeguards get the anthropic-beta header they need, beside any of yours. The reply is a dict, as in LiteLLM, that also reads by attribute (resp.content[0].text). With stream=True you get the server-sent events as bytes, as in LiteLLM.

Agent Router serves the Messages API for Claude, so other providers are refused with BadRequestError; use completion for them. Routes are refused too, because the local gateway answers a route in the OpenAI format; use completion with a route. completion also sends Claude in the Messages format, and translates it to and from OpenAI's shape; use anthropic.messages.create when your code already speaks Anthropic's format.

Embeddings

resp = theagentrouter.embedding(model="openai/text-embedding-3-small", input=["refunds", "returns"])
resp.data[0]["embedding"]               # the first vector; resp.data[0].embedding works too
resp.usage.prompt_tokens
resp._hidden_params["response_cost"]    # US dollars, from the model list

aembedding is the async version. dimensions, encoding_format and user are passed to the provider. Embeddings go where a completion for the same model goes: through the local gateway or one you set. OpenAI, Gemini, Mistral, Together and Bedrock's Titan models have embeddings; Anthropic has no embeddings API.

What a call returns

What completion returns reads like LiteLLM's, both as attributes and as a dict:

resp.choices[0].message.content
resp["choices"][0]["message"]["content"]
resp.usage.total_tokens
resp.to_dict()

model_dump(mode="json", exclude_none=True) works too: it leaves out optional fields that are None, and keeps null values inside payload dictionaries and lists. Serialization differences when migrating.

resp._hidden_params says where the call went and what it cost, as LiteLLM's does:

Key	Example
`route`	`local gateway`, `gateway` (one you set) or `api_base` (an endpoint named on the call)
`sent_to`	`http://127.0.0.1:52413/anthropic/v1/messages`: the address the SDK sent the request to, here the local gateway
`llm_provider`	`anthropic`
`model_sent`	`anthropic/claude-sonnet-5-5`: the model name in the request, as you wrote it. The gateway passes the provider its own name
`response_cost`	`0.000123`, in US dollars, from the model list. `completion_cost(resp)` gives the same. `None` when the model has no known price, such as a route name on a gateway you set, as in LiteLLM; `completion_cost(resp)` then warns and returns 0
`served_by`, `route_name`	For a route: the model that answered, and the route's name

Errors

Errors keep LiteLLM's names and carry status_code, model and llm_provider:

from theagentrouter import exceptions

try:
    theagentrouter.completion(model="openai/gpt-5.4-mini", messages=messages)
except exceptions.ContextWindowExceededError:
    ...
except exceptions.RateLimitError:
    ...

Error	When
`UnsupportedParamsError`	An option cannot be translated to the selected request format, or the model refuses it
`BadRequestError`	The request is wrong, for example a model name the SDK cannot place with a provider (`acme-model`), or a bare `gemini-…` name
`ContextWindowExceededError`	The messages are too long for the model
`ContentPolicyViolationError`	The provider refused the content
`AuthenticationError`	The key is missing or wrong
`PermissionDeniedError`	The key cannot use this model
`NotFoundError`	The model does not exist
`RateLimitError`	Too many requests
`BudgetExceededError`	A gateway's budget is used up. It is a kind of `RateLimitError`
`Timeout`	No answer within `timeout`
`APIConnectionError`	The provider or gateway could not be reached, or the local gateway could not start (the message says why)
`InternalServerError`, `BadGatewayError`, `ServiceUnavailableError`	The provider or gateway failed

Cost and model details

theagentrouter.completion_cost(resp)                      # US dollars for this response
theagentrouter.get_model_info("openai/gpt-5.4-mini")       # context window, prices, features
theagentrouter.supports_function_calling("anthropic/claude-sonnet-5-5")
theagentrouter.model_cost["openai/gpt-5.4-mini"]           # the raw entry, as in litellm.model_cost

These read the model list that ships with each release (theagentrouter/agent-router-models): OpenAI, Anthropic, Gemini and Groq models, and DeepInfra and xAI models for prices only. It has no Bedrock, Mistral, DeepSeek or Together models. max_tokens and max_output_tokens are the output limit where the list has one, and None where it has not. For OpenAI models without an explicit input limit, max_input_tokens is the context window less the output limit, since OpenAI's window includes the output. The SDK never downloads the list while it runs. get_model_info raises NotFoundError for a model that is not on the list.

The list also marks models deprecated. A model it still offers has a deprecation_date (the date it retires, LiteLLM's name for it), and the first call to it in a program warns with the list's reason, such as anthropic/claude-sonnet-4-5 is deprecated and retires on 2026-11-30: replaced by claude-sonnet-5-5. A model it no longer offers is not served by the local gateway: a call raises NotFoundError with the reason, such as deepinfra/Qwen/Qwen3-Max is no longer offered: Retired on DeepInfra on 2026-10-10. Replacement: Qwen/Qwen3.8-Max. A gateway you set decides for itself.

Adding models

register_model adds models or changes their details, in LiteLLM's format, as litellm.register_model does:

theagentrouter.register_model({
    "ft:gpt-5.4-mini:acme": {"litellm_provider": "openai", "mode": "chat", "input_cost_per_token": 3e-07, "output_cost_per_token": 1.2e-06},
})

# Or LiteLLM's whole list, about 4,000 models from every provider:
theagentrouter.register_model("https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json")

# Or a newer copy of the list this SDK ships, one file per provider:
theagentrouter.register_model("https://raw.githubusercontent.com/theagentrouter/agent-router-models/main/openai.json")

A registered model gets prices and details, a rule in the local gateway when it next starts, and a rule in local_gateway.export_config(). Registrations last for the program that makes them.

Streams put back together

theagentrouter.stream_chunk_builder(chunks) turns the chunks of a streamed call back into one response, as LiteLLM's does: each choice on its own, with text and reasoning joined, tool calls joined by their index, thinking blocks kept, the last finish reason, and the last usage.

Callbacks

Code that sets LiteLLM's callback lists keeps working through theagentrouter.compat.litellm:

from theagentrouter.compat import litellm

def log(kwargs, response, start_time, end_time):
    print(kwargs["model"], kwargs["response_cost"], response.usage.total_tokens)

litellm.success_callback = [log]     # also failure_callback, and callbacks for logger objects

A function is called as fn(kwargs, response, start_time, end_time); a logger object gets log_success_event and log_failure_event, or their async_ versions under acompletion. They run in your program after each completion, after a stream ends (with the response put back together), and on errors. A callback that fails warns and does not stop the call. LiteLLM's built-in integrations, named by string ("langfuse"), are not here: the gateway's OpenTelemetry traces and metrics record calls.

Counting tokens

theagentrouter.token_counter(model="anthropic/claude-sonnet-5-5", messages=messages, tools=tools)
theagentrouter.token_counter(model="openai/gpt-5.4-mini", text="Summarise yesterday's refunds")

LiteLLM's arguments. For OpenAI and Anthropic models, with or without the provider in the name, the gateway counts: Agent Router asks the provider (/responses/input_tokens, /messages/count_tokens), so the number is what the provider will charge for. Every other provider, a name the SDK cannot place with a provider, and a count the gateway cannot give are estimated on this computer the way LiteLLM does: its rules for messages and tools, with tiktoken. Install it with pip install theagentrouter[tokens]; without it the estimate is about four characters a token, and the SDK warns once.

The gateway's count is a request each time. Code that counts before every turn can set AGENTROUTER_TOKEN_COUNT=estimate to always estimate, with no request, as LiteLLM does. litellm.disable_token_counter = True makes it return 0, as in LiteLLM.

Tests

mock_response works as in LiteLLM: the call is answered on your computer, so no gateway starts and no provider is called.

resp = theagentrouter.completion(model="openai/gpt-5.4-mini", messages=messages, mock_response="Offline reply")
resp.choices[0].message.content          # "Offline reply"; usage is 10 prompt and 20 completion tokens

theagentrouter.completion(model="openai/gpt-5.4-mini", messages=messages, mock_response="litellm.RateLimitError")  # raises RateLimitError

It also takes an exception to raise, a dict or ModelResponse as the whole response, stream=True (the text comes three characters at a time, with usage on the last chunk) and acompletion. mock_tool_calls adds tool calls, mock_timeout=True with a timeout raises Timeout, and mock_delay waits first. Callbacks run as for any call. To test the request the SDK sends, use a test server with api_base instead.

Gateways

The SDK never calls a provider itself. A call goes to the first of these that applies:

api_base on the call. That endpoint, for this call only, in the provider's own format (LiteLLM's argument).
A gateway you set. With AGENTROUTER_BASE_URL, every call goes there.
The local gateway. Agent Router on this computer, started by the first call (AGENTROUTER_AUTOSTART=0 turns that off). If the local gateway is not installed, as on Windows, the call fails with a message that says to set AGENTROUTER_BASE_URL.

A gateway you set and the local gateway get the same requests, so code that works with one works with the other:

Call	Path on the gateway	Format
`completion` to Claude (`anthropic/…` or `claude-…`)	`/anthropic/v1/messages`	Anthropic Messages, so prompt caching and thinking work
`completion` to any other model	`/v1/chat/completions`	OpenAI Chat Completions
`embedding`	`/v1/embeddings`	OpenAI
`responses`	`/v1/responses`	OpenAI Responses
`anthropic.messages.create`	`/anthropic/v1/messages`	Anthropic Messages, as you wrote it
`token_counter`	`/anthropic/v1/messages/count_tokens` for Claude, `/v1/responses/input_tokens` for OpenAI

The model name is sent as you wrote it: anthropic/claude-sonnet-5-5 or claude-sonnet-5-5. The gateway passes the provider its own name.

A gateway you set

A remote gateway: an Agent Router your team deploys, or your company's Agent Router Enterprise.

export AGENTROUTER_BASE_URL=https://gateway.acme.example/v1
export AGENTROUTER_API_KEY=...      # the gateway's key, not a provider's

Set the gateway's address with or without the /v1 that OpenAI clients add; the SDK adds the paths above. Every call goes there and the local gateway never starts. Provider keys stay on the gateway, so this computer needs none. The key goes as Authorization: Bearer on OpenAI-format calls and as x-api-key on Claude calls, as Anthropic's own clients send it. Routes defined in code set up the local gateway; that gateway uses its own routes of the same name: see Routes and a gateway you set.

The local gateway

It comes with pip install theagentrouter on macOS 13 or newer on Apple chips and on Linux with glibc 2.36 or newer (not with theagentrouter-core, the SDK alone). There is nothing else to run. The first call starts Agent Router's aigw on a free port chosen at start, answering only programs on this computer, and says so once:

theagentrouter: started the local gateway on http://127.0.0.1:52413 (providers from your environment: anthropic, bedrock, openai).

The port changes from start to start, so another Agent Router gateway on 1975 (your own aigw run, for example) never blocks it. Every program using the SDK finds the running gateway by itself; to reach it from another tool, read the port from that line or from port in ~/.agentrouter/local-gateway/state.json. Two Agent Router gateways still cannot run on one computer: aigw keeps two internal ports fixed (1061 and 18000), so with another Agent Router gateway running the SDK says so and asks you to stop it or set AGENTROUTER_BASE_URL.

The SDK sets the gateway up from the provider keys in your environment. The gateway holds the keys: each is in a private file it reads (mode 0600), never in its config file.
It starts with a rule for every model in the SDK's bundled model list for those providers. A model name without a rule restarts a running gateway with one, as LiteLLM sends any name to the provider. Register custom models with register_model before the gateway starts to avoid the restart.
AGENTROUTER_AUTOSTART=0 stops the SDK from starting it.
A gateway set with AGENTROUTER_BASE_URL always wins, and then the SDK starts nothing.
Its files, including its log aigw.log, are in ~/.agentrouter/local-gateway/.
The very first start downloads Envoy (about 100 MB, into ~/.local/share/aigw), so it needs the internet and can take a minute.
Adding a model or provider needs the keys of every provider already running; without them the SDK raises and leaves the gateway running.
It keeps running after your program ends. To stop it: python -c "from theagentrouter import local_gateway; local_gateway.stop()". The next call starts it again.
To load a new SDK version or a rotated provider key, stop it when other programs have finished; the next call starts it again.

The local gateway is for development. Its known limits, such as a new route, model or provider restarting it and interrupting other programs, are listed in How the local gateway works.

Provider	Set in the environment
AWS Bedrock	`AWS_PROFILE`, or `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`; and `AWS_REGION`
Azure OpenAI	`AZURE_API_KEY` and `AZURE_API_BASE`; optionally `AZURE_API_VERSION`
OpenAI, Anthropic, Gemini, Groq, Mistral, DeepSeek, Together	The key from Models and keys

Routes: fallbacks, retries and names of your own

A route is a model name of your own with the models behind it. The local gateway serves routes.

import theagentrouter as ar

ar.route(
    "chat",
    primary="openai/gpt-5.4",
    fallbacks=["anthropic/claude-sonnet-5-5"],
    retry=ar.Retry(attempts=2, on=(429, 500, 502, 503, 504)),
)
resp = ar.completion(model="chat", messages=messages)
resp._hidden_params["served_by"]   # the model that answered
resp._hidden_params["route_name"]  # "chat"

The local gateway does the retries and the fallbacks, as Agent Router Enterprise does: the SDK sends one request, named after the route.

Retry: the gateway tries each tier attempts more times on the listed status codes and on connection failures, with backoff seconds before the first retry, doubling up to 10 seconds, and an optional per-try timeout.
Fallback: when a tier still fails with one of those codes, or cannot be reached, the gateway moves to the next tier. A stream falls back only before its reply starts.
Sharing: primary=["openai/gpt-5.4", "openai/gpt-5.4-mini"] puts two models in one tier, which share the traffic equally. A fallback can be a list too. LiteLLM's Router shares a group by weight, rpm or tpm, as LiteLLM does: see Coming from LiteLLM.
served_by is read from the model name in the reply, so it is missing when a provider answers with a name that does not start with the model's own.
Routes are kept in ~/.agentrouter/routes.json and shared by every program on the computer, like the local gateway. Defining a route again replaces it; ar.list_routes() and ar.remove_route(name) list and remove them.
A route can take a model's name, such as openai/gpt-5.4 or gpt-5.4, as a group in the LiteLLM proxy's model list can. Every program's calls to that model then go through the route.
Define routes without local provider keys or a companion package when exporting configuration. The machine that starts the gateway needs credentials for every provider in the saved routes.
A route sends one request, and any of its models may get it. In a route of OpenAI or Azure with Anthropic, Gemini, Groq or Bedrock models, max_tokens goes as max_completion_tokens, which they all take. Any other route sends what you wrote. If an OpenAI model refuses max_tokens, the SDK makes the call once more with max_completion_tokens. The gateway removes temperature and top_p before a request reaches a newer Claude model on Anthropic or Bedrock (see Parameters), and an effort level before it reaches Claude on Bedrock.
thinking and top_k have no field in the OpenAI format, so they are not sent in a route, and the SDK warns once. extra_body reaches every model in the route.
A route reaches Anthropic through its OpenAI-compatible endpoint, which has its own rules: a JSON schema in response_format needs "strict": true, and prompt caching does not apply. A forced tool call is still refused on Sonnet 5.5, Opus 5.5 and Fable 5.1, even with drop_params.
Limit: limit=ar.Limit(60, per="minute") lets at most 60 requests a minute through the route; the gateway answers the rest with 429 (RateLimitError). per is "second", "minute", "hour" or "day". The count covers every program on the computer that uses the route. It counts requests, not tokens (see Known limits).

LiteLLM's Router works on top of routes: see Coming from LiteLLM.

Running the same gateway for a team

local_gateway.export_config() gives the config the SDK writes for the local gateway, to run on a server. Developers then set AGENTROUTER_BASE_URL to that server, and the same code works there.

from pathlib import Path
from theagentrouter import local_gateway

Path("config.yaml").write_text(local_gateway.export_config(port=1975))

aigw is the program inside the local gateway package, and it is not on your PATH. On a server, pip install theagentrouter-local-gateway and find it as below, or download it from Agent Router's releases.

export AGENTROUTER_API_KEY=...             # the server client key; set it on clients too
AIGW=$(python -c "from theagentrouter_local_gateway import binary_path; print(binary_path())")
ENVOY_VERSION=1.39.4 "$AIGW" run config.yaml   # provider keys come from the server environment

What the file holds:

Providers: those whose keys are set on your computer, or the ones you name: export_config(providers=["openai", "anthropic"]).
Models: a rule for every model in the SDK's bundled model list for those providers, with and without the provider in the name, plus any you added with register_model. The server serves only these. The bundled list has no Bedrock, Mistral, DeepSeek or Together models, so for those providers name the models (export_config(models=["mistral/mistral-large-latest"])) or register LiteLLM's list first (see Adding models). Export again when the models change.
Routes: the routes defined on your computer, with their fallbacks, retries and limits, under the names code calls them by: a LiteLLM Router group's route is named after the group, chat, not router-1a2b3c4d:chat. Two routes that would get the same name raise BadRequestError; remove one with ar.remove_route(). A client cannot see which models are behind a route's name, so max_tokens goes as written. If an OpenAI model refuses it, the SDK makes the call once more with max_completion_tokens. Pass max_completion_tokens yourself to save that extra request. A route name shaped like a Claude model, such as claude-chat-5, goes to the gateway as Claude, so use another name for a route.
No key values: only references such as ${OPENAI_API_KEY}, which aigw fills in from the server's environment.

Exported configurations require a client key by default. Set AGENTROUTER_API_KEY on the server and on each client. The gateway accepts Authorization: Bearer <key> or x-api-key: <key>, refuses a missing or wrong key with 403, and removes the client key before forwarding. A server started without AGENTROUTER_API_KEY refuses every request. The SDK raises that refusal as PermissionDeniedError, and its message names AGENTROUTER_API_KEY on both sides. export_config(authentication=False) explicitly exports an open endpoint. Terminate TLS in front of the server when callers use an untrusted network.

Routes and a gateway you set

Routes defined in code set up the local gateway only. With AGENTROUTER_BASE_URL set, the same code runs unchanged:

route() and LiteLLM's Router set up nothing on this computer and write nothing to routes.json.
A call to a route sends its name to that gateway, and the SDK warns once. That gateway's own route of that name, with its fallbacks, retries and limit, applies. A Router sends the group name, chat.
A gateway made with export_config() has this computer's routes under those names, so code tested against the local gateway calls the same routes there.

Settings

Setting	What it does
`AGENTROUTER_BASE_URL`	Send every call through this gateway instead of the local one
`AGENTROUTER_API_KEY`	The gateway's key
`AGENTROUTER_HOME`	Folder for the SDK's files (default `~/.agentrouter`)
`AGENTROUTER_AUTOSTART`	`0` stops the SDK from starting the local gateway
`AGENTROUTER_TOKEN_COUNT`	`estimate` makes `token_counter` estimate on this computer instead of asking the gateway

AGENTROUTER_BASE_URL, AGENTROUTER_API_KEY and AGENTROUTER_TOKEN_COUNT can also be kept in AGENTROUTER_HOME/config.json, as base_url, api_key and token_count. Environment variables win over the file. When AGENTROUTER_BASE_URL names a different gateway from the file, the file's key is not used, so it never goes to a gateway it was not made for.

To check that the keys a model needs are set, use LiteLLM's validate_environment(model). For the local gateway it checks what the gateway needs for that provider (for Bedrock: a profile, or a key and secret, and a region); with a gateway set, it checks AGENTROUTER_API_KEY.

Navigation Menu

Repository navigation

usage.md

usage.md

Usage

Contents

A first call

Models and keys

Parameters

Streaming

Async

Tools

The Responses API

The Messages API

Embeddings

What a call returns

Errors

Cost and model details

Adding models

Streams put back together

Callbacks

Counting tokens

Tests

Gateways

A gateway you set

The local gateway

Routes: fallbacks, retries and names of your own

Running the same gateway for a team

Routes and a gateway you set

Settings

FilesExpand file tree

usage.md

Latest commit

History

usage.md

File metadata and controls

Usage

Contents

A first call

Models and keys

Parameters

Streaming

Async

Tools

The Responses API

The Messages API

Embeddings

What a call returns

Errors

Cost and model details

Adding models

Streams put back together

Callbacks

Counting tokens

Tests

Gateways

A gateway you set

The local gateway

Routes: fallbacks, retries and names of your own

Running the same gateway for a team

Routes and a gateway you set

Settings