Browse the results: onthebench.ai - sortable tables, protocol matrix, charts, methodology.
A fair, reproducible benchmark for self-hostable AI gateways - LiteLLM (Rust & Python), Bifrost, Portkey, Kong, Helicone, GoModel, Busbar, and whatever else you drop in. Same box, same mock, same load, same cpu pin, for every gateway. One command runs it; the charts regenerate from raw results; every source ref is pinned in the open (container images by tag) and the built commit or image tag is stamped into the output.
Chart bars are coloured by implementation language (a neutral property), never by rank or brand, so
the colour can't be misread as favouring the sponsor. Every number regenerates from committed JSON. If a
gateway can't serve the endpoint, the result says served: false with the evidence, instead of quietly
dropping it. Add your gateway (or fix how we run yours) with a one-file manifest.
On the Bench is built and operated by the Busbar team, and busbar is one of the entrants. It stays honest structurally: one shared harness with no per-gateway special-casing, every number from the committed JSON, fully open source. Don't take our word for it - read the code and re-run it.
Ran on: AWS m7g.4xlarge Graviton3 (ARM64, 16 cores - Graviton does not hyperthread, so
1 vCPU = 1 core), Ubuntu 24.04. The gateway under test is pinned to 4 cores (an
m7g.xlarge-class slice, the 4-vCPU class gateways commonly publish against); the load generator
gets 6 and the mock gets 6, so neither can starve the other or bottleneck the gateway. One
dedicated box per gateway. The exact instance type, core split and arch are recorded in every
snapshot and shown on the board.
The board is at onthebench.ai - the complete field, live, regenerated
from the committed snapshots in results/snapshots/. Every gateway that could
not serve a pairing is marked there rather than hidden, and every absent number carries the reason it
is absent.
It ranks at any of the declared tail-latency bounds and re-sorts in front of you, which is the part a static page cannot do: the ranking depends on the bound you accept, and there is no single answer to bake in.
Numbers land as runs complete. Re-run
run-on-ec2.shand the board updates from the new snapshots.
To run locally on your own box:
- Rust (
cargo) - builds the mock (mock/, a hyper server that answers all six wire protocols; the runner checks its baseline against the frozen rig calibration), the engine (including its ownotb loadgenload generator), plus the gateways compiled from source (LiteLLM-Rust, Helicone). Source builds also needcmake,clang, andprotobuf-compiler. - Docker - for the container-based gateways (Bifrost, Kong, GoModel, One-API, …).
- Python 3 - runs the auditors and verifiers (
bench-audit.py,verify-*.py). No plotting library is needed: the charts are drawn by the site in the browser (the static matplotlib pipeline was retired in 704fcf6). - Container gateways use official ARM64 images pinned by tag and digest. Native builds pin named release commits.
Cloud runs use Python's AWS SDK, bash 4+, SSH and rsync. Install lib/aws-sdk-requirements.txt in a virtual environment and set OTB_AWS_PYTHON to that interpreter. Set OTB_AWS_ACCOUNT_ID explicitly and select OTB_AWS_PROFILE, standard SDK credentials, or an authorized workload role. No credentials belong in the repository. See frozen rig and credential setup.
run-field.py queries current on-demand compute, gp3 and public IPv4 prices. --placement cheapest can distribute identical boxes across eligible regions; --placement same-region selects the cheapest region with enough quota for the whole field. Owned VPCs, subnets and routing are created and torn down with the run. Other workloads' networks are never adopted.
export OTB_AWS_ACCOUNT_ID=YOUR_ACCOUNT_ID
export OTB_AWS_PROFILE=YOUR_PROFILE
export OTB_AWS_PYTHON=/path/to/venv/bin/python
"$OTB_AWS_PYTHON" run-field.py --placement cheapest --state-dir /path/to/private-state --rig-dir /path/to/verified-rig
# Later, run just one updated gateway with the same frozen rig:
"$OTB_AWS_PYTHON" run-field.py busbar --placement same-region --state-dir /path/to/another-private-state --rig-dir /path/to/verified-rigClone, then run one script. Everything is at the repo root, and every gateway provisions itself
from the ref pinned in its own gateways/<name>/ directory - a Docker image tag in
definition.json, or (for native gateways) a source commit in build.sh. Nothing to fetch
by hand for any of them.
git clone https://github.com/GetBusbar/benchmarking && cd benchmarking
./run-on-ec2.sh # every gateway, one fresh box each, in parallel
./run-on-ec2.sh litellm-rust bifrost # a subset
./run-on-ec2.sh harvest # pull results from boxes a dead run left behind
./run-on-ec2.sh kill # terminate every box nowThe lower-level runner requires explicit SDK credentials, a verified rig, ssh and rsync. Every gateway gets its own fresh
m7g.4xlarge, so no gateway inherits another's page cache or disk state, and the wall clock is
the slowest single gateway rather than the sum of all of them.
One run measures latency, throughput, memory and cost for every gateway on the same box. Out
comes one snapshot per gateway in results/snapshots/ - every rung of every sweep, every RSS sample,
and the engine commit that measured it - which is what the board is generated from.
run-on-ec2.sh launches a Graviton box, installs everything, runs the full suite, pulls results
back, and terminates the box. Use run-field.py above for credential verification and automatic network management.
./run-on-ec2.sh # every gateway, one-click (Graviton/arm64)
./run-on-ec2.sh litellm-rust bifrost # a subset
ARCH=x86 ./run-on-ec2.sh # the whole field on Intel instead - one flipArchitecture is one knob. ARCH=arm64 (default) runs the field on Graviton (m7g); ARCH=x86
runs the same field on Intel (m7i). One switch picks the instance family and the matching Ubuntu
AMI; every gateway builds/pulls for that arch on its own box, and the arch is recorded in each result
so runs from different arches never get conflated. Every gateway here runs natively on both -
including Helicone and One-API, which publish x86-only images (we build Helicone from source and pin
One-API to its arm64 tag), so nothing is quietly arm64-only or x86-only.
A gateway that can't be stood up (unreachable, or needs infra a single container can't provide) is
recorded served: false and shown as such - never silently dropped. To pin a different build of any
gateway, edit launch.image in its own gateways/<name>/definition.json, or for a source-built one the
COMMIT in its build.sh (which also reads a <NAME>_COMMIT override); the image tag or built commit
is stamped into every result.
The current suite checks 36 ingress/upstream protocol pairs for each gateway. Served pairs can each run throughput searches, memory soaks, streaming checks, added-latency windows and CPU measurements. A single gateway can therefore take several hours; unsupported pairs finish much sooner. The number of supported pairs and search convergence matter more than build time alone.
The full field runs on separate boxes in parallel, so its elapsed time follows the slowest gateway. A later single-gateway rerun avoids repeating the other entrants, but still runs that gateway's full protocol matrix on a fresh box. Follow the per-gateway phase logs and the completed-result list for progress; recorded phase durations provide evidence for future estimates.
The runner terminates each box after harvesting its results and arms a separate maximum-lifetime shutdown as a backstop. Keep the frozen rig and calibration when rerunning one gateway; changing the instrument calls for a new field-wide experiment.
Passthrough perf (folded into matrix/) - what the system can do (the metrics that matter most):
-
added latency (µs) - p99 the gateway adds over the upstream at concurrency 1 (gateway p99 − direct-to-mock p99). Microseconds, because at this scale ms hides the story.
-
the throughput frontier - requests/sec at each tail latency you are willing to accept: 1 ms, 5 ms, 10 ms, 50 ms, 100 ms, plus one reading with no latency bound at all. Each is the most req/s the cell carried while 99% of requests finished under that bound AND it failed none it accepted. Read as: "18,995 req/s while 99% of requests finished under 10 ms."
This replaced a single "RPS ceiling with p99 under a chosen bar", and the reason is that a scalar cannot express a tradeoff. Throughput and tail latency rise together with concurrency, so "the throughput" is a POINT ON A CURVE and picking the point for the reader hides the shape. Two real cells from the same board: agentgateway carries 23,630 req/s at a 1 ms tail and gains only 7% by dropping the bound entirely, while apisix nearly DOUBLES between 1 ms and 5 ms. Published as one number those looked comparable; they are not the same machine.
All six readings come off ONE concurrency sweep, published in full beside them, so every reading is re-derivable from the rungs rather than taken on trust. The sequence is monotone by construction: relaxing a bound only adds rungs to the set the maximum is taken over.
-
cost per request (µs of CPU) - user plus system time across the gateway's whole process tree, sampled as the difference across one load window and divided by the requests that window completed. Measured at ONE concurrency held identical for every gateway, published beside it, because matched load is impossible across a field spanning double digits to five figures of req/s.
This exists because peak throughput is a SATURATION number: once a gateway fills the cores it was given, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money. On the current board it spans 89 µs to 199,333 µs per request, a range peak throughput cannot express.
Published with it: the utilisation of the pinned cores over that same window, which is what makes a peak interpretable. Near 100% the peak is the gateway's own CPU wall; well below it, something else is holding it and more CPU would not help. A window with any failure publishes no cost at all (CPU divided by only the successes would describe the failures, not the work), and a window on a box taking major page faults has its cost flagged as a harness fault, because what was timed is the disk.
The matrix's best same-dialect diagonal cell IS this passthrough measurement (the retired standalone
perf/ suite is gone; gen-data projects the board's headline perf from the matrix cell).
stream/ (opt-in: SUITES="stream matrix" ./run-on-ec2.sh) - what the gateway adds to a
token stream. The mock answers stream:true with a valid SSE stream: a role chunk, then 64
content deltas paced at 20 ms, then finish + [DONE] (Anthropic event shape on /messages).
Against that fixed pace, per gateway:
- added TTFT (µs) - time to the first SSE event through the gateway (not necessarily a content token: a role chunk or other scaffolding event counts) minus direct-to-mock, at concurrency 1.
- added inter-frame latency (µs) - p50/p99 of the gateway's inter-event gap over every SSE event of the stream (scaffolding events included) minus the direct-to-mock gap. Both sides carry the mock's 20 ms pace and the same timer jitter, so the subtraction isolates the gateway's per-frame overhead.
- streams sustained - the highest concurrent stream count where every expected content frame
arrives, no stream stalls past 10x the mock's pacing interval, and the stream error rate stays
under 0.1%; plus the frames/sec carried there. A proxy that drops a frame has dropped a user's
token, so the sustained ceiling is the last concurrency before anything is lost. Beside it, the
mock's own frames/sec at the operating concurrency (derived from its declared pacing) is published
as
streams_sustained_mock_ceiling, withstreams_sustained_headroom= carried / ceiling (a ratio that may exceed 1.0); nothing is flagged or withheld on it.
A gateway that answers 200 but buffers the stream (never frames) is recorded with its
stream_served naming the absence reason ("untestable", "rig_limited", and the rest of the
Absent vocabulary), not with a bare false - this engine never writes false here, since that
would assert the gateway does not stream, which no observation establishes. false is still
representable in results/stream/<gateway>.json only so older artifacts predating this vocabulary
still parse. The stream_* fields are additive; existing result files stay valid. Knobs: STREAM_CHUNKS,
STREAM_INTERVAL_MS, STREAM_CHUNK_BYTES, STALL_X, SWEEP, SWEEP_DUR.
xlate/ (opt-in: SUITES="xlate matrix" ./run-on-ec2.sh) measures protocol translation.
The client speaks Anthropic (POST /v1/messages, a Messages body, anthropic-version and
x-api-key headers) while the upstream mock speaks OpenAI on the manifest's GW_PATH, so the
gateway must translate the request out and the response back. The mock is untouched; that is the
point. The lane repeats the c1 added-latency measurement and the sustained-RPS-@20ms sweep on the
translation path and writes results/xlate/<gateway>.json (xlate_added_latency_p99_us, and the
translation cell's own throughput frontier). One honest asymmetry, recorded in the JSON as
xlate_baseline_shape: openai: the mock does not translate, so the direct baseline is the OpenAI
shape straight to the mock, and the added-latency figure therefore includes the translation work,
which is exactly what this lane exists to price. Many gateways cannot serve Anthropic ingress
against an OpenAI upstream at all; one probe decides, and a non-2xx, a non-Anthropic body, or the
mock's own canned /messages body (proof the path was proxied verbatim, not translated) is
recorded xlate_served: false with the probe status and body snippet as evidence, never a crash.
Manifests may override GW_ANTHROPIC_PATH (default /v1/messages) and add
GW_ANTHROPIC_AUTH_HEADER; the load generator sends the token as both Authorization: Bearer and
x-api-key, so most manifests need nothing.
matrix/ (opt-in: GATEWAY=<name> matrix/run.sh) is the protocol support matrix, a
capability suite rather than a latency suite. One gateway is probed across six ingress protocol
shapes (OpenAI chat completions, OpenAI Responses, Anthropic Messages, Gemini generateContent,
Cohere v2 chat with a v1 fallback, Bedrock Converse) while the upstream mock stays fixed on the
OpenAI shape, so every non-OpenAI cell is a translation claim: the gateway must convert the request
out and the response back. One probe per cell decides it: the engine's probe accepts any 2xx status
and does NOT validate the reply envelope (it does not check for choices[0].message, a Responses
envelope, an Anthropic content array, candidates[0].content and so on), so a served cell means "the
gateway answered 2xx on that pairing", not "the reply was a well-formed translation". The xlate passthrough guard
generalizes to every cell: the mock answers all six protocols by path, so a gateway that proxies an
ingress path verbatim gets a plausible 200 from the mock's canned constant; every translation cell
rejects that canned body as untranslated passthrough. Bedrock gets one extra honesty rule: real
Bedrock clients sign with AWS SigV4, and a gateway that answers 401/403 to the probe's bearer token
records "unprobed_auth" (distinct from false) with the evidence, because the harness does not
forge signatures and a red it did not earn would be a lie. Each cell writes
{served, status, verdict_note, body_snippet} to results/matrix/<gateway>.json, valid JSON
always, exit 0 always. The full grid is no longer future work: every served cell now carries its own
added latency, its own throughput frontier, its own memory window and its own cost, measured against
every upstream dialect - which is why busbar, the only entrant serving all 36 pairings, takes about
ten hours while a gateway serving one takes minutes. Manifests may override GW_MATRIX_PATH_OPENAI,
GW_MATRIX_PATH_RESPONSES, GW_MATRIX_PATH_ANTHROPIC (defaults to the shared
GW_ANTHROPIC_PATH), GW_MATRIX_PATH_GEMINI, GW_MATRIX_PATH_COHERE, GW_MATRIX_PATH_BEDROCK;
most need nothing. The same guard (rejecting an untranslated passthrough body as a false positive) is
covered directly against the mock's own request_shape_ok in mock/src/main.rs's test suite, gated
in CI by the mock-shape job.
Memory (folded into matrix/) - resident memory across a request's life (matters most at GB scale):
- idle RSS - the MEDIAN over the cold-idle window on a freshly restarted process, sampled before it serves any request (a warm post-sweep process would measure "recovered", not "idle").
- steady-state RSS - the level the RSS settles at while the identical fixed load runs on that cell. A cell whose RSS is not steady over the trailing window when the fixed load ends has no steady state, and publishes its growth rate (MiB/min over the final window) instead: that rate IS the leak rate.
- plateaued / time to plateau / growth rate - whether the RSS was steady over the trailing window at the END of the load, when it first read flat (recorded only; it does not end the load), and how fast it was still moving. The growth rate is published whether or not the cell plateaued, because any threshold admits a leak slower than itself.
- recovered RSS - the RSS at the END of the recovery window after the load stops: does it release,
or stay pinned? A gateway that pools memory and never returns it looks bounded on a boot-time
docker statsbut stays pinned at peak under sustained load. (post_load_rss_mibis a back-compat alias of this field.)
Machine. m7g.4xlarge - 16 real Graviton3 cores (Graviton doesn't hyperthread: 1 vCPU = 1
core). The gateway under test is pinned to 4 cores (= an m7g.xlarge, the 4-vCPU class gateways
commonly publish against); the load generator gets 6 and the mock gets 6, each isolated on
its own cores. That is stricter than a co-located 4-vCPU run where the load tool steals cycles from
the gateway - here the gateway gets a clean 4 cores and neither the harness nor the upstream can
bottleneck it. The pinning is enforced with --cpuset-cpus (containers) and taskset (native), and
verified: a container inspects as CpusetCpus: 0-3 with a matching host affinity. All loopback; no
network noise.
The mock. A deterministic Rust server (mock/) that answers all six wire protocols by path and
holds hundreds of thousands of concurrent requests so it's never the limit. One knob: MOCK_TTFT_MS,
a per-request delay simulating the model doing work.
Latency - instant mock. Added latency is gateway p99 − direct-to-mock p99 at concurrency 1
against a zero-delay mock. Zero base keeps the overhead a clean microsecond delta; a 20 ms base
would just add noise to a sub-millisecond number.
Throughput - a frontier, not a number. A single throughput figure invites "you picked the flattering metric", and it deserved the accusation: whichever tail latency you allow decides the answer, and any one choice is arbitrary. So every cell reports its rate at six declared bounds (1/5/10/50/100 ms and unbounded) off one sweep, and the reader ranks at whichever bound matters to them. Nothing is suppressed and nothing is chosen on the reader's behalf.
This replaced two scalars that were the same sweep summarised twice under a chosen ceiling - and which could invert against each other, because two different algorithms read one set of windows: the "maximum" came out BELOW the "sustained" figure on real cells (aisix 16,232 vs 16,610; bifrost 5,113 vs 5,174). A maximum another reading of the same windows beats is not a maximum. Six maxima over sets that only grow makes that unrepresentable rather than merely policed.
Each reading carries its own evidence: the concurrency it was observed at, the tail it actually produced (never the bound - 4 ms under a 100 ms bound is a different finding from 99 ms), and the concurrency above it that stopped qualifying. When nothing above it was probed at all, the rate is a floor and says so, rather than being discarded for failing to prove maximality.
A rig limit is never charged to the gateway: connections this host could not open (ephemeral ports or descriptors exhausted) are counted separately and the window is discarded unmeasured, so our own port range can never be published as the gateway's ceiling.
Memory. A per-cell memory window, not a synthetic burst (the old standalone 150 KB x 1500c
suite is deleted - it mislabelled itself as 6x6 provenance). Every served cell gets its own window
and no cell is selected: the gateway is cold-started for that cell, its cold idle RSS is sampled
before it serves a single request, then the identical fixed load (same concurrency and payload for
every gateway and every cell, c=32) runs for a fixed MEMORY_LOAD_S (300 s) on every cell, and RSS
is sampled through a recovery window after the load stops. The plateau verdict is taken at the end
of that load, over the trailing 60 s (drift and range both under their thresholds); the first flat
trailing minute is only recorded, as time_to_plateau_s, and never shortens the load - a load whose
length depended on how settled a gateway looked would ask different gateways different questions. Not
plateaued at the end is a published result (plateaued: false plus the growth rate), not an error.
One process lifecycle per cell, so the idle -> steady -> recovered arc is a real curve. The idle,
load, recovery and plateau windows and the load concurrency are engine constants (not environment
tunables); the protocol string and the window lengths travel in the result, so every published label
renders the durations the run actually used. Any RSS the sampler cannot obtain is null, never a fabricated 0, and a rig that
cannot read RSS at all withholds the plateau verdict as null rather than asserting false.
The website and engine share this repository. site/gen-data.mjs projects the measured
snapshots; site/build.mjs validates that bundle and renders the existing dashboard into
complete HTML with the same renderer the browser uses. It also publishes gateway evidence,
Markdown, CSV, JSON, schemas, a sitemap, and immutable raw runs. No benchmark rerun is needed
for a website-only change. The API documentation is at /gateways/data.
npm ci --prefix site --ignore-scripts
node site/build.mjs "$PWD" /path/to/empty/build-directory
OTB_TEST_ARTIFACT=/path/to/empty/build-directory node --test site/publishing.test.mjs
node site/serve.mjs /path/to/empty/build-directoryProduction is built from main, including its committed results and full Git history.
The GitLab workflow on dev is the production controller: trigger it with
DEPLOY_WEBSITE=1 (also scheduled nightly, which alone refreshes GitHub stars). It clones main,
runs checks, and invokes site/publish.mjs. The publisher deploys a Worker with static assets
on onthebench.ai/*; it does not deploy a JavaScript-only Pages shell. The existing analytics
relay routes remain separate. site/wrangler.jsonc declares the route and the PUBLICATIONS
KV namespace, which retains immutable dataset versions across deployments. Exact raw run
versions are recovered from Git history and addressed by SHA-256.
For an explicit release, OTB_BUILD_ROOT=/path/to/scratch node site/publish.mjs runs the same
build and targeted HTTP checks before deploying with Wrangler's configured credentials.
Use --dry-run to build and bundle without publishing. The token needs Workers Scripts,
Workers Routes and Workers KV write permissions for the declared account and zone.
CI runs the existing full site suites; local verification should use the targeted publishing
tests and the actual generated artifact. Website changes pushed to dev run verification
automatically. An API or web pipeline on dev with VERIFY_WEBSITE_REF set to a full commit
SHA checks that release candidate without publishing it. Production promotion still follows
the repository's protected-branch approval policy.
Default comparison pages and their JSON are also built ahead of time. Arbitrary dashboard
query combinations use the shared renderer at request time and can exceed Workers Free's
CPU allowance; production should use Workers Paid for reliable dynamic requests. The
Cloudflare zone must permit automated clients (including Browser Integrity Check settings).
Deploy authorization alone does not grant permission to edit those zone settings. A release
must check both ordinary crawler user agents and a default HTTP-library client against the
live domain. A descriptive user agent can work even when a default library user agent is
blocked before the Worker executes.
site/publish.mjs runs site/check-live.mjs after deployment and reports a failed release
check if the CDN blocks an agent or changes the raw evidence bytes. This is a post-deployment
check: an error does not mean that the previous deployment remains active.
Drop a directory under gateways/ with a definition.json manifest. The runners are
gateway-agnostic; there is nothing else to edit. See gateways/README.md.
- Source refs are config, not defaults buried in a script. Each gateway's ref is pinned in its
own directory, next to its disclosure of what it deviates from and why: a container image tag in
definition.json(tag only, not a digest), or a source commit inbuild.sh(overridable from the environment). The image tag, or the actual commit built, is written into each result'sbuildfield. "You used an old branch" is answerable by pointing at the manifest and the recorded build. - Each gateway is launched the only way it actually serves the endpoint. For example,
LiteLLM-Rust's
/v1/messagesroute only serves theazure_aiprovider and only serves at all under itspython-configreader (the lean env config returns400) - verified against its own source. We launch it that way and record what it costs, rather than quoting an idle number from a config that doesn't serve. The reasoning is ingateways/litellm-rust/definition.json(matrix_note) andgateways/litellm-rust/why. - The mock is deterministic and dumb - it answers any path with a fixed small body (OpenAI shape,
or Anthropic shape for
/messages), so the number is the gateway's cost, not the upstream's. - The chart colors by language, not by name or rank. Every bar takes its gateway's implementation-language colour, the same palette every other surface uses, so no colour marks a winner and none marks the sponsor; order and length carry the measurement.
Published gateway numbers are often hard to reproduce - the hardware isn't disclosed, the config may not actually serve the endpoint, and the chart can't be regenerated from raw data. This repo is built to be the opposite: disclosed hardware, configs that serve (or are recorded as not serving), and every number regenerating from committed JSON. Clone it, run it, and check the work - including ours.