[arekay-nv]

What does this PR do?

Draft until the Linux performance runs and the live-endpoint checks are done.

Implements #540. Records the endpoint's reported token usage
(usage.prompt_tokens / usage.completion_tokens, or SGLang meta_info)
beside the client tokenizer's counts on every run, and lets one setting choose
which basis is the headline. The default stays on the client tokenizer, so
every existing number is unchanged; the server counts and a paired per-sample
comparison (osl_paired) make the #540 gap (inline <think>…</think> vs
structured reasoning_content) visible on every run.

Item	Behavior
Default basis	settings.token_count_source: client (--token-count-source client). Every unsuffixed number, pass/fail decision and exit code matches main for a well-formed endpoint.
Server numbers	Reported beside the primary every run, with a _server suffix (isl_server, osl_server, tpot_ns_server; input_sequence_lengths_server, output_sequence_lengths_server, tpot_server, tps_server). They never decide pass/fail under the default.
Flip	--token-count-source server makes server the primary (unsuffixed, gates the run) and names the other basis *_client. The flip is one schema default (config/schema.py); no key name or code path changes.
Usage request	Streaming OpenAI chat and /v1/completions requests send stream_options: {"include_usage": true} by default. --no-stream-include-usage (YAML settings.client.stream_include_usage: false) opts out; request bytes then equal main's.
Setup error	--token-count-source server together with the opt-out on a streaming OpenAI run raises SetupError (every streamed sample would miss usage and the run would exit 4 after doing all the work).
Other endpoints	SGLang meta_info and non-streaming usage already carry counts. api_type: videogen always resolves to client (INFO log). n > 1 rows are rejected only under a server primary.

Changes from the earlier draft (server default)

Before	Now
Default server; client counts as *_client	Default client (identical to main); server counts as *_server
Basis passed as strings / bool flags in places	TokenCountSource everywhere, keyword-only; names derived by one basis_key rule
Each consumer decided "missing usage" its own way	One rule (classify_server_usage / read_server_usage) shared by the aggregator, post-run OSL, scorer and replays
Run basis read from run_config.token_count_source	Top-level token_count_source in result_summary.json, read by one reader (token_count_source_for_run)
Stubs always sent a usage frame	Stubs send it only when the request asks, in vLLM's order

Out of scope: a live (TUI) display of the other basis, early stopping and
steady-state detection on the other basis, and the RECV_FIRST vs COMPLETE clock
mismatch.

How to review

Size

Category	Files	Added	Removed	Share of added lines
Implementation (src/, scripts/)	39	3,188	664	24%
↳ of which hot path¹	12	573	90	4%
Tests: code	59	8,102	470	62%
Tests: fixtures (JSON/JSONL)	7	1,179	0	9%
Docs (docs/, *.md, examples)	20	674	186	5%
Generated config templates	3	9	3	<1%
Total	128	13,152	1,323

About 2.5 lines of test code per line of implementation. The stub servers
(src/inference_endpoint/testing/, about 320 added lines) count as
implementation here but are test infrastructure.

¹ openai/, sglang/, load_generator/, plus worker.py, adapter_protocol.py and metrics_table.py.

Commits, in reading order

The history is ten commits, one per layer, oldest first (the order of the
Commits tab). Each commit holds its layer's final code and tests, so no
intermediate rework appears.

#	Commit	Layer	Impl +/−	Tests +/−	What to check
1	040d48c8	Archived pre-usage run, pinned on main	—	+200 (+982 fixtures)	Skim. Its assertions are the numbers that must not move. Commit 8 changes only call sites in this test: git diff 040d48c8 e9f2db57 -- tests/unit/metrics/test_archived_run_compat.py.
2	fdb91a44	Core types	+157/−7	+244	basis_key is the only naming rule. A server count is kept only if it is a non-bool int in [0, TOKEN_HDR_HIGH]. EventRecord counts default to UNSET, so a record without them encodes byte-identically to an archived line.
3	df28e3eb	Config	+37/−2	+51/−1	The default is client; flipping it is this one line plus the regenerated templates.
4	28cf4286	Request and response (hot path)	+369/−96	+1,349/−63	The per-frame cost on the OpenAI SSE path is one usage is None test. stream_options goes only on streaming requests with n ≤ 1. A malformed count never drops a frame. SGLang validates prompt_tokens only when it changes. session.py copies counts without re-validating.
5	87e7dfd5	Stub servers and test fixtures	+319/−68	+716/−3	Test infrastructure; skim. Usage frame only when requested, in vLLM's order. The stream layout is pinned byte for byte, so roofline numbers stay comparable.
6	0bbccc22	Aggregator and the usage rule (hot path: metrics_table.py)	+784/−124	+1,626/−118 (+197 fixture)	ServerUsageTrigger runs once per COMPLETE and does no tokenizer work. classify_server_usage is the only definition of missing usage. Primary series keep main's names and registration order. Server TPOT rule and its skips.
7	dff68380	Report, pipeline, setup checks	+521/−82	+2,250/−148	Under the default, every unsuffixed field equals main's and report.txt keeps main's rows, with the other basis after them. Completeness follows the primary basis. Setup checks: resolve_token_count_source, _check_multi_choice_rows.
8	e9f2db57	Post-run tools	+1,001/−285	+1,650/−139	Every tool goes through read_server_usage and replay_basis. Basis resolution order: summary, then config.yaml, then client. A malformed log fails loudly.
9	36265fa7	Docs	—	—	+639/−180 of prose; check it matches the code.
10	f07b0d2b	Roofline opt-out option	—	+19/−1	Perf-suite flag only.

Every commit imports cleanly, but only the head is tested as a whole:
tests/test_helpers.py (commit 5) imports names that commits 6 and 7 add.

If time is short

1. Commit 4, then metrics_table.py in commit 6: the hot path.
2. Commit 7: what a default run reports.
3. Commit 2: the types everything else uses.

Skim or skip: the JSON fixtures (commits 1 and 6), the generated templates
(commit 3), the stub servers (commit 5) and the docs (commit 9).

Checking it locally

git fetch origin feat/server-token-counts && git checkout FETCH_HEAD
uv sync --extra dev --extra test

# Parity with main: archived run, report, replay and event-record pins (14 tests)
uv run pytest --no-cov tests/unit/metrics/test_archived_run_compat.py \
  tests/unit/metrics/test_report_builder.py \
  tests/unit/metrics/test_steady_state_diagnostics.py tests/unit/core/test_record.py \
  -k "archived or client_primary_unsuffixed or mains_up_to or identical_with_and_without or byte_identical_to_archived"

# Both bases on a real run. The echo server counts characters, as the char tokenizer does.
uv run python -m inference_endpoint.testing.echo_server --port 8765 &
uv run inference-endpoint benchmark offline --endpoints http://localhost:8765 \
  --model test-model --dataset tests/assets/datasets/dummy_1k.jsonl,parser.prompt=text_input \
  --tokenizer tests/assets/tokenizers/char --streaming on --report-dir /tmp/token_basis_run
# On macOS, also pass: --workers 2 --no-cpu-affinity --max-connections 32
# report.txt shows main's rows, then "Other token basis (server):" with
# "OSL paired (client - server): mean 0.00 tokens/sample, ratio 1.0000 over 1000 samples"

For the full suite run uv run pytest.

Naming rule

The primary basis keeps the base name; the other basis appends
_<basis.value> (pure append, so series tpot_ns becomes tpot_ns_server and
report key tpot becomes tpot_server).

Never suffixed (basis-independent): osl_paired, the osl_paired_* counters,
n_samples_missing_usage, n_samples_missing_usage_empty_output,
n_missing_usage, the full-run n_undecodable / n_missing,
client_token_tasks_pending, token_count_source.

n_errors counts tokenizer failures, so it exists only on the client basis
(n_errors_client under a server primary).

What a default run writes, versus main

Additions only. Every key main writes is still there, and under a client
primary each unsuffixed value equals main's on the same inputs (pinned by
tests that rebuild reports and replays from identical snapshots and event
logs).

Artifact	Added
performance/result_summary.json	token_count_source, n_samples_missing_usage, n_samples_missing_usage_empty_output, client_token_tasks_pending, osl_paired, and the other basis as input_sequence_lengths_server, output_sequence_lengths_server, tpot_server, tps_server (a block is null when no usable sample was recorded; "partial": true inside it when some samples lacked usage; tps_server is null while partial)
final_snapshot.json	isl_server, osl_server, tpot_ns_server series, osl_paired_* and missing-usage counters, registered after main's series so a default snapshot is a strict superset; wire schema unchanged
events.jsonl	prompt_tokens / completion_tokens on sample.complete records (omitted when the endpoint sent none)
report.txt	An Other token basis (server): block after main's summary rows (token count source, TPS (server), OSL paired, a partial-reason sentence), and (server) sections after main's latency-breakdown sections, titled (server) (partial) when some samples lacked usage. main's rows keep their order and wording under a client primary
Full-run OSL block (output_sequence_lengths_full_run)	output_sequence_lengths_server, n_turns_counted_server, n_empty_server; its top-level partial still reflects only the primary basis
accuracy/accuracy_results.json	top-level token_count_source; per dataset output_sequence_lengths_server, n_osl_samples_server, n_missing_usage

Server TPOT uses LoadGen's rule, (complete - first chunk) / (completion_tokens - 1).
No server series records a value below 1.

Deliberate divergences from main under the default

Change	Why
stream_options sent by default on streaming OpenAI requests	Needed for the _server metrics. --no-stream-include-usage restores main's request bytes.
Live latency and completion timestamps shift by the usage frame on default-on streams	The stream ends one frame later. The usage-only frame yields no StreamChunk, so TTFT and chunk_delta sample counts are unchanged. A live A/B cannot match bit for bit; parity is tested on identical inputs.
A malformed usage object no longer fails a non-streaming sample	Server usage must never change client results. Each count is kept only if it is a non-bool int in [0, TOKEN_HDR_HIGH]; one warning per worker process names the first bad type.
A non-streaming /v1/completions body with no usage key no longer fails the sample (TextCompletionResponse.usage gains a default)	Same rule.
parse_sse_chunk drops only the malformed frame, not the rest of the read	Bug fix.
SGLang MetaInfo accepts the current meta_info shape (null finish_reason, etc.)	Bug fix; needed to read SGLang counts.
Audit and pipeline messages for an incomplete run print Report.incomplete_reason() (the audit ExecutionError, the pipeline "Report is incomplete: ..." log) instead of main's fixed text	Wording only. Same condition, same exit path.
Steady-state and early-stopping replay CLIs fail loudly on an unreadable summary or config.yaml, and on a non-compact event log; they report undecodable sample.complete lines	A run whose basis cannot be determined is never silently replayed on the wrong one. Archived runs that record no basis resolve to client with a stderr note, and --token-count-source overrides.
The accuracy scorer skips non-COMPLETE lines by their leading bytes instead of decoding them, and fails loudly (EventLogFormatError) on an event log that is not compact msgspec JSON	JSONLWriter always writes the compact form the filter needs; a log in another encoding (for example spaced json.dumps) would otherwise score nothing silently.
*_server blocks and report.txt lines listed above	Additive only.

n > 1 under a client primary: the adapter omits stream_options and
non-streaming decode copies usage only for a single choice, so _server is
partial for those rows. A server that forces usage on every stream (for example
vLLM --enable-force-include-usage) still returns summed counts.

Measured

macOS (arm64, 10 cores, Python 3.12). These numbers are indicative only; a
Linux idle-host run is the authoritative check and is pending.

OpenAIMsgspecAdapter.decode_sse_message (test_decode_sse)

Branch vs main (4b385eb5), 5 interleaved runs, median of the per-run
medians. Budget: within +/-3%.

Case	main	Branch	Delta
empty	209 ns	250 ns	+40 ns (see note)
100 chars	200.0 ns	204.8 ns	+2.4%
1k chars	397.7 ns	403.8 ns	+1.5%
8k chars	1709 ns	1722 ns	+0.8%
32k chars	6083 ns	6125 ns	+0.7%

Note: macOS's timer ticks in 41.7 ns steps, so the empty case flips between 209
and 250 ns. Its minimum is 165.8 ns on both trees. A many-iteration timeit of
the same input (3 interleaved runs, medians) gives empty 184.6 vs 188.7 ns
(+2.2%), 100 chars +1.8%, 1k +1.9%, 8k +0.9%, 32k +0.9%. The per-frame cost on the
default path is one usage is None test.

CLI roofline (test_e2e_perf.py, streaming rows, stub servers)

Stub server 3 workers, client 4 workers, all on the same host; main runs its
own stub, so each side uses its own tree end to end. The client runs without
CPU pinning (pin_loadgen is Linux-only). QPS, higher is better. "Opt-out" is
the branch with --roofline-no-stream-include-usage, which makes the stub send
no usage frame; "default" is the branch with the usage request on.

Row	main median	Branch, opt-out (ratio to main)	Branch, default (ratio to main)
concurrency c=1000, stream (n=6)	36,522	37,042 (1.014x)	36,108 (0.989x)
concurrency c=4000, stream (n=6)	35,116	34,874 (0.993x)	35,552 (1.012x)
max_throughput 2M burst, stream (n=1)	11,930	12,316 (1.032x)	12,353 (1.035x)

Turning the usage request off recovers main's rate within noise on every row,
and the default path shows no measurable cost on this host. The host is a shared
laptop: main alone varies 32.0k to 38.7k QPS across the six c=1000 runs, so
differences under about 5% are not resolvable here, and the 0.99x opt-out bar
(opt-out QPS versus main) is below this host's noise floor. The concurrency rows are six interleaved rounds, each
a separate pytest session, in rotating order (main, opt-out, default). The
burst row ran once per mode (about three minutes each), in the same session as
a first pair of concurrency runs; those first-session concurrency runs were
noisier (c=1000: opt-out 0.80x, default 1.02x; c=4000: 0.93x and 0.90x) and
are not in the medians above. A Linux idle-host run is the authoritative check
and is pending.

Follow-ups (outside this change)

• AgenticInferenceDataset.load: a JSONL stream column with missing values
  loads as float64, so requests carry "stream": 1.0/0.0; coerce to bool.

Type of change

• New feature

Related issues

Implements #540.

🤖 Generated with Claude Code

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅