Repository navigation
Expand file tree
/
Copy pathbench-audit.py
More file actions
executable file
·1854 lines (1665 loc) · 111 KB
/
Copy pathbench-audit.py
File metadata and controls
executable file
·1854 lines (1665 loc) · 111 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
#!/usr/bin/env python3
# SPDX-License-Identifier: Apache-2.0
# Copyright (C) 2026 Busbar Inc and contributors
#
# EVERY CROSS-METRIC INVARIANT A PUBLISHED BOARD MUST HOLD, as a program that exits non-zero.
#
# This file exists because of a specific failure, and the failure was not a bad number - it was a bad
# ANSWER TO "IS THE DATA GOOD?". Auditing a run meant writing throwaway python in a scratch directory,
# eyeballing the output, and forming an opinion. An opinion does not survive the next run, cannot be
# handed to anyone, and cannot be re-checked after a change. When asked "would you publish this?" the
# honest answer had to be "the checks I happened to think of, passed" - which is not a verdict.
#
# So the checks live here, they run over the artifacts, and "the audit is done" means this exits 0.
#
# It is the same lesson the engine spent a week learning about its own guards. `transient_budget()`
# was called by nothing. `box_qualify` always seeded. `history/append.py` wrote zero rows. Twenty-seven
# site tests asserted against an empty board. Every one of them was a check that existed in someone's
# head and nowhere in the code, and every one of them passed by doing nothing at all. A gate that
# cannot fail is not a gate, and a gate nobody wrote is worse.
#
# ./bench-audit.py the newest engine present in results/snapshots
# ./bench-audit.py --engine dc7a53c a specific engine, to audit one board
# ./bench-audit.py --gateway kong narrow to one gateway while investigating
#
# Runnable from any directory: every path it reads is anchored to this file, not to the caller's cwd.
#
# Exit 0 = every invariant held. Exit 1 = at least one did not, and each violation names the cell.
import argparse
import collections
import datetime
import glob
import json
import os
import re
import sys
# ── where the board lives ─────────────────────────────────────────────────────────────────────────
#
# ANCHORED TO THIS FILE, NOT TO THE INVOKER'S CWD (ledger TOOL-03). `glob.glob("results/snapshots/…")`
# resolves against whatever directory the audit was started from, so running it from anywhere but the
# repo root found zero files. That used to be a silent vacuous pass; the empty-board skip in main()
# now prints and exits 0, which is only marginally better - it still reports "nothing published" about
# a board that is right there on disk. An audit whose answer depends on the caller's shell state is
# not a verdict, so the paths come off this script's own location and the tool is runnable from any
# cwd, which is also what CI, a git hook and an operator poking at it from ~ all want.
#
# HERE is a module global rather than a constant baked into each call precisely so the tests can point
# the whole audit at a fixture tree: a check that can only be exercised against the real repo is a
# check whose RED half cannot be written.
HERE = os.path.dirname(os.path.abspath(__file__))
def snapshot_paths():
"""Every snapshot file on this board, sorted, resolved against HERE rather than the cwd."""
return sorted(glob.glob(os.path.join(HERE, "results", "snapshots", "result_*.json")))
# ── the bars ──────────────────────────────────────────────────────────────────────────────────────
#
# Named, not inlined, because a reader deciding whether to trust a violation needs to see what it was
# measured against. Each carries the reasoning that set it where it is.
# (The C6 sustained-vs-peak bar that used to open this section is gone with the two fields it policed;
# the note below records why, and what replaced it. No C6 constant is declared in this file any more.)
#
# THE GROSS-INVERSION CEILING IS RETIRED, and with it the cross-language drift gate that policed it.
#
# `C6_GROSS_PCT = 5.0` capped how much window noise could excuse a `rps_sustained_20ms` figure sitting
# ABOVE the `rps_max_proxy` it was meant to sit under. Both fields are deleted. The inversion is not
# merely bounded now, it is UNREPRESENTABLE: the frontier is six maxima over sets that only grow as the
# bound relaxes, so a looser reading cannot come out below a tighter one for any input at all.
#
# The gate went with it, and that is the point rather than an omission. `check_c6_bar_agrees_with_the_site`
# parsed the site's own `export const C6_GROSS_PCT = 5;` and failed on disagreement - a good mechanism,
# for a constant that governed something. Kept after the constant stopped governing anything, it would
# have been a gate policing agreement between two dead literals: exactly the decoration this file's
# whole job is to find, and which it has caught twice elsewhere (a phantom phase weight, a trigger
# listing a step no workflow ran).
#
# What replaced the invariant: `check_frontier_rates_are_monotone_in_the_bound` asserts the ordering
# over all six readings, and `check_frontier_is_rederivable_from_its_sweep` recomputes every one of them
# from the raw rungs - which catches a mis-ordered pair and much more besides, with no tolerance to tune.
# The engine's own declaration of the field list ABSENCE_CARRYING_FIELDS claims to mirror. This is
# python parsing a sibling in another language rather than importing it, because there is no build
# step to share the list through. See check_absence_fields_mirror_the_engine() for what happens when the two disagree.
RECORD_RS_PATH = os.path.join("engine", "src", "record.rs")
RECORD_RS_STRUCTS = {"perf": "CellPerf", "stream": "CellStream", "memory": "CellMemory"}
_RUST_IDENT_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
# ── the frontier ──────────────────────────────────────────────────────────────────────────────────
#
# The tail-latency bounds every served cell must publish a reading at, in microseconds, ascending, with
# the UNBOUNDED reading (`p99_bound_us: null`, failures only, no latency claim) last. This is the
# engine's `frontier::P99_BOUNDS_US`, restated here for the same reason ABSENCE_CARRYING_FIELDS restates
# record.rs's field lists: there is no build step to share a constant across the two languages. And for
# the same reason it is CROSS-CHECKED rather than trusted - `check_frontier_bounds_agree_with_the_engine`
# parses the rust declaration and fails on drift, because a python copy that fell behind would let a
# board publish five columns while this file cheerfully audited four.
#
# WHY A LIST AND NOT "whatever the artifact carries": the length and the ordering are the invariant. A
# cell publishing four readings, or the same five bounds in a different order, silently changes what the
# board means without any single number being wrong - see `check_frontier_is_complete`.
P99_BOUNDS_US = [1_000, 5_000, 10_000, 50_000, 100_000]
FRONTIER_RS_PATH = os.path.join("engine", "src", "frontier.rs")
FRONTIER_RS_RE = re.compile(r"pub const P99_BOUNDS_US:\s*\[u64;\s*\d+\]\s*=\s*\[([^\]]*)\]\s*;")
# ── the dialect axis ──────────────────────────────────────────────────────────────────────────────
#
# The order a definition.json `matrix` is read in: row = ingress, column = egress, both in the engine's
# `Dialect::ALL` order (`manifest::matrix_declared_capable`). Restated here for the same reason as
# P99_BOUNDS_US and CROSS-CHECKED for the same reason - `check_dialect_order_agrees_with_the_engine`
# parses ingress.rs and fails on drift. A python copy that fell out of order would not crash anything:
# it would silently name every declared cell after the wrong dialect, so the declared-vs-untestable
# check would compare the wrong pairs and go quiet on a real collision.
DIALECTS = ["openai", "openai-responses", "anthropic", "gemini", "cohere", "bedrock"]
INGRESS_RS_PATH = os.path.join("engine", "src", "ingress.rs")
INGRESS_ALL_RE = re.compile(r"pub const ALL:\s*\[Dialect;\s*\d+\]\s*=\s*\[([^\]]*)\]\s*;")
INGRESS_AS_STR_RE = re.compile(r"pub fn as_str\(&self\)\s*->\s*&'static str\s*\{(.*?)\n \}", re.S)
INGRESS_ARM_RE = re.compile(r"Dialect::(\w+)\s*=>\s*\"([^\"]+)\"")
# A rate this far above its own concurrency is not a proxy measurement.
#
# One connection cannot issue 20000 requests per second against a real socket; a number that says so
# is a units error or a counted retry, not a gateway. Deliberately loose - this catches the class of
# defect where a rate is divided by the wrong thing, not marginal optimism.
MAX_RPS_PER_CONNECTION = 20_000
def _instrument_map():
"""commit sha -> instrument id, from site/instrument-equivalence.json.
THE INSTRUMENT, NOT THE COMMIT - the same grouping C8 and gen-data use. Two commits whose built
binaries are byte-identical are one instrument, so pinning the audit to a raw sha would drop rows
that were measured by the very same binary. That is how busbar-150 fell out of this audit while
being on the board: it was stamped 80030c2f and the rest 7fd350ed, commits that build identical
bytes. An entry is only honoured here if it carries the artifact evidence its own file demands.
"""
p = os.path.join(HERE, "site", "instrument-equivalence.json")
out = {}
try:
with open(p) as fh:
doc = json.load(fh)
except FileNotFoundError:
return out
except Exception as e:
# NOT SILENT. An unreadable equivalence file narrows every instrument to its raw commit, so
# gateways measured by a byte-identical binary drop out of the audited set into NOT AUDITED -
# the reader must be told the pinning lost its evidence, not left to guess why.
print(f" unreadable {p}: {e} - pinning to raw engine commits only", file=sys.stderr)
return out
if not isinstance(doc, dict):
print(f" {p} is not a JSON object - pinning to raw engine commits only", file=sys.stderr)
return out
for inst in doc.get("instruments") or []:
commits = inst.get("commits") or []
ev = ((inst.get("evidence") or {}).get("otb_release_sha256")) or {}
hashes = list(ev.values())
if not inst.get("id") or not commits:
continue
# No artifact evidence, or hashes that disagree, means the entry proves nothing.
if len(hashes) < len(commits) or len(set(hashes)) != 1:
continue
for c in commits:
out[c] = inst["id"]
return out
def _same_instrument(sha, engine, imap):
"""Does `sha` belong to the instrument identified by `engine` (a sha prefix or an instrument id)?"""
if not engine:
return True
if sha.startswith(engine):
return True
mine = imap.get(sha)
if mine is None:
return False
# `engine` may be an instrument id, or any sha belonging to that instrument.
if mine == engine:
return True
return any(mine == inst for c, inst in imap.items() if c.startswith(engine))
def load(engine=None, gateway=None):
"""The newest snapshot per gateway, pinned to one INSTRUMENT so a board is audited as a board."""
imap = _instrument_map()
by_gw = {}
for f in snapshot_paths():
try:
d = json.load(open(f))
except Exception as e:
print(f" unreadable snapshot {f}: {e}", file=sys.stderr)
continue
gw = d.get("gateway")
sha = ((d.get("rig") or {}).get("engine") or {}).get("commit") or ""
if not gw or (gateway and gw != gateway):
continue
if not _same_instrument(sha, engine, imap):
continue
by_gw[gw] = (f, d, sha)
return by_gw
def newest_engine():
"""The engine commit of the snapshot with the newest `measured_at`, so the default audits the
current board. Recency is the snapshot's own timestamp, not filename order - the files sort
alphabetically by gateway, so "last file" is just "last gateway in the alphabet". A snapshot
whose `measured_at` is missing or unparseable falls back to its file mtime - but ranks BELOW every
snapshot that carries a real stamp, because after a git checkout the mtime is the checkout time and
would otherwise beat every honest timestamp on the board.
"""
best = None # ((has_stamp, when), sha)
for f in snapshot_paths():
try:
d = json.load(open(f))
except Exception:
continue
sha = ((d.get("rig") or {}).get("engine") or {}).get("commit") or ""
if not sha:
continue
when = None
raw = d.get("measured_at")
if isinstance(raw, str):
try:
when = datetime.datetime.fromisoformat(raw.rstrip("Zz"))
except ValueError:
when = None
stamped = when is not None
if when is None:
when = datetime.datetime.fromtimestamp(os.path.getmtime(f), datetime.timezone.utc)
# Normalise to naive UTC so a mix of offset-carrying stamps and mtime fallbacks compares.
if when.tzinfo is not None:
when = when.astimezone(datetime.timezone.utc).replace(tzinfo=None)
key = (stamped, when)
if best is None or key > best[0]:
best = (key, sha)
return best[1] if best else None
def served_cells(d):
"""Every cell the gateway actually served, with its coordinates for naming a violation."""
for eg, blk in ((d.get("matrix") or {}).get("upstreams") or {}).items():
for ing, c in (blk.get("cells") or {}).items():
if c.get("served") is True:
yield f"{ing}>{eg}", c
# ── the invariants ────────────────────────────────────────────────────────────────────────────────
#
# Each takes one served cell and yields a string per violation. A check that can never yield is a
# check that is not doing anything, which is the defect class this whole file is about: if one of
# these stops firing on data that used to trip it, that is a finding, not a pass.
# REMOVED: check_sustained_not_above_peak. It compared `perf.rps_sustained_20ms` against
# `perf.rps_max_proxy` and allowed C6_GROSS_PCT of window noise between them. BOTH FIELDS ARE DELETED,
# and the state it guarded - a gated reading coming out ABOVE the ungated one over the same windows - is
# now unrepresentable rather than unchecked. It was reachable because the two numbers came from two
# different algorithms over the same rungs (a plateau search that quit on three flat rungs, and a gate
# bisection that walked past where the plateau search had stopped), so the "maximum" could be a max over
# a SUBSET of what the "sustained" figure searched. It fired for real: aisix openai-responses>anthropic
# published 16,232 max against 16,610 sustained, and bifrost openai-responses>openai-responses 5,113
# against 5,174. The frontier is six readings of ONE rung set, each a maximum over a set that only GROWS
# as the bound relaxes, so the inversion class is structural nonsense now - and the ordering it implied
# is asserted anyway, over all six readings rather than two, by
# `check_frontier_rates_are_monotone_in_the_bound`. See `engine/src/frontier.rs`'s module note.
# REMOVED: check_peak_came_from_its_own_sweep. It held `perf.rps_max_proxy` to `max(sweep_max_proxy)`,
# which is one direction of one comparison against one deleted scalar. Its successor is strictly
# stronger and not a rewrite of it: `check_frontier_is_rederivable_from_its_sweep` recomputes EVERY
# reading from the rungs - the qualifying set, the maximum over it, the concurrency it was observed at,
# the tail that came with it and the boundary rung above it - and demands equality in both directions,
# where this only caught a peak that was too HIGH. A peak that was too LOW (the actual field defect: a
# plateau search that stopped early) passed this check every time.
def check_sweep_carries_its_latency(name, c):
"""Every throughput window must publish the p99 it ran at.
THIS IS THE REGRESSION GUARD FOR THE DEFECT THAT CAUSED THE RERUN. The load generator computed a
p99 for every window and always had; `SweepProbe::probe` narrowed its result to a rate and a
verdict and dropped it. Because the latency was gone, the sustained figure could not be read off
the throughput sweep, so the engine measured the cell a SECOND time to get it - minutes later,
across a gateway restart. Thirty-three readings per cell were paid for and discarded in every run
the project ever published. If this check ever passes vacuously again, the second search comes
back with it.
"""
pts = (c.get("perf") or {}).get("sweep_max_proxy") or []
if not pts:
return
withp99 = sum(1 for r in pts if r.get("p99_us") is not None)
if withp99 == 0:
yield f"{name}: all {len(pts)} throughput windows published without the p99 they measured"
# REMOVED: check_ttft_percentiles_are_ordered. Its own docstring carved out the exact reason it was
# unsound - "distinct from a difference-of-percentiles, which genuinely has no ordering:
# added_gap_p99_us below added_gap_p50_us is legal, because each is an independent difference between
# two legs' matched percentiles" - and then applied the rule to `added_ttft`, which is built the
# identical way (`added_ttft_at` subtracts the direct leg's percentile from the gateway leg's, one
# percentile at a time). The carve-out was right; it just was not applied consistently.
#
# It fired on litellm-rust anthropic>anthropic in the 2026-08-01 run (p99 177 below p50 182) - real,
# correct data - and, because CI gates on this file, would have failed the build for a board that was
# fine. That is the third copy of this same unsound rule found in this project; the other two were
# deleted from audit-every-metric.py and verify-latency.py for the same reason.
#
# Nothing replaces it because there is nothing sound left to point it at: every percentile this
# artifact publishes (added_ttft_p50/p99, added_gap_p50/p99, added_latency_p50/p99) is a DIFFERENCE.
# The raw single-population legs are not on the wire, so no ordering invariant exists to check. A
# guard that cannot be stated soundly is worse than no guard - it fails honest boards.
# REMOVED: check_rate_and_concurrency_travel_together. It paired `rps_sustained_20ms` with
# `rps_sustained_20ms_concurrency` and `rps_max_proxy` with `conc_at_peak`, and the failure it guarded was
# a rate published with the concurrency it was observed at MISSING - unre-derivable, unchartable. All
# four fields are deleted, and the state is unreachable by construction rather than by policing: a rate
# and its concurrency are no longer two sibling keys that can drift apart, they are two fields of ONE
# `FrontierReading` object emitted from one `frontier::Reading`, which cannot exist without both. What
# CAN still go wrong is a reading whose halves disagree with the rungs, and that is checked head-on by
# `check_frontier_is_rederivable_from_its_sweep` (the concurrency must name a rung that actually carried
# the winning rate at that bound), plus `check_every_absent_frontier_reading_has_a_reason` for the
# absent case.
# REMOVED (RE-POINTED, NOT DROPPED): check_rate_is_physically_possible read `rps_max_proxy / conc_at_peak`
# against MAX_RPS_PER_CONNECTION. Both fields are deleted, but the defect class - a rate divided by the
# wrong thing - is a property of ANY published rate/concurrency pair and did not go away with them, so it
# is re-pointed at all six readings by `check_frontier_rate_is_physically_possible` below. That is more
# coverage than before, not less: the old check looked at one pair per cell, the new one at up to six.
# REMOVED: check_frames_have_a_stream_behind_them. It fired when `stream.cpu_fps` was published beside a
# MEASURED `streams_sustained` of 0 - a frames/sec rate over a population of zero streams. `cpu_fps` and
# its `cpu_fps_concurrency` / `sweep_cpu_fps` are deleted, so there is no longer any frame rate in the
# artifact that can be divided by a stream count: `streams_sustained_fps` is the frame rate OF
# `streams_sustained` itself, measured in the same window rather than by a separate climb, so it cannot
# be a rate over a population it was not measured against. Note the check had already been narrowed once
# (litellm-rust anthropic>anthropic 2026-07-29) to a MEASURED 0 rather than a reasoned absence - the
# residual risk that survived that narrowing was exactly the two-searches-two-populations shape, and the
# second search is gone.
# THE DEFINITION OF DONE, as fields. Every metric a served cell's block may publish; a null on any
# of these with no `absences` entry beside it is a bare hole, which the board's owner has ruled out:
# "either this cell is measured and all data must be reported, or this cell wasn't tested and empty
# is expected - not a combo". Mirrors the engine's `absences()` lists in `engine/src/record.rs`
# (CellPerf, CellStream, CellMemory), field for field.
ABSENCE_CARRYING_FIELDS = {
"perf": [
# THE THROUGHPUT SCALARS ARE NOT LISTED HERE ANY MORE, and their absence is checked, not
# assumed: `rps_sustained_20ms` / `rps_sustained_20ms_concurrency` / `conc_at_sustained` /
# `rps_max_proxy` / `rps_max_proxy_concurrency` / `conc_at_peak` are deleted from `CellPerf`, so
# keeping them here would make `check_absence_fields_mirror_the_engine` fail as "policing fields
# the engine no longer defines" - which is exactly what it did before this list was cut, and is
# the reason that check exists.
#
# WHAT REPLACES THEM IS NOT A FIELD ON THIS LIST. `perf.frontier` is a Vec, unreachable by the
# engine's `absences_of!` macro (which walks scalar `Measurement` fields), so `CellPerf::absences`
# populates its keys in a hand-written loop under BOUND-keyed names - `perf.frontier.10ms.rps`,
# `perf.frontier.unbounded.rps`. Adding "frontier" here would break the mirror check in the other
# direction and would police the wrong shape anyway; the frontier's own null-carries-a-reason
# invariant is `check_every_absent_frontier_reading_has_a_reason`.
"added_latency_p50_us", "added_latency_p99_us", "gateway_c1_p99_us", "direct_c1_p99_us",
# THE COST GROUP. Every one of these can be absent for a reason a reader must see, and two of
# those reasons are REFUSALS rather than gaps: a window with any failure publishes no cost
# (CPU divided by only the successes would describe the failures, not the work), and a window
# on a swapping box has its cost marked a harness fault. A null with the reason missing reads
# as "not implemented" when the truth is "measured, and deliberately withheld".
#
# This list fell behind the engine when the fields were added, and the mirror check caught it
# exactly as designed - reporting that every check built on this list was silently blind to
# nine fields. That is the second cross-file mirror the cost work has broken (bench-dashboard
# PHASE_ORDER was the first), and both were caught by their own guard rather than by review.
"cpu_us_per_request", "rps_per_cpu_second", "cost_window_conc", "cost_window_ok",
"cost_window_rps", "cost_core_utilisation", "cost_threads",
"cost_nonvol_ctxt_per_request", "cost_majflt",
],
"stream": [
# `cpu_fps` and `cpu_fps_concurrency` are deleted from `CellStream` (with `sweep_cpu_fps`), same
# reasoning as the perf block above: a field the engine no longer defines cannot be policed here.
"added_ttft_p50_us", "added_ttft_p99_us", "added_gap_p50_us", "added_gap_p99_us",
"streams_sustained", "streams_sustained_fps",
# The WEIGHT behind the two added-TTFT percentiles. A failed probe used to be dropped inside a
# filter_map, so a p99 over three lucky samples published identically to one over a hundred -
# and with a single survivor the p50 and p99 ranks collapse to the same index, which reads as a
# coherent pair. These carry the count so a reader can weigh the percentile beside them.
"ttft_gw_samples", "ttft_direct_samples",
],
"memory": [
"idle_rss_mib", "steady_state_rss_mib", "recovered_rss_mib", "peak_rss_mib",
"peak_rss_hwm_mib", "time_to_plateau_s", "growth_rate_mib_per_min",
# Newly coverable: these were bare `Option`s that collapsed the metric group's reason on the
# way out, so a memory window that could not judge the plateau published two nulls nothing
# could explain. They are `Measurement`s now and ride in the cell's absences map like every
# other number, which is what lets this list hold them to the same bar.
"plateaued", "load_s",
# Absent BECAUSE the measurement succeeded: these describe HOW a window failed to settle, so a
# window that DID settle has no shape to publish and says exactly that. They are listed here
# for the reason every other field is - a null must carry a reason - and NOT held to
# check_no_bare_absence's "a served cell publishes a number", which is why that check reads
# SHAPE_FIELDS below and lets a reasoned absence stand.
"shape", "idle_shape",
# The idle window's own verdict and its fitted slope. The memory group measured both on every
# served cell and CellMemory had nowhere to put them, so they reached the artifact as neither
# a number nor a null - a missing key, invisible to every check built on this list. They are
# ordinary numeric metrics (a settled idle window publishes 1.0 and its slope), so unlike the
# shape fields they ARE held to 'a served cell publishes a number'.
"idle_static", "idle_growth_rate_mib_per_min",
],
}
# The fields whose absence is a RESULT rather than a gap. See the note in ABSENCE_CARRYING_FIELDS.
SHAPE_FIELDS = {"memory": {"shape", "idle_shape"}}
# The absence reasons that legitimately excuse a CAPACITY metric from being a number. Everything
# else must be a number - a gateway that failed the gate at every rung is a measured 0, not a hole.
CAPACITY_ABSENCE_OK = {"untestable", "rig_limited", "search_exhausted", "harness_error", "not_served"}
def check_no_bare_absence(name, c):
"""0 is a number; a bare n/a is not. Every null metric on a served cell carries its reason.
The 2026-07-28 board's defining defect: holes with no stated cause, indistinguishable from cells
that never ran. The engine has always written the reason into the cell's `absences` map; this
pins that the map actually covers every null the cell publishes, so no consumer ever has to
render an unexplained blank.
"""
absences = c.get("absences") or {}
for block, fields in ABSENCE_CARRYING_FIELDS.items():
blk = c.get(block) or {}
for f in fields:
if f in blk and blk[f] is None and f"{block}.{f}" not in absences:
yield f"{name}: {block}.{f} is null with NO reason in absences (a bare hole)"
def check_declared_fields_are_carried(name, c, known=None):
"""A served cell must CARRY every field it declares - as a number, or as a null with a reason.
THE HOLE THIS CLOSES IS THE ONE `check_no_bare_absence` CANNOT SEE (ledger TOOL-04). That check
reads `f in blk and blk[f] is None`, so it polices a field that is PRESENT and null. A field
OMITTED FROM THE BLOCK ENTIRELY is invisible to it - and to every other check here, all of which
reach for values with `.get()` and get None back whether the serializer wrote a null or wrote
nothing at all. "Key missing" and "measured" are indistinguishable to the whole file, which means
the audit's answer to "is every metric accounted for?" was really "is every metric that happened
to be serialized accounted for?".
That is not hypothetical arithmetic: today the engine serializes all of these unconditionally (no
`skip_serializing` anywhere in record.rs), so the board is honest. One `#[serde(skip_serializing_if
= "Option::is_none")]` on a Measurement field - the single most ordinary thing anyone would add to
trim an artifact - would drop the key, and the entire absence discipline would evaporate silently
while this audit kept printing PASS. The board's rule is "either this cell is measured and all
data must be reported, or this cell wasn't tested"; a missing key is neither, so it is a
violation, and the shape of the artifact stops being a matter of which code path the serializer
took.
The block itself is held to the same standard. A served cell with no `stream` object at all is not
a quieter version of a cell whose streaming legs failed - it is the same claim with the evidence
deleted, and `c.get(block) or {}` elsewhere in this file would read it as a block full of nulls
with nothing to explain them.
"""
for block, fields in ABSENCE_CARRYING_FIELDS.items():
blk = c.get(block)
if not isinstance(blk, dict):
yield (f"{name}: served cell publishes NO {block} block at all - a served cell carries "
f"its declared fields or states why they are absent, it does not omit them")
continue
for f in fields:
if f not in blk:
# A field the PRODUCING ENGINE never had is a different thing from one it dropped.
# main() computes `known` POOLED across every snapshot of the audited instrument: a
# field no cell on that board carries was not in the engine that wrote it, and
# demanding it would be demanding that yesterday's artifact contain tomorrow's field.
# Pooled rather than per snapshot, so a serializer that dropped a key on SOME cells -
# or on EVERY cell of one gateway - still fails loudly, because other cells from the
# same producer carry it. A board that omits a field everywhere is DISCLOSED at the
# end of the run instead, never silently forgiven.
if known is not None and f not in known.get(block, ()):
continue
yield (f"{name}: {block}.{f} is OMITTED from the block (not null-with-reason, "
f"absent) - key-missing and measured are indistinguishable to every check")
def check_stream_capacity_is_a_number(name, c):
"""A streaming cell's capacity metrics are numbers (0 included), or a rig-class absence.
The yield gate: streams_sustained and cpu_fps produced values (or measured zeroes) on every
served streaming cell once the gate published failures as 0. An absence whose reason is
`not_measured` here means a search quietly stopped producing - the exact silent-yield defect
that shipped a board with cpu_fps on 1 of 16 served cells.
`cpu_fps` IS NO LONGER IN THE FIELD LIST BELOW - it is deleted from `CellStream`. The silent-yield
defect it was watched for is still live for `streams_sustained`, which is a bisection that can stop
producing, so this check keeps its job with a shorter list rather than being removed. The historical
1-of-16 board is what motivated it and is left in the paragraph above because it is why the bar is
"a number or a reasoned absence" rather than "a value if the search felt like it".
"""
st = c.get("stream") or {}
if st.get("stream_served") is not True:
return
absences = c.get("absences") or {}
for f in ("streams_sustained",):
if st.get(f) is None:
entry = absences.get(f"stream.{f}") or {}
reason, detail = entry.get("reason"), entry.get("detail")
if reason in CAPACITY_ABSENCE_OK:
continue
# AN ABSENCE THAT EXPLAINS ITSELF IS NOT A SILENT YIELD. The defect this catches is a
# search that stopped producing and said nothing - the board that shipped cpu_fps on 1 of
# 16 served cells. "The bisection proved c=6144 and could not reconfirm it" is the
# opposite: a search that ran, failed to establish a ceiling, and published why. Requiring
# a measured 0 there would force a number onto a question that was genuinely not settled,
# which is the fabrication this whole file exists to prevent.
if detail:
continue
yield (f"{name}: stream.{f} is absent with reason {reason!r} and NO detail on a served "
f"streaming cell - a search that stops producing must say why")
# ── the SEARCH TRACE's invariants ────────────────────────────────────────────────────────────────
#
# EVERY CHECK ABOVE READS THE PUBLISHED NUMBER. None of them read `sweep_streams`, the trace of the
# search that produced it - and that is exactly how this file returned PASS ten times over a board in
# which eight cells had silently given up. `check_stream_capacity_is_a_number` even accepts those on
# purpose: an absence carrying prose ("the bisection proved c=6144 and could not reconfirm it") reads
# as a search that ran, failed honestly, and said why. It cannot tell that apart from a search that
# quit with most of its budget unspent, because it never looks at what the search actually did.
#
# These checks close that. The bar is the same as the frontier's: re-derive from the evidence and
# demand agreement, rather than trust the summary. The trace is published on every streaming cell, so
# the search's own behaviour is auditable from committed JSON with no rig and no box - which is where
# every defect of this class should have been caught, and was not.
#
# Old snapshots predate the typed-error and host-state fields; a check that needs them skips rather
# than fails, because absent evidence is not evidence of a defect.
def stream_trace(c):
"""The rungs a streaming cell's search actually probed, in order, or None."""
st = c.get("stream") or {}
if st.get("stream_served") is not True:
return None
sw = st.get("sweep_streams")
return sw if isinstance(sw, list) and sw else None
def proven_clean_top(rungs):
"""The highest concurrency the UNCONTAMINATED ascending prefix carried - every rung from the
first up to and including it passed, before anything in this cell had failed.
This is the ASCENDING PREFIX and nothing else: it deliberately stops at the first failed window,
so the concurrency it returns is one the cell reached before anything in it had gone wrong. What
counts as a violation relative to that top is decided by the caller - see the note there on why a
failure AT the top is the search working and only a failure BELOW it is a finding.
"""
top = 0
for r in rungs:
if r.get("passed") is not True:
break
top = max(top, r.get("conc") or 0)
return top
# Mirrors engine/src/run.rs's MAX_CEILING_STEPDOWNS. A copy rather than a read because this file
# audits committed artifacts and must not depend on the engine source that produced them.
MAX_CEILING_STEPDOWNS = 4
def step_downs_taken(rungs, clean):
"""How many step-downs the confirmation loop actually spent, as the trace records it.
The trace is ascending sweep, then bisection probes, then the bisected ceiling's confirmation
windows (its SECOND appearance - a bisection never probes one concurrency twice), then one seed
window per step-down, each strictly below the ceiling (`run::sweep_streams_cell`). The ceiling is
the highest concurrency above the clean prefix that passed a window.
NOT every distinct concurrency above the clean prefix: that counted the ascending failure and every
bisection probe as spent budget, so any real bisection over a bracket wider than ~16 rungs read as
"budget exhausted" and the check exempted itself on every cell it was written for.
"""
passed_above = [r.get("conc") or 0 for r in rungs
if r.get("passed") is True and (r.get("conc") or 0) > clean]
if not passed_above:
return 0
ceiling = max(passed_above)
seen = [i for i, r in enumerate(rungs) if (r.get("conc") or 0) == ceiling]
if len(seen) < 2:
return 0 # no confirmation window was recorded, so no step-down can follow one
after = rungs[seen[1] + 1:]
return len({r.get("conc") for r in after if clean < (r.get("conc") or 0) < ceiling})
def check_search_spent_its_budget(name, c):
"""A search that publishes nothing must have RUN OUT, not stopped early.
plano `anthropic>anthropic` is the case: the ascending sweep carried c=64, the bisection settled
on c=79, confirmation failed, the step-down bisected to c=71, its first window failed - and the
search ended there, with every concurrency between 64 and 71 untried and most of its step-down
budget unspent. A rung is confirmed by MAJORITY, not by its first window (one-api's published 266
came from `[pass, pass, fail]`), so one failing window is a vote, not a verdict.
The signature is a gap: the search stopped at a rung strictly above a concurrency it had already
carried, having probed nothing in between. That is room it declined to use, and the cell paid for
it with an absence where a number was available.
"""
rungs = stream_trace(c)
if not rungs:
return
st = c.get("stream") or {}
if st.get("streams_sustained") is not None:
return
clean = proven_clean_top(rungs)
if clean <= 0:
return
# WHERE THE SEARCH STOPPED, not the smallest rung it ever probed. Taking the minimum over every
# rung included the ascending prefix's own c=1 and c=2, so `lowest` was always <= clean + 1 and
# this check returned early on every cell in existence - it had never fired once, including on
# the plano cell it was written from. Caught by its own reject fixture in bench-audit_test.py,
# which is the entire reason that file demands one.
lowest = rungs[-1].get("conc") or 0
if lowest <= clean + 1:
return # it walked down to (or below) what it had carried
# A SEARCH THAT SPENT ITS DECLARED BUDGET DID NOT GIVE UP - it ran out, which is a different
# finding and not a defect. plano anthropic>anthropic on the 2026-08-01 run is the case: the
# bisection proved c=224, confirmation failed, and the step-down walked 176 -> 152 -> 140 -> 134,
# four rungs, exactly MAX_CEILING_STEPDOWNS. This check could only see the gap between the last
# rung and the carried one, so it read a fully-spent budget as an abandoned one and would have
# stopped a 14-box run for it.
#
# Counting distinct concurrencies probed BELOW the bisected ceiling, AFTER its confirmation began,
# is the budget as the trace shows it. At the cap the honest reading is "our search range was too
# small for this cell", which belongs in the run's write-up, not in a defect report.
if step_downs_taken(rungs, clean) >= MAX_CEILING_STEPDOWNS:
return
between = [r for r in rungs if clean < (r.get("conc") or 0) < lowest]
if not between:
yield (f"{name}: the search published nothing but stopped at c={lowest} while it had already "
f"carried c={clean}, probing none of the {lowest - clean - 1} concurrencies between - "
f"it gave up with budget unspent, and a majority-confirmed rung may well be in there")
def check_no_rung_fails_below_one_already_carried(name, c):
"""A rung cannot fail below one the same cell has already carried cleanly - unless something
other than the gateway changed, and then the absence must SAY so.
This is the signature that took six cells off the 2026-07-31 board. It is either our rig failing
to drain between windows or a gateway that stopped serving after an overload, and both are
findings; what is not acceptable is publishing it as an ordinary failure with no attribution.
"""
rungs = stream_trace(c)
if not rungs:
return
clean = proven_clean_top(rungs)
if clean <= 0:
return
# BELOW, NOT AT. This read `<= clean`, which flags the search's own terminating condition: the
# ascending sweep probes a rung once, that probe passes, confirmation at the SAME concurrency then
# fails, and the engine steps down and publishes the lower rung. apisix anthropic>anthropic did
# exactly that - c=16384 passed the sweep, lost both confirmation windows, and 8192 was published
# after holding 4 of 4 - and it was reported as "impossible for the gateway alone".
#
# A failure AT the top is how a ceiling is found. A failure strictly BELOW a concurrency the cell
# already carried is the thing that has no gateway-only explanation, which is what the docstring
# above has always said ("cannot fail BELOW one the same cell has already carried") and what the
# 2026-07-31 six-cell case actually looked like. The comparison now matches the claim.
below = [r.get("conc") for r in rungs if r.get("passed") is not True and (r.get("conc") or 0) < clean]
if not below:
return
absences = c.get("absences") or {}
detail = ((absences.get("stream.streams_sustained") or {}).get("detail") or "").lower()
# WORDS, NOT SUBSTRINGS: a bare `"rig" in detail` matched "original", "trigger" and "bridge", so an
# absence that named nothing at all could still excuse the impossible signature.
named = re.search(r"already carried|did not recover|\bdrain|\brig\b|\brestart", detail) is not None
if not named:
yield (f"{name}: rung(s) {sorted(set(below))} failed below c={clean}, which this cell had "
f"already carried cleanly - impossible for the gateway alone, and the absence does not "
f"name what else changed")
def check_published_rung_held_a_majority(name, c):
"""The published sustained figure must be a rung whose windows in the trace actually held.
The engine confirms by majority and the trace records every window, so this is re-derivable. A
published rung that lost its own windows would mean the summary and the evidence disagree - the
one defect the design cannot rule out structurally.
"""
rungs = stream_trace(c)
if not rungs:
return
st = c.get("stream") or {}
sus = st.get("streams_sustained")
if sus is None:
return
at = [r.get("passed") is True for r in rungs if (r.get("conc") or 0) == sus]
if not at:
yield (f"{name}: publishes streams_sustained={sus} but the trace contains no window at that "
f"concurrency - the number did not come from this sweep")
return
if sum(at) * 2 <= len(at):
yield (f"{name}: publishes streams_sustained={sus} on {sum(at)} of {len(at)} windows - the "
f"engine's own rule is a majority, so this rung was not confirmed")
def check_a_wedged_gateway_is_named_as_one(name, c):
"""A gateway that stops serving and never resumes must be reported as that, not as a bare absence.
aisix carried every rung to c=8,192, was pushed to c=16,384, then failed seventeen consecutive
windows including c=4,096 which it had just served. "It does not recover from overload" is the
single most useful thing a reader could learn from that cell, and it was published as silence.
"""
rungs = stream_trace(c)
if not rungs or len(rungs) < 6:
return
tail = [r.get("passed") is True for r in rungs[-5:]]
if any(tail):
return
# A WEDGE IS A FAILURE BELOW WHAT THE CELL ALREADY CARRIED, not merely a run of failures at the
# end. A budget-exhausted step-down ALWAYS ends on a run of failures - that is its shape - and
# agentgateway anthropic>openai showed it: proven clean to c=2048, then five failures walking
# 3170 -> 2118, every one of them ABOVE 2048. The gateway never stopped serving anything it had
# held; it just did not converge above what it carried, which is ordinary non-convergence and is
# already reported as such.
#
# This fired mid-run as a violation, the second false positive of its kind. The impossible
# signature - and the only one that means the process stopped serving - is a tail failure strictly
# below the top of the uncontaminated ascending prefix.
clean = proven_clean_top(rungs)
if clean <= 0 or not any((r.get("conc") or 0) < clean for r in rungs[-5:]):
return
entry = (c.get("absences") or {}).get("stream.streams_sustained") or {}
detail = (entry.get("detail") or "").lower()
# AN ABSENCE FILED AS THE RIG'S HAS ALREADY ANSWERED THIS. The engine's own contamination guard
# reaches the same conclusion by a stricter route - it saw a rung fail below one the cell had
# carried and said so in the reason token - so demanding the words "recover" or "restart" on top
# of that is this check grading prose rather than reading the finding. busbar
# anthropic>anthropic came back reason=harness_error with the impossibility spelled out, and this
# still called it unexplained: the third false positive of the same family, all of them from
# keyword-matching a sentence instead of asking what the absence claims.
if entry.get("reason") in ("harness_error", "rig_limited"):
return
if "recover" not in detail and "restart" not in detail and "already carried" not in detail:
yield (f"{name}: the last 5 rungs all failed - the gateway stopped serving and did not come "
f"back - but the absence does not say so")
def check_no_rig_side_error_is_charged_to_the_gateway(name, c):
"""A connection this HOST could not make is never the gateway's error.
The engine discards a whole window on `RigExhausted`, so a counted connect-failure means the peer
refused - but the typed breakdown is what makes that checkable rather than assumed. Skipped on
snapshots that predate the typed fields.
"""
rungs = stream_trace(c)
if not rungs:
return
if not any("stream_errors_connect_failed" in r for r in rungs):
return
bad = [(r.get("conc"), r.get("stream_errors_connect_failed")) for r in rungs
if (r.get("stream_errors_connect_failed") or 0) > 0]
for conc, n in bad:
yield (f"{name}: c={conc} counted {n} connect-failure(s) against the gateway - a connection "
f"that was never made is not a stream the gateway failed to serve")
TRACE_CHECKS = [
check_search_spent_its_budget,
check_no_rung_fails_below_one_already_carried,
check_published_rung_held_a_majority,
check_a_wedged_gateway_is_named_as_one,
check_no_rig_side_error_is_charged_to_the_gateway,
]
# ── the frontier's invariants ─────────────────────────────────────────────────────────────────────
#
# `perf.frontier` is SIX READINGS OF ONE RUNG SET, published beside the rung set itself
# (`perf.sweep_max_proxy`), which is what makes every one of them re-derivable rather than asserted -
# and re-derivation is the check that matters here. The old scalars could only be sanity-checked
# (is the peak at least as big as the sustained figure? is the peak a rate some window produced?)
# because the algorithm that produced them had thrown away the evidence: a plateau search's stopping
# point is not in the artifact. The frontier's arithmetic is `max(rps) over {rungs that qualify}`, every
# input to it is published, so this file can run that arithmetic itself and demand the same answer.
#
# The bar for these is deliberately "recompute and compare", not "looks plausible". A summary that
# disagrees with the rungs it claims to summarise is the one defect the frontier design cannot rule out
# structurally - `frontier.rs` derives monotonicity from the algorithm, but nothing in the artifact
# proves the published readings CAME from that algorithm over these rungs. That proof is here.
def bound_key(us):
"""The name a reading's absences are filed under: `10ms`, or `unbounded` for the failure-only one.
Mirrors `CellPerf::absences`'s `format!("{}ms", us / 1000)` exactly, because a key this file computes
differently from the engine would look like a missing reason for every absent reading on the board -
a false FAIL, which costs as much trust as a false pass.
"""
return "unbounded" if us is None else f"{us // 1000}ms"
def frontier_of(c):
"""The cell's frontier readings, or [] when it has none (a pre-frontier artifact, or a shrunk one -
`check_frontier_is_complete` is the only place allowed to tell those two apart)."""
fr = (c.get("perf") or {}).get("frontier")
return fr if isinstance(fr, list) else []
def sweep_rungs(c):
"""The rungs the frontier is read from. Every reading must be re-derivable from exactly these."""
return [r for r in ((c.get("perf") or {}).get("sweep_max_proxy") or []) if isinstance(r, dict)]
def rung_served_cleanly(r):
"""Did the gateway serve every request it accepted at this rung? `frontier::Rung::served_cleanly`.
ZERO FAILURES, NOT A TOLERANCE - the engine's rule, and the reason is that the rig's own refused
connects never reach the rung (`GenStats::rig_refused` discards those windows), so a failure here is
the gateway failing a request it accepted.
EXACT NOW: `ok > 0 and fail == 0`, both read directly off the rung, because `SweepPoint` publishes
`ok`. It did not used to: `SweepPoint` carried conc/rps/p99_us/fail only, so this file had to
APPROXIMATE the engine's `ok > 0` half - first as `rps > 0` alone, which produced a real FALSE
POSITIVE on live data. Plano at c=256 published `rps: 0, p99_us: 3398432, fail: 0`, and `rps > 0`
called that window dirty. But a percentile cannot exist without a completed, timed request - so
`ok >= 1`, the window served cleanly, and its rate merely rounded down through the engine's `as
i64` (one request over a four-second window is 0.25 rps, published as 0). The engine was right and
this check was wrong, and it mattered: the approximation then re-derived a
`first_disqualified_conc` the engine had correctly left absent, and the disagreement between the
two checkers had to be settled BY HAND. `SweepPoint` publishing `ok` is what removes that class of
disagreement rather than merely widening the approximation again - there is no approximation left
to widen.
AND WHEN `ok` IS ABSENT ENTIRELY - a snapshot measured before the field existed - this FALLS BACK to
the p99 rule rather than refusing. Refusing was the first shape of this fix, and it was worse: with
`ok` unavailable every rung on such a board fails cleanliness-for-lack-of-proof, so the re-derivation
check had to be skipped wholesale, and the STRONGEST invariant in this file went unaudited on every
cell of the board about to ship. A skipped check is weaker than an approximate one. The fallback is
the p99-as-proof-of-completion rule, which is WIDER than the engine's (a completion always leaves a
latency sample; a rate can round away), so its residual error is a missed catch and never a false
alarm - which is the only direction an approximation may err in a tool whose warnings must be
trusted.
An ABSENT `fail` is still not a clean rung: "measured nothing" and "measured no failures"
are different facts, and a rung this file cannot prove clean does not get to count as one. Every
snapshot on disk as of 2026-07-29 predates `ok`, so this returns False for ALL of their rungs - see
`producer_knew_ok` for how that is kept from reading as "every old rung was dirty" at the one place
(`check_frontier_is_rederivable_from_its_sweep`) that would otherwise turn it into a false alarm.
"""
ok, fail, p99 = r.get("ok"), r.get("fail"), r.get("p99_us")
if not isinstance(fail, (int, float)) or fail != 0:
return False
if isinstance(ok, (int, float)):
return ok > 0
# `ok` absent: this rung predates the field. Fall back rather than refuse - see the docstring.
return (isinstance(r.get("rps"), (int, float)) and r.get("rps") > 0) or isinstance(p99, (int, float))
def rung_qualifies(r, bound_us):
"""Does this rung count toward the reading at `bound_us`? `None` = the failure-only reading.
`frontier::Rung::qualifies`, restated: clean, and under the bound STRICTLY (`p99 < bound`, so a rung
sitting exactly on a bound does not clear it - "under 1 ms" means under). A rung with no p99 is
disqualified from every BOUNDED reading, because a rung with no latency reading has not earned a
claim about latency - but not from the unbounded one, which makes no latency claim to earn.
"""
if not rung_served_cleanly(r):
return False
if bound_us is None:
return True
p99 = r.get("p99_us")
return isinstance(p99, (int, float)) and p99 < bound_us
def top_probed_conc(rungs):
"""The highest concurrency the sweep asked for, qualifying or not - `Reading::top_probed_conc`. A
rung that failed is still a rung we looked at, and is exactly what proves we did not stop early."""
concs = [r.get("conc") for r in rungs if isinstance(r.get("conc"), (int, float))]
return max(concs) if concs else None
def check_frontier_is_complete(name, c, frontier_known=False):
"""Every served cell publishes ALL SIX readings: the declared bounds ascending, unbounded last.
A CELL PUBLISHING FOUR WOULD SILENTLY SHRINK THE BOARD. Nothing else here would notice: every other
check iterates whatever readings are present and would pass over a cell missing its 1ms and 5ms
columns, while the site would render the gateway as though those bounds had never been asked about.
The engine writes an absent-with-reason reading for a bound nothing qualified at, precisely so that
"no throughput under 1ms" and "we did not report 1ms" stay distinguishable - and this is the check
that makes the difference visible instead of merely intended.
THE ORDER IS PART OF THE INVARIANT, not presentation. `frontier.rs` derives monotonicity in the
BOUND; a reader (and `check_frontier_rates_are_monotone_in_the_bound`) reads that off the sequence
positionally, so a permuted sequence turns a true property into a false alarm or hides a real one.
`frontier_known` IS HOW A PRE-FRONTIER ARTIFACT IS TOLD FROM A SHRUNK ONE - the same mechanism, and
the same reasoning, as `fields_the_producer_knew`: ask the artifact. A snapshot where NO served cell
carries a frontier was written by an engine that did not have the metric, and demanding it would be
demanding that yesterday's artifact contain tomorrow's field; those are disclosed by count at the end
of the run, never silently forgiven. A snapshot where SOME cell carries one knew how, so a cell
without it dropped it, which is the defect.
"""
fr = frontier_of(c)
if not fr:
# WITHHELD IS NOT DROPPED, and conflating them made this check cry wolf on the one cell on the
# board where the harness behaved best.
#
# aisix openai-responses>openai answers 200, so `served` is True and its siblings all carry a
# frontier - which is exactly the shape this check calls a drop. But EGRESS RE-VERIFICATION
# PROVED THE GATEWAY NEVER TRANSLATED: the request arrived on the mock's openai-responses
# endpoint and nothing arrived on openai, so it forwarded the ingress request unchanged. Every
# number taken there describes a wire that is not openai-responses>openai, and the engine
# therefore withheld the whole perf group and published `egress_reverified: false` plus the
# evidence in `perf_dropped`. An empty frontier is the CORRECT artifact for that cell; demanding
# one would be demanding a throughput figure for a translation that did not happen.
#
# So the exemption is not "trust the producer" - it is keyed on the producer having published
# the DISCLOSURE. A cell that simply lost its frontier has no `egress_reverified: false` and
# still fails, which is the case this check exists for.
perf = (c.get("perf") or {}) if isinstance(c, dict) else {}
if perf.get("egress_reverified") is False:
return
if frontier_known:
yield (f"{name}: served cell publishes NO frontier while other cells in the same snapshot "
f"do - the producing engine had the metric, so this cell's throughput answer was "
f"dropped rather than never measured")
return
expected = list(P99_BOUNDS_US) + [None]
got = [r.get("p99_bound_us") if isinstance(r, dict) else "<not an object>" for r in fr]
if got != expected:
yield (f"{name}: frontier publishes bounds {got} but the board is defined at {expected} "
f"(the declared bounds ascending, then the unbounded reading) - a frontier of a "
f"different shape is a different board")
# EVERY FIELD PRESENT, not merely non-null. Same reasoning as check_declared_fields_are_carried: a
# key the serializer omitted and a key it wrote as null are indistinguishable to every `.get()` in
# this file, so a `skip_serializing_if` on any of these would evaporate the absence discipline while
# the audit kept printing PASS.
for r in fr:
if not isinstance(r, dict):
continue
for f in ("p99_bound_us", "rps", "concurrency", "p99_us", "first_disqualified_conc",
"lower_bound"):
if f not in r:
yield (f"{name}: frontier reading at {bound_key(r.get('p99_bound_us'))} OMITS `{f}` - "
f"key-missing and measured are indistinguishable to every check here")
def check_frontier_rates_are_monotone_in_the_bound(name, c):
"""Relaxing the bound can never lower the reading.
THIS IS STRUCTURAL IN THE ENGINE AND ASSERTED ANYWAY. A rung qualifies at bound B if its tail is
under B and it failed nothing, so relaxing B only ADDS rungs to the qualifying set, and each reading
is a maximum over that set - a max over a superset cannot be smaller. `frontier.rs` says so in its
module note and its own test walks hostile inputs. So a violation here does NOT mean "the gateway
behaved strangely"; it means the published readings did not come from the rungs they claim to
summarise, which is the one thing the structure cannot rule out. An invariant nothing checks is an
invariant nobody notices breaking - and the inversion class it descends from (`rps_max_proxy` below
`rps_sustained_20ms`) shipped on two real cells before it was structural.
An ABSENT looser reading beside a PRESENT tighter one is the same violation in its degenerate form:
the looser bound's qualifying set contains the tighter one's, so it cannot be empty when that one
was not.
"""
prev_rps = prev_key = None
for r in frontier_of(c):
if not isinstance(r, dict):
continue
key = bound_key(r.get("p99_bound_us"))
rps = r.get("rps")
if rps is None:
if prev_rps is not None:
yield (f"{name}: frontier.{key} published NO rate while the tighter frontier."
f"{prev_key} published {prev_rps} - a looser bound's qualifying set contains "
f"the tighter one's, so it cannot be empty when that one was not")
continue
if prev_rps is not None and rps < prev_rps:
yield (f"{name}: frontier.{key} reads {rps} rps, BELOW frontier.{prev_key}'s {prev_rps} - "
f"relaxing the bound lowered the reading, which the max-over-a-superset structure "
f"makes impossible, so these readings are not the rungs they claim")
prev_rps, prev_key = rps, key
def check_frontier_is_rederivable_from_its_sweep(name, c, ok_known=True):
"""Recompute every reading from `sweep_max_proxy` and demand the published one matches.
THE STRONGEST CHECK IN THIS FILE, and the only one that verifies the engine's ARITHMETIC rather than
the plausibility of its summary. Each reading is `max(rps)` over the rungs that qualify at its bound,
the concurrency that maximum was observed at, the tail THAT rung produced, and the lowest
concurrency above it that stopped qualifying. All four inputs are published per rung, so all four
outputs are recomputable, and a summary that disagrees with the rungs behind it fails here.
It replaces `check_peak_came_from_its_own_sweep`, which only caught a peak that was too HIGH. The
actual field defect was the opposite - a plateau search that stopped three flat rungs in and reported
a maximum BELOW what its own sibling search reached - and that passed the old check every time.
ONE PLACE WHERE THE ARTIFACT IS GENUINELY SHORT OF INFORMATION, handled rather than assumed away:
rungs may repeat a concurrency, so ties are matched against the SET of rungs carrying the winning
rate at the published concurrency rather than against one assumed argmax.
`ok_known` IS NOW DISCLOSURE ONLY - IT NO LONGER SKIPS ANYTHING, and that reversal is the point.
It first gated the whole re-derivation, on the reasoning that `rung_served_cleanly` reads `ok` and
every snapshot predating that field would fail cleanliness for LACK OF PROOF, emptying every
qualifying set and flagging honest rates as unbacked.
That was true of the first shape of the fix and it traded one failure for a worse one: THE STRONGEST