Skip to content

bench: factor the fresh-process probe harness into bench/_probe.py - #169

Merged
sbryngelson merged 1 commit into
sbryngelson:mainfrom
axiom-of-choice:feat-163-probe-helper
Aug 4, 2026
Merged

bench: factor the fresh-process probe harness into bench/_probe.py#169
sbryngelson merged 1 commit into
sbryngelson:mainfrom
axiom-of-choice:feat-163-probe-helper

Conversation

@axiom-of-choice

Copy link
Copy Markdown
Contributor

Fixes #163.

The helper

bench/_probe.py with the surface you suggested: probe_cell(build_graph, feed) compiles and dispatches one graph in-process with the breaker disabled, probe_isolated(argv, script) runs one in a fresh interpreter and parses the verdict. Both take callables rather than values so nothing imports aneforge before the env is set, and probe_isolated ignores stderr on purpose since E5RT writes compiler noise there.

Also moved short() and chip() over, since every table consumer wants them.

Behaviour is unchanged

--affine-scan 512 is byte-identical before and after, by diff:

$ diff /tmp/affine_before.txt /tmp/affine_after.txt && echo IDENTICAL
IDENTICAL

The other three modes still print the same shapes; --scan-d 512 reproduces the same failing set I reported on #149 ([1024, 1536, 2560, 4096, 10240, 12288, 16384]).

The second consumer

bench/rms_norm_compile_probe.py, which is about 60 lines because the harness is inherited. It doubles as the control for #149: rms_norm has a single affine slot, so it can never reach the asymmetric one-term-live state your affine finding points at. On M2 Pro:

$ python bench/rms_norm_compile_probe.py
Apple M2 Pro: rms_norm at R=512 seed=13, one fresh process per cell, breaker disabled

  [  512,   512]  OK
  ... every cell ...
  [  512, 16384]  OK

15/15 pass. No failures, as the single-affine-slot control predicts (#149).

15/15 clean, including the 7 cells where layer_norm fails at the same R and seed. So the helper generalizes and the first thing it produced supports the theory.

One subtlety in the port, worth a look

The original drew gamma, beta and the input from one RNG stream in that order. Splitting the work into a build_graph and a feed made it easy to re-derive default_rng(seed) inside each callable, which silently shifts the stream and changes the input tensor for a given seed. My first version did exactly that and I only caught it by comparing arrays.

The fix is to materialize all three up front in the original order and close over them, so a seed keeps producing byte-identical tensors:

rng = np.random.default_rng(seed)
g_rand = ...   # gamma first
b_rand = ...   # then beta
x_in   = ...   # then the input, at the stream position the original reached

Verified with np.array_equal against the original ordering, not just by the table matching.

Verification

Apple M2 Pro (Mac14,12 Mac mini), macOS 26.5.2.

Check Result
--affine-scan 512 before vs after byte-identical
--scan-d, --one --repeat, default matrix unchanged
bench/rms_norm_compile_probe.py 15/15 pass
ruff check bench/ clean
pyright on the three files 0 errors
.githooks/pre-commit OK

Nothing under aneforge/ is touched, so there is no library behaviour to regress and the corpus is unaffected.

probe_cell's annotations are string-quoted under TYPE_CHECKING so the module still imports without pulling in aneforge or numpy at load time; pyright flagged the looser object signature I started with, which is what pushed me to the concrete types.

Implements sbryngelson#163. The reusable core of the layer_norm probe was welded to that op:
one compile per fresh process with the circuit breaker disabled, and outcomes
staged as OK / C-FAIL / D-FAIL. Both matter, since a compile failure otherwise
paces the next cell and makes later cells look broken, and a compile failure has a
different cause than a dispatch failure.

probe_cell takes a build_graph and a feed so a caller supplies only its graph;
probe_isolated runs one in a subprocess and parses the verdict, ignoring stderr
because E5RT writes compiler noise there.

layer_norm_compile_probe.py now consumes the helper with no behavior change:
--affine-scan 512 is byte-identical before and after, verified by diff, and the
other three modes still print the same shapes.

Adds rms_norm_compile_probe.py as the second consumer. It doubles as the control
for sbryngelson#149: rms_norm has a single affine slot so it can never reach the asymmetric
one-term-live state, and it passes 15/15 on M2 Pro where layer_norm fails 7 of the
same cells.

One subtlety in the port: the original drew gamma, beta and the input from one RNG
stream in that order, so the tensors are materialized up front in the same order
rather than re-derived inside each callable, which would have shifted the stream
and changed the inputs for a given seed. Confirmed byte-identical.
@sbryngelson
sbryngelson merged commit 9ace07c into sbryngelson:main Aug 4, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bench: reusable per-op compile/execute probe helper

2 participants