bench: factor the fresh-process probe harness into bench/_probe.py - #169
Merged
sbryngelson merged 1 commit intoAug 4, 2026
Merged
Conversation
Implements sbryngelson#163. The reusable core of the layer_norm probe was welded to that op: one compile per fresh process with the circuit breaker disabled, and outcomes staged as OK / C-FAIL / D-FAIL. Both matter, since a compile failure otherwise paces the next cell and makes later cells look broken, and a compile failure has a different cause than a dispatch failure. probe_cell takes a build_graph and a feed so a caller supplies only its graph; probe_isolated runs one in a subprocess and parses the verdict, ignoring stderr because E5RT writes compiler noise there. layer_norm_compile_probe.py now consumes the helper with no behavior change: --affine-scan 512 is byte-identical before and after, verified by diff, and the other three modes still print the same shapes. Adds rms_norm_compile_probe.py as the second consumer. It doubles as the control for sbryngelson#149: rms_norm has a single affine slot so it can never reach the asymmetric one-term-live state, and it passes 15/15 on M2 Pro where layer_norm fails 7 of the same cells. One subtlety in the port: the original drew gamma, beta and the input from one RNG stream in that order, so the tensors are materialized up front in the same order rather than re-derived inside each callable, which would have shifted the stream and changed the inputs for a given seed. Confirmed byte-identical.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #163.
The helper
bench/_probe.pywith the surface you suggested:probe_cell(build_graph, feed)compiles and dispatches one graph in-process with the breaker disabled,probe_isolated(argv, script)runs one in a fresh interpreter and parses the verdict. Both take callables rather than values so nothing imports aneforge before the env is set, andprobe_isolatedignores stderr on purpose since E5RT writes compiler noise there.Also moved
short()andchip()over, since every table consumer wants them.Behaviour is unchanged
--affine-scan 512is byte-identical before and after, bydiff:The other three modes still print the same shapes;
--scan-d 512reproduces the same failing set I reported on #149 ([1024, 1536, 2560, 4096, 10240, 12288, 16384]).The second consumer
bench/rms_norm_compile_probe.py, which is about 60 lines because the harness is inherited. It doubles as the control for #149:rms_normhas a single affine slot, so it can never reach the asymmetric one-term-live state your affine finding points at. On M2 Pro:15/15 clean, including the 7 cells where
layer_normfails at the same R and seed. So the helper generalizes and the first thing it produced supports the theory.One subtlety in the port, worth a look
The original drew gamma, beta and the input from one RNG stream in that order. Splitting the work into a
build_graphand afeedmade it easy to re-derivedefault_rng(seed)inside each callable, which silently shifts the stream and changes the input tensor for a given seed. My first version did exactly that and I only caught it by comparing arrays.The fix is to materialize all three up front in the original order and close over them, so a seed keeps producing byte-identical tensors:
Verified with
np.array_equalagainst the original ordering, not just by the table matching.Verification
Apple M2 Pro (Mac14,12 Mac mini), macOS 26.5.2.
--affine-scan 512before vs after--scan-d,--one --repeat, default matrixbench/rms_norm_compile_probe.pyruff check bench/pyrighton the three files.githooks/pre-commitNothing under
aneforge/is touched, so there is no library behaviour to regress and the corpus is unaffected.probe_cell's annotations are string-quoted underTYPE_CHECKINGso the module still imports without pulling in aneforge or numpy at load time; pyright flagged the looserobjectsignature I started with, which is what pushed me to the concrete types.