[midagedev]

Metal encode trace의 버퍼 번호가 병렬 가중치 로드 순서와 상관없이 매 실행 같아져, 같은 커밋끼리의 비교가 identical(rc 0)이 됩니다.

What and why

The weights load on several threads (arena.rs, in_order). The encode trace named a buffer #n by the order buffers were made, so a weight buffer's number changed from run to run. A tools/metal-compare.sh of a commit against itself renumbered about half the buffers, and --compare-trace returned 6 ("identical up to buffer numbering"). That is a diagnostic, not a proof, so no Metal change could get a trace proof.

in_order now runs each job under a trace scope. A buffer made inside job j is placed (j, k). When the load ends, the jobs below the earliest failure are numbered from the counter at the load's start, in job order: the numbers a one-thread load would give. Buffers made after the load continue from there.

A buffer with no deterministic place adds to a refused REASON: N line, and the compare refuses that trace (rc 2). The cases are:

• a buffer made on another thread outside a load;
• a buffer made by no job inside a load;
• a nested load;
• a stale job.

The trace format moves to v2. A v1/v2 pair goes through the renumbering path with a note and never returns 0. The engine does not change: without the encode-trace feature the scope is a no-op.

This also corrects a claim in #1139's description, which is struck through and corrected there. The renumbering seen for #1137 came from this load order, not from an extra buffer.

Release notes: internal — Metal's plain-decode compare can prove a change leaves the dispatch unchanged.

Code shape

• No second copy. The trace owns the naming. in_order only opens the scope.
• Model code holds only architecture facts.
• Dead code is gone.
• No silent failure. A buffer the rule cannot place is counted on a refused line, and the compare refuses that trace by name.
• Comments state what is true now.
• No relaxed gate. The compare's rc 6 path is unchanged. A v1/v2 pair cannot return 0.
• cargo fmt, just mac-metal-lint (0 warnings), just mac-check, just mac-lint (ratchet 0).

Duplication found or removed: none.

What I ran

On the device (second Mac, M1 Pro), tools/metal-compare.sh BASE --here TREE with BASE and TREE the same commit (an A/A):

before (main 0d6f89cb8 against itself):
  summary: added=0 removed=0 changed=0 same=381
  identical up to buffer numbering: 52 of 109 buffers renumbered    (depth 0)
  identical up to buffer numbering: 58 of 125 buffers renumbered    (depth 600)
  metal-compare: rc 3 (pipelines rc 0; trace depth ~0 rc 6, depth 600 rc 6; ids differ 0)
this change (1886a4a2b against itself):
  summary: added=0 removed=0 changed=0 same=381
  identical    generated ids: identical (134895,160927,412,163951)
  identical    generated ids: identical (211607,215232,220750,132615)
  metal-compare: rc 0 (pipelines rc 0, added 0; trace depth ~0 rc 0, depth 600 rc 0; ids differ 0)

Host (lead's Mac):

cargo test -p bloomery-metal --lib                                         ok. 160 passed
cargo test -p bloomery-metal --lib --features encode-trace                 ok. 166 passed
cargo test -p bloomery-metal --features pipeline-scan --bin metal_scan     ok. 33 passed
cargo test -p bloomery-metal --features pipeline-scan,encode-trace --bin metal_scan   ok. 34 passed

The encode-trace lib line failed once, on its first run after the build. Its output was overwritten before I read it. The next 130 runs all passed: 30 and then 100 in a loop.

FAIL-first (the round's temporary mutants, removed afterwards):

• A global counter in place of the scoped names fails 6 tests, including both interleaving tests. On 4 workers, job 3's buffers took qwen38 seat hangs with the engine thread spinning (RTX 5090, WSL2), with or without the MTP draft #1–v0.2.6: Qwen3.8 with the MTP draft stops with "MTP window: a step after rows whose hidden rows the step's own arena held" on the second ~29.6K-token request (RTX 5090, WSL2) #3.
• Each single mutant fails exactly its own test:
  • the earlier buffer at an address keeps the name;
  • counted is ignored;
  • an off-thread buffer gets a number;
  • Drop for Load is a no-op;
  • the compare ignores the version.

After the first review and CI (52943de)

• CI check-recipes failed in orphan-tests. It found 7 tests that no recipe runs: the trace's numbering tests and metal_scan's version test. All of them are behind encode-trace.
  • lab-metal now runs the lib and metal_scan tests with encode-trace. The 160 default-feature lib tests are among the 166.
  • metal_scan's features are now the ones metal-compare builds it with.
  • python3 tools/recipes.py orphan-tests → ok (5171 tests, each run by a gate-* or lab-* recipe).
• The review's must-fix (R14). MADE, address and the mixed order now live once, in trace::testing, and both test modules import them.
  • cargo test -p bloomery-metal --lib --features encode-trace → ok. 166 passed.
  • --features pipeline-scan,encode-trace --bin metal_scan → ok. 34 passed.
  • just mac-metal-lint 0 warnings; check-comments ok.
• The engine code is unchanged since 1886a4a, so the device A/A above stands.

AI use: Claude Code wrote this change (a Sonnet round, reviewed and run on the device by the lead session).

🤖 Generated with Claude Code

https://claude.ai/code/session_01CjHkkntc2L1xRN21xB39mj

[github-actions]

Verdict: MUST_FIX: The new tests duplicate a shared load fixture.

Must fix

1. Give the scripted load fixture one owner

crates/metal/src/arch/qwen3moe/arena.rs:1523 and crates/metal/src/trace.rs:548 duplicate the job counts and synthetic address rule (R14). The mixed event order is also copied at arena.rs:1578 and trace.rs:626.
This is a kind B defect under R14. Sharing these now removes two definitions and one event list before more cases depend on both copies, reusing the existing test module without adding a layer.

Move the fixtures into the existing trace::testing module and delete both local MADE/address definitions:

pub(crate) const MADE: [u64; 5] = [2, 1, 0, 3, 2];
pub(crate) const MIXED: [(usize, u64); 8] = [
    (4, 0), (3, 0), (0, 0), (4, 1),
    (1, 0), (3, 1), (0, 1), (3, 2),
];

pub(crate) fn address(j: usize, k: u64) -> usize {
    0x10_0000 + j * 0x1000 + usize::try_from(k).expect("a small count") * 0x10
}

In trace::tests, replace its shared import with:

use super::testing::{MADE, MIXED, Script, address, names, refused, session};

Delete its local mixed array and use ("mixed", MIXED) in the cases. In arena::in_order_tests, add:

#[cfg(feature = "encode-trace")]
use crate::trace::testing::{MADE, MIXED, address};

Replace the successful parallel-load test's literal script with:

let script = Script::new(MIXED.to_vec());

Checked

Duplicates: both fixture copies and the shared trace::testing::Script owner confirmed on added lines.

Complexity: version compatibility and diagnostics add branches to compare::traces; no separate finding.

Silent failure: invalid buffer scopes produce refused records, which the comparison rejects with rc 2.

Tests (static inspection): bypassing job scopes breaks worker-numbering assertions; counting every job breaks the failed-load expectation; ignoring versions breaks the mixed-format assertion.

Gates: no pins or acceptance thresholds moved; mixed versions cannot return rc 0. The PR body reports FAIL-first mutant results.

Escape catalog: C1–C4 do not match this fixture finding. No ledger lessons are in force.

Notes

Validation was static; no Metal tests were run.

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[github-actions]

Since the last review: finding 1 is fixed. trace.rs:472 owns the job counts, event order and address rule. Both test modules import them.

Verdict: CLEAN: The fixture duplication is resolved, and the new commits introduce no further findings.

Checked

Duplicates: 3 relocated items checked; each has one owner and is imported by both test modules.

Complexity: the new commits relocate fixtures and change test recipes; production control flow is unchanged.

Silent failure: 0 new suppression sites. The shared address conversion retains an explicit panic.

Tests: static inspection of trace.rs:150: global-counter numbering assigns the next number to job 4 instead of job 0, contradicting both interleaving assertions. The shared fixtures preserve all events and unique addresses; imports match feature guards.

Gates: justfile:101 enables encode-trace for the library and scan tests. No pins, thresholds or test cases were removed.

Escape catalog: no C1–C4 patterns matched.

Notes

Metal tests were not run here. Validation was static, including fixture and recipe checks.

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}