feat(training): replay gradient-checkpointing schedules, and run the verified CCT one in CI

[runwangdl]

Makes gradient-checkpointing schedules replayable from a supported entry point, and
puts the one verified CCT schedule into CI.

Stacked on #54: based on its branch, so the diff below is against it.

Why

The decision of which activations to drop and regenerate comes from a solve over the
training graph. Replaying that decision on the graph Deeploy actually deploys was
only possible from outside the tree, as a sitecustomize patch over
trainingUtils._mockScheduler driven by SEQ_PATH / SHARE_CONST / TIGHTEN
environment variables. Nothing about it could run in CI.

What

testMVPTraining.py --recomputeSchedule <json> takes {"seq": [[node, is_recompute]]}
in run order. Each recompute becomes a short-lived clone, and consumers scheduled
after it read the clone, so the original's output dies at its last forward use
instead of staying live across the backward pass. The flag is threaded through
deeployTrainingRunner and the GAP9 tiled runner, and create_test_config grows a
training_recompute_schedule so CI configs can set it.

The parts that are not obvious, each of which was a failure before it was handled:

• Lowering can leave a stale same-named node with its outputs cleared. Scheduling it
  puts an empty layer into layerBinding and fails far from the cause, so name
  lookup prefers a node that still produces something.
• The scheduler is called twice, for binding and for the tiler. The second call
  replays the order cached from the first; cloning again would produce a second set
  of clones for the same recomputes.
• Nodes the schedule does not name still have to run, and cannot go at the end. A
  constant duplicated per consumer during folding produces a tensor its consumer
  reads, so appending it after that consumer leaves the input unresolved and context
  lookup fails.
• A solve can place a recompute whose consumers linearise before the clone, or rewire
  every consumer of an original to a clone. Either leaves a node nothing reads, whose
  output cleanup strips. Both are pruned to a fixpoint, since dropping one can orphan
  its producer.
• A clone shares the original's constants rather than copying them. A copy charges
  the full constant again per clone, so recomputing a node that reads a weight grows
  the persistent region instead of shrinking the peak: on the int8 QLoRA CCT, 117
  clones took persistent from 662 KB to 938 KB.

The failure mode this refuses

A schedule is keyed on node names and lowering renames nodes. Replaying a solve
against a graph it did not come from drops every recompute and quietly runs the
default order under a name claiming to be checkpointed, which a pass/fail CI result
cannot distinguish from the real thing. A replay matching under 90% of its entries
raises instead.

CI

model	schedule	Errors	cycles/step
CCT (exact ILP, 250 entries, 6 recomputes)	recompute_checkmate.json	0	68.63M vs 64.24M baseline (+6.8%)

The extra cycles are the point: checkpointing buys peak memory with compute, so this
runs alongside the ordinary singlebuffer entry rather than replacing it, in its own
job so a failure is attributable.

Not included

The four operating points from the memory-budget sweep (peak 1669 / 1575 / 1383 /
1255 KB at budget fractions 0.9 to 0.6) are not here. Their activation figures are
LP predictions validated against codegen, and their latencies are summed per-op
cycles; neither came from a gvsoc run, and the sequences themselves were never
emitted, so there is nothing to replay. Adding them would install unverified
configurations as regression baselines. Producing them needs a solve per point plus a
gvsoc run each.

🤖 Generated with Claude Code