Proposal
Give sweep_sparse_probe a partial-result path so one rejected fit doesn't destroy the sweep (sparse_probing.py:419), and surface per-fit diagnostics on SparseProbeControl (:86).
Motivation
Context first, since the acceptance rule changed after #1774 merged and that PR's description still describes the earlier one: a fit now runs LBFGS then damped Newton refinement, and is accepted only when the Newton decrement is at most decrement_tolerance. A fit that misses it raises, and the sweep has no partial-result path, so everything already computed goes with it. This includes the main probes, which are the expensive part and are typically fine:
X = torch.randn(200, 32, generator=torch.Generator().manual_seed(0))
y = (torch.rand(200, generator=torch.Generator().manual_seed(1)) < 0.5).long()
X[:, 3] += y.float() * 1.5
kw = dict(seed=0, max_refinement_steps=0, decrement_tolerance=1e-14)
sweep_sparse_probe(X, y, ks=[1, 2, 4], n_random_subsets=0, n_label_shuffles=0, **kw) # all three converge
sweep_sparse_probe(X, y, ks=[1, 2, 4], n_random_subsets=10, n_label_shuffles=10, **kw) # RuntimeError on a control
Since the refinement landed this no longer fires at shipped defaults, so it's robustness rather than routine loss. But the failure mode is still "lose everything" for a control that stopped short.
Related, and why that's hard to reason about from a result: stop_reason, newton_decrement and refinement_steps are on SparseProbeResult but not on controls, and on the sweep path most fits are controls.
Pitch
- A rejected fit doesn't discard completed ones; the sweep reports which fits were rejected and why.
SparseProbeControl carries per-repeat diagnostics, or the guide says plainly that it doesn't.fit_sparse_probe keeps raising.
Acceptance:
Checklist
Proposal
Give
sweep_sparse_probea partial-result path so one rejected fit doesn't destroy the sweep (sparse_probing.py:419), and surface per-fit diagnostics onSparseProbeControl(:86).Motivation
Context first, since the acceptance rule changed after #1774 merged and that PR's description still describes the earlier one: a fit now runs LBFGS then damped Newton refinement, and is accepted only when the Newton decrement is at most
decrement_tolerance. A fit that misses it raises, and the sweep has no partial-result path, so everything already computed goes with it. This includes the main probes, which are the expensive part and are typically fine:Since the refinement landed this no longer fires at shipped defaults, so it's robustness rather than routine loss. But the failure mode is still "lose everything" for a control that stopped short.
Related, and why that's hard to reason about from a result:
stop_reason,newton_decrementandrefinement_stepsare onSparseProbeResultbut not on controls, and on the sweep path most fits are controls.Pitch
SparseProbeControlcarries per-repeat diagnostics, or the guide says plainly that it doesn't.fit_sparse_probekeeps raising.Acceptance:
make unit-testpassesuv run mypy .passesChecklist