Skip to content

Sample one run per precursor for first-pass SVM training - #66

Merged
brendanx67 merged 1 commit into
mainfrom
feature/train-pick-run-parity
Aug 21, 2026
Merged

brendanx67 merged 1 commit into
mainfrom
feature/train-pick-run-parity

Conversation

@brendanx67

Copy link
Copy Markdown
Collaborator

Brings Rust in line with the C# implementation, which shipped this change first
(ProteoWizard/pwiz#4593). Without it the two diverge substantially on Astral, so
this restores the cross-impl gate.

The problem

The first-pass training subsample represented each precursor by its BEST
observation across all files. Every training row was therefore a maximum over
however many runs were batched together, while the model it produces scores
ordinary per-run rows. The mismatch grows with the batch: mean coelution_sum
is 1.19 at one file and 5.75 at 82, against 1.18 in the population actually
being scored.

The change

Represent each precursor by ONE uniformly sampled run, contributing that run's
best candidate peak, at the same row count and the same memory. A size-1
reservoir decides: the k-th run a precursor appears in takes the slot with
probability 1/k, drawn from a SplitMix64 mix of (base_id, k, seed) rather than
a shared RNG, so thread scheduling cannot move the choice and a re-run trains on
the same rows.

Both halves matter. Pass 1 is pre-compaction, so a precursor carries several
candidate-peak rows per run. Drawing per ROW rather than per RUN would weight a
run by how many candidates it produced and leave a random candidate as the
training row, which measured -17.4% identifications on a 3-file test in C#.

Targets and decoys of one base_id are separate precursors that share it, so
both the counter key and the draw take the decoy bit -- masking it off would make
a target and its paired decoy decide identically at every k and land on the same
run for essentially every precursor.

No opt-out switch. C# keeps one only as an A/B lever for the change itself; Rust
tracks the shipped default.

Results

82 SEA-AD Astral files, matched true entrapment FDP, experiment scope:

pass 1 pass 2
baseline (cross-run maximum) 34,552 35,327
one sampled run 45,943 52,911
+33.0% +49.8%

A wash at 3 files on every dataset carrying a true-FDP oracle, so this is a
large-batch effect, as the mechanism predicts.

Cross-impl parity

Before this change, Rust and C# diverged on the 3-file Astral set by 2,096,935
of 6,226,744 scores, 454 precursors, and 85 protein-group winner flips.

Compare-EndToEnd-Crossimpl.ps1 now reports PASS at 1e-9 on both Stellar and
Astral
-- precursors, blib content, Stage 7 protein FDR, and FDR sidecars.

Also: pearson_correlation_raw on degenerate input

A constant series has zero variance and no defined correlation. Clamping the
denominator to 1e-30 and dividing by its root manufactures an arbitrary tiny
value out of numerator noise. This returns 0.0 instead, matching both C# helpers
(PearsonCorrelation.Pearson and TukeyMedianPolish.PearsonCorrelationRaw).

It is in this PR because it has no independent motivation -- it earns its place
as the last thing the parity gate needed. It surfaced as
median_polish_residual_correlation = -8.47e-7 against C#'s exactly 0 on one
precursor whose every fragment pair was degenerate, moving one SVM score by
8.6e-8 (~48 million ULPs at that magnitude, so not rounding noise).

test_pearson_constant_input previously asserted only is_finite(), which the
clamped form satisfies. It now pins the value, plus the both-constant and
all-zeros cases.

Testing

  • cargo fmt --check, cargo clippy -D warnings, cargo test -- all pass
  • New reservoir_takes_slot_matches_cross_impl_vectors: 21 pinned
    (base_id, seen, expected) values shared with C#, including decoy-bit-set
    cases and true values at seen=5/7/82 so a broken implementation cannot pass
    by returning false
  • New reservoir_spreads_winners_evenly_over_runs: mirrors the C# uniformity
    test over 82 runs and 200,000 precursors
  • Compare-EndToEnd-Crossimpl.ps1 on Stellar and Astral, both PASS at 1e-9

The training subsample represented each precursor by its BEST observation
across all files. Every training row was therefore a maximum over however
many runs were batched together, while the model it produces scores ordinary
per-run rows. The mismatch grows with the batch: mean coelution_sum is 1.19 at
one file and 5.75 at 82, against 1.18 in the population actually being scored.

Represent each precursor by ONE uniformly sampled run instead, contributing
that run's best candidate peak, at the same row count and the same memory. A
size-1 reservoir decides: the k-th run a precursor appears in takes the slot
with probability 1/k, drawn from a SplitMix64 mix of (base_id, k, seed) rather
than a shared RNG, so thread scheduling cannot move the choice and a re-run
trains on the same rows.

Both halves matter. Pass 1 is pre-compaction, so a precursor carries several
candidate-peak rows per run; drawing per row rather than per run would weight a
run by how many candidates it produced and leave a random candidate as the
training row, which measured -17.4% identifications on a 3-file test.

Measured on 82 SEA-AD Astral files at matched true entrapment FDP: +33.0%
identifications at pass 1 and +49.8% at pass 2. A wash at 3 files on every
dataset carrying a true-FDP oracle.

Also fixes pearson_correlation_raw on degenerate input. A constant series has
zero variance and no defined correlation; clamping the denominator to 1e-30 and
dividing by its root manufactures an arbitrary tiny value out of numerator
noise. Return 0.0 instead, which is both the defensible answer and what both C#
helpers do -- PearsonCorrelation.Pearson and TukeyMedianPolish's private
PearsonCorrelationRaw.

This restores end-to-end cross-impl agreement with the C# implementation, which
shipped the run-level sampling first. Before this change the two diverged on
2,096,935 of 6,226,744 scores, 454 precursors, and 85 protein-group winner
flips on the 3-file Astral set.

There is deliberately no opt-out switch here. The C# side keeps one only as an
A/B lever for the change itself; Rust tracks the shipped default.
@brendanx67
brendanx67 merged commit 7343ffd into main Aug 21, 2026
3 checks passed
@brendanx67
brendanx67 deleted the feature/train-pick-run-parity branch August 21, 2026 22:53
brendanx67 pushed a commit that referenced this pull request Sep 27, 2026
…69)

The first-pass grid search kept the strict maximum of the inner-CV passing
counts, but neighboring C values usually pass within noise of each other: on
3-file Stellar, C = 0.1, 1 and 10 passed 5,025, 5,037 and 5,002 targets. The
pick then flips on tiny input changes, and it matters. A weakly regularized
C = 1 fit splits weight between correlated spectral features (apex-scan xcorr
and median_polish_cosine against the multi-scan sg_weighted_cosine) in a way
that scores the first pass alike but the second pass, which reuses the frozen
first-pass model on reconciled peaks, much worse. Two Stellar libraries that
differed only by 1e-4 GPU/CPU rounding gave 21,176 and 28,309 experiment
precursors at the same entrapment FDP.

grid_search_c now keeps the smallest C whose count is within
PercolatorConfig::c_selection_tolerance (default 0.01) of the best, through a
new svm::select_c; 0 keeps the strict maximum with the first C in grid order
winning a tie, as before. In C# the same rule took the two libraries above to
30,316 and 30,485, and moved the regression datasets' experiment precursors
by +16% (Stellar), +1% (Stellar library decoys), +6% (Stellar generated decoys
with entrapment, whose measured FDP stayed within one standard error of 1%)
and 0% (Astral).

Cross-impl: mirrors C# PercolatorTrainer.SelectC, which shipped the rule
first; test_select_c_tolerance runs the same cases as C#
FdrTest.TestSvmCSelectionTolerance. As with #66 there is no opt-out here;
C# keeps OSPREY_SVM_C_TOLERANCE only for A/B work.

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant