Two-sided span counting: precision per prediction span, recall per annotation

[omri374]

Summary

SpanEvaluator counted predictions through the annotations they overlapped: a prediction span overlapping two annotations entered num_predicted twice, and a group of same-type spans jointly covering one annotation entered it once. num_predicted therefore drifted both above and below the model's actual output depending on gold layout, an annotation could be counted as both FN and TP, and a blob prediction could earn double TP credit at lenient thresholds (a sloppy span covering two golds scored P=1.0, R=1.0 by being counted twice on both sides).

This PR replaces that with two-sided counting — each metric counted in its natural unit, each unit counted exactly once:

• Recall pass (per annotation) — semantics unchanged: each annotation independently checks whether same-type predictions cover it at IoU ≥ threshold (pairwise for a single span, combined IoU for several) → TP or FN. Predictions are not consumed: one span may satisfy several annotations, so no verdict depends on matching order.
• Precision pass (per prediction span) — num_predicted = actual span count; every span is either credited (participated in ≥1 successful match) or an FP. Precision = (num_predicted − FP) / num_predicted.

The two numerators may legitimately differ (one wide span covering two annotations = 2 recall hits, 1 credited prediction), which is why precision is no longer TP / num_predicted.

Counting contract (all covered by tests)

Scenario	Recall side	Precision side
2 preds jointly cover 1 gold, combined IoU ≥ τ	1 TP	np=2, both credited
2 preds jointly fail	1 FN	np=2, 2 FP (was 1 per group)
2 standalone preds	—	2 FP
1 pred covers 2 golds, each ≥ τ	2 TP	np=1, credited (was np=2, tp=2)
1 pred covers 2 golds, each < τ	2 FN	np=1, 1 FP (was 2)
1 pred matches long gold, swallows short one	1 TP, 1 FN	np=1, credited, 0 FP (was 1 — no more double penalty)

Invariants: TP + FN == num_annotated; num_predicted == actual emitted span count.

Also in this PR

• Single-span coverage uses exact pairwise Span.iou; the combined-IoU path (slightly inflated at boundaries) is reserved for genuine multi-span coverage. Borderline single-span matches can flip at exact threshold values (e.g. true IoU 0.4706 previously computed as 0.50 no longer passes τ=0.5).
• One confusion-matrix cell per span: a wrong-type detection ≥ τ is a single (ann_type, pred_type) cell representing both the gold and the prediction — no fallback (ann_type, "O") / ("O", pred_type) entries.
• ~370 lines of matching machinery removed: the single/multiple-overlap scenario dispatch (_compare_single_overlaps, _compare_multiple_overlaps) collapsed into one uniform rule; dead helpers (_find_best_match, _check_if_matched_already, _handle_unmatched_predictions, _update_wrong_entities, _add_to_processed_predictions) deleted.
• Docs: docs/span_evaluation.md and docs/span_matching_strategies.md corrected where they described the old counting (precision formula, prediction counting, multi-span FP accounting) plus a short section on one prediction overlapping multiple annotations; typed docstrings added throughout span_evaluator.py.
• CHANGELOG: new "Behavior Changes" section under 0.3.2.

Verification

• New tests: test_two_sided_counting_semantics (the 6-scenario contract table, incl. the P = (np−fp)/np ≠ tp/np regression case and both ledger invariants) and test_single_prediction_overlapping_multiple_annotations_counted_once (the original double-counting repro). 681 unit tests pass; 8 pre-existing expectations updated, each a direct consequence of the rules above.
• Differential run vs old code (300 real sentences × 18 types × τ ∈ {0.5, 0.75, 0.9}, seeded perturbed predictions): recall side bit-for-bit identical except 3 verdicts, each traced to an old-code defect (1 annotation counted FN+TP; 2 borderline matches passing only via the combined-IoU off-by-one); num_predicted now equals the independently-counted actual span total everywhere (old code was wrong on 6 of 18 types, in both directions).

Downstream impact

• Anyone computing precision as tp/predicted from per_type counts (or pii_true_positives / pii_predicted) must switch to (predicted − fp)/predicted — tp/predicted can now exceed 1.
• Reported numbers shift on datasets with fragmented or over-wide predictions; that's the correction, not a regression.
• Remaining known issue, deliberately out of scope: the combined-IoU char math has an off-by-one (end+1 phantom character, shifted ranges for subsequent spans) — now isolated to the genuine multi-span path and a candidate for a follow-up fix.

🤖 Generated with Claude Code