Add model-agreement stats to Taxa views

[mihow]

Follow-up to #1316 / #1317 (now scoped to verification status only) and #1307 (parents_json rollup performance).

Goal

Expose per-taxon model-agreement metrics on the taxa list and detail endpoints, rolled up across descendants via parents_json. Two distinct signals:

• model_agreed_with_prediction_count — occurrences whose chosen Identification used the "Agree with prediction" UI shortcut (i.e. Identification.agreed_with_prediction_id is set). Measures explicit user endorsement of the model's pick.
• model_agreed_exact_count — occurrences whose final determination_id equals the top machine Classification.taxon_id for that occurrence, regardless of how the user got there (typed-in name vs. Agree shortcut). Broader signal — also captures cases where a user typed a name that happens to match the model output.

Both counts roll up descendant occurrences (a Family/Order row aggregates its species), and both are restricted to verified occurrences (those with at least one non-withdrawn Identification).

Proposed API

• New query param: ?with_model_agreement=true (gated; default off to keep the list endpoint cheap).
• List endpoint (GET /api/v2/taxa/?with_model_agreement=true) — adds the two annotations to each row.
• Detail endpoint (GET /api/v2/taxa/<id>/) — always returns both counts unconditionally.
• model_agreed_with_prediction_count and model_agreed_exact_count added to ordering_fields.

Why this is its own PR

PR #1317 originally bundled these counts with the verification UX. The two stories have different audiences (naturalist trust vs. ML evaluation) and the model-agreement surface had no FE consumer at merge time. Splitting keeps each PR focused.

Implementation guidance — patterns proven in PR #1317

Most of the heavy lifting already exists on worktree-taxa-verification-counts (now trimmed to verification only). The model-agreement implementation should reuse the same patterns. Detailed reference: docs/claude/reference/hierarchical-rollup-query-performance.md.

Computation strategy

1. Use the sparse CASE-from-map pattern, not a correlated parents_json subquery.
   A per-taxon correlated parents_json @> [{"id": OuterRef("id")}] subquery cannot use the GIN index (main_taxon_parents_json_gin_idx, migration 0087) because GIN jsonb_path_ops only serves containment with a constant RHS, not an OuterRef. PR Add verification status to Taxa views #1317 measured a 30s timeout on a ~1k-taxa / ~17k-occurrence project with that shape.

   Instead, iterate the sparse verified-occurrence set once in Python, build {taxon_id: count} dicts (incrementing the determination taxon and every ancestor in parents_json), and apply as constant-time CASE annotations. The result is DB-sortable, paginatable, and stripped from the pagination COUNT.

2. Sparse maps only. Dense CASE-from-map (one When per observed taxon × multiple columns) breaks past sqlparse's 10000-token limit. The verified subset is bounded by human review effort, so it stays sparse — safe.

3. Extend TaxonQuerySet.with_verification_counts (ami/main/models.py). The method already accepts an include_agreement flag and annotates _best_machine_taxon_id via a Classification subquery ordered by BEST_MACHINE_PREDICTION_ORDER. Reintroduce the two count maps walking the same iterator over the verified-occurrence values, emit two extra CASE annotations.

4. No change needed to the hybrid subquery/aggregation dispatch in get_taxa_observed. Model-agreement counts are computed on top of the verification base, not parallel to the observation aggregates.

Gotchas to preserve

5. Detections fan-out under ?collection=<id>. When occurrence_filters joins to detections, a single occurrence yields one .values() row per matching detection, which inflates the count maps. PR Add verification status to Taxa views #1317 fix (commit 10c72cbd): include pk in .values(...) and chain .distinct(). Regression test: test_verified_count_not_inflated_by_collection_join. Reuse the same dedup; add an equivalent regression test for each new count.

6. parents_json round-trip through django-pydantic-field. Elements may come back as dict or as TaxonParent instances depending on the query path. Read defensively:

   parent_id = parent.get("id") if isinstance(parent, dict) else getattr(parent, "id", None)

7. Distinguish from human verification in naming. Use the model_agreed_* prefix to avoid conflating ML-eval signals with the human-trust verified_count. The bare agreed_* was rejected on PR Add verification status to Taxa views #1317 review for this reason.

8. Gate behind ?with_model_agreement=true on the list endpoint. The Classification subquery is heavier than the verification base; the default taxa list should not pay the cost. Detail view always includes both counts.

9. Single boolean parser. PR Add verification status to Taxa views #1317 had two parsers for with_agreement — a custom string match in serializers.py and BooleanField().clean() in views.py. mihow flagged the duplication. Pass include_model_agreement from the view to the serializer via context, parse once.

10. Serializer field-pop pattern. On the detail serializer, declare both fields unconditionally. On the list serializer, pop them in __init__ unless the context flag is set.

Tests to port

PR #1317 had these tests covering the bundled-in agreed_* counts (deleted as part of the trim). Port them, renamed and updated for model_agreed_*:

• test_agreed_with_prediction_counts_only_chosen_identification — only the chosen (best non-withdrawn) Identification matters, not all Identification rows on an occurrence.
• test_agreed_exact_count_on_detail — exact-match count present on detail.
• test_agreed_exact_count_gated_on_list — absent unless with_model_agreement=true.
• Hierarchical rollup (species verifies → genus / family rows increment).
• Distinct-dedup under ?collection= (regression for the detections fan-out).

Relationship to #1307

#1307 investigated parents_json rollup performance and flagged the missing GIN index as a blocker. PR #1317 added that index (migration 0087, jsonb_path_ops, CONCURRENTLY / IF NOT EXISTS). The index does not back the rollup itself (OuterRef RHS); it serves the pre-existing literal-RHS consumers (occurrence-list ?taxon=<id>, the project default-taxa filter). The Python-pass rewrite is what actually resolved the rollup timeout.

Performance budget

PR #1317 final timings on the verification scope on a production-scale project (~1k taxa / 17k occurrences):

path	ms
default limit=25	~0.9s (target after hybrid restore)
verified=true	~0.8s
verified=false	~1.2s
?collection=<id>	~1.8s

Adding the two model-agreement counts in the gated case should add negligible cost — same iterator over the same sparse set, two additional dict accumulators. The Classification subquery is already paid for via _best_machine_taxon_id. Bench before / after to confirm no regression on the gated path; the ungated default path should be untouched.

Future scaling

If the precompute pass ever becomes the dominant cost, denormalise into a TaxonObserved model per (project, taxon) holding verified / agreed / observation aggregates, refreshed via the cached-count pattern. Sketched but not scoped — only worth pursuing if measurements show the in-request pass is hot.