From cae65a7a085da8779014dbe9c84d4f5d8f2808fb Mon Sep 17 00:00:00 2001 From: Michael Bunsen Date: Thu, 2 Jul 2026 13:58:19 -0700 Subject: [PATCH 1/4] docs: add Proposal A implementation plan for regional taxa lists (#1364) [skip ci] Staged, TDD-oriented implementation plan for building a project taxa list from a geographic region so class masking works out of the box. Covers the reusable core service, the union-with-provenance source merge (sources union, never intersect), Site/Project data-model fields, region derivation for backfill, the class-masking auto-resolution order, the five surfaces, a test plan, and a phased rollout. Refs #1364, #999, #1289 Co-Authored-By: Claude --- ...roposal-a-regional-taxa-lists-impl-plan.md | 450 ++++++++++++++++++ 1 file changed, 450 insertions(+) create mode 100644 docs/claude/planning/2026-07-02-proposal-a-regional-taxa-lists-impl-plan.md diff --git a/docs/claude/planning/2026-07-02-proposal-a-regional-taxa-lists-impl-plan.md b/docs/claude/planning/2026-07-02-proposal-a-regional-taxa-lists-impl-plan.md new file mode 100644 index 000000000..a26b0aaaf --- /dev/null +++ b/docs/claude/planning/2026-07-02-proposal-a-regional-taxa-lists-impl-plan.md @@ -0,0 +1,450 @@ +# Implementation plan — Proposal A: build a project taxa list from a region + +**Status:** planning (not started) +**Issue:** [#1364](https://github.com/RolnickLab/antenna/issues/1364) — "Let users build a project taxa list from a region, so class masking works out of the box" +**Design doc (authoritative):** `docs/claude/planning/2026-07-02-regional-taxa-lists-class-masking.md` +**Depends on / relates to:** #999 (class masking), #1289 (post-processing framework, merged), #1094 (configurable taxa lists), #939 (import taxa), #1020 (closed — browser GBIF taxon-select, UI reference), #1293 (taxa list CSV export) +**Author:** Claude (agent), 2026-07-02 + +This document turns the design doc's "Proposal A", "reusable core", "data model", and "resolution order" sections into a staged, test-first implementation plan with concrete file paths, function signatures, and anchors into existing code to reuse. It is a public-repo planning doc: full prose, and every claim about an external API is labelled as unverified until it has been exercised from this codebase. + +--- + +## 1. What we are building and why + +Class masking (#999) keeps only the classifier predictions whose taxon appears in a chosen taxa list, cutting a global classifier down to the species that actually occur at a site. The blocker is that the taxa list has to be hand-curated one taxon at a time, and nothing ties a list to the place a project monitors. + +Proposal A adds one capability — **"given a region, produce a taxa list"** — written once as a service function and surfaced five ways (management command, unit test, Django admin action, DRF endpoint, main UI). It fetches the species recorded in a geographic region from one or more external biodiversity databases (GBIF, iNaturalist), maps them onto Antenna `Taxon` rows, optionally reports how many fall inside a classifier's label set, and saves a project-scoped `TaxaList`. It also adds region/list fields to `Site` and `Project` and a resolution order so the class-masking task can pick the right list automatically per occurrence. + +### 1.1 The one design decision to get right: multiple sources UNION, they do not INTERSECT + +When more than one source is queried (GBIF **and** iNaturalist, say), a species present in **any** source is a candidate for the regional list. The sources are combined with a **wide join / concatenation**, not an intersection: + +- The merged intermediate is a **wide table**: one row per canonical species, carrying per-source provenance columns — which source(s) contributed the species, each source's native key (`gbif_taxon_key`, `inat_taxon_id`, …), and each source's observation count. +- Sources are **never intersected against each other**. Querying two sources can only grow the candidate set, never shrink it. +- The **only** intersection in the whole flow is the *optional, final, reporting-only* step against a classifier's label set. That intersection does not remove species from the saved list — it only reports "of the N species in this regional list, M are actually in classifier X's labels, so masking will affect M of them." It is a usefulness metric, not a filter. + +Every abstraction below (`SourceSpecies`, the source protocol, the merge function, the `Result` summary) is shaped around this union-with-provenance model. If a future requirement wants "species seen in at least K sources", that is a *post-merge* filter over the provenance columns, not a change to how sources combine. + +--- + +## 2. Architecture: one core service, five thin surfaces + +``` + generate_regional_taxa_list() ← the only place the logic lives + │ + ┌──────────────┬─────────────┼──────────────┬───────────────┐ + mgmt command unit tests admin action DRF endpoint main UI + (+ --all-projects) (stub source) (Project/Site) (POST …/regional/) (button → API) +``` + +Internally the service is a short pipeline: + +``` +region code ─▶ [Source clients] fetch_species() per source → list[SourceSpecies] + ─▶ merge_source_species() WIDE UNION on canonical key, provenance preserved + ─▶ map_to_taxa() match/create Antenna Taxon rows + ─▶ (optional) intersect_classifier() REPORT-ONLY coverage against a classifier's labels + ─▶ save / update TaxaList idempotent, project-scoped + ─▶ Result counts + unmatched names for human review +``` + +**New module:** `ami/main/services/regional_taxa.py` (new `services` package under `ami/main/`, per the "processing logic extraction" note in `CLAUDE.md`; add `ami/main/services/__init__.py`). + +**Reuses (do not reinvent):** +- `ami/utils/requests.py::create_session()` (`:14`) — all outbound HTTP, for retry/backoff. +- `ami/main/management/commands/import_taxa.py::create_taxon()` and `get_or_create_root_taxon()` — the rank-hierarchy builder for creating missing `Taxon` rows under the `Arthropoda` root. This logic currently lives inside the command module; see §5.3 for extracting it so the service can call it without importing a management command. +- `TaxaList.objects.get_or_create_for_project(name, project=None)` (`ami/main/models.py:4598`) — idempotent list creation, project-scoped. +- `AlgorithmCategoryMap.with_taxa()` (`ami/ml/models/algorithm.py:105`) — the name-keyed category→taxon resolution masking already uses; reused for the classifier-coverage report. +- `Project.objects.all()` loop shape from `ami/main/management/commands/assign_roles.py:79` — the `--all-projects` backfill template. + +--- + +## 3. Data model changes + +Minimal columns; polygons/geometry are explicitly out of scope (design doc Proposal C). Every field ships with its migration in the same PR (`makemigrations --check --dry-run` must pass — `CLAUDE.md` domain invariant). + +### 3.1 `Site` (`ami/main/models.py:654`) + +| Field | Type | Notes | +|---|---|---| +| `region_source` | `CharField(choices=RegionSource.choices, blank=True)` | `gbif_gadm` \| `inat_place`; blank = unset | +| `region_code` | `CharField(max_length=64, blank=True)` | a GADM GID (`USA.11_1`) or an iNat `place_id` string | +| `taxa_list` | `ForeignKey("TaxaList", null=True, blank=True, on_delete=SET_NULL, related_name="+")` | the designated list for this site | + +### 3.2 `Project` (`ami/main/models.py:289`) + +| Field | Type | Notes | +|---|---|---| +| `region_source` | `CharField(choices=RegionSource.choices, blank=True)` | fall-back source when a site has none | +| `region_code` | `CharField(max_length=64, blank=True)` | fall-back region code | +| `default_taxa_list` | `ForeignKey("TaxaList", null=True, blank=True, on_delete=SET_NULL, related_name="+")` | fall-back list when a site has none | + +`RegionSource` is a `models.TextChoices` enum defined once (suggested home: `ami/main/models.py` near `TaxaList`, or a small `ami/main/services/regional_taxa.py` constant re-exported — keep the enum with the model so migrations reference a stable path). + +**Relationship note.** `TaxaList↔Project` (`related_name="taxa_lists"`) and `TaxaList↔Taxon` (`related_name="lists"`) stay M2M. The new `taxa_list` / `default_taxa_list` are single FKs *layered on top*: a project still associates with many lists, but names one default. `related_name="+"` avoids reverse-accessor clutter on `TaxaList`. + +**Migrations:** one migration in `ami/main/migrations/` adding all six fields (they are all additive, nullable/blank, no data change). If `Project` and `Site` edits are cleaner as two migrations, that is fine; either way they are state+schema migrations, not data migrations. + +--- + +## 4. Source abstraction + +### 4.1 `SourceSpecies` — the per-source record (the "wide table" row, pre-merge) + +```python +# ami/main/services/regional_taxa.py +import dataclasses + +@dataclasses.dataclass(frozen=True) +class SourceSpecies: + """One species as reported by ONE source for a region. + + The merge step (§4.4) concatenates these across sources and deduplicates on a + canonical key, unioning provenance. Fields are deliberately source-agnostic so + a new source only has to populate what it knows. + """ + source: str # RegionSource value, e.g. "gbif_gadm" + scientific_name: str # canonical binomial as the source spells it + rank: str | None = None # "SPECIES", "SUBSPECIES", … when the source gives it + gbif_taxon_key: int | None = None + inat_taxon_id: int | None = None + observation_count: int | None = None # source's record/observation count in the region + raw: dict | None = None # untouched source payload for debugging / future fields +``` + +### 4.2 The source protocol + +```python +import typing + +class RegionalSpeciesSource(typing.Protocol): + source_key: str # a RegionSource value + + def fetch_species( + self, region_code: str, taxon_scope: "TaxonScope" + ) -> list[SourceSpecies]: + """Return every species the source records in `region_code`, within + `taxon_scope` (e.g. Lepidoptera). Paginates internally; all HTTP goes + through create_session(). Raises on transport/HTTP error — never returns + a partial list silently (CLAUDE.md: raise, don't return sentinels).""" +``` + +`TaxonScope` is a tiny value object holding the source-specific root-taxon identifiers, so the caller says "Lepidoptera" once and each source translates it: + +```python +@dataclasses.dataclass(frozen=True) +class TaxonScope: + label: str # "Lepidoptera", for logging + gbif_taxon_key: int | None # e.g. 797 (Lepidoptera) — CANDIDATE, verify live + inat_taxon_id: int | None # e.g. 47157 (Lepidoptera) — CANDIDATE, verify live +``` + +Default scope: Lepidoptera (moth/butterfly platform). Keep it a parameter so a project could scope to Arthropoda or a family later. + +### 4.3 Concrete sources (endpoints are CANDIDATE, UNVERIFIED) + +No live GBIF/iNat code exists in `ami/` today — these fields are only ever populated from files. The endpoints below are from published API docs and must be exercised against the live services (a recorded-response fixture per source) before they are trusted. Rate limits and exact faceting behaviour are the first things to verify (§10, §11). + +**`GBIFRegionalSource`** — GBIF occurrence search, faceted by species. +- Candidate endpoint: `GET https://api.gbif.org/v1/occurrence/search` +- Candidate params: `facet=speciesKey`, `facetLimit=`, `facetOffset=`, `gadmGid=` (or `country=` when the code is a country), `taxonKey=`, `hasCoordinate=true`, `limit=0` (we want facet buckets, not records). +- Pagination: page `facetOffset`/`facetLimit` until a facet page returns fewer than `facetLimit` buckets. GBIF caps facet depth, so very species-rich regions may need the "download" API instead — note as a known limit, not a v1 blocker. +- Each facet bucket gives a `speciesKey` and a count; resolve the key to a scientific name via `GET /v1/species/{speciesKey}` (batchable, cache per key). Populates `gbif_taxon_key`, `scientific_name`, `observation_count`. + +**`INaturalistRegionalSource`** — iNat species counts by place. +- Candidate endpoint: `GET https://api.inaturalist.org/v1/observations/species_counts` +- Candidate params: `place_id=`, `taxon_id=`, `quality_grade=research`, `per_page=200`, `page=`. +- Pagination: increment `page` until `results` is short of `per_page` or `total_results` is reached. +- Each result has `taxon.id`, `taxon.name`, `taxon.rank`, `count`. Populates `inat_taxon_id`, `scientific_name`, `rank`, `observation_count`. + +Both share an HTTP helper built on `create_session()`; add a small on-disk/Redis response cache keyed by `(source, region_code, taxon_scope)` because these queries are slow and rate-limited, and the same region is re-queried on every idempotent re-run. Cache TTL on the order of days is fine (regional checklists change slowly). + +### 4.4 The wide merge — `merge_source_species()` + +```python +def merge_source_species( + per_source: list[list[SourceSpecies]], +) -> list["MergedSpecies"]: + """Concatenate species across sources and deduplicate on a canonical key, + UNIONING provenance. This is a wide join, never an intersection: a species in + ANY source survives. Two source rows collapse into one MergedSpecies when they + share a canonical key. + + Canonical key precedence (first that both rows share wins): + 1. gbif_taxon_key + 2. inat_taxon_id + 3. normalized scientific_name (casefold + collapse whitespace) + """ +``` + +```python +@dataclasses.dataclass +class MergedSpecies: + scientific_name: str + rank: str | None + gbif_taxon_key: int | None + inat_taxon_id: int | None + sources: set[str] # provenance: which sources contributed + observation_counts: dict[str, int] # provenance: per-source count + contributing: list[SourceSpecies] # raw rows, for audit +``` + +Merge rules: +- **Dedup key** as above. Name-only collision without a shared external key is the fragile case — normalize (`casefold`, strip authorship if present, collapse whitespace) and, when two rows merge on name but disagree on an external key, keep both keys and log a provenance warning rather than silently dropping one. +- **Provenance union:** `sources = ∪ row.source`; `observation_counts[row.source] = row.observation_count`; keep both external keys if each source supplied one; prefer the GBIF name spelling when present, else the first. +- Output is stable-ordered (e.g. by `-max(observation_counts)`, then name) so runs are reproducible and diffs are readable. + +--- + +## 5. Mapping merged species → Antenna `Taxon` + +### 5.1 Match precedence — `map_to_taxa()` + +```python +def map_to_taxa( + merged: list[MergedSpecies], *, create_missing: bool, dry_run: bool, +) -> "MappingOutcome": + """Resolve each MergedSpecies to a Taxon. Match precedence: + 1. Taxon.gbif_taxon_key == merged.gbif_taxon_key + 2. Taxon.inat_taxon_id == merged.inat_taxon_id + 3. Taxon.name == merged.scientific_name (exact; Taxon.name is unique) + On no match: create via the rank-hierarchy builder when create_missing, else + record the name as unmatched for human review. Never mutate on dry_run.""" +``` + +`MappingOutcome` carries: `matched: list[tuple[MergedSpecies, Taxon]]`, `created: list[Taxon]`, `unmatched_names: list[str]`. Do the external-key matches as two bulk `filter(..__in=[...])` queries (not per-row) — `CLAUDE.md`: no queries in loops. + +### 5.2 The name-join coupling (why exact match matters here) + +`Taxon.name` is globally unique (`models.py:4346`), and class masking joins list membership to classifier labels by **name** (`AlgorithmCategoryMap.with_taxa()`, keyed on `taxon.name` at `algorithm.py:135`). So: +- Matching a source species to an existing `Taxon` by external key and then trusting its `.name` is correct — the list stores `Taxon` rows, and masking will later match those same names to classifier labels. +- A source name that differs from both any `Taxon.name` and the classifier's label spelling will silently fail to mask even after we add it to the list. This is the **name-match fragility** open question (§10); the service surfaces `unmatched_names` and the classifier-coverage count precisely so a human can see the miss rate before trusting auto-masking. + +### 5.3 Extract `create_taxon` for reuse + +`create_taxon()` and `get_or_create_root_taxon()` live inside `ami/main/management/commands/import_taxa.py` today (module-level functions, so importable, but coupling the service to a command module is awkward). Extract them into `ami/main/services/taxonomy.py` (or a shared helper the command re-imports) so both the command and the new service call one implementation. This is a mechanical move-and-reimport; keep the command's behaviour identical and covered by its existing tests. + +--- + +## 6. The core service function + +```python +# ami/main/services/regional_taxa.py + +@dataclasses.dataclass +class RegionalTaxaResult: + region_source: str + region_code: str + taxa_list_id: int | None # None on dry_run + list_created: bool # False when an existing list was updated + union_size: int # distinct species after the wide merge + per_source_counts: dict[str, int] # {"gbif_gadm": 812, "inat_place": 640} + matched_existing: int # merged species already present as a Taxon + created_taxa: int # new Taxon rows created (0 when create_missing=False) + in_classifier_labels: int | None # coverage against classifier (None if no classifier given) + not_in_classifier: int | None # union_size - in_classifier_labels + unmatched_names: list[str] # source names with no Taxon and not created + dry_run: bool + + +def generate_regional_taxa_list( + *, + region_source: str, + region_code: str, + project: "Project | None" = None, + classifier: "Algorithm | None" = None, + taxon_scope: TaxonScope | None = None, # defaults to Lepidoptera + sources: "list[RegionalSpeciesSource] | None" = None, # defaults per region_source; DI seam for tests + create_missing: bool = True, + name: str | None = None, # defaults to f"{region_code} ({region_source})" + dry_run: bool = False, +) -> RegionalTaxaResult: + ... +``` + +Behaviour: +1. Resolve `sources` (default: the one client matching `region_source`; a caller may pass several to union). The `sources` parameter is the **dependency-injection seam** — tests pass a stubbed client, no HTTP. +2. `fetch_species()` each source → `merge_source_species()` → wide union. +3. `map_to_taxa(create_missing, dry_run)`. +4. If `classifier` given: compute coverage against its label set (reporting only) — reuse `classifier.category_map.with_taxa()` or a name-set intersection against the labels; do not filter the list by it. +5. Unless `dry_run`: `get_or_create_for_project(name, project)`, then set the list's taxa (idempotent — update the existing list's M2M rather than creating a duplicate, mirroring the manager's contract). `update_calculated_fields()` on the list if it has cached counts. +6. Return `RegionalTaxaResult`. + +**Idempotency:** a second run for the same `(name, project)` updates the same `TaxaList` (the manager's `get_or_create_for_project` guarantees one row; the service replaces/refreshes its taxa set). Re-running never creates duplicate lists or duplicate `Taxon` rows (name-unique + external-key match handle that). + +**No sentinels, no asserts:** failure paths raise (`CLAUDE.md`); the service returns a `RegionalTaxaResult` only on success. Callers translate exceptions to their surface (DRF 4xx, command error, admin message). + +--- + +## 7. Region derivation (A3) — powers `--all-projects` + +Most existing projects have no region code but do have deployment coordinates (`Deployment.latitude` `:770`, `longitude` `:771`). `derive_region_code()` reverse-geocodes a representative point to a region code so backfill needs no manual entry: + +```python +def derive_region_code( + project_or_site, *, region_source: str, +) -> str | None: + """Reverse-geocode a representative deployment coordinate to a region code. + Representative point = the centroid or first non-null (lat, lon) among the + object's deployments (Site.boundary_rect() at models.py:670 already aggregates + the extent). Returns None when no deployment has coordinates.""" +``` + +- For `gbif_gadm`: candidate endpoint `GET https://api.gbif.org/v1/geocode/reverse?lat=<>&lng=<>` returning GADM GIDs (UNVERIFIED — confirm the response shape and which GADM level to take). Fall back to a bundled GADM lookup if the endpoint is unreliable. +- For `inat_place`: iNat has no clean reverse-geocode to a single place; prefer GBIF/GADM for derivation even when the chosen *fetch* source is iNat, or require manual `place_id` entry. Note this asymmetry. + +A3 is what makes "backfill every known project" feasible: the `--all-projects` command path calls `derive_region_code()` when a project has no `region_code`, then `generate_regional_taxa_list()`. Whether derivation is reliable enough to run unattended is an explicit open verification item (§11) — until confirmed, `--all-projects` should default to `--dry-run` and print what it *would* create. + +--- + +## 8. Auto-apply masking — resolution order + +Extend `ClassMaskingConfig` (on branch `rework/999-class-masking-on-framework`, `ami/ml/post_processing/class_masking.py:17`) so the task can resolve the taxa list per occurrence instead of requiring an explicit `taxa_list_id`. **This change lands on the #999 branch (or a follow-up stacked on it), not on `main` today**, because `class_masking.py` does not exist on `main` yet. + +### 8.1 Config change (additive, backwards-compatible) + +Today `taxa_list_id: int` is required (`:25`). Make the list selection a discriminated choice so the explicit path is untouched: + +```python +class ClassMaskingConfig(pydantic.BaseModel): + source_image_collection_id: int | None = None + occurrence_id: int | None = None + algorithm_id: int + reweight: bool = True + + # List selection: explicit id, OR auto-resolution per occurrence. + taxa_list_id: int | None = None + taxa_list_mode: typing.Literal["explicit", "auto"] = "explicit" + + @pydantic.root_validator(skip_on_failure=True) + def _validate_list_selection(cls, values): + if values["taxa_list_mode"] == "explicit" and values.get("taxa_list_id") is None: + raise ValueError("explicit mode requires taxa_list_id") + # auto mode ignores taxa_list_id and resolves per occurrence + return values +``` + +Keep the existing `_exactly_one_scope` validator. The explicit path in `run()` (`class_masking.py:~305`, `TaxaList.objects.get(pk=config.taxa_list_id)`) is unchanged; a new `auto` branch resolves per occurrence. + +### 8.2 Resolution order (from the design doc) + +``` +occurrence.deployment.research_site.taxa_list + ↳ else research_site.region_code → generate/lookup regional list + ↳ else project.default_taxa_list + ↳ else project.region_code → generate/lookup regional list + ↳ else no masking (log occurrence.pk and skip) +``` + +Implement as `resolve_taxa_list_for_occurrence(occurrence) -> TaxaList | None`, a helper (home: `ami/main/services/regional_taxa.py`, or a thin `ami/ml/post_processing/masking_resolution.py` that imports the service). When a step hits a `region_code` with no linked list, it calls `generate_regional_taxa_list()` (cache-backed, so it is cheap after the first run) and stores the result on the site/project so subsequent runs short-circuit to the FK. + +**Batching (avoid N+1):** in `auto` mode, group the scoped occurrences by `(research_site, project)` first and resolve one list per group, not per occurrence — the scoped queryset is already built in `_scoped_classifications` (`class_masking.py:~280`). This keeps masking's existing batched-commit loop intact. + +**Safety:** `auto` mode is a no-op when nothing resolves (log + skip), so a pipeline can enable it by default without masking projects that have no region/list configured. This preserves the existing explicit behaviour bit-for-bit for anyone still passing `taxa_list_id`. + +### 8.3 How it plugs into the framework + +The task is still triggered through the #1289 framework (`make_post_processing_action`, `ami/ml/post_processing/admin/actions.py:177`) and the admin form (`class_masking_form.py`). The form grows a "list mode" choice (explicit list dropdown vs. auto). The `build_jobs`/`default_build_jobs` path (`actions.py:87,184`) is unchanged — config still validates against `ClassMaskingConfig` before enqueue. Whether "auto" is a per-`ProjectPipelineConfig` toggle or a pipeline property is an open question (§10); the config-level `taxa_list_mode` is the minimum that unblocks manual/admin use. + +--- + +## 9. The five surfaces (each a thin wrapper) + +### 9.1 Management command — `ami/main/management/commands/generate_regional_taxa_list.py` +``` +python manage.py generate_regional_taxa_list \ + --project --region-source gbif_gadm --region \ + [--classifier ] [--scope lepidoptera] \ + [--no-create-missing] [--all-projects] [--dry-run] +``` +- Single project: calls the service, prints the `RegionalTaxaResult` as a table. +- `--all-projects`: loops `Project.objects.all()` (shape from `assign_roles.py:79`); for each project without a `region_code`, calls `derive_region_code()` (A3); **defaults to `--dry-run`** until derivation reliability is confirmed (§11). Prints a per-project summary. +- Raises `CommandError` on failure; no sentinels. + +### 9.2 Unit tests — call the service directly with a stubbed source +The primary regression surface (see §10). Stubbed `RegionalSpeciesSource` passed via the `sources=` DI seam; no network. + +### 9.3 Django admin action — on `Project` and `Site` changelists +Built with `make_post_processing_action`-style plumbing is **not** required here (this is not a post-processing Job); instead a plain admin action + intermediate confirmation form (region source + code, optional classifier, dry-run checkbox) that calls the service and shows the `RegionalTaxaResult` via `messages`. Mirrors the confirmation-page pattern in `ami/ml/post_processing/admin/actions.py::render_confirmation` for consistency, but runs synchronously (regional fetch is slow — consider enqueuing a Job if it exceeds request timeout; note as a refinement). + +### 9.4 DRF endpoint +`POST /api/v2/projects/{project_pk}/taxa-lists/regional/` as an `@action(detail=True, methods=["post"])` on `ProjectViewSet` (`ami/main/api/views.py:158`, already `ProjectMixin`), plus optionally a `Site` variant. Template: the `regroup_sessions` / `sync` actions (`views.py:339,363`) that validate, act, and return `202`. +- Body validated by a request serializer (`region_source`, `region_code`, optional `classifier_id`, `dry_run`); parse every param via `SingleParamSerializer` (`ami/base/serializers.py:108`) / a DRF serializer so bad input returns 400 not 500 (`CLAUDE.md` DoD). +- Permissions: `require_project`, object access through `get_object()`/`check_object_permissions()` — never a raw pk lookup (`CLAUDE.md` DoD). Only project members with the right role may generate (align with membership model; §11 verification item). +- Because a live fetch is slow, the endpoint should enqueue a Job and return `{"job_id": …}` (like `sync`), with `dry_run` running synchronously for preview. Decide sync-vs-async here explicitly. + +### 9.5 Main UI button — later slice +"Create taxa list for region" in Project settings and the Site editor, calling 9.4. Reference the closed **#1020** GBIF taxon-select React component for the region/place picker. Add a React Query mutation hook under `ui/src/data-services/hooks/` (follow `ui/AGENTS.md`). Explicitly a **later phase** (§Phase 4). + +--- + +## 10. Test plan (TDD — write these first) + +All tests use factories (`ami/*/tests/factories.py`) and the stubbed-source DI seam; **no test hits a live API**. Record one real GBIF and one real iNat response as a fixture (small, scrubbed) for the source-client parsing tests. + +1. **Source-client parsing** — feed each concrete source a recorded/canned JSON payload (monkeypatched `create_session`/response) and assert it yields the expected `SourceSpecies` list, including pagination termination and key/name/count extraction. One test per source. +2. **Wide merge — union + provenance** (the load-bearing test for the core decision): + - Two sources sharing a species by `gbif_taxon_key` collapse to one `MergedSpecies` with `sources == {both}` and both counts. + - A species in only one source survives (union, not intersection). + - Name-only collision with conflicting external keys keeps both keys and logs a warning. + - Dedup-key precedence (gbif → inat → name) exercised explicitly. +3. **Mapping + create** — existing `Taxon` matched by external key; matched by name; `create_missing=True` creates via the hierarchy builder; `create_missing=False` records `unmatched_names`. Assert no duplicate `Taxon` on re-run (name-unique). +4. **Idempotency** — run the service twice for the same `(name, project)`; assert one `TaxaList`, taxa set refreshed, no duplicate lists/taxa. +5. **Classifier coverage is report-only** — with a `classifier`, assert `in_classifier_labels` / `not_in_classifier` are populated and that the saved list's taxa are **not** filtered by the classifier (a species outside the classifier's labels still appears in the list). +6. **Region derivation (A3)** — `derive_region_code()` returns a code for a project with deployment coords (stubbed reverse-geocode) and `None` when no deployment has coordinates. +7. **Masking resolution order** — parametrize the five-step ladder: site list wins over site region-code wins over project default list wins over project region-code wins over skip. Assert `auto` mode is a no-op (skips, does not raise) when nothing resolves, and that explicit `taxa_list_id` still behaves exactly as before. +8. **Masking `auto` batching** — `assertNumQueries` with a **multi-row** fixture (occurrences across two sites) proving list resolution is grouped, not per-occurrence (single-row fixtures cannot catch N+1 — `CLAUDE.md` DoD; template `ami/ml/tests.py:1006`). Strict `==` counts. +9. **Endpoint permission matrix** — member / non-member / anonymous / superuser against `POST …/regional/` (template `ami/main/tests.py:1532`); `?…=abc` bad-param returns 400. +10. **Command** — `--dry-run` creates nothing; `--all-projects --dry-run` iterates and reports; failure raises `CommandError`. + +Repo rules to honour while writing: migration in the same PR; no `assert` in production code; raise don't return sentinels; test docstrings state the invariant, not "this PR added X". + +--- + +## 11. Phased rollout (internal-first) + +Order the slices so the internal path is usable and validated before the API/UI. Each phase is independently shippable and mergeable. + +**Phase 0 — De-risk the external APIs (spike, no merge).** +Exercise the candidate GBIF and iNat endpoints from a scratch script against 2–3 real regions (one where a project already exists — e.g. Quebec/Vermont for the moths classifier). Confirm: faceting/pagination behaviour, rate limits, name/key/count fields, and reverse-geocode response shape. **Gate:** measure how many of a real classifier's labels a generated regional list actually covers (the design doc's key unknown). This decides whether Proposal A is useful alone or needs Proposal B's manual curation. Output: a short findings note + recorded response fixtures for the parsing tests. *De-risks: the whole premise (coverage) and every "CANDIDATE, UNVERIFIED" endpoint above.* + +**Phase 1 — Core service + one source + model fields (backend, no UI).** +`SourceSpecies`/`MergedSpecies`/protocol, one concrete source (whichever won Phase 0 on moth coverage), `merge_source_species`, `map_to_taxa`, `generate_regional_taxa_list`, the `Site`/`Project` fields + migration, and `create_taxon` extraction (§5.3). Tests 1–5. *De-risks: the union-with-provenance core and the data model, behind unit tests, with zero external dependency in CI.* + +**Phase 2 — Management command + admin action + A3.** +Command (incl. `--all-projects --dry-run` backfill), `derive_region_code`, admin actions on Project/Site. Tests 6, 10. *De-risks: unattended backfill and gives internal users a way to generate lists before any API exists.* + +**Phase 3 — Masking auto-resolution (on the #999 branch / stacked follow-up).** +`ClassMaskingConfig` `taxa_list_mode`, resolution ladder, batched grouping, admin form "list mode". Tests 7, 8. **Requires #999 to have landed or be the base branch** — coordinate with that PR. *De-risks: the "works out of the box" promise; gated behind no-op-when-unconfigured so it is safe to enable.* + +**Phase 4 — API endpoint + main UI.** +`POST …/regional/` (sync-vs-async decided), request serializer, permission matrix (test 9), then the React button reusing #1020's picker. *De-risks: external/self-serve use once the service output is trusted internally.* + +Second source (union path) can slot in whenever Phase 0 shows it adds coverage — the merge already supports it; adding a source is `+1` client + `+1` parsing test. + +--- + +## 12. Open questions (carried from the design doc + implementation-level) + +Design-doc decisions still open: +- **GBIF vs iNaturalist first** — decided empirically in Phase 0 by moth coverage + key cleanliness. +- **GADM granularity** — level 1 (state/province) vs level 2 (county). Make it configurable per site/project; default level 1, revisit after Phase 0. +- **Create-vs-skip unmatched taxa** — exposed as `create_missing` (default True). Confirm we want to grow the taxonomy from external sources vs. only intersect with existing taxa. +- **TaxaList scope** — regional lists are project-scoped (not global) so one project's region does not pollute another's picker. Confirmed direction; `get_or_create_for_project(project=...)`. +- **How the pipeline auto-applies masking** — per-`ProjectPipelineConfig` toggle vs. pipeline property. `taxa_list_mode="auto"` at the config level is the minimum; the pipeline-wiring decision aligns with #1289/#999. +- **Name-match fragility** — masking joins on `Taxon.name`; source/classifier spelling mismatches silently drop species. Measured in Phase 0; surfaced per-run via `unmatched_names` + `not_in_classifier`. + +Implementation-level questions surfaced here: +- **Sync vs. async for the endpoint/admin action** — regional fetch can exceed request timeouts; likely enqueue a Job (like `sync`), keep `dry_run` synchronous. Decide in Phase 2/4. +- **Cache backend + TTL** for source responses (Redis vs. on-disk; days-scale TTL). Needed once real fetch latency is known (Phase 0). +- **Where `RegionSource` and the masking-resolution helper live** — enum with the model (stable migration path); resolver in `services/` vs. a thin `ml/post_processing/` shim. +- **Reverse-geocode reliability for unattended backfill** — until confirmed, `--all-projects` stays `--dry-run` by default (§7). + +## 13. What we still need to verify before building + +- Exact GBIF endpoint + params returning a species list for a GADM region, its facet-depth cap, and rate limits — and the iNat equivalent (`species_counts` by `place_id`). Both are on-paper only. +- The GBIF reverse-geocode response shape and which GADM level to take for A3. +- Label-coverage of a generated regional list against a **real** classifier (Quebec/Vermont moths) — the go/no-go for Proposal A standing alone. +- Permissions story for the new fields and the generate action (who may set a region, who may trigger generation) consistent with the project-membership model. +``` From 2e9e06c6f6e9decb023b3b9ede2b8e5e0f760f9f Mon Sep 17 00:00:00 2001 From: Michael Bunsen Date: Thu, 2 Jul 2026 14:10:11 -0700 Subject: [PATCH 2/4] docs: add model & DB awareness to regional taxa-list plan (#1364) [skip ci] Fold the model/DB-awareness requirement into the Proposal A plan as a first-class section. The regional-list generator now, by default, subsets the union of source species to those a classifier can actually predict (name in some AlgorithmCategoryMap label set), with an opt-in flag to also create uncovered regional species flagged as not classifiable. Confirmed by code reading that no persisted Taxon-to-Algorithm/CategoryMap link exists today (with_taxa() resolves names live, unpersisted), so the plan adds a persisted relationship (category-map-anchored M2M plus a denormalized Taxon.is_classifiable boolean) and a refresh path keyed on labels_hash. Updates the Result dataclass with explicit buckets, the data-model and test-plan sections, and the open-questions list. Refs #1364, #999, #1289 Co-Authored-By: Claude --- ...roposal-a-regional-taxa-lists-impl-plan.md | 204 ++++++++++++------ 1 file changed, 142 insertions(+), 62 deletions(-) diff --git a/docs/claude/planning/2026-07-02-proposal-a-regional-taxa-lists-impl-plan.md b/docs/claude/planning/2026-07-02-proposal-a-regional-taxa-lists-impl-plan.md index a26b0aaaf..84ec193ab 100644 --- a/docs/claude/planning/2026-07-02-proposal-a-regional-taxa-lists-impl-plan.md +++ b/docs/claude/planning/2026-07-02-proposal-a-regional-taxa-lists-impl-plan.md @@ -14,7 +14,7 @@ This document turns the design doc's "Proposal A", "reusable core", "data model" Class masking (#999) keeps only the classifier predictions whose taxon appears in a chosen taxa list, cutting a global classifier down to the species that actually occur at a site. The blocker is that the taxa list has to be hand-curated one taxon at a time, and nothing ties a list to the place a project monitors. -Proposal A adds one capability — **"given a region, produce a taxa list"** — written once as a service function and surfaced five ways (management command, unit test, Django admin action, DRF endpoint, main UI). It fetches the species recorded in a geographic region from one or more external biodiversity databases (GBIF, iNaturalist), maps them onto Antenna `Taxon` rows, optionally reports how many fall inside a classifier's label set, and saves a project-scoped `TaxaList`. It also adds region/list fields to `Site` and `Project` and a resolution order so the class-masking task can pick the right list automatically per occurrence. +Proposal A adds one capability — **"given a region, produce a taxa list"** — written once as a service function and surfaced five ways (management command, unit test, Django admin action, DRF endpoint, main UI). It fetches the species recorded in a geographic region from one or more external biodiversity databases (GBIF, iNaturalist), maps them onto Antenna `Taxon` rows, restricts (by default) to the species some classifier can actually predict, and saves a project-scoped `TaxaList`. It also adds region/list fields to `Site` and `Project` and a resolution order so the class-masking task can pick the right list automatically per occurrence. ### 1.1 The one design decision to get right: multiple sources UNION, they do not INTERSECT @@ -22,7 +22,7 @@ When more than one source is queried (GBIF **and** iNaturalist, say), a species - The merged intermediate is a **wide table**: one row per canonical species, carrying per-source provenance columns — which source(s) contributed the species, each source's native key (`gbif_taxon_key`, `inat_taxon_id`, …), and each source's observation count. - Sources are **never intersected against each other**. Querying two sources can only grow the candidate set, never shrink it. -- The **only** intersection in the whole flow is the *optional, final, reporting-only* step against a classifier's label set. That intersection does not remove species from the saved list — it only reports "of the N species in this regional list, M are actually in classifier X's labels, so masking will affect M of them." It is a usefulness metric, not a filter. +- The **only** intersections in the whole flow are applied *after* the union and are separate axes from source provenance: (i) an *optional, reporting-only* comparison against one specific classifier's label set, and (ii) the **model-coverage** restriction described in §6 (does any classifier cover this species at all). Neither removes species based on which source they came from; they act on the taxon after mapping. See §6 for how model-coverage is a separate axis from source provenance. Every abstraction below (`SourceSpecies`, the source protocol, the merge function, the `Result` summary) is shaped around this union-with-provenance model. If a future requirement wants "species seen in at least K sources", that is a *post-merge* filter over the provenance columns, not a change to how sources combine. @@ -44,9 +44,10 @@ Internally the service is a short pipeline: region code ─▶ [Source clients] fetch_species() per source → list[SourceSpecies] ─▶ merge_source_species() WIDE UNION on canonical key, provenance preserved ─▶ map_to_taxa() match/create Antenna Taxon rows - ─▶ (optional) intersect_classifier() REPORT-ONLY coverage against a classifier's labels + ─▶ apply_model_coverage() annotate/subset by classifier coverage (§6) + ─▶ (optional) intersect_classifier() REPORT-ONLY coverage against ONE named classifier ─▶ save / update TaxaList idempotent, project-scoped - ─▶ Result counts + unmatched names for human review + ─▶ Result bucketed counts + unmatched names for human review ``` **New module:** `ami/main/services/regional_taxa.py` (new `services` package under `ami/main/`, per the "processing logic extraction" note in `CLAUDE.md`; add `ami/main/services/__init__.py`). @@ -55,7 +56,7 @@ region code ─▶ [Source clients] fetch_species() per source - `ami/utils/requests.py::create_session()` (`:14`) — all outbound HTTP, for retry/backoff. - `ami/main/management/commands/import_taxa.py::create_taxon()` and `get_or_create_root_taxon()` — the rank-hierarchy builder for creating missing `Taxon` rows under the `Arthropoda` root. This logic currently lives inside the command module; see §5.3 for extracting it so the service can call it without importing a management command. - `TaxaList.objects.get_or_create_for_project(name, project=None)` (`ami/main/models.py:4598`) — idempotent list creation, project-scoped. -- `AlgorithmCategoryMap.with_taxa()` (`ami/ml/models/algorithm.py:105`) — the name-keyed category→taxon resolution masking already uses; reused for the classifier-coverage report. +- `AlgorithmCategoryMap.with_taxa()` (`ami/ml/models/algorithm.py:105`) — the name-keyed category→taxon resolution masking already uses; reused for both the classifier-coverage report and the persisted model-coverage relationship (§6). - `Project.objects.all()` loop shape from `ami/main/management/commands/assign_roles.py:79` — the `--all-projects` backfill template. --- @@ -84,7 +85,17 @@ Minimal columns; polygons/geometry are explicitly out of scope (design doc Propo **Relationship note.** `TaxaList↔Project` (`related_name="taxa_lists"`) and `TaxaList↔Taxon` (`related_name="lists"`) stay M2M. The new `taxa_list` / `default_taxa_list` are single FKs *layered on top*: a project still associates with many lists, but names one default. `related_name="+"` avoids reverse-accessor clutter on `TaxaList`. -**Migrations:** one migration in `ami/main/migrations/` adding all six fields (they are all additive, nullable/blank, no data change). If `Project` and `Site` edits are cleaner as two migrations, that is fine; either way they are state+schema migrations, not data migrations. +### 3.3 Model-coverage relationship (the "are the models aware of this taxon" field) + +This is the persisted relationship required by §6. Storing it here (not computing it live) is a deliberate decision — see §6.2 for the options considered and the recommendation. + +- **`AlgorithmCategoryMap.taxa`** — `ManyToManyField("main.Taxon", related_name="category_maps", blank=True)`. The persisted membership: which `Taxon` rows a label set resolves to (by the same name-match `with_taxa()` uses). The label set is the real owner of coverage (many `Algorithm` rows share one `category_map` via the FK at `ami/ml/models/algorithm.py:212`), so anchoring the M2M on the category map avoids duplicating rows across algorithms. +- **`Taxon.is_classifiable`** — `BooleanField(default=False, db_index=True)`. Denormalized flag: `True` iff the taxon is in at least one category map's `taxa`. Gives cheap filtering and a simple UI signal without a join. + +"Which algorithm(s) cover this taxon" is then a derived query, not a stored column: +`Algorithm.objects.filter(category_map__taxa=taxon)` — richer than a boolean, and free once the M2M exists. + +**Migrations:** the M2M through table lives with `AlgorithmCategoryMap` (an `ami/ml/` migration); the `is_classifiable` boolean is an `ami/main/` migration on `Taxon`. Both are additive. Neither backfills automatically — the refresh command (§6.3) populates them, and the same migration PR should either run a data migration calling that refresh or document that the command must be run once after deploy. --- @@ -144,7 +155,7 @@ Default scope: Lepidoptera (moth/butterfly platform). Keep it a parameter so a p ### 4.3 Concrete sources (endpoints are CANDIDATE, UNVERIFIED) -No live GBIF/iNat code exists in `ami/` today — these fields are only ever populated from files. The endpoints below are from published API docs and must be exercised against the live services (a recorded-response fixture per source) before they are trusted. Rate limits and exact faceting behaviour are the first things to verify (§10, §11). +No live GBIF/iNat code exists in `ami/` today — these fields are only ever populated from files. The endpoints below are from published API docs and must be exercised against the live services (a recorded-response fixture per source) before they are trusted. Rate limits and exact faceting behaviour are the first things to verify (see Phase 0, §12, and §14). **`GBIFRegionalSource`** — GBIF occurrence search, faceted by species. - Candidate endpoint: `GET https://api.gbif.org/v1/occurrence/search` @@ -195,6 +206,8 @@ Merge rules: - **Provenance union:** `sources = ∪ row.source`; `observation_counts[row.source] = row.observation_count`; keep both external keys if each source supplied one; prefer the GBIF name spelling when present, else the first. - Output is stable-ordered (e.g. by `-max(observation_counts)`, then name) so runs are reproducible and diffs are readable. +Note that model-coverage (§6) is **not** part of the merge — it is applied after mapping to `Taxon`, as a separate axis. Merging is only about combining sources. + --- ## 5. Mapping merged species → Antenna `Taxon` @@ -219,7 +232,9 @@ def map_to_taxa( `Taxon.name` is globally unique (`models.py:4346`), and class masking joins list membership to classifier labels by **name** (`AlgorithmCategoryMap.with_taxa()`, keyed on `taxon.name` at `algorithm.py:135`). So: - Matching a source species to an existing `Taxon` by external key and then trusting its `.name` is correct — the list stores `Taxon` rows, and masking will later match those same names to classifier labels. -- A source name that differs from both any `Taxon.name` and the classifier's label spelling will silently fail to mask even after we add it to the list. This is the **name-match fragility** open question (§10); the service surfaces `unmatched_names` and the classifier-coverage count precisely so a human can see the miss rate before trusting auto-masking. +- A source name that differs from both any `Taxon.name` and the classifier's label spelling will silently fail to mask even after we add it to the list. This is the **name-match fragility** open question (§13); the service surfaces `unmatched_names`, the model-coverage buckets (§6), and the classifier-coverage count precisely so a human can see the miss rate before trusting auto-masking. + +Because model-coverage (§6) is keyed on the same name-match, a mapped `Taxon` and its coverage flag are consistent with what masking will actually do at run time, by construction. ### 5.3 Extract `create_taxon` for reuse @@ -227,7 +242,52 @@ def map_to_taxa( --- -## 6. The core service function +## 6. Model & DB awareness + +The generator must be aware of two things about every candidate species before it saves a list: + +- **DB presence** — does a `Taxon` row already exist (matched by external key or name, §5.1)? +- **Model coverage** — is the taxon covered by at least one classifier, i.e. does its name appear in some `AlgorithmCategoryMap` label set (the same `Taxon.name == label` join class masking uses)? + +These two facts drive the default behaviour, the opt-in richer behaviour, and the `Result` breakdown. Model-coverage is a **separate axis** from source provenance (§1.1): the wide merge still unions all sources; coverage is applied afterwards, per mapped `Taxon`. Do not conflate "in multiple sources" with "model-aware" — they are orthogonal. + +### 6.1 Default subsets to model-known species; opt-in creates the rest, flagged + +- **Default:** the saved list is `regional-species-from-sources ∩ taxa-covered-by-at-least-one-classifier`. Rationale: class masking can only ever affect taxa in some classifier's label set, so a regional species no model can predict is inert for masking. Keeping the default tight makes the list predictable and keeps "why does my 1500-species region only mask to N" answerable. +- **Opt-in (`include_uncovered=True`, exposed as `--include-uncovered` / a form checkbox):** also keep regional species with no model coverage. Under this mode, species absent from the DB are created via the hierarchy builder (§5.3) but each carries an honest coverage indicator — `is_classifiable=False` and no `category_maps` membership — so the list/UI can show "in the region, but no model can predict it yet". Many valid regional species have no training data and will never appear in a prediction list; the model must not pretend otherwise. + +A note on interaction with `create_missing` (§5.1): under the default (model-covered-only) scope, a species is kept only if its name matches some label set; creating its `Taxon` when missing is safe because, by construction, it *is* classifiable. Under `include_uncovered`, `create_missing` also governs whether uncovered species get `Taxon` rows created (default yes) or are only reported as unmatched. + +### 6.2 The persisted relationship — options and recommendation + +Confirmed by code reading: there is **no persisted `Taxon ↔ Algorithm` or `Taxon ↔ AlgorithmCategoryMap` link today**. `Algorithm.category_map` is an FK to `AlgorithmCategoryMap` (`ami/ml/models/algorithm.py:212`, `related_name="algorithms"`); the label set lives in `data` (JSON) + `labels` (`ArrayField`) with a `labels_hash` BigInt; and the only label→`Taxon` path is `with_taxa()` (`algorithm.py:105`), which resolves names **on the fly, unpersisted**. So a persisted relationship has to be added — the user explicitly wants "a field relationship to show if the models are aware of them", not a live computation. + +| Option | Shape | Verdict | +|---|---|---| +| (a) boolean | `Taxon.is_classifiable` denorm, refreshed on category-map change | Cheap, filterable, good UI signal — but cannot say *which* model covers a taxon. Acceptable **MVP** on its own. | +| (b) through relationship | persisted `Taxon ↔ Algorithm` (or `↔ AlgorithmCategoryMap`) membership | Answers "which model(s) cover this taxon", supports per-classifier questions. The user asked for a relationship → this is the **target**. | +| (c) compute-only | keep using `with_taxa()` live, no schema | Cheapest, but no persisted field and slow to filter/UI. **Rejected** — it is exactly today's state, and the user wants persistence. | + +**Recommendation: (b), anchored on the category map, plus (a) as a denormalized convenience.** Because many algorithms share one `category_map`, storing membership as `AlgorithmCategoryMap.taxa` (§3.3) avoids duplicating rows per algorithm, and "which algorithms" derives for free via `Algorithm.objects.filter(category_map__taxa=taxon)`. The `Taxon.is_classifiable` boolean rides alongside for cheap filtering and a simple "models know this species" badge. This satisfies the "which models are aware" requirement while staying normalized. Whether the M2M should instead be anchored directly on `Algorithm` is an open question (§13) — anchoring on the map is the normalized choice given the shared-map schema, but an `Algorithm`-anchored M2M reads more directly if maps are rarely shared in practice. + +If (b) is too much for the first slice, ship (a) alone as the MVP (boolean only), and add the M2M in a follow-up — the service and `Result` treat coverage as a single predicate either way, so the upgrade does not change their contract. + +### 6.3 Refresh / consistency — coverage is derived data + +Classifier label sets change whenever an algorithm or category map is added or updated, so model-coverage is derived and needs an explicit refresh path tied to the same name-resolution `with_taxa()` uses (so coverage always matches masking's join): + +- **`refresh_category_map_taxa(category_map)`** — recompute `category_map.taxa` from its labels via the `with_taxa()` resolution, then update `Taxon.is_classifiable` for the affected taxa (True iff still in any map's `taxa`). Set-based, bulk, no per-row queries. +- **Hook on change:** recompute on `AlgorithmCategoryMap` save/import when the label set changes. `labels_hash` (`algorithm.py:68`) is the ready-made "did the label set change" signal — compare before/after and only recompute when it moves. Also trigger on the algorithm-registration / category-map-import code path that first creates a map. +- **Full rebuild:** a management command `refresh_taxon_model_coverage` loops all category maps, rebuilds membership, and recomputes `is_classifiable` — used for the initial backfill (the migration in §3.3) and as a repair tool. +- **When it recomputes** is called out explicitly so readers are not surprised by staleness: on category-map create/update (hook), and on demand (command). It is *not* recomputed on every occurrence write — masking still resolves names live via `with_taxa()`, so a brief lag in the denormalized flag never causes a wrong mask, only a slightly stale "is_classifiable" badge until the next refresh. + +### 6.4 How the service uses coverage + +In `generate_regional_taxa_list()` (§7): after `map_to_taxa()`, `apply_model_coverage()` partitions the mapped taxa into model-covered vs. uncovered using the persisted `is_classifiable` / `category_maps` relationship (not a live recompute — the relationship is the source of truth, kept fresh by §6.3). The default scope keeps only the covered partition; `include_uncovered` keeps both. The optional `classifier=` argument is a *narrower, reporting-only* overlay on top: "of the kept species, how many are in *this specific* model's labels" — it never changes which taxa are saved. + +--- + +## 7. The core service function ```python # ami/main/services/regional_taxa.py @@ -238,12 +298,20 @@ class RegionalTaxaResult: region_code: str taxa_list_id: int | None # None on dry_run list_created: bool # False when an existing list was updated - union_size: int # distinct species after the wide merge + # --- source union (§1.1) --- + regional_total: int # distinct species after the wide merge (union across sources) per_source_counts: dict[str, int] # {"gbif_gadm": 812, "inat_place": 640} - matched_existing: int # merged species already present as a Taxon - created_taxa: int # new Taxon rows created (0 when create_missing=False) - in_classifier_labels: int | None # coverage against classifier (None if no classifier given) - not_in_classifier: int | None # union_size - in_classifier_labels + # --- DB presence & model coverage (§6) --- + already_in_db: int # merged species already present as a Taxon + created_taxa: int # new Taxon rows created this run + model_covered: int # kept by default: covered by >=1 classifier + regional_no_model_coverage: int # in region but no model covers them + # (created+flagged under include_uncovered, else skipped) + saved_list_size: int # taxa actually written to the TaxaList + # --- optional single-classifier report (§6.4) --- + in_classifier_labels: int | None # coverage vs the ONE classifier passed (None if none given) + not_in_classifier: int | None # saved_list_size - in_classifier_labels + # --- review --- unmatched_names: list[str] # source names with no Taxon and not created dry_run: bool @@ -253,11 +321,12 @@ def generate_regional_taxa_list( region_source: str, region_code: str, project: "Project | None" = None, - classifier: "Algorithm | None" = None, - taxon_scope: TaxonScope | None = None, # defaults to Lepidoptera + classifier: "Algorithm | None" = None, # OPTIONAL single-model report only (§6.4) + taxon_scope: TaxonScope | None = None, # defaults to Lepidoptera sources: "list[RegionalSpeciesSource] | None" = None, # defaults per region_source; DI seam for tests + include_uncovered: bool = False, # default subsets to model-covered species (§6.1) create_missing: bool = True, - name: str | None = None, # defaults to f"{region_code} ({region_source})" + name: str | None = None, # defaults to f"{region_code} ({region_source})" dry_run: bool = False, ) -> RegionalTaxaResult: ... @@ -265,11 +334,12 @@ def generate_regional_taxa_list( Behaviour: 1. Resolve `sources` (default: the one client matching `region_source`; a caller may pass several to union). The `sources` parameter is the **dependency-injection seam** — tests pass a stubbed client, no HTTP. -2. `fetch_species()` each source → `merge_source_species()` → wide union. -3. `map_to_taxa(create_missing, dry_run)`. -4. If `classifier` given: compute coverage against its label set (reporting only) — reuse `classifier.category_map.with_taxa()` or a name-set intersection against the labels; do not filter the list by it. -5. Unless `dry_run`: `get_or_create_for_project(name, project)`, then set the list's taxa (idempotent — update the existing list's M2M rather than creating a duplicate, mirroring the manager's contract). `update_calculated_fields()` on the list if it has cached counts. -6. Return `RegionalTaxaResult`. +2. `fetch_species()` each source → `merge_source_species()` → wide union (`regional_total`). +3. `map_to_taxa(create_missing, dry_run)` → matched/created/unmatched (`already_in_db`, `created_taxa`, `unmatched_names`). +4. `apply_model_coverage()` (§6.4): partition mapped taxa into covered / uncovered via the persisted relationship (`model_covered`, `regional_no_model_coverage`). Default keeps covered only; `include_uncovered` keeps both (creating + flagging uncovered per §6.1). +5. If `classifier` given: compute the single-model report against its label set (reporting only) — reuse `classifier.category_map.with_taxa()` or a name-set intersection against the labels; do not filter the list by it. +6. Unless `dry_run`: `get_or_create_for_project(name, project)`, then set the list's taxa (idempotent — update the existing list's M2M rather than creating a duplicate, mirroring the manager's contract). `update_calculated_fields()` on the list if it has cached counts. +7. Return `RegionalTaxaResult`. **Idempotency:** a second run for the same `(name, project)` updates the same `TaxaList` (the manager's `get_or_create_for_project` guarantees one row; the service replaces/refreshes its taxa set). Re-running never creates duplicate lists or duplicate `Taxon` rows (name-unique + external-key match handle that). @@ -277,7 +347,7 @@ Behaviour: --- -## 7. Region derivation (A3) — powers `--all-projects` +## 8. Region derivation (A3) — powers `--all-projects` Most existing projects have no region code but do have deployment coordinates (`Deployment.latitude` `:770`, `longitude` `:771`). `derive_region_code()` reverse-geocodes a representative point to a region code so backfill needs no manual entry: @@ -294,15 +364,15 @@ def derive_region_code( - For `gbif_gadm`: candidate endpoint `GET https://api.gbif.org/v1/geocode/reverse?lat=<>&lng=<>` returning GADM GIDs (UNVERIFIED — confirm the response shape and which GADM level to take). Fall back to a bundled GADM lookup if the endpoint is unreliable. - For `inat_place`: iNat has no clean reverse-geocode to a single place; prefer GBIF/GADM for derivation even when the chosen *fetch* source is iNat, or require manual `place_id` entry. Note this asymmetry. -A3 is what makes "backfill every known project" feasible: the `--all-projects` command path calls `derive_region_code()` when a project has no `region_code`, then `generate_regional_taxa_list()`. Whether derivation is reliable enough to run unattended is an explicit open verification item (§11) — until confirmed, `--all-projects` should default to `--dry-run` and print what it *would* create. +A3 is what makes "backfill every known project" feasible: the `--all-projects` command path calls `derive_region_code()` when a project has no `region_code`, then `generate_regional_taxa_list()`. Whether derivation is reliable enough to run unattended is an explicit open verification item (§14) — until confirmed, `--all-projects` should default to `--dry-run` and print what it *would* create. --- -## 8. Auto-apply masking — resolution order +## 9. Auto-apply masking — resolution order Extend `ClassMaskingConfig` (on branch `rework/999-class-masking-on-framework`, `ami/ml/post_processing/class_masking.py:17`) so the task can resolve the taxa list per occurrence instead of requiring an explicit `taxa_list_id`. **This change lands on the #999 branch (or a follow-up stacked on it), not on `main` today**, because `class_masking.py` does not exist on `main` yet. -### 8.1 Config change (additive, backwards-compatible) +### 9.1 Config change (additive, backwards-compatible) Today `taxa_list_id: int` is required (`:25`). Make the list selection a discriminated choice so the explicit path is untouched: @@ -327,7 +397,7 @@ class ClassMaskingConfig(pydantic.BaseModel): Keep the existing `_exactly_one_scope` validator. The explicit path in `run()` (`class_masking.py:~305`, `TaxaList.objects.get(pk=config.taxa_list_id)`) is unchanged; a new `auto` branch resolves per occurrence. -### 8.2 Resolution order (from the design doc) +### 9.2 Resolution order (from the design doc) ``` occurrence.deployment.research_site.taxa_list @@ -343,43 +413,45 @@ Implement as `resolve_taxa_list_for_occurrence(occurrence) -> TaxaList | None`, **Safety:** `auto` mode is a no-op when nothing resolves (log + skip), so a pipeline can enable it by default without masking projects that have no region/list configured. This preserves the existing explicit behaviour bit-for-bit for anyone still passing `taxa_list_id`. -### 8.3 How it plugs into the framework +### 9.3 How it plugs into the framework -The task is still triggered through the #1289 framework (`make_post_processing_action`, `ami/ml/post_processing/admin/actions.py:177`) and the admin form (`class_masking_form.py`). The form grows a "list mode" choice (explicit list dropdown vs. auto). The `build_jobs`/`default_build_jobs` path (`actions.py:87,184`) is unchanged — config still validates against `ClassMaskingConfig` before enqueue. Whether "auto" is a per-`ProjectPipelineConfig` toggle or a pipeline property is an open question (§10); the config-level `taxa_list_mode` is the minimum that unblocks manual/admin use. +The task is still triggered through the #1289 framework (`make_post_processing_action`, `ami/ml/post_processing/admin/actions.py:177`) and the admin form (`class_masking_form.py`). The form grows a "list mode" choice (explicit list dropdown vs. auto). The `build_jobs`/`default_build_jobs` path (`actions.py:87,184`) is unchanged — config still validates against `ClassMaskingConfig` before enqueue. Whether "auto" is a per-`ProjectPipelineConfig` toggle or a pipeline property is an open question (§13); the config-level `taxa_list_mode` is the minimum that unblocks manual/admin use. --- -## 9. The five surfaces (each a thin wrapper) +## 10. The five surfaces (each a thin wrapper) -### 9.1 Management command — `ami/main/management/commands/generate_regional_taxa_list.py` +### 10.1 Management command — `ami/main/management/commands/generate_regional_taxa_list.py` ``` python manage.py generate_regional_taxa_list \ --project --region-source gbif_gadm --region \ [--classifier ] [--scope lepidoptera] \ - [--no-create-missing] [--all-projects] [--dry-run] + [--include-uncovered] [--no-create-missing] [--all-projects] [--dry-run] ``` -- Single project: calls the service, prints the `RegionalTaxaResult` as a table. -- `--all-projects`: loops `Project.objects.all()` (shape from `assign_roles.py:79`); for each project without a `region_code`, calls `derive_region_code()` (A3); **defaults to `--dry-run`** until derivation reliability is confirmed (§11). Prints a per-project summary. +- Single project: calls the service, prints the `RegionalTaxaResult` bucket breakdown as a table (regional-total / already-in-db / model-covered / no-model-coverage / saved). +- `--all-projects`: loops `Project.objects.all()` (shape from `assign_roles.py:79`); for each project without a `region_code`, calls `derive_region_code()` (A3); **defaults to `--dry-run`** until derivation reliability is confirmed (§14). Prints a per-project summary. - Raises `CommandError` on failure; no sentinels. -### 9.2 Unit tests — call the service directly with a stubbed source -The primary regression surface (see §10). Stubbed `RegionalSpeciesSource` passed via the `sources=` DI seam; no network. +A sibling command `refresh_taxon_model_coverage` (§6.3) rebuilds the coverage relationship + `is_classifiable`; it is independent of region generation but shares the coverage plumbing. -### 9.3 Django admin action — on `Project` and `Site` changelists -Built with `make_post_processing_action`-style plumbing is **not** required here (this is not a post-processing Job); instead a plain admin action + intermediate confirmation form (region source + code, optional classifier, dry-run checkbox) that calls the service and shows the `RegionalTaxaResult` via `messages`. Mirrors the confirmation-page pattern in `ami/ml/post_processing/admin/actions.py::render_confirmation` for consistency, but runs synchronously (regional fetch is slow — consider enqueuing a Job if it exceeds request timeout; note as a refinement). +### 10.2 Unit tests — call the service directly with a stubbed source +The primary regression surface (see §11). Stubbed `RegionalSpeciesSource` passed via the `sources=` DI seam; no network. -### 9.4 DRF endpoint +### 10.3 Django admin action — on `Project` and `Site` changelists +Built with `make_post_processing_action`-style plumbing is **not** required here (this is not a post-processing Job); instead a plain admin action + intermediate confirmation form (region source + code, optional classifier, include-uncovered, dry-run checkbox) that calls the service and shows the `RegionalTaxaResult` via `messages`. Mirrors the confirmation-page pattern in `ami/ml/post_processing/admin/actions.py::render_confirmation` for consistency, but runs synchronously (regional fetch is slow — consider enqueuing a Job if it exceeds request timeout; note as a refinement). + +### 10.4 DRF endpoint `POST /api/v2/projects/{project_pk}/taxa-lists/regional/` as an `@action(detail=True, methods=["post"])` on `ProjectViewSet` (`ami/main/api/views.py:158`, already `ProjectMixin`), plus optionally a `Site` variant. Template: the `regroup_sessions` / `sync` actions (`views.py:339,363`) that validate, act, and return `202`. -- Body validated by a request serializer (`region_source`, `region_code`, optional `classifier_id`, `dry_run`); parse every param via `SingleParamSerializer` (`ami/base/serializers.py:108`) / a DRF serializer so bad input returns 400 not 500 (`CLAUDE.md` DoD). -- Permissions: `require_project`, object access through `get_object()`/`check_object_permissions()` — never a raw pk lookup (`CLAUDE.md` DoD). Only project members with the right role may generate (align with membership model; §11 verification item). +- Body validated by a request serializer (`region_source`, `region_code`, optional `classifier_id`, `include_uncovered`, `dry_run`); parse every param via `SingleParamSerializer` (`ami/base/serializers.py:108`) / a DRF serializer so bad input returns 400 not 500 (`CLAUDE.md` DoD). +- Permissions: `require_project`, object access through `get_object()`/`check_object_permissions()` — never a raw pk lookup (`CLAUDE.md` DoD). Only project members with the right role may generate (align with membership model; §14 verification item). - Because a live fetch is slow, the endpoint should enqueue a Job and return `{"job_id": …}` (like `sync`), with `dry_run` running synchronously for preview. Decide sync-vs-async here explicitly. -### 9.5 Main UI button — later slice -"Create taxa list for region" in Project settings and the Site editor, calling 9.4. Reference the closed **#1020** GBIF taxon-select React component for the region/place picker. Add a React Query mutation hook under `ui/src/data-services/hooks/` (follow `ui/AGENTS.md`). Explicitly a **later phase** (§Phase 4). +### 10.5 Main UI button — later slice +"Create taxa list for region" in Project settings and the Site editor, calling 10.4. Reference the closed **#1020** GBIF taxon-select React component for the region/place picker. Add a React Query mutation hook under `ui/src/data-services/hooks/` (follow `ui/AGENTS.md`). Explicitly a **later phase** (§12 Phase 4). The list detail/UI should surface the model-coverage flag so uncovered species are visibly marked (§6.1). --- -## 10. Test plan (TDD — write these first) +## 11. Test plan (TDD — write these first) All tests use factories (`ami/*/tests/factories.py`) and the stubbed-source DI seam; **no test hits a live API**. Record one real GBIF and one real iNat response as a fixture (small, scrubbed) for the source-client parsing tests. @@ -391,41 +463,44 @@ All tests use factories (`ami/*/tests/factories.py`) and the stubbed-source DI s - Dedup-key precedence (gbif → inat → name) exercised explicitly. 3. **Mapping + create** — existing `Taxon` matched by external key; matched by name; `create_missing=True` creates via the hierarchy builder; `create_missing=False` records `unmatched_names`. Assert no duplicate `Taxon` on re-run (name-unique). 4. **Idempotency** — run the service twice for the same `(name, project)`; assert one `TaxaList`, taxa set refreshed, no duplicate lists/taxa. -5. **Classifier coverage is report-only** — with a `classifier`, assert `in_classifier_labels` / `not_in_classifier` are populated and that the saved list's taxa are **not** filtered by the classifier (a species outside the classifier's labels still appears in the list). -6. **Region derivation (A3)** — `derive_region_code()` returns a code for a project with deployment coords (stubbed reverse-geocode) and `None` when no deployment has coordinates. -7. **Masking resolution order** — parametrize the five-step ladder: site list wins over site region-code wins over project default list wins over project region-code wins over skip. Assert `auto` mode is a no-op (skips, does not raise) when nothing resolves, and that explicit `taxa_list_id` still behaves exactly as before. -8. **Masking `auto` batching** — `assertNumQueries` with a **multi-row** fixture (occurrences across two sites) proving list resolution is grouped, not per-occurrence (single-row fixtures cannot catch N+1 — `CLAUDE.md` DoD; template `ami/ml/tests.py:1006`). Strict `==` counts. -9. **Endpoint permission matrix** — member / non-member / anonymous / superuser against `POST …/regional/` (template `ami/main/tests.py:1532`); `?…=abc` bad-param returns 400. -10. **Command** — `--dry-run` creates nothing; `--all-projects --dry-run` iterates and reports; failure raises `CommandError`. +5. **Single-classifier coverage is report-only** — with a `classifier`, assert `in_classifier_labels` / `not_in_classifier` are populated and that the saved list's taxa are **not** filtered by that classifier beyond the default model-coverage rule. +6. **Model-coverage default subset (§6.1)** — a region whose species are a mix of model-covered and uncovered: default run saves only the covered ones; `Result.model_covered` and `regional_no_model_coverage` buckets are correct; an uncovered species is absent from the saved list. +7. **Opt-in create-and-flag (§6.1)** — with `include_uncovered=True`, uncovered regional species are kept, their `Taxon` rows created (under `create_missing`), and each is flagged `is_classifiable=False` with no `category_maps` membership. Assert the covered species remain flagged `True`. +8. **Coverage refresh (§6.3)** — adding a category map whose labels match a taxon sets that taxon's `is_classifiable=True` and its `category_maps` membership; removing/renaming the label (changing `labels_hash`) drops coverage on the next refresh; `Algorithm.objects.filter(category_map__taxa=taxon)` returns the covering algorithm. Exercise both the hook and the `refresh_taxon_model_coverage` command. +9. **Region derivation (A3)** — `derive_region_code()` returns a code for a project with deployment coords (stubbed reverse-geocode) and `None` when no deployment has coordinates. +10. **Masking resolution order (§9.2)** — parametrize the five-step ladder: site list wins over site region-code wins over project default list wins over project region-code wins over skip. Assert `auto` mode is a no-op (skips, does not raise) when nothing resolves, and that explicit `taxa_list_id` still behaves exactly as before. +11. **Masking `auto` batching** — `assertNumQueries` with a **multi-row** fixture (occurrences across two sites) proving list resolution is grouped, not per-occurrence (single-row fixtures cannot catch N+1 — `CLAUDE.md` DoD; template `ami/ml/tests.py:1006`). Strict `==` counts. +12. **Endpoint permission matrix** — member / non-member / anonymous / superuser against `POST …/regional/` (template `ami/main/tests.py:1532`); `?…=abc` bad-param returns 400. +13. **Command** — `--dry-run` creates nothing; `--all-projects --dry-run` iterates and reports; failure raises `CommandError`. Repo rules to honour while writing: migration in the same PR; no `assert` in production code; raise don't return sentinels; test docstrings state the invariant, not "this PR added X". --- -## 11. Phased rollout (internal-first) +## 12. Phased rollout (internal-first) Order the slices so the internal path is usable and validated before the API/UI. Each phase is independently shippable and mergeable. **Phase 0 — De-risk the external APIs (spike, no merge).** Exercise the candidate GBIF and iNat endpoints from a scratch script against 2–3 real regions (one where a project already exists — e.g. Quebec/Vermont for the moths classifier). Confirm: faceting/pagination behaviour, rate limits, name/key/count fields, and reverse-geocode response shape. **Gate:** measure how many of a real classifier's labels a generated regional list actually covers (the design doc's key unknown). This decides whether Proposal A is useful alone or needs Proposal B's manual curation. Output: a short findings note + recorded response fixtures for the parsing tests. *De-risks: the whole premise (coverage) and every "CANDIDATE, UNVERIFIED" endpoint above.* -**Phase 1 — Core service + one source + model fields (backend, no UI).** -`SourceSpecies`/`MergedSpecies`/protocol, one concrete source (whichever won Phase 0 on moth coverage), `merge_source_species`, `map_to_taxa`, `generate_regional_taxa_list`, the `Site`/`Project` fields + migration, and `create_taxon` extraction (§5.3). Tests 1–5. *De-risks: the union-with-provenance core and the data model, behind unit tests, with zero external dependency in CI.* +**Phase 1 — Core service + one source + model fields + coverage relationship (backend, no UI).** +`SourceSpecies`/`MergedSpecies`/protocol, one concrete source (whichever won Phase 0 on moth coverage), `merge_source_species`, `map_to_taxa`, `apply_model_coverage`, `generate_regional_taxa_list`, the `Site`/`Project` fields + migration, the **model-coverage relationship + `is_classifiable` + refresh hook/command (§3.3, §6)**, and `create_taxon` extraction (§5.3). Tests 1–8. *De-risks: the union-with-provenance core, the data model, and the "are the models aware" relationship, behind unit tests with zero external dependency in CI. The coverage relationship is Phase-1 material because the default list behaviour depends on it.* **Phase 2 — Management command + admin action + A3.** -Command (incl. `--all-projects --dry-run` backfill), `derive_region_code`, admin actions on Project/Site. Tests 6, 10. *De-risks: unattended backfill and gives internal users a way to generate lists before any API exists.* +Command (incl. `--all-projects --dry-run` backfill), `derive_region_code`, admin actions on Project/Site. Tests 9, 13. *De-risks: unattended backfill and gives internal users a way to generate lists before any API exists.* **Phase 3 — Masking auto-resolution (on the #999 branch / stacked follow-up).** -`ClassMaskingConfig` `taxa_list_mode`, resolution ladder, batched grouping, admin form "list mode". Tests 7, 8. **Requires #999 to have landed or be the base branch** — coordinate with that PR. *De-risks: the "works out of the box" promise; gated behind no-op-when-unconfigured so it is safe to enable.* +`ClassMaskingConfig` `taxa_list_mode`, resolution ladder, batched grouping, admin form "list mode". Tests 10, 11. **Requires #999 to have landed or be the base branch** — coordinate with that PR. *De-risks: the "works out of the box" promise; gated behind no-op-when-unconfigured so it is safe to enable.* **Phase 4 — API endpoint + main UI.** -`POST …/regional/` (sync-vs-async decided), request serializer, permission matrix (test 9), then the React button reusing #1020's picker. *De-risks: external/self-serve use once the service output is trusted internally.* +`POST …/regional/` (sync-vs-async decided), request serializer, permission matrix (test 12), then the React button reusing #1020's picker and surfacing the model-coverage flag. *De-risks: external/self-serve use once the service output is trusted internally.* Second source (union path) can slot in whenever Phase 0 shows it adds coverage — the merge already supports it; adding a source is `+1` client + `+1` parsing test. --- -## 12. Open questions (carried from the design doc + implementation-level) +## 13. Open questions (carried from the design doc + implementation-level) Design-doc decisions still open: - **GBIF vs iNaturalist first** — decided empirically in Phase 0 by moth coverage + key cleanliness. @@ -433,18 +508,23 @@ Design-doc decisions still open: - **Create-vs-skip unmatched taxa** — exposed as `create_missing` (default True). Confirm we want to grow the taxonomy from external sources vs. only intersect with existing taxa. - **TaxaList scope** — regional lists are project-scoped (not global) so one project's region does not pollute another's picker. Confirmed direction; `get_or_create_for_project(project=...)`. - **How the pipeline auto-applies masking** — per-`ProjectPipelineConfig` toggle vs. pipeline property. `taxa_list_mode="auto"` at the config level is the minimum; the pipeline-wiring decision aligns with #1289/#999. -- **Name-match fragility** — masking joins on `Taxon.name`; source/classifier spelling mismatches silently drop species. Measured in Phase 0; surfaced per-run via `unmatched_names` + `not_in_classifier`. +- **Name-match fragility** — masking joins on `Taxon.name`; source/classifier spelling mismatches silently drop species. Measured in Phase 0; surfaced per-run via `unmatched_names` + the model-coverage buckets + `not_in_classifier`. + +Model & DB awareness questions (§6): +- **Boolean flag vs. through-model** — recommendation is the category-map-anchored M2M (`AlgorithmCategoryMap.taxa`) + a denormalized `is_classifiable` boolean, with (a) boolean-only as an acceptable MVP. Confirm this over an `Algorithm`-anchored M2M (more direct if maps are rarely shared) — see §6.2. +- **When to recompute coverage** — on category-map create/update (hook keyed on `labels_hash`) plus an on-demand `refresh_taxon_model_coverage` command. Confirm no other write path (algorithm import, pipeline registration) needs to trigger it, and whether a periodic Celery Beat refresh is warranted as a safety net. +- **DB-presence vs. occurrence-presence** — `is_classifiable` marks "a model could predict this taxon". Separately, we may want to mark "this taxon has actually been observed in this project" (occurrence-presence) — a different, project-scoped signal. Decide whether the regional-list UI needs both, and keep them distinct (a taxon can be model-covered but never observed, or observed but not model-covered). Implementation-level questions surfaced here: - **Sync vs. async for the endpoint/admin action** — regional fetch can exceed request timeouts; likely enqueue a Job (like `sync`), keep `dry_run` synchronous. Decide in Phase 2/4. - **Cache backend + TTL** for source responses (Redis vs. on-disk; days-scale TTL). Needed once real fetch latency is known (Phase 0). - **Where `RegionSource` and the masking-resolution helper live** — enum with the model (stable migration path); resolver in `services/` vs. a thin `ml/post_processing/` shim. -- **Reverse-geocode reliability for unattended backfill** — until confirmed, `--all-projects` stays `--dry-run` by default (§7). +- **Reverse-geocode reliability for unattended backfill** — until confirmed, `--all-projects` stays `--dry-run` by default (§8). -## 13. What we still need to verify before building +## 14. What we still need to verify before building - Exact GBIF endpoint + params returning a species list for a GADM region, its facet-depth cap, and rate limits — and the iNat equivalent (`species_counts` by `place_id`). Both are on-paper only. - The GBIF reverse-geocode response shape and which GADM level to take for A3. -- Label-coverage of a generated regional list against a **real** classifier (Quebec/Vermont moths) — the go/no-go for Proposal A standing alone. +- Label-coverage of a generated regional list against a **real** classifier (Quebec/Vermont moths) — the go/no-go for Proposal A standing alone. This is also what validates the §6 default-subset behaviour: if model coverage of a region is very low, the default list will be small and the `include_uncovered` path becomes the common case. +- The cost of refreshing the coverage relationship on large classifiers — how many taxa a real category map resolves to, and whether the per-save hook is cheap enough or should defer to an async task. - Permissions story for the new fields and the generate action (who may set a region, who may trigger generation) consistent with the project-membership model. -``` From 17f330379181b80ec50ac9e3ef4b3c285e846a1f Mon Sep 17 00:00:00 2001 From: Michael Bunsen Date: Thu, 2 Jul 2026 14:14:25 -0700 Subject: [PATCH 3/4] docs: align model-coverage relationship naming with the requested spec (#1364) [skip ci] Rename the persisted model-coverage relationship to the through-model the requester asked for: Taxon.covered_by_algorithms (M2M to Algorithm) so the list and UI can show which model is aware of a taxon, with Taxon.has_model_coverage as the denormalized boolean MVP. The category-map-anchored variant is retained as a noted deduplication alternative and open question, since many algorithms share one category map. Updates the data-model section, the options table and recommendation, the refresh helpers, the Result-consuming step, the test plan, and the open questions accordingly. Refs #1364, #999, #1289 Co-Authored-By: Claude --- ...roposal-a-regional-taxa-lists-impl-plan.md | 39 ++++++++++--------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/docs/claude/planning/2026-07-02-proposal-a-regional-taxa-lists-impl-plan.md b/docs/claude/planning/2026-07-02-proposal-a-regional-taxa-lists-impl-plan.md index 84ec193ab..60e8097a0 100644 --- a/docs/claude/planning/2026-07-02-proposal-a-regional-taxa-lists-impl-plan.md +++ b/docs/claude/planning/2026-07-02-proposal-a-regional-taxa-lists-impl-plan.md @@ -89,13 +89,14 @@ Minimal columns; polygons/geometry are explicitly out of scope (design doc Propo This is the persisted relationship required by §6. Storing it here (not computing it live) is a deliberate decision — see §6.2 for the options considered and the recommendation. -- **`AlgorithmCategoryMap.taxa`** — `ManyToManyField("main.Taxon", related_name="category_maps", blank=True)`. The persisted membership: which `Taxon` rows a label set resolves to (by the same name-match `with_taxa()` uses). The label set is the real owner of coverage (many `Algorithm` rows share one `category_map` via the FK at `ami/ml/models/algorithm.py:212`), so anchoring the M2M on the category map avoids duplicating rows across algorithms. -- **`Taxon.is_classifiable`** — `BooleanField(default=False, db_index=True)`. Denormalized flag: `True` iff the taxon is in at least one category map's `taxa`. Gives cheap filtering and a simple UI signal without a join. +- **`Taxon.covered_by_algorithms`** — `ManyToManyField("ml.Algorithm", related_name="covered_taxa", blank=True)`. The persisted through-relationship: which classifier(s) can predict this taxon, i.e. whose category-map label set contains the taxon's name (the same name-match `with_taxa()` uses). This is what lets the list/UI show *which* model is aware of a taxon, not just whether one is. +- **`Taxon.has_model_coverage`** — `BooleanField(default=False, db_index=True)`. Denormalized flag: `True` iff `covered_by_algorithms` is non-empty. Gives cheap filtering and a simple UI badge without a join. Acceptable as the MVP on its own if the M2M is deferred (see §6.2). -"Which algorithm(s) cover this taxon" is then a derived query, not a stored column: -`Algorithm.objects.filter(category_map__taxa=taxon)` — richer than a boolean, and free once the M2M exists. +"Which algorithm(s) cover this taxon" is then a direct read — `taxon.covered_by_algorithms.all()` — and "which taxa can this model predict" is `algorithm.covered_taxa.all()`. -**Migrations:** the M2M through table lives with `AlgorithmCategoryMap` (an `ami/ml/` migration); the `is_classifiable` boolean is an `ami/main/` migration on `Taxon`. Both are additive. Neither backfills automatically — the refresh command (§6.3) populates them, and the same migration PR should either run a data migration calling that refresh or document that the command must be run once after deploy. +**Implementation note (dedup):** because many `Algorithm` rows share one `category_map` (the FK at `ami/ml/models/algorithm.py:212`), the refresh (§6.3) computes membership once per *category map* from its label set and then fans it out to the algorithms that use that map. Whether the persisted M2M is materialized directly as `Taxon ↔ Algorithm` (more rows, direct read) or normalized as `Taxon ↔ AlgorithmCategoryMap` with algorithm-level answers derived via the FK is left as an open question (§13); the recommended default is the direct `Taxon ↔ Algorithm` relationship the user asked for. + +**Migrations:** the M2M through table is an `ami/ml`-or-`ami/main` migration (either app can host a `Taxon ↔ Algorithm` M2M — put it where `Taxon` lives, `ami/main`, referencing `ml.Algorithm`); the `has_model_coverage` boolean is an `ami/main/` migration on `Taxon`. Both are additive. Neither backfills automatically — the refresh command (§6.3) populates them, and the same migration PR should either run a data migration calling that refresh or document that the command must be run once after deploy. --- @@ -254,7 +255,7 @@ These two facts drive the default behaviour, the opt-in richer behaviour, and th ### 6.1 Default subsets to model-known species; opt-in creates the rest, flagged - **Default:** the saved list is `regional-species-from-sources ∩ taxa-covered-by-at-least-one-classifier`. Rationale: class masking can only ever affect taxa in some classifier's label set, so a regional species no model can predict is inert for masking. Keeping the default tight makes the list predictable and keeps "why does my 1500-species region only mask to N" answerable. -- **Opt-in (`include_uncovered=True`, exposed as `--include-uncovered` / a form checkbox):** also keep regional species with no model coverage. Under this mode, species absent from the DB are created via the hierarchy builder (§5.3) but each carries an honest coverage indicator — `is_classifiable=False` and no `category_maps` membership — so the list/UI can show "in the region, but no model can predict it yet". Many valid regional species have no training data and will never appear in a prediction list; the model must not pretend otherwise. +- **Opt-in (`include_uncovered=True`, exposed as `--include-uncovered` / a form checkbox):** also keep regional species with no model coverage. Under this mode, species absent from the DB are created via the hierarchy builder (§5.3) but each carries an honest coverage indicator — `has_model_coverage=False` and an empty `covered_by_algorithms` — so the list/UI can show "in the region, but no model can predict it yet". Many valid regional species have no training data and will never appear in a prediction list; the model must not pretend otherwise. A note on interaction with `create_missing` (§5.1): under the default (model-covered-only) scope, a species is kept only if its name matches some label set; creating its `Taxon` when missing is safe because, by construction, it *is* classifiable. Under `include_uncovered`, `create_missing` also governs whether uncovered species get `Taxon` rows created (default yes) or are only reported as unmatched. @@ -264,11 +265,11 @@ Confirmed by code reading: there is **no persisted `Taxon ↔ Algorithm` or `Tax | Option | Shape | Verdict | |---|---|---| -| (a) boolean | `Taxon.is_classifiable` denorm, refreshed on category-map change | Cheap, filterable, good UI signal — but cannot say *which* model covers a taxon. Acceptable **MVP** on its own. | -| (b) through relationship | persisted `Taxon ↔ Algorithm` (or `↔ AlgorithmCategoryMap`) membership | Answers "which model(s) cover this taxon", supports per-classifier questions. The user asked for a relationship → this is the **target**. | +| (a) boolean | `Taxon.has_model_coverage` denorm, refreshed on category-map change | Cheap, filterable, good UI badge — but cannot say *which* model covers a taxon. Acceptable **MVP** on its own. | +| (b) through relationship | persisted `Taxon.covered_by_algorithms` M2M → `Algorithm` | Answers "which model(s) cover this taxon", supports per-classifier questions. The user asked for a relationship → this is the **target**. | | (c) compute-only | keep using `with_taxa()` live, no schema | Cheapest, but no persisted field and slow to filter/UI. **Rejected** — it is exactly today's state, and the user wants persistence. | -**Recommendation: (b), anchored on the category map, plus (a) as a denormalized convenience.** Because many algorithms share one `category_map`, storing membership as `AlgorithmCategoryMap.taxa` (§3.3) avoids duplicating rows per algorithm, and "which algorithms" derives for free via `Algorithm.objects.filter(category_map__taxa=taxon)`. The `Taxon.is_classifiable` boolean rides alongside for cheap filtering and a simple "models know this species" badge. This satisfies the "which models are aware" requirement while staying normalized. Whether the M2M should instead be anchored directly on `Algorithm` is an open question (§13) — anchoring on the map is the normalized choice given the shared-map schema, but an `Algorithm`-anchored M2M reads more directly if maps are rarely shared in practice. +**Recommendation: (b), the direct `Taxon ↔ Algorithm` M2M (`covered_by_algorithms`), plus (a) `has_model_coverage` as a denormalized convenience.** The M2M reads directly — `taxon.covered_by_algorithms.all()` names the covering models with no join — which is exactly the "show which models are aware" capability the user asked for. The boolean rides alongside for cheap filtering and a simple "models know this species" badge. To avoid recomputing per algorithm (many share one `category_map`), the refresh resolves membership once per category map and fans it out to that map's algorithms (§3.3, §6.3). Whether to instead normalize as `Taxon ↔ AlgorithmCategoryMap` (fewer rows when maps are shared, algorithm answers derived via the FK) is an open question (§13); the recommended default stays the direct through-model. If (b) is too much for the first slice, ship (a) alone as the MVP (boolean only), and add the M2M in a follow-up — the service and `Result` treat coverage as a single predicate either way, so the upgrade does not change their contract. @@ -276,14 +277,14 @@ If (b) is too much for the first slice, ship (a) alone as the MVP (boolean only) Classifier label sets change whenever an algorithm or category map is added or updated, so model-coverage is derived and needs an explicit refresh path tied to the same name-resolution `with_taxa()` uses (so coverage always matches masking's join): -- **`refresh_category_map_taxa(category_map)`** — recompute `category_map.taxa` from its labels via the `with_taxa()` resolution, then update `Taxon.is_classifiable` for the affected taxa (True iff still in any map's `taxa`). Set-based, bulk, no per-row queries. +- **`refresh_algorithm_coverage(algorithm)`** — resolve the taxa for the algorithm's `category_map` label set once via the `with_taxa()` resolution, set `algorithm.covered_taxa` to that set, then update `Taxon.has_model_coverage` for the affected taxa (True iff `covered_by_algorithms` is still non-empty). Because algorithms sharing a `category_map` resolve to the same taxa, the label→taxa resolution is done per map and fanned out to each algorithm using it. Set-based, bulk, no per-row queries. - **Hook on change:** recompute on `AlgorithmCategoryMap` save/import when the label set changes. `labels_hash` (`algorithm.py:68`) is the ready-made "did the label set change" signal — compare before/after and only recompute when it moves. Also trigger on the algorithm-registration / category-map-import code path that first creates a map. -- **Full rebuild:** a management command `refresh_taxon_model_coverage` loops all category maps, rebuilds membership, and recomputes `is_classifiable` — used for the initial backfill (the migration in §3.3) and as a repair tool. -- **When it recomputes** is called out explicitly so readers are not surprised by staleness: on category-map create/update (hook), and on demand (command). It is *not* recomputed on every occurrence write — masking still resolves names live via `with_taxa()`, so a brief lag in the denormalized flag never causes a wrong mask, only a slightly stale "is_classifiable" badge until the next refresh. +- **Full rebuild:** a management command `refresh_taxon_model_coverage` loops all category maps / algorithms, rebuilds the `covered_by_algorithms` membership, and recomputes `has_model_coverage` — used for the initial backfill (the migration in §3.3) and as a repair tool. +- **When it recomputes** is called out explicitly so readers are not surprised by staleness: on category-map create/update (hook), and on demand (command). It is *not* recomputed on every occurrence write — masking still resolves names live via `with_taxa()`, so a brief lag in the denormalized flag never causes a wrong mask, only a slightly stale `has_model_coverage` badge until the next refresh. ### 6.4 How the service uses coverage -In `generate_regional_taxa_list()` (§7): after `map_to_taxa()`, `apply_model_coverage()` partitions the mapped taxa into model-covered vs. uncovered using the persisted `is_classifiable` / `category_maps` relationship (not a live recompute — the relationship is the source of truth, kept fresh by §6.3). The default scope keeps only the covered partition; `include_uncovered` keeps both. The optional `classifier=` argument is a *narrower, reporting-only* overlay on top: "of the kept species, how many are in *this specific* model's labels" — it never changes which taxa are saved. +In `generate_regional_taxa_list()` (§7): after `map_to_taxa()`, `apply_model_coverage()` partitions the mapped taxa into model-covered vs. uncovered using the persisted `has_model_coverage` / `covered_by_algorithms` relationship (not a live recompute — the relationship is the source of truth, kept fresh by §6.3). The default scope keeps only the covered partition; `include_uncovered` keeps both. The optional `classifier=` argument is a *narrower, reporting-only* overlay on top: "of the kept species, how many are in *this specific* model's labels" — it never changes which taxa are saved. --- @@ -432,7 +433,7 @@ python manage.py generate_regional_taxa_list \ - `--all-projects`: loops `Project.objects.all()` (shape from `assign_roles.py:79`); for each project without a `region_code`, calls `derive_region_code()` (A3); **defaults to `--dry-run`** until derivation reliability is confirmed (§14). Prints a per-project summary. - Raises `CommandError` on failure; no sentinels. -A sibling command `refresh_taxon_model_coverage` (§6.3) rebuilds the coverage relationship + `is_classifiable`; it is independent of region generation but shares the coverage plumbing. +A sibling command `refresh_taxon_model_coverage` (§6.3) rebuilds the coverage relationship + `has_model_coverage`; it is independent of region generation but shares the coverage plumbing. ### 10.2 Unit tests — call the service directly with a stubbed source The primary regression surface (see §11). Stubbed `RegionalSpeciesSource` passed via the `sources=` DI seam; no network. @@ -465,8 +466,8 @@ All tests use factories (`ami/*/tests/factories.py`) and the stubbed-source DI s 4. **Idempotency** — run the service twice for the same `(name, project)`; assert one `TaxaList`, taxa set refreshed, no duplicate lists/taxa. 5. **Single-classifier coverage is report-only** — with a `classifier`, assert `in_classifier_labels` / `not_in_classifier` are populated and that the saved list's taxa are **not** filtered by that classifier beyond the default model-coverage rule. 6. **Model-coverage default subset (§6.1)** — a region whose species are a mix of model-covered and uncovered: default run saves only the covered ones; `Result.model_covered` and `regional_no_model_coverage` buckets are correct; an uncovered species is absent from the saved list. -7. **Opt-in create-and-flag (§6.1)** — with `include_uncovered=True`, uncovered regional species are kept, their `Taxon` rows created (under `create_missing`), and each is flagged `is_classifiable=False` with no `category_maps` membership. Assert the covered species remain flagged `True`. -8. **Coverage refresh (§6.3)** — adding a category map whose labels match a taxon sets that taxon's `is_classifiable=True` and its `category_maps` membership; removing/renaming the label (changing `labels_hash`) drops coverage on the next refresh; `Algorithm.objects.filter(category_map__taxa=taxon)` returns the covering algorithm. Exercise both the hook and the `refresh_taxon_model_coverage` command. +7. **Opt-in create-and-flag (§6.1)** — with `include_uncovered=True`, uncovered regional species are kept, their `Taxon` rows created (under `create_missing`), and each is flagged `has_model_coverage=False` with an empty `covered_by_algorithms`. Assert the covered species remain flagged `True`. +8. **Coverage refresh (§6.3)** — adding a category map whose labels match a taxon sets that taxon's `has_model_coverage=True` and its `covered_by_algorithms` membership; removing/renaming the label (changing `labels_hash`) drops coverage on the next refresh; `taxon.covered_by_algorithms.all()` returns the covering algorithm(s). Exercise both the hook and the `refresh_taxon_model_coverage` command. 9. **Region derivation (A3)** — `derive_region_code()` returns a code for a project with deployment coords (stubbed reverse-geocode) and `None` when no deployment has coordinates. 10. **Masking resolution order (§9.2)** — parametrize the five-step ladder: site list wins over site region-code wins over project default list wins over project region-code wins over skip. Assert `auto` mode is a no-op (skips, does not raise) when nothing resolves, and that explicit `taxa_list_id` still behaves exactly as before. 11. **Masking `auto` batching** — `assertNumQueries` with a **multi-row** fixture (occurrences across two sites) proving list resolution is grouped, not per-occurrence (single-row fixtures cannot catch N+1 — `CLAUDE.md` DoD; template `ami/ml/tests.py:1006`). Strict `==` counts. @@ -485,7 +486,7 @@ Order the slices so the internal path is usable and validated before the API/UI. Exercise the candidate GBIF and iNat endpoints from a scratch script against 2–3 real regions (one where a project already exists — e.g. Quebec/Vermont for the moths classifier). Confirm: faceting/pagination behaviour, rate limits, name/key/count fields, and reverse-geocode response shape. **Gate:** measure how many of a real classifier's labels a generated regional list actually covers (the design doc's key unknown). This decides whether Proposal A is useful alone or needs Proposal B's manual curation. Output: a short findings note + recorded response fixtures for the parsing tests. *De-risks: the whole premise (coverage) and every "CANDIDATE, UNVERIFIED" endpoint above.* **Phase 1 — Core service + one source + model fields + coverage relationship (backend, no UI).** -`SourceSpecies`/`MergedSpecies`/protocol, one concrete source (whichever won Phase 0 on moth coverage), `merge_source_species`, `map_to_taxa`, `apply_model_coverage`, `generate_regional_taxa_list`, the `Site`/`Project` fields + migration, the **model-coverage relationship + `is_classifiable` + refresh hook/command (§3.3, §6)**, and `create_taxon` extraction (§5.3). Tests 1–8. *De-risks: the union-with-provenance core, the data model, and the "are the models aware" relationship, behind unit tests with zero external dependency in CI. The coverage relationship is Phase-1 material because the default list behaviour depends on it.* +`SourceSpecies`/`MergedSpecies`/protocol, one concrete source (whichever won Phase 0 on moth coverage), `merge_source_species`, `map_to_taxa`, `apply_model_coverage`, `generate_regional_taxa_list`, the `Site`/`Project` fields + migration, the **model-coverage relationship (`covered_by_algorithms` + `has_model_coverage`) + refresh hook/command (§3.3, §6)**, and `create_taxon` extraction (§5.3). Tests 1–8. *De-risks: the union-with-provenance core, the data model, and the "are the models aware" relationship, behind unit tests with zero external dependency in CI. The coverage relationship is Phase-1 material because the default list behaviour depends on it.* **Phase 2 — Management command + admin action + A3.** Command (incl. `--all-projects --dry-run` backfill), `derive_region_code`, admin actions on Project/Site. Tests 9, 13. *De-risks: unattended backfill and gives internal users a way to generate lists before any API exists.* @@ -511,9 +512,9 @@ Design-doc decisions still open: - **Name-match fragility** — masking joins on `Taxon.name`; source/classifier spelling mismatches silently drop species. Measured in Phase 0; surfaced per-run via `unmatched_names` + the model-coverage buckets + `not_in_classifier`. Model & DB awareness questions (§6): -- **Boolean flag vs. through-model** — recommendation is the category-map-anchored M2M (`AlgorithmCategoryMap.taxa`) + a denormalized `is_classifiable` boolean, with (a) boolean-only as an acceptable MVP. Confirm this over an `Algorithm`-anchored M2M (more direct if maps are rarely shared) — see §6.2. +- **Boolean flag vs. through-model, and how to anchor it** — recommendation is the direct `Taxon.covered_by_algorithms` M2M → `Algorithm` + a denormalized `has_model_coverage` boolean, with (a) boolean-only as an acceptable MVP. The remaining choice is whether to store the M2M directly as `Taxon ↔ Algorithm` (recommended; direct read of which model covers a taxon) or normalize it as `Taxon ↔ AlgorithmCategoryMap` to dedup when many algorithms share a map (algorithm answers then derived via the FK) — see §6.2. - **When to recompute coverage** — on category-map create/update (hook keyed on `labels_hash`) plus an on-demand `refresh_taxon_model_coverage` command. Confirm no other write path (algorithm import, pipeline registration) needs to trigger it, and whether a periodic Celery Beat refresh is warranted as a safety net. -- **DB-presence vs. occurrence-presence** — `is_classifiable` marks "a model could predict this taxon". Separately, we may want to mark "this taxon has actually been observed in this project" (occurrence-presence) — a different, project-scoped signal. Decide whether the regional-list UI needs both, and keep them distinct (a taxon can be model-covered but never observed, or observed but not model-covered). +- **DB-presence vs. occurrence-presence** — `has_model_coverage` marks "a model could predict this taxon". Separately, we may want to mark "this taxon has actually been observed in this project" (occurrence-presence) — a different, project-scoped signal. Decide whether the regional-list UI needs both, and keep them distinct (a taxon can be model-covered but never observed, or observed but not model-covered). Implementation-level questions surfaced here: - **Sync vs. async for the endpoint/admin action** — regional fetch can exceed request timeouts; likely enqueue a Job (like `sync`), keep `dry_run` synchronous. Decide in Phase 2/4. From 574f9e302acf6a07066ed6afe92eb48208069994 Mon Sep 17 00:00:00 2001 From: Michael Bunsen Date: Thu, 2 Jul 2026 14:39:58 -0700 Subject: [PATCH 4/4] docs: add design writeup + Phase 0 coverage findings (#1364) [skip ci] Phase 0 spike verified GBIF/iNat regional endpoints and the A3 reverse-geocode path against a live run, and measured 70% coverage of the 2497-label Quebec & Vermont classifier from a Vermont region list. Verdict: GO for Proposal A. Co-Authored-By: Claude --- ...07-02-phase0-regional-coverage-findings.md | 44 ++++ .../phase0_regional_coverage_spike.py | 198 ++++++++++++++++++ ...07-02-regional-taxa-lists-class-masking.md | 153 ++++++++++++++ 3 files changed, 395 insertions(+) create mode 100644 docs/claude/analysis/2026-07-02-phase0-regional-coverage-findings.md create mode 100644 docs/claude/analysis/phase0_regional_coverage_spike.py create mode 100644 docs/claude/planning/2026-07-02-regional-taxa-lists-class-masking.md diff --git a/docs/claude/analysis/2026-07-02-phase0-regional-coverage-findings.md b/docs/claude/analysis/2026-07-02-phase0-regional-coverage-findings.md new file mode 100644 index 000000000..07dba393a --- /dev/null +++ b/docs/claude/analysis/2026-07-02-phase0-regional-coverage-findings.md @@ -0,0 +1,44 @@ +# Phase 0 findings — regional taxa-list coverage spike (#1364) + +**Date:** 2026-07-02 +**Verdict:** GO for Proposal A. The candidate GBIF and iNaturalist endpoints work with the parameters the plan assumed, the A3 reverse-geocode path resolves cleanly, and a region-derived list covers a useful majority of a real classifier's labels. +**Script:** `docs/claude/analysis/phase0_regional_coverage_spike.py` (run inside the django container; reads DB + live APIs, writes nothing). + +## Setup + +- Region: Vermont (chosen to match the Quebec & Vermont classifier). +- Taxon scope: Lepidoptera (GBIF taxonKey 797, iNat taxon_id 47157). +- Classifier under test: local `Algorithm` pk 10, "Quebec & Vermont Species Classifier - Apr 2024", **2497 labels**. +- Name match: exact string on the scientific name (same join class masking uses — `Taxon.name` == label). + +## Results (measured, this run) + +| Metric | Value | +|---|---| +| Q&V classifier labels | 2497 | +| GBIF VT Lepidoptera species | 2164 | +| iNat VT Lepidoptera species | 1922 | +| Region union (GBIF ∪ iNat) | 2299 | +| Q&V ∩ GBIF | 1743 (69.8% of labels) | +| Q&V ∩ iNat | 1527 (61.2% of labels) | +| **Q&V ∩ union** (default masking-list size) | **1749 (70.0% of labels)** | +| Region species not in Q&V (no-coverage bucket) | 550 (23.9% of the union) | + +Reverse-geocode check (A3): point `(44.26, -72.58)` → GADM level-1 gid `USA.46_1` ("Vermont"). The `/geocode/reverse` response carries GADM ids at every level, so deriving a region code from a deployment's stored `latitude`/`longitude` is straightforward — the `--all-projects` backfill path is viable. + +## Interpretation + +- **A region-derived default list is worth building.** For Vermont it keeps 1749 of 2497 classes and masks ~748 (30%) that neither GBIF nor iNat records in the state — a meaningful, defensible reduction, which is exactly what class masking is for. +- **GBIF is the primary source; iNat is secondary.** GBIF alone reaches 69.8%; adding iNat lifts the *intersection* by only 6 species (to 70.0%). iNat's larger contribution is to the regional *union* (the uncovered bucket), not to classifier coverage. This supports starting Phase 1 with a **GBIF-only** source client and adding iNat later — the wide-union merge still holds, it just starts with one source. +- **The `include_uncovered` opt-in is a real, non-trivial set** (550 species for Vermont). The default-subset-to-model-known behaviour matters: without it, a naive regional list would add 550 taxa no model can predict. + +## Caveats / what still needs verifying (carried into Phase 1) + +- **Name-join fragility is the top risk.** 748 Q&V labels (30%) are absent from the region union. Some are genuine regional absences; some are likely name-format or synonym mismatches (authorship, subspecies, GBIF/iNat vs. our backbone naming). Before trusting the default list, audit a sample of those 748 to estimate the true-absence vs. mismatch split, and decide whether to widen matching (e.g. `Taxon.search_names`, synonym resolution) — this is the single measurement that most affects list quality. +- **GBIF name resolution is slow** (2164 speciesKey→name lookups took ~92s at 16 threads). Production must cache resolved keys and/or resolve lazily; not a blocker, but the service needs a cache layer. +- **Granularity was GADM level 1 (state).** Level 2 (county) would tighten the list but risk under-including; the region-code field should carry the level, and the right default is an open question. +- **Single region / single classifier tested.** Vermont + Q&V is the friendliest case (the classifier was built for this region). Coverage for a mismatched region (e.g. a tropical site against Q&V) will be far lower — which is the correct behaviour, but worth confirming the numbers degrade sensibly before enabling auto-masking broadly. + +## Recommendation for Phase 1 + +Proceed. Build the core service with a **GBIF-first** source client (iNat behind the same protocol, added second), the wide-union merge, the model-coverage relationship, and the default-subset behaviour — all behind unit tests with a stubbed source (no network in CI). Fold a name-mismatch audit of the uncovered Q&V labels into the Phase 1 work, since it gates how much to invest in fuzzy matching. diff --git a/docs/claude/analysis/phase0_regional_coverage_spike.py b/docs/claude/analysis/phase0_regional_coverage_spike.py new file mode 100644 index 000000000..e33f0dc27 --- /dev/null +++ b/docs/claude/analysis/phase0_regional_coverage_spike.py @@ -0,0 +1,198 @@ +"""Phase 0 spike (#1364) — verify GBIF/iNat regional endpoints and measure real +classifier label coverage. Throwaway measurement script, run inside the django +container so it has DB + network + the repo's create_session(). + + docker compose exec -T django python docs/claude/analysis/phase0_regional_coverage_spike.py + +Region: Vermont (matches the Quebec & Vermont classifier). Taxon scope: Lepidoptera. +Nothing is written to the DB. Output is a coverage report on stdout. +""" + +from __future__ import annotations + +import concurrent.futures +import sys +import time + +from ami.ml.models.algorithm import Algorithm +from ami.utils.requests import create_session + +GBIF = "https://api.gbif.org/v1" +INAT = "https://api.inaturalist.org/v1" +LEP_GBIF = 797 # Lepidoptera, GBIF backbone taxonKey +LEP_INAT = 47157 # Lepidoptera, iNaturalist taxon_id +VT_POINT = (44.26, -72.58) # Montpelier, VT — for reverse-geocode (also exercises A3) +QV_ALGORITHM_PK = 10 # "Quebec & Vermont Species Classifier - Apr 2024" (2497 labels) + + +def norm(name: str) -> str: + return (name or "").strip() + + +def load_classifier_labels() -> set[str]: + algo = Algorithm.objects.get(pk=QV_ALGORITHM_PK) + labels = algo.category_map.labels or [] + print(f"[classifier] {algo.name!r} — {len(labels)} labels") + return {norm(x) for x in labels if norm(x)} + + +def gbif_reverse_gadm1(session, lat, lng) -> str | None: + """A3 check: reverse-geocode a point to its GADM level-1 gid.""" + r = session.get(f"{GBIF}/geocode/reverse", params={"lat": lat, "lng": lng}, timeout=30) + r.raise_for_status() + gadm1 = None + for item in r.json(): + gid = item.get("id", "") + # GADM level-1 ids look like "USA.46_1"; level-0 "USA", level-2 "USA.46.14_1" + if item.get("source", "").lower().find("gadm") >= 0 or gid.startswith("USA"): + parts = gid.split(".") + if len(parts) == 2 and gid.endswith("_1"): + gadm1 = gid + print(f"[gbif] reverse-geocode {lat},{lng} -> GADM1 {gid} ({item.get('title')})") + return gadm1 + + +def gbif_vt_species(session, gadm_gid: str, cap: int = 3000) -> set[str]: + """Distinct Lepidoptera species in the region via speciesKey facet, resolved to names.""" + keys: list[int] = [] + offset = 0 + page = 1000 + while True: + r = session.get( + f"{GBIF}/occurrence/search", + params={ + "taxonKey": LEP_GBIF, + "gadmGid": gadm_gid, + "hasCoordinate": "true", + "facet": "speciesKey", + "facetLimit": page, + "facetOffset": offset, + "limit": 0, + }, + timeout=60, + ) + r.raise_for_status() + facets = r.json().get("facets", []) + counts = facets[0]["counts"] if facets else [] + if not counts: + break + keys.extend(int(c["name"]) for c in counts) + offset += page + if len(counts) < page or len(keys) >= cap: + break + keys = keys[:cap] + print(f"[gbif] {len(keys)} distinct Lepidoptera speciesKeys in {gadm_gid} (cap={cap}); resolving names...") + + def resolve(key: int) -> str | None: + s = create_session() + try: + rr = s.get(f"{GBIF}/species/{key}", timeout=30) + if rr.status_code != 200: + return None + d = rr.json() + return norm(d.get("canonicalName") or d.get("species") or d.get("scientificName")) + except Exception: + return None + + names: set[str] = set() + t0 = time.time() + with concurrent.futures.ThreadPoolExecutor(max_workers=16) as ex: + for nm in ex.map(resolve, keys): + if nm: + names.add(nm) + print(f"[gbif] resolved {len(names)} species names in {time.time() - t0:.0f}s") + return names + + +def inat_place_id(session, q: str) -> int | None: + r = session.get(f"{INAT}/places/autocomplete", params={"q": q, "per_page": 10}, timeout=30) + r.raise_for_status() + results = r.json().get("results", []) + # admin_level 10 == state/province in iNat + for p in results: + if p.get("admin_level") == 10 and norm(p.get("name")) == q: + print(f"[inat] place {q!r} -> place_id {p['id']} (admin_level 10)") + return p["id"] + if results: + p = results[0] + print(f"[inat] place {q!r} -> place_id {p['id']} ({p.get('name')}, admin_level {p.get('admin_level')}) [fallback]") + return p["id"] + return None + + +def inat_vt_species(session, place_id: int) -> set[str]: + """Species-level Lepidoptera in the place; names come back inline (fast).""" + names: set[str] = set() + page = 1 + while True: + r = session.get( + f"{INAT}/observations/species_counts", + params={ + "place_id": place_id, + "taxon_id": LEP_INAT, + "quality_grade": "research", + "hrank": "species", + "per_page": 500, + "page": page, + }, + timeout=60, + ) + r.raise_for_status() + data = r.json() + results = data.get("results", []) + for row in results: + t = row.get("taxon", {}) + if t.get("rank") == "species" and t.get("name"): + names.add(norm(t["name"])) + total = data.get("total_results", 0) + if page * 500 >= total or not results: + break + page += 1 + print(f"[inat] {len(names)} species-rank Lepidoptera in place {place_id}") + return names + + +def pct(a: int, b: int) -> str: + return f"{(100.0 * a / b):.1f}%" if b else "n/a" + + +def main() -> int: + session = create_session() + labels = load_classifier_labels() + + gadm1 = gbif_reverse_gadm1(session, *VT_POINT) + gbif_names: set[str] = set() + if gadm1: + gbif_names = gbif_vt_species(session, gadm1) + else: + print("[gbif] could not resolve GADM1 gid; skipping GBIF") + + place = inat_place_id(session, "Vermont") + inat_names = inat_vt_species(session, place) if place else set() + + union = gbif_names | inat_names + print("\n==================== COVERAGE REPORT ====================") + print(f"Q&V classifier labels : {len(labels)}") + print(f"GBIF VT Lepidoptera species : {len(gbif_names)}") + print(f"iNat VT Lepidoptera species : {len(inat_names)}") + print(f"Region union (GBIF ∪ iNat) : {len(union)}") + print("--------------------------------------------------------") + cov_g = labels & gbif_names + cov_i = labels & inat_names + cov_u = labels & union + print(f"Q&V ∩ GBIF : {len(cov_g):5d} ({pct(len(cov_g), len(labels))} of labels)") + print(f"Q&V ∩ iNat : {len(cov_i):5d} ({pct(len(cov_i), len(labels))} of labels)") + print(f"Q&V ∩ union : {len(cov_u):5d} ({pct(len(cov_u), len(labels))} of labels) <-- default masking-list size") + print("--------------------------------------------------------") + only_region = union - labels + print(f"Region species NOT in Q&V labels (no-model-coverage bucket): {len(only_region)}") + print(f" ({pct(len(only_region), len(union))} of the regional union — these are the include_uncovered opt-in rows)") + print("========================================================") + # A few examples of the intersection and the uncovered set for eyeballing name-match quality. + print("sample Q&V∩union :", sorted(list(cov_u))[:8]) + print("sample region-only:", sorted(list(only_region))[:8]) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/claude/planning/2026-07-02-regional-taxa-lists-class-masking.md b/docs/claude/planning/2026-07-02-regional-taxa-lists-class-masking.md new file mode 100644 index 000000000..d7479ccb3 --- /dev/null +++ b/docs/claude/planning/2026-07-02-regional-taxa-lists-class-masking.md @@ -0,0 +1,153 @@ +# Let users build a project taxa list from a region, so class masking works out of the box + +**Labels:** enhancement, needs design, backend, ml +**Related:** #999 (class masking), #1289 (post-processing framework), #1094 (configurable taxa lists), #939 (import taxa from external lists), #1020 (closed — GBIF taxon select in the UI), #1293 (taxa list CSV export) + +## Summary + +Class masking (#999) lets an operator keep only the classifier predictions that fall inside a chosen taxa list, which cuts a global classifier down to the species that actually occur at a site. The feature is nearly ready, and the taxa list editor is merged (#1094). The missing piece is the taxa list itself: today someone has to hand-curate one taxon at a time, which is too tedious to expect from project owners, and nothing connects a list to the place a project monitors. + +This ticket proposes a way for a user to generate a regional species list automatically from an external biodiversity database (GBIF and/or iNaturalist) by giving a region code, so that curating a project's masking list becomes a single action instead of hundreds. It also proposes where region codes and taxa lists should live on the data model (on the `Site`, with a fall-back on the `Project`), and how the first class-masking pipeline can pick the right list automatically from the occurrence's site rather than making the operator choose one every time. + +The core capability — "given a region, produce a taxa list" — is written once as a service function and surfaced through a management command, a Django admin action, an API endpoint, and the main UI, so the same logic can also backfill regional lists for every project already in Antenna. + +We should ship the simplest useful version first. My recommendation is Proposal A (generate from a region code) over Proposal B (spreadsheet import with a matching UI); B is more flexible but much more work, and A covers the common case. + +--- + +## Background — current state of the code + +Grounded in the current tree so the proposals below reference real names. + +**Models** +- There is a `Site` model (verbose name "Research Site", `ami/main/models.py:654`) that groups deployments. It has `name`, `description`, `project` only — **no coordinates, region, or country**. Its geographic extent is derived on the fly from its deployments via `boundary_rect()` (`models.py:670`). +- Coordinates live on `Deployment` (`models.py:763`): `latitude` (`:770`) and `longitude` (`:771`). Deployment has **no `country` or `region` field** either. +- `Occurrence` → site chain: `Occurrence.deployment` (`models.py:3598`) → `Deployment.research_site` (`models.py:803`) → `Site`. `Occurrence` also carries a direct `project` FK (`models.py:3599`). +- `TaxaList` (`models.py:4646`): M2M `taxa` → `Taxon` (`:4652`) and M2M `projects` → `Project` (`:4653`); a list with `projects=None` is a global list. Manager helper `TaxaList.objects.get_or_create_for_project(name, project=None)` (`models.py:4598`). +- `Project` (`models.py:289`): **no default-taxa-list field, no region/country, no geo bounds.** It does have include/exclude *filter* taxa M2Ms on `ProjectSettingsMixin` (`ami/main/models_future/projects.py:42,52`), but those are for the occurrence default-filter system, not a linked `TaxaList`. +- `Taxon` (`models.py:4342`): `name` is globally unique (`:4346`); external keys already exist — `gbif_taxon_key` (`:4364`), `bold_taxon_bin` (`:4365`), `inat_taxon_id` (`:4366`); hierarchy via `parent` (`:4349`) and denormalized `parents_json` (`:4356`). + +**Class masking (branch `feat/postprocessing-class-masking`, PR #999; reworked onto the framework in `rework/999-class-masking-on-framework`)** +- `ClassMaskingTask` (`ami/ml/post_processing/class_masking.py:228`), config `ClassMaskingConfig` (`:17`) requires an explicit `taxa_list_id` (`:25`). +- The list is chosen by an operator from an admin dropdown over **all** taxa lists (`ami/ml/post_processing/admin/class_masking_form.py:25`), loaded by PK at run time (`class_masking.py:306`). +- Matching taxa-list membership to classifier categories is effectively **`Taxon.name == classifier label`**, resolved through `AlgorithmCategoryMap.with_taxa()` (`ami/ml/models/algorithm.py:105`, keyed by `taxon.name` at `:135`). `search_names__overlap` widens the DB query but the returned map is name-keyed. +- **Nothing currently selects a list by region, site, or project.** The operator picks a list by hand, and nothing constrains it to the classifier's region or to the project of the rows being processed. + +**Post-processing framework (merged, #1289)** +- `make_post_processing_action(...)` (`ami/ml/post_processing/admin/actions.py:177`) builds the admin action; a `build_jobs` hook (`actions.py:87,184`) is the escape hatch for row→Job fan-out. Task params are configured via an admin form only — there is no API surface for triggering post-processing yet. + +**Taxa import / external APIs** +- `import_taxa` command (`ami/main/management/commands/import_taxa.py`) ingests a CSV/JSON file (local path or URL), builds the rank hierarchy under an `Arthropoda` root, and creates a **global** list via `get_or_create_for_project(project=None)` (`:240`). It matches taxa **by name only** (`:321`) and has no `--project` argument yet (`@TODO` at `:169`). +- **No live GBIF / iNaturalist / GADM integration exists** anywhere in `ami/`. The `gbif_taxon_key` / `inat_taxon_id` fields are populated from files, never fetched. (PR #1020, closed, did prototype a GBIF taxon-select React component that called the GBIF API from the browser — a useful reference for the UI side.) +- HTTP with retries: `ami/utils/requests.py::create_session()` (`:14`) is the repo convention for outbound API calls. +- All-projects backfill template: `assign_roles.py` loops `Project.objects.all()` (`:79`) — the shape to mirror. + +--- + +## The reusable core (independent of which proposal we pick) + +The one thing every proposal needs is a single function: + +> **`generate_regional_taxa_list(region, project=None, classifier=None, name=None, dry_run=False) -> Result`** +> Given a region code, fetch the species that occur there from an external source, map them onto Antenna `Taxon` rows, optionally intersect with a classifier's label set, and create (or preview) a `TaxaList` scoped to the project. Returns a summary: matched / unmatched-from-source / not-in-classifier / created counts, plus the unmatched names so a human can review. + +Suggested home: `ami/main/services/regional_taxa.py` (new services module — see the "processing logic extraction" note in `CLAUDE.md`). + +This function is the single source of truth surfaced five ways, exactly as required: + +| Surface | How | +|---|---| +| Management command | `python manage.py generate_regional_taxa_list --project --region [--classifier ] [--all-projects] [--dry-run]` — also the backfill tool | +| Unit test | Calls the function directly with a stubbed source client; asserts match counts and idempotency | +| Django admin action | Action on `Project` and/or `Site` changelist → confirmation form (region source + code, optional classifier) → runs the function | +| API endpoint | `POST /api/v2/projects/{id}/taxa-lists/regional/` (and/or a `Site` variant) → returns the preview/result | +| Main UI | "Create taxa list for region" button in Project settings and in the Site editor → calls the API endpoint | + +Idempotency matters here: re-running for the same region should update the existing list, not create duplicates (mirror `get_or_create_for_project`). External calls go through `create_session()` for retry/backoff, and results should be cached — GBIF/iNat regional queries are slow and rate-limited. + +--- + +## Proposed data model changes + +Minimal columns; polygons and richer geometry come later. + +- **`Site`**: add `region_source` (choices: `gbif_gadm`, `inat_place`), `region_code` (char — a GADM GID like `USA.11_1`, or an iNat `place_id`), and `taxa_list` (nullable FK → `TaxaList`). This is where the user's "each site has a region code and a taxa list" lands. +- **`Project`**: add `region_source` + `region_code` and `default_taxa_list` (nullable FK → `TaxaList`) as the fall-back when a site has none. + +Note that `TaxaList↔Project` and `TaxaList↔Taxon` are M2M today; the new `taxa_list` / `default_taxa_list` are single FKs layered on top (a project can still be associated with many lists, but has one designated default). Each new field needs a migration in the same PR. + +### Resolution order when masking runs + +For "the first pipeline that uses class masking applies it automatically", the class-masking task grows an `auto` mode (resolve the list per occurrence) in addition to today's explicit `taxa_list_id`: + +``` +occurrence.deployment.research_site.taxa_list + ↳ else research_site.region_code → generate/lookup regional list + ↳ else project.default_taxa_list + ↳ else project.region_code → generate/lookup regional list + ↳ else no masking (log and skip) +``` + +This keeps class masking a no-op until a site or project has a region/list configured, so it is safe to enable by default on a pipeline. + +--- + +## Proposals (simplest first) + +### Proposal A — Generate a regional list from a region code *(recommended first)* + +User provides a region code (or we derive it, see A3); we fetch the regional species list, intersect it with the classifier's labels, and save a `TaxaList`. One button, no per-taxon work. + +Sub-options for the **source API** (can support more than one; start with whichever is easiest to get good moth coverage from): + +- **A1 — GBIF occurrence facets by GADM region.** Query GBIF's occurrence search faceted by `speciesKey`, filtered by `gadmGid` (GADM administrative region, level 0/1/2) or `country`, scoped to the relevant taxon (e.g. Lepidoptera key), `hasCoordinate=true`. GADM gives clean nested region codes and pairs naturally with "add polygons later". Species come back with GBIF keys, so mapping to `Taxon.gbif_taxon_key` is exact; fall back to name match. +- **A2 — iNaturalist species counts by place.** `GET /v1/observations/species_counts?place_id=&taxon_id=&quality_grade=research` returns a ranked species list for a place. iNat `place_id`s are easy to look up and map to `Taxon.inat_taxon_id`. Good community coverage; place boundaries are iNat-specific rather than GADM. +- **A3 — Derive the region code from existing coordinates.** Projects already have deployment `latitude`/`longitude`. Reverse-geocode a representative point to a GADM GID (GBIF exposes a reverse-geocode endpoint) or an iNat place, so we can generate lists for **all existing projects** with no manual region entry — this is what makes the "backfill every known project" line item feasible. + +**Matching / review.** Because our masking join is `Taxon.name == label`, the value of a regional list is its intersection with the classifier's category map. The function should report, per run: how many source species matched an existing `Taxon`, how many are new (create them via the `import_taxa` hierarchy builder), and how many of the matched taxa are actually in the classifier's labels (only those affect masking). Surface the not-in-classifier count so users understand why a large regional list still only masks to N species. + +**Effort:** medium. New service + one source client + model fields + admin action + one API endpoint + a thin UI button. Reuses `create_session`, `import_taxa.create_taxon`, `get_or_create_for_project`. + +### Proposal B — Spreadsheet import with a matching interface + +User uploads a spreadsheet of species; we match each row against the global classifier's class list and show an intermediate review screen (matched / fuzzy / unmatched, with manual override) before saving the `TaxaList`. + +- More flexible than A (handles curated/authoritative lists, non-GBIF sources, local common names). +- Much more work: file parsing, fuzzy matching UI, per-row conflict resolution, persistence of decisions. This is essentially a new interactive workflow. +- Partially foreshadowed by existing pieces: `import_taxa` already parses CSV/JSON and matches by name; #1293 already emits a taxa-list CSV (the reverse direction). A matching-review UI is the new part. + +**Recommendation:** defer. A regional generator (A) removes most of the tedium for the common case; B is the power-user path once A is in and we see what still needs manual curation. + +### Proposal C — Polygons / geometry *(explicitly future)* + +Store real geometry on `Site` (GeoDjango — there is already a commented-out TODO at `models.py:666`) and compute regional lists from a drawn boundary rather than an administrative code. Out of scope here; the `region_source`/`region_code` fields in A are designed so a `polygon` source can be added later without reworking the resolution order. + +--- + +## What we'd ship first (proposed slice) + +1. Service function `generate_regional_taxa_list(...)` + one source client (A1 **or** A2 — pick based on moth coverage) behind `create_session`. +2. Model fields on `Site` and `Project` + migration. +3. Management command wrapping the service, including `--all-projects` backfill using A3 reverse-geocoding. +4. Admin action on `Project`/`Site` (fastest path to internal use; matches the #1289 admin-first pattern). +5. Wire the class-masking `auto` resolution order so a pipeline can apply masking without an operator choosing a list each run. + +API endpoint + main-UI button follow once the service and its output are validated internally. + +--- + +## Open questions / decisions to make + +- **GBIF vs iNaturalist as the first source.** Which gives better regional moth coverage and cleaner keys for our classifiers? (Leaning GBIF+GADM for A1 because keys and nested regions are cleaner, but iNat place counts may match community reality better.) +- **Region granularity.** GADM level 1 (state/province) vs level 2 (county/district) — too coarse over-includes species, too fine under-includes. Configurable per site? +- **Create-vs-skip unmatched taxa.** When a source species has no `Taxon` row, do we create it (grows the taxonomy from an external source) or only keep intersections with existing taxa? Creating is more complete but imports taxa we may never classify. +- **`TaxaList` scope.** Regional lists are project-specific by nature; confirm they should be project-scoped lists (not global) so one project's region doesn't pollite another's list picker. +- **How the pipeline "auto-applies" masking.** Is it a per-`ProjectPipelineConfig` toggle, or a property of the pipeline itself? Needs to align with the #1289 framework and #999's config. +- **Name-match fragility.** Since masking joins on `Taxon.name`, mismatches between GBIF/iNat names and our classifier labels silently drop species. We should measure the miss rate on a real classifier before trusting auto-masking. + +## What we still need to verify (before building) + +- Confirm the exact GBIF endpoint + params that return a species list for a GADM region with acceptable rate limits, and the iNat equivalent — on paper both exist, but neither has been exercised from this codebase. +- Measure, for one real classifier (e.g. the Quebec/Vermont moths model), how many of its labels a generated regional list actually covers — this decides whether A is useful on its own or needs B's manual curation. +- Confirm reverse-geocoding deployment coordinates to a region code is reliable enough to backfill existing projects unattended, or whether it needs human confirmation per project. +- Decide the migration/permissions story for the new `Site`/`Project` fields (who can set a region, who can trigger generation) consistent with the project-membership model.