Skip to content

Let users submit identifications in bulk instead of one request per occurrence #1358

Description

@mihow

Summary

The only way to create an identification today is POST /api/v2/identifications/, one request per occurrence (config/api_router.py registers a single IdentificationViewSet; there is no bulk/batch route). A user verifying or identifying many occurrences — for example bulk-confirming a model's predictions across thousands of occurrences — has to fire one HTTP request per occurrence. Each request is a full DB transaction (under ATOMIC_REQUESTS) that withdraws the user's prior identifications on that occurrence and recomputes the occurrence determination. For a few thousand occurrences this is slow for the user, makes their client hammer the API serially or in parallel, surfaces 4xx/5xx under load that the client then retries (creating churn), and concentrates database-connection demand.

A bulk endpoint would let a client submit many identifications in one request, processed in far fewer transactions — a better experience for the user and materially less load on the system.

Motivation (observed)

On 2026-06-29 a member doing legitimate bulk verification submitted roughly 6,800 identifications across about 5,700 occurrences, at ~150–340 requests/min over ~90 minutes — one POST per occurrence. During a window of database-connection pressure, ~1,900 of those POSTs returned 4xx/5xx and the client retried them. Data integrity held (the per-user withdraw-on-save logic self-corrected the retries), but the one-request-per-occurrence access pattern made the load worse and the experience poor. The connection pressure had other contributing causes; the relevant point here is simply that no bulk path exists, so high-volume identification is forced into thousands of individual requests.

Proposed direction (to discuss — not a fixed design)

  • A bulk route, e.g. POST /api/v2/identifications/bulk/ (or have the existing create accept a list), taking a list of items: {occurrence, taxon, agreed_with_prediction_id?, agreed_with_identification_id?}.
  • Server-side: validate all items, then create in as few transactions as practical (chunked, e.g. 100/transaction, rather than one giant transaction). Run the withdraw + determination recompute per occurrence, but batched.
  • Enforce the same ObjectPermission per item — do not skip the per-object permission check, which is a classic bulk-endpoint bug.
  • Return per-item results (created / withdrew-prior / error) so the client learns what landed without N follow-up round-trips.

Open questions

  • Endpoint shape: new @action vs list-accepting create.
  • Transaction granularity: one transaction (long locks, large rollback blast radius) vs chunked. Chunked is likely safer.
  • Determination recompute is the expensive per-occurrence step — can it be deferred or batched into a single pass at the end?
  • Maximum batch size and per-request validation limits.
  • Does the verify UI adopt this, or is it primarily for scripted/automated clients?

Acceptance criteria

  • A client can submit ≥100 identifications in a single request.
  • Per-object permissions are enforced per item (a denied item fails just that item, not the batch silently).
  • Resulting determinations are identical to issuing N single POSTs.
  • Documented, with tests for the happy path, partial failure, and a permission-denied item.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions