Add required permissions to processing service users for processing jobs

[mihow]

Summary

Several @action endpoints on ViewSets (notably JobViewSet.tasks / JobViewSet.result) delegate to the model's check_custom_permission(user, action), which builds a permission codename like {action}_{model} (or {action}_{job_type}_job for Job) and calls user.has_perm(codename, project). A number of these codenames are never registered — there is no matching Permission row in the database, and no Role in ami/users/roles.py grants them.

The practical consequence: only Django superusers can call these endpoints. Any non-superuser, including a user holding the MLDataManager (or any other) project role, receives 403 Forbidden. This is a silent gap — the action decorator and view code look complete, but the permission is unreachable except via the superuser bypass.

Currently observed in production: an external ML worker authenticating with a non-superuser API token cannot POST to /api/v2/jobs/{id}/tasks/ to pull queued batches, so that worker has been in a tight 403-retry loop with no work being done. Making the worker's backing user a superuser restores service, but that's a workaround, not a fix — it's also what the co-located worker user currently does, which is why the bug hadn't surfaced until a second worker with a non-superuser token was introduced.

Predecessors

This is in the same family as:

• feat(roles): allow ML Data Manager to run batch ML jobs #1212 — feat(roles): allow ML Data Manager to run batch ML jobs — added run_ml_job to the MLDataManager role after discovering batch ML runs were superuser-only.
• feat(roles): grant ML Data Manager collection and pipeline config perms #1215 — feat(roles): grant ML Data Manager collection and pipeline config perms — expanded the same role with collection + pipeline config perms.

Both fixes added permissions to Project.Permissions and wired them into MLDataManager. This issue is the same pattern, for the remaining custom-action codenames.

Missing permission codenames (audit)

Ran against a current production DB: all registered Permission.codename values, cross-referenced against codenames that check_custom_permission logic would construct for the custom actions on each ViewSet. Codenames with no matching Permission row:

Worker-side (external ML workers — highest priority):

• tasks_ml_job — JobViewSet.tasks, called by workers to pull queued NATS tasks
• result_ml_job — JobViewSet.result, called by workers to POST back inference results

Job lifecycle (user-initiated):

• cancel_ml_job
• cancel_data_export_job
• cancel_data_storage_sync_job
• cancel_populate_captures_collection_job

Read/dashboard actions:

• charts_project — ProjectViewSet.charts
• timeline_project — ProjectViewSet.timeline

User-marking / taxa management:

• unstar_sourceimage — asymmetric with star_sourceimage, which IS registered
• remove_taxalist — asymmetric with add_taxalist, which IS registered
• suggest_taxon

The full set of custom actions was enumerated from @action(... methods=...) decorators in ami/jobs/views.py, ami/main/api/views.py, ami/ml/views.py, ami/labelstudio/views.py. Ones whose codenames are registered (so they work for non-superusers): star_sourceimage, populate_sourceimagecollection, sync_deployment, test_s3storagesource, add_taxalist.

Each missing codename should be verified independently against its actual view — some ViewSets may override permission_classes, use a different permission path, or have auth handled upstream. A couple of the read actions (charts, timeline) probably don't need a dedicated perm and could collapse to view_project; the fix for those might be to rewrite the check rather than register a new perm. But for the job-lifecycle and worker-side ones, registration + role assignment is the clear fix.

Proposed direction

Following the pattern in #1212 / #1215:

1. Add the missing codenames to Project.Permissions (enum in ami/main/models.py, alongside RUN_ML_JOB et al.). The post_migrate signal that builds the guardian Permission rows + role groups will pick them up on next deploy.
2. Assign them to appropriate roles in ami/users/roles.py:
   • tasks_ml_job and result_ml_job → probably a new ProcessingServiceWorker role (narrowly scoped to worker auth tokens) rather than MLDataManager, because granting these to human data managers is broader than needed. Alternatively, add to MLDataManager if the team prefers one-role-per-user.
   • cancel_*_job → MLDataManager (matches run_*_job scope)
3. Re-examine the read/charts/timeline actions — may not need a dedicated perm; view_project may suffice after a small refactor.
4. Tests: extend the pattern in ami/users/tests that already covers TestMLDataManagerCanRunBatchMLJob to assert the new codenames flow to the right roles. Add an API test that mimics an external worker token (non-superuser) successfully POSTing /api/v2/jobs/{id}/tasks/ and /result/.
5. Migration: roles picked up via post_migrate. Existing assigned memberships will get the new perms automatically on next run.

Out of scope

• Removing the superuser workaround on existing worker user(s) should wait until the new role is rolled out + assigned + verified in staging. That's a deployment follow-up, not part of the permission-registration fix itself.
• Per-pipeline or per-project scoping of worker perms (e.g. "this token can only pull tasks for projects X/Y/Z") — worth discussing separately; guardian supports it, but the current architecture treats processing-service auth as project-membership-bound already.

How to reproduce

1. Create a non-superuser user.
2. Assign them the MLDataManager (or any existing) role on any project with an async_api job.
3. Generate an auth token for that user.
4. curl -H "Authorization: Token <token>" -X POST <base_url>/api/v2/jobs/<id>/tasks/ -d '{"batch_size":1}'
5. Observe 403 Forbidden. Promote the user to is_superuser=True and the same request returns 200 (or 405 on GET, proving auth itself works and only the object-permission layer was blocking).

[mihow]

Claude says: Related in-flight work: PR #1194 (API key auth + identity for processing services) and #1201 (API key management UI), plus ADC #136 (switch ADC workers to API-key auth). These register the processing-service permissions and identity this issue needs. Linking for tracking.