Research and tune download failures across API and Celery

[patphongs]

What we’re after

Research and implement better handling for download failures across the API and Celery apps.

Logs show jobs can hit DB statement timeouts. The task writes download-failed:<task_id> for the API flow, but Celery reports the task as succeeded.

Action item(s)

• Decide whether Download failed should re-raise so Celery marks the task failed.
• Add a distinct metric or structured log for download failures, especially DB statement timeouts.

Completion criteria

• API and Celery failure behavior is documented.
• Download failures are visible through Celery logs.
• When API download fails, Celery should fail.

References/resources/technical considerations

• webservices/tasks/download.py: generic Exception logs Download failed, writes Redis failure state, and does not re-raise.
• webservices/resources/download.py: API polling reads download-failed:<task_id> and returns HTTP 500.