Lazy thumbnail generation can saturate the web worker pool during cold gallery loads

[mihow]

Summary

Since lazy thumbnail generation shipped (#1306, merged 2026-06-16), production APM shows that the thumbnail endpoint can transiently saturate the gunicorn worker pool during cold gallery loads. A single user opening a gallery whose thumbnails are not yet cached fans out to ~80 thumbnail requests; each cold request blocks a worker for several seconds while it fetches, decodes, resizes, and re-uploads the image. In the worst burst observed, demand reached ~11.7 worker-equivalents against a pool of 8 workers — i.e. the pool was fully saturated and requests queued for about two minutes, and other traffic's p99 latency rose ~9.4× during that window.

Two things keep this from being an emergency: it caused zero errors, timeouts, or worker restarts (the queue drained within the request timeout), and the web host is never CPU- or memory-bound (CPU peaked ~18%; only half its cores are serving workers). Absolute volume is also tiny — a few hundred thumbnail hits/day, well under 0.2% of traffic. This is a bursty worker-occupancy problem, not a capacity emergency. #1331 directly removes the largest slice of it (the warm path); this issue records the measurements so it's clear what #1331 buys and what is left over.

Numbers below are measured from production APM and a read-only check of the live host unless labelled otherwise. One caveat up front: retained APM transaction data for this service only goes back to ~2026-06-15, one day before #1306 merged, so a true before/after comparison is not possible — everything here is the absolute post-deploy picture.

Observations

Thumbnail endpoint latency (7 days, measured): 2,425 hits, p50 104 ms, p95 14.3 s, p99 17.5 s, max 90.4 s. Time is dominated by external (object-store) duration (avg ~7.7 s); database time is negligible (~55 ms).

Latency distribution:

Bucket	Hits	Share
Warm (<0.5 s)	1,345	55%
0.5–2 s	188	8%
2–10 s	390	16%
>10 s	502	21%

• Warm hit (cached row exists): avg 72 ms, but still ~178 ms of object-store round-trip because the endpoint serves a redirect to storage rather than the URL directly. This is exactly the cost Serve direct thumbnail URLs to reduce load on main app server #1331 removes.
• Cold hit (≥2 s, n=892): avg 9.9 s, almost entirely image fetch + generate + upload — i.e. ~10 s of worker-hold per cold thumbnail. Serve direct thumbnail URLs to reduce load on main app server #1331 does not change this path (a thumbnail still has to be generated once).
• Cumulative worker-hold across the week was ~9,160 worker-seconds — small in aggregate; the problem is how it concentrates.

Worst burst (measured), 2026-06-21: at 11:43 UTC, 81 hits consumed ~700 worker-seconds within 60 s, i.e. ~11.7 worker-equivalents of demand. The pool is 8 workers, so this fully saturated the pool and queued requests for ~2 minutes.

Collateral latency, same 10-minute window (measured): non-thumbnail traffic avg 38 → 98 ms (2.6×), p95 52 → 336 ms (6.5×), p99 176 → 1,656 ms (9.4×) — consistent with requests waiting for a free worker. 0% error rate in both the quiet and burst windows.

No worker restarts during the bursts (measured): the web container logs (covering both burst windows), the host kernel log, and APM logs show zero worker timeouts, SIGKILL/SIGSEGV, OOM-kills, or worker recycling. Workers booted once at container start and ran continuously. The saturation was pure queueing, not crash-and-respawn.

Host resource use during the burst (measured): web host CPU averaged ~0.7% → ~2.1% (max ~18%); memory ~18% → ~21%; load average well under 1. Over 8 days the web host CPU max stayed below 20%. The web tier is therefore worker-slot-bound, not CPU/RAM-bound — additional worker slots are cheap, and the host (16 cores) currently runs only 8 workers.

Database connections (measured): the web tier connects to Postgres directly (no connection pooler in that path). max_connections is 500; a spot check during normal load showed ~140 active backends (~28%, ~357 free). Connection exhaustion is therefore not a near-term risk at current load — though this is a single snapshot, not a burst peak (see below).

Why the worker-hold happens

The web server runs async workers (uvicorn.workers.UvicornWorker). With an async worker, a blocking call — here the synchronous PIL decode/resize and the object-store GET/PUT — blocks that worker's entire event loop for the duration, not just one coroutine. So each in-flight cold thumbnail effectively removes one worker from rotation until it finishes. The pool is currently 8 workers on a 16-core host (an explicit WEB_CONCURRENCY setting below the core count), so 8 concurrent cold thumbnails are enough to saturate it.

Does the cold rate decline as caches warm?

No — checked explicitly. The daily cold share (hits >2 s) since 2026-06-15 is volatile and not trending down (it ranged roughly 9%–68% day to day with no downward drift). Each new project or freshly browsed gallery re-introduces cold generation, so the burst risk recurs rather than self-limiting as the cache fills. This is why warm-path-only relief is necessary but not sufficient.

What #1331 fixes (and what it doesn't)

• Fixes: the warm path (55% of hits today). Serve direct thumbnail URLs to reduce load on main app server #1331 serves the storage URL straight from the serializer, so a cached thumbnail no longer holds a worker at all — the browser's <img> goes directly to storage. On an S3-backed deployment this removes the warm-path worker-hold entirely.
• Does not fix: the cold path. The first time any thumbnail is requested it still has to be generated on a web worker (~10 s hold, the 21% >10 s tail). The cold burst is what saturates the pool, and the cold rate does not self-limit, so Serve direct thumbnail URLs to reduce load on main app server #1331 alone shrinks but does not eliminate the saturation.

Suggested directions (ordered by effort/risk, for discussion)

1. Merge Serve direct thumbnail URLs to reduce load on main app server #1331 — removes the warm-path worker-hold (the majority of hits). Low effort, low risk, already mergeable.
2. Raise the worker count (e.g. WEB_CONCURRENCY 8 → 16). The host has 16 cores and runs ~18% CPU at peak, so adding worker slots is cheap headroom on existing hardware and directly addresses the saturation the bursts cause. Re-measure memory and DB connection use after the change.
3. Move cold thumbnail generation off the web worker — generate asynchronously (background task, or eagerly near upload) and serve a placeholder until ready. Because the cold rate does not decline on its own, this is the durable fix rather than an optional follow-up: it removes the multi-second worker-hold from the request path entirely.
4. Adding web replicas is not warranted — the host is not resource-bound, and the cheaper worker-count bump provides the same burst headroom.

Separate, unrelated observation (likely its own issue)

The single largest whole-app latency event in the window (2026-06-20 ~21:00 UTC, app p99 ~74 s) was not thumbnail-driven. It was a database-bound /api/v2/captures/collections/ query (p99 ~80 s, ~12 s of DB time) running concurrently with job-result ingestion, with the database host pinned near 100% CPU. Thumbnails were 2 hits that hour. This is a DB-contention problem independent of #1306 and probably deserves a separate ticket.

Verification still open

• Peak database connection use during a saturation burst — only an instantaneous snapshot was available (~28% used); the burst peak was not captured.
• Whether slow cold hits cluster on specific large projects/galleries — the thumbnail transactions carry no project attribute, so this isn't answerable from APM without adding one.
• Worker concurrency is inferred from worker-seconds, not directly measured — the request-queue-time metric is not currently captured, so "~12 worker-equivalents" remains an estimate (the queueing it implies is corroborated by the measured 9.4× p99 rise and the 8-worker pool size).

[mihow]

Claude says: The "separate, unrelated observation" above (the /api/v2/captures/collections/ DB-contention event) already has a fix in flight — it does not need a new ticket. Open PR #1301 denormalizes the three always-on SourceImageCollection source-image counts into cached columns and drops the live COUNT(DISTINCT) annotations from the capture-set list viewset, which is the recurring cost behind that event (on 2026-06-20 a request hit p99 ~80s with the database host near 100% CPU, concurrent with job-result ingestion).

The catch is that #1301 was never merged and has gone stale (currently conflicting with main, migration numbers now collide). It needs reviving rather than rewriting. I've left a coverage caveat on #1301 describing exactly what it does and doesn't denormalize, plus the rebase/renumber steps.

[mihow]

Should be closed by #1331 and an increase of workers on live