Close the observability gaps that hid an incident's root cause and a silent email outage

[mihow]

Summary

A 2026-06-29 production incident took longer to root-cause than it should have, and in the process we discovered an external-email failure that had been happening silently for a long time. None of the items below are application-logic bugs — they are metrics/alerting gaps. Fixing them makes the next incident faster to diagnose and makes silent dependency failures visible.

Gaps and proposals

1. Database connection-pool saturation is invisible (highest value)

The incident's root cause was Postgres connection-slot exhaustion, but neither APM nor error tracking exposes connection counts — we had to query pg_stat_activity live to find it.

• Add a metric for active / idle / total Postgres connections against the configured maximum (a DB on-host integration, or a small periodic gauge exported to the metrics backend).
• Alert when total connections exceed ~80% of the configured maximum.
• This would have surfaced the root cause in seconds instead of after manual DB inspection.

2. Silent external-call failures go undetected (high value)

External email sends were failing with HTTP 401 and had been for a long time, completely silent because the mail path uses fail_silently=True, which swallows ESP errors. Users noticed only indirectly (password resets / confirmations never arriving).

• Add an alert on non-2xx responses to external dependencies, starting with the email ESP. The APM external-span status code is already captured, so an alert on email-ESP status ≥ 400 (or rate of non-success > 0) is straightforward.
• Consider surfacing fail_silently failures rather than swallowing them — e.g. log at ERROR (so error tracking sees them) or wire an ESP delivery webhook/signal — so a broken mail credential cannot stay invisible.

3. Error transactions lose their endpoint in APM

Requests that fail before view resolution get bucketed under the CSRF middleware function with a null request URI, so APM error analysis cannot tell which endpoint is erroring (this is why an unrelated external-call spike initially looked like it came from "CSRF middleware").

• Set the transaction name earlier / explicitly so 5xx transactions retain their route. Improves every future error investigation.

4. (Optional, lower priority) No request tracing attached to errors

Performance-trace sampling is effectively off, so error tracking carries no per-request trace context. A small non-zero trace sample rate would attach traces to a fraction of errors. This overlaps with APM, so it is low priority.

Impact

Items 1 and 2 are the high-value ones: #1 would have cut this incident's diagnosis time dramatically, and #2 would have caught the silent email outage long before users were affected.

Acceptance criteria

• A Postgres connection-count metric exists, and an alert fires before the pool is exhausted.
• An alert fires when email/ESP calls return non-2xx.
• 5xx transactions in APM carry their endpoint name rather than a null URI.