Skip to content

Bound mirror subrequest stalls with a 300s abort timeout - #8

Merged
molocule merged 1 commit into
richard/lossy-subrequestfrom
claudia/mirror-subrequest-timeout
Aug 19, 2026
Merged

Bound mirror subrequest stalls with a 300s abort timeout#8
molocule merged 1 commit into
richard/lossy-subrequestfrom
claudia/mirror-subrequest-timeout

Conversation

@molocule

Copy link
Copy Markdown

Problem

FPRS domain-proxy pods leak RSS at multiple GB/hour (us-west-2 pods reached 62GB before dying). Heap flamegraphs (via the now-fixed /debug/pprof/heap/flamegraph) show ~24% of live heap under process_subrequest — tens of thousands of live mirror subrequest sessions on a 24-minute-old pod, each pinning its 64KB FixedBuffer retry buffer plus three tasks.

Mechanism: since #5, small single-poll request bodies (the vast majority) take feed_mirror_body's clean-EOF path instead of the hard-abort path. That path awaited the drain task with no timeout: drain only completes when the subrequest pipeline drops its sender, so any stalled mirror pipeline (hung mirror upstream, unconsumed body) permanently leaked the feeder task, drain task, subrequest task, session, and buffered body. Additionally, once a fork is finish()ed, wait_for_abort can never fire, so a parked body send had no escape either.

Fix

Bound every wait on subrequest progress with MIRROR_SUBREQUEST_STALL_TIMEOUT (300s):

  • body-chunk sends on a finished fork (new select! arm),
  • the EOF send,
  • the post-EOF drain wait.

On expiry the subrequest is aborted — same cleanup as the existing abort path. A healthy pipeline consumes each chunk and finishes its upstream exchange well within 300s, so the timeout only fires when the mirror data is already lost; worst-case leaked state becomes stall_rate × 300s instead of unbounded.

Tests

  • eof_wait_on_stalled_pipeline_aborts_subrequest — stalled pipeline after clean EOF is aborted and the feeder exits (paused-time).
  • stalled_body_send_on_finished_fork_aborts_subrequest — parked send on a finished fork (abort can no longer fire) is unblocked by the stall timeout.
  • Existing finished_fork_sends_clean_eof_without_cancelling_subrequest and aborted_fork_cancels_subrequest_while_body_send_is_blocked unchanged and passing.

Adds tokio/test-util as a dev-dependency for paused-time tests.

🤖 Generated with Claude Code

A mirror subrequest is fire-and-forget: nothing consumes its response, so
nothing external bounds its lifetime. Since #5, small single-poll request
bodies take feed_mirror_body's clean-EOF path, which awaited the drain task
with no timeout — a stalled subrequest pipeline permanently leaked the
feeder, drain, and subrequest tasks plus the session and its 64KB retry
buffer. Heap profiles on flash-proxy domain-proxy pods show ~24% of leaked
memory under process_subrequest.

Bound every wait on subrequest progress (post-finish body sends, the EOF
send, and the drain wait) with a 300s stall timeout that aborts the
subrequest. A healthy pipeline consumes chunks and completes well within
it, so it only fires when the mirror data is already lost.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@molocule
molocule merged commit da2803e into richard/lossy-subrequest Aug 19, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant