Bound mirror subrequest stalls with a 300s abort timeout - #8
Merged
molocule merged 1 commit intoAug 19, 2026
Merged
Conversation
A mirror subrequest is fire-and-forget: nothing consumes its response, so nothing external bounds its lifetime. Since #5, small single-poll request bodies take feed_mirror_body's clean-EOF path, which awaited the drain task with no timeout — a stalled subrequest pipeline permanently leaked the feeder, drain, and subrequest tasks plus the session and its 64KB retry buffer. Heap profiles on flash-proxy domain-proxy pods show ~24% of leaked memory under process_subrequest. Bound every wait on subrequest progress (post-finish body sends, the EOF send, and the drain wait) with a 300s stall timeout that aborts the subrequest. A healthy pipeline consumes chunks and completes well within it, so it only fires when the mirror data is already lost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
FPRS domain-proxy pods leak RSS at multiple GB/hour (us-west-2 pods reached 62GB before dying). Heap flamegraphs (via the now-fixed
/debug/pprof/heap/flamegraph) show ~24% of live heap underprocess_subrequest— tens of thousands of live mirror subrequest sessions on a 24-minute-old pod, each pinning its 64KBFixedBufferretry buffer plus three tasks.Mechanism: since #5, small single-poll request bodies (the vast majority) take
feed_mirror_body's clean-EOF path instead of the hard-abort path. That path awaited the drain task with no timeout:drainonly completes when the subrequest pipeline drops its sender, so any stalled mirror pipeline (hung mirror upstream, unconsumed body) permanently leaked the feeder task, drain task, subrequest task, session, and buffered body. Additionally, once a fork isfinish()ed,wait_for_abortcan never fire, so a parked body send had no escape either.Fix
Bound every wait on subrequest progress with
MIRROR_SUBREQUEST_STALL_TIMEOUT(300s):select!arm),On expiry the subrequest is aborted — same cleanup as the existing abort path. A healthy pipeline consumes each chunk and finishes its upstream exchange well within 300s, so the timeout only fires when the mirror data is already lost; worst-case leaked state becomes stall_rate × 300s instead of unbounded.
Tests
eof_wait_on_stalled_pipeline_aborts_subrequest— stalled pipeline after clean EOF is aborted and the feeder exits (paused-time).stalled_body_send_on_finished_fork_aborts_subrequest— parked send on a finished fork (abort can no longer fire) is unblocked by the stall timeout.finished_fork_sends_clean_eof_without_cancelling_subrequestandaborted_fork_cancels_subrequest_while_body_send_is_blockedunchanged and passing.Adds
tokio/test-utilas a dev-dependency for paused-time tests.🤖 Generated with Claude Code