Skip to content

fix(relay): wait for the peer's Close before dropping any WebSocket - #8426

Merged
wpfleger96 merged 18 commits into
mainfrom
duncan/ws-terminal-close-delivery
Oct 19, 2026
Merged

wpfleger96 merged 18 commits into
mainfrom
duncan/ws-terminal-close-delivery

Conversation

@wpfleger96

@wpfleger96 wpfleger96 commented Oct 18, 2026 •

Copy link
Copy Markdown
Member

🤖 A NIP-42 login refused for a NIP-FI key mismatch could reach the client as an abnormal close (1006, no reason) instead of the NOTICE "restricted: authorization denied" followed by close 1008 that every other authorization denial produces.

The relay did write the NOTICE and the 1008 Close, but the connection task dropped TCP as soon as the writer finished, before the peer finished the close handshake. The client could then lose frames it hadn't read yet. It depends on timing, so it showed up intermittently and is not specific to key pairing.

Relay route

  • recv_loop now hands back the read half even while a frame handler is still running. Frames are handled one at a time while the connection is live. Once it is cancelled, an in-flight handler (for example a Nostr CLOSE waiting on the subscription lock behind a REQ stuck in Redis) is spawned to finish instead of holding the reader. It is never dropped partway, so subscription retirement stays atomic, and cleanup joins it before the expiry barrier and application cleanup.
  • As soon as the writer finishes, await_client_close keeps the transport open until the client's Close arrives or the stream ends. It runs concurrently with joining the heartbeat, auth-timeout, expiry and auth-cancel tasks and the connection's application cleanup (subscriptions released, registry entry removed, presence cleared). Cleanup never waits on the peer, so a nonresponsive client cannot delay registry counts, presence or has_admitted_connection. The socket never waits on cleanup, so a stalled Redis (the pool has no wait timeout) or an expiry join blocked on its quiescence barrier cannot hold the socket and its file descriptor open past the bounded wait. The active-connection gauge and the connection permit are released after both finish.
  • The connection permit (BUZZ_MAX_CONNECTIONS) is now taken at the start of handle_connection, before the NIP-FI gate and the community-active bootstrap check, and held through the active handler or the refusal drain and its bounded wait. Previously a socket in bootstrap, or refused there, held no permit, so a burst of refusals could open sockets beyond the limit. Over the limit, the socket is still dropped with no frames. buzz_ws_connections_active still counts only connections that reached the active handler.

Huddle-audio route

  • The relay and audio bootstrap drains share one bounded helper. A socket refused before admission waits after its community registry entry is released.

  • Every exit of the audio handler gets the bounded peer wait. Exits whose cleanup follows their terminal frames (room_full, room_ended, version mismatch, archived channel, owner rejection or unreachable, failed commit confirmation) and the normal teardown hand the socket to a background task that waits for the client's Close, so lease releases, 48102 writes and room archiving on Postgres and Redis never hold the socket open. The remaining exits send their frames last, and handle_active_audio_connection waits once they return. The wrapper joins the background task before returning, so the audio connection permit is held until the socket ends, as on the relay route.

  • Audio error exits now follow their existing JSON error with a Close frame:

    Exit Close code
    huddle_relay_draining 1012 service restart
    join_rejected (owner resolution failed), huddle_owner_unreachable, failed channel lookup 1011 internal error
    huddle_audio_unavailable, unsupported_version, upgrade_required, room_full, room_ended, archived channel, owner rejection, NIP-FI authorization denial 1008 policy

    Audio login failures and authorization refusals outside NIP-FI authorization_denied still send only their existing error frame, then get the bounded wait.

  • The audio read half is an enum, so the hand-off has no expect() that can panic.

Timing

  • The peer wait is bounded by WS_TERMINAL_FLUSH_TIMEOUT (1s), and drains to end-of-stream so a client-initiated Close gets its reply. The terminal writer flush has its own 1s budget, so the worst case for terminal transport work is about 2s per refused or closed socket.
  • Terminal frame writes (flush_terminal_frames, send_exit_frames_bounded) still give up on the first failure, but a failed or timed-out send is now logged at debug instead of being discarded.
  • The graceful-shutdown comment in main.rs now describes per-stage budgets (restart jitter, restart-Close acknowledgement, terminal flush, peer wait) rather than a per-socket total. The 30s hard drain bounds every socket, and the 35s process shutdown budget fits within the chart's 60s terminationGracePeriodSeconds.

Tests

Real-socket tests in the infra-free relay lane of just test-unit and scripts/run-tests.sh unless marked PostgreSQL:

  • The key-mismatch login and a session-expiry denial (as a control) both assert NOTICE then 1008 / authorization denied on the wire.
  • A raw-TCP client that never answers Close: the relay holds the socket for at least the bounded wait, has deregistered the connection by the time the socket ends, and ends it on its own under a 10s watchdog.
  • A raw-TCP client that withholds its Close until the connection has deregistered: the relay ends the socket in under the 1s cap, which fails if cleanup waits on the peer.
  • A raw-TCP client that never answers while final cleanup is stalled at a test pause point: the socket still ends after the bounded wait.
  • A raw-TCP client that never answers Close on the relay bootstrap drain, the audio bootstrap drain and an early audio-handler exit: the relay holds each socket for the bounded wait.
  • On the relay route with a one-slot budget through the real handle_connection: a refused bootstrap holds the slot until its socket ends, and a bootstrap stalled on the community check already holds the slot before the AUTH challenge is sent.
  • An established relay session whose reader is blocked in a Nostr CLOSE handler on a held subscription lock when the session expires: the socket ends within the bounded wait while the lock is held, and cleanup completes with the subscription retired once it is released.
  • An established relay session whose expiry task is stalled behind a held session effect permit: the socket still ends within the bounded wait.
  • PostgreSQL: an established audio session whose teardown is stalled after its Close, and a stalled archived-channel exit, both end within the bounded wait from the hand-off.
  • PostgreSQL: a fast refusal (unsupported protocol version) through the real handle_audio_connection with a one-slot permit budget keeps the permit taken until that socket ends.
  • PostgreSQL: huddle_audio_unavailable sends its error and then Close 1008 as the very next frame.

Duncan and others added 3 commits October 18, 2026 13:41
The relay dropped TCP right after writing its terminal Close. The client's
Close reply then hit a closed socket, the kernel answered with RST, and the
client could lose the unread NOTICE and 1008 Close, surfacing as 1006. The
connection now holds the read half until the client answers Close, bounded by
the existing terminal deadline, and logs terminal frame send failures.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
The new module matched no exact-name filter, so CI never ran it. The raw
client now walks frame headers instead of scanning for a 0x88 byte.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Registry, presence and subscriptions no longer lag a nonresponsive peer by
the close-handshake wait. Tests now bound the wait from above and cover a
client that answers Close.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96 wpfleger96 changed the title fix(relay): wait for the client's Close reply before dropping the socket fix(relay): wait for the peer's Close before dropping the socket Oct 18, 2026
@wpfleger96
wpfleger96 marked this pull request as ready for review October 18, 2026 19:26
@wpfleger96
wpfleger96 requested a review from a team as a code owner October 18, 2026 19:26
@wpfleger96
wpfleger96 deployed to codex-review October 18, 2026 19:26 — with GitHub Actions Active
@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Oct 18, 2026
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

🔐 Codex Security Review

Status: review required for the current range.

The current range is af7979e1ca0c40af2ba4a8d05edad6bb5796c584...59cdf0f5c0ce78298efdb9ca6f4d02a376cb8b94.
A new review must complete for this exact range. When manual authorization
is required, a user with write access must comment exactly
@buzz-security-review 59cdf0f5c0ce78298efdb9ca6f4d02a376cb8b94 to authorize a new review.
Any previous review applies only to its recorded range.

Time from before connect so the hold lower bound and prompt upper bound
cannot be consumed by scheduler delay; replace per-read 3s timeouts with
one generous watchdog.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@github-actions github-actions Bot removed the codex-security-review-current The posted Codex security review matches its recorded range. label Oct 18, 2026
@wpfleger96
wpfleger96 deployed to codex-review October 18, 2026 19:37 — with GitHub Actions Active
@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Oct 18, 2026
Duncan and others added 2 commits October 18, 2026 15:47
The client withholds its Close reply until the connection deregisters, so a relay that cleans up after the peer wait now fails deterministically.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Holding the read half through presence cleanup let a stalled Redis (the pool has no wait timeout) keep the socket open indefinitely. Running both concurrently caps the socket at the 1s peer wait while cleanup still never waits on the client.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@github-actions github-actions Bot removed the codex-security-review-current The posted Codex security review matches its recorded range. label Oct 18, 2026
@wpfleger96
wpfleger96 deployed to codex-review October 18, 2026 20:15 — with GitHub Actions Active
@wpfleger96

Copy link
Copy Markdown
Member Author

🤖 Addressed in a22ea8b. Connection cleanup (subscription release, deregistration, presence clear) now runs concurrently with the bounded wait for the client's Close via tokio::join!, so a stalled Redis or exhausted pool can no longer hold the socket open past WS_TERMINAL_FLUSH_TIMEOUT. The completed wait branch drops ws_recv immediately, closing the transport while cleanup is still pending. stalled_cleanup_does_not_hold_socket_open stalls cleanup indefinitely and checks that the socket still ends.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 deployed to codex-review October 18, 2026 20:38 — with GitHub Actions Active
@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Oct 18, 2026
The bootstrap drains on the relay and audio routes, and every audio
handler exit, dropped the socket right after the server's Close. Unread
client input then turns the close into a reset that discards frames the
client has not read. Each path now holds the read half for the bounded
Close wait after its cleanup finishes.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@github-actions github-actions Bot removed the codex-security-review-current The posted Codex security review matches its recorded range. label Oct 18, 2026
@wpfleger96
wpfleger96 deployed to codex-review October 18, 2026 21:00 — with GitHub Actions Active
@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Oct 18, 2026
@wpfleger96 wpfleger96 changed the title fix(relay): wait for the peer's Close before dropping the socket fix(relay): wait for the peer's Close before dropping any WebSocket Oct 18, 2026
Exits that send their terminal frames and then await Postgres, Redis or
the remote owner kept the socket open for as long as that cleanup took.
Those exits and the normal teardown now hand the socket to a task that
waits, bounded by WS_TERMINAL_FLUSH_TIMEOUT, for the client's Close.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 deployed to codex-review October 18, 2026 21:28 — with GitHub Actions Active
@github-actions github-actions Bot added codex-security-review-current The posted Codex security review matches its recorded range. and removed codex-security-review-current The posted Codex security review matches its recorded range. labels Oct 18, 2026
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 deployed to codex-review October 18, 2026 21:55 — with GitHub Actions Active
@github-actions github-actions Bot added codex-security-review-current The posted Codex security review matches its recorded range. and removed codex-security-review-current The posted Codex security review matches its recorded range. labels Oct 18, 2026
Duncan and others added 2 commits October 19, 2026 10:54
… writer exit

A relay socket refused during bootstrap never held a connection slot, and
the close-delivery wait now keeps it open up to one more second, so refusal
bursts could exceed BUZZ_MAX_CONNECTIONS. The slot is now taken before
bootstrap and held through the refusal drain.

The peer wait also started only after the expiry task join, which blocks
on its quiescence barrier while a handler holds an effect permit across
Redis with no deadline. It now starts as soon as the writer finishes.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…ose-delivery

* origin/main:
  feat(relay): resolve NIP-FI issuer bindings from the database (#8377)
  test(mobile): remove flaky channel collection assertion (#8440)
  fix(ios): supply missing push notification sound at handoff (#8443)
  feat(release): support version-matched mobile stable branches (#8430)
  fix(mobile): default community push notifications to on (#8425)
  fix(push-gateway): request default notification sound (#8431)
  feat(mobile): add swipe-to-reply message gesture (#8436)
  chore(deny): check buzz-mesh-smoke in the root graph again (#8424)
  fix(mobile): bump Flutter 3.47, resolving memory leak (#8419)
  fix(mobile): bound search result rendering work (#8418)
  Polish mobile message layout and navigation controls (#8427)
  feat(relay): start a channel with its members in one create (#8421)
  fix(desktop): bump zip to 9 for RUSTSEC-2026-0367/0368/0369 (#8412)
  fix(deps): bump negentropy to 0.5.2 for RUSTSEC-2026-0365 (#8409)

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@github-actions github-actions Bot removed the codex-security-review-current The posted Codex security review matches its recorded range. label Oct 19, 2026
Duncan and others added 2 commits October 19, 2026 11:40
A Nostr CLOSE handled inline could wait forever on the subscription lock
behind a REQ stuck in Redis, so recv_loop never returned and the bounded
peer wait never started. On cancellation the in-flight handler is now
spawned to finish and joined before the expiry barrier and cleanup.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 deployed to codex-review October 19, 2026 16:20 — with GitHub Actions Active
Audio admission errors now send Close 1008 (refusal) or 1011 (server
failure), the peer wait drains to end-of-stream so a client-initiated
Close gets its reply, both bootstrap drains share one bounded helper, and
the audio read half becomes an enum so no expect() can panic. Adds a
falsifying assertion for the in-flight join and a PostgreSQL test for a
stalled archived-channel exit.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 deployed to codex-review October 19, 2026 18:10 — with GitHub Actions Active
@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Oct 19, 2026
These refusals sent only error text, so a silent client waited out the peer wait and was dropped without a Close. Draining closes 1012 so clients reconnect; the owner-resolve failure closes 1011; an unsupported deployment closes 1008. The shutdown comment now states per-stage budgets instead of a per-socket total the code does not guarantee.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@github-actions github-actions Bot removed the codex-security-review-current The posted Codex security review matches its recorded range. label Oct 19, 2026
@wpfleger96
wpfleger96 deployed to codex-review October 19, 2026 18:52 — with GitHub Actions Active
@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Oct 19, 2026
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@github-actions github-actions Bot removed the codex-security-review-current The posted Codex security review matches its recorded range. label Oct 19, 2026
@wpfleger96
wpfleger96 merged commit 91e7b46 into main Oct 19, 2026
104 of 105 checks passed
@wpfleger96
wpfleger96 deleted the duncan/ws-terminal-close-delivery branch October 19, 2026 20:13
wpfleger96 pushed a commit that referenced this pull request Oct 19, 2026
* origin/main:
  fix(relay): wait for the peer's Close before dropping any WebSocket (#8426)
  fix(desktop): release the live tail after a single scroll back to bottom (#8465)

Signed-off-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz>

# Conflicts:
#	crates/buzz-relay/src/connection.rs
wpfleger96 added a commit that referenced this pull request Oct 19, 2026
…gent-admission

* origin/main:
  feat(relay): tag every membership notice with its channel type (#8406)
  fix(relay): wait for the peer's Close before dropping any WebSocket (#8426)
  fix(desktop): release the live tail after a single scroll back to bottom (#8465)
  test(auth): read the clock once in NIP-FI until test (#8457)
  fix(mobile): reset community notification preferences (#8455)
  fix(release): keep mobile candidates pinned as branches advance (#8454)

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>

This branch was successfully deployed

No deployments
codex-review — 59cdf0f5 Deployed Oct 19, 2026 by wpfleger96 via Run Codex Security Review #9022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants