[jeremydmiller]

Fixes #5358. This supersedes #5359 by @erdtsieck. Her commit is cherry-picked unchanged, so she stays its author, and she's credited as co-author on the follow-up commit.

The problem

When a local queue's receiving block faulted terminally (#506), the queue rejected every later message until the process restarted. Listener-backed receivers are rebuilt by ListeningAgent/BackPressureAgent, but nothing owns a local queue that way. On the agents queue, that left a node heartbeating and getting agents assigned while it couldn't run a single agent command (seen in production after a managed OutOfMemoryException).

Changes

From #5359 (Anne Erdtsieck), unchanged: BufferedReceiver builds its block in buildReceivingBlock(). When RebuildsFaultedBlock is set (only BufferedLocalQueue sets it), the next enqueue swaps in a new block under a lock. A late fault from a block that was already replaced is ignored.

Follow-up:

• BufferedReceiver.HasFaulted and DurableReceiver.HasFaulted are backed by volatile fields. They're set on the faulting block's thread and read without a lock on every enqueue.
• When a buffered queue replaces a faulted block, the warning says how many buffered envelopes were lost (the faulted block's frozen Count).
• DurableLocalQueue had the same gap, and it's closed here. On the next enqueue, a faulted DurableReceiver is taken out the way a pause takes a receiver out:
  • The queue is latched, so anything published in the meantime is stored for recovery rather than held against rows that are about to be released.
  • The faulted receiver is drained, which releases its rows for recovery. The rows are durable, so unlike the buffered case nothing is lost.
  • It's disposed asynchronously, as ListeningAgent disposes a faulted receiver.
  • A new receiver is started through the existing startAsync, which waits for that drain (L1) before it claims rows. If a pause takes over in the meantime, its Restarter owns the restart instead.
• DurableReceiver.HoldsWork ignores a faulted block's frozen Count. Otherwise a replaced faulted receiver would read as holding work for as long as it lived, and every later blanket release at its address would fall back to releasing by id.

Tests

• faulted_buffered_local_queue (from Rebuild a buffered local queue whose receiving block faulted (#5358) #5359): a message published after the fault is handled, and the agents queue accepts envelopes again.
• faulted_durable_local_queue (new): a message published after the fault is handled by a rebuilt receiver, the queue is Accepting, and the rebuilt receiver keeps working. Both tests fail without the DurableLocalQueue change (checked by reverting it) and pass with it.

Run locally on net10.0:

• The faulted_*_local_queue classes: 4/4 pass.
• The CoreTests.Transports.Local, Runtime.WorkerQueues and Runtime.Partitioning namespaces: 389/389 pass.
• Full CoreTests: 4007 passed, 0 failed, 8 skipped.
• Wolverine builds with -f net9.0.
• PostgreSQL suites weren't run locally (the local port 5433 belongs to another project's container). CIPostgresql covers them.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U