[jeremydmiller]

The bug

A scheduled retry of a message received on a database-backed queue (ScheduleRetry in an error policy) deletes the message's inbox row and writes the envelope to the queue's scheduled table. On four of the five stores those two writes were not atomic:

Store	Before	Atomic?
SQL Server	delete ...; merge ... in one batch, no transaction	no
MySQL	DELETE ...; INSERT ... in one command, autocommit	no
SQLite	DELETE ...; INSERT ... in one command, autocommit	no
Oracle	two separate commands	no
PostgreSQL	two statements in one Npgsql command, which runs as one implicit transaction	yes

When the scheduled write failed (a command timeout, a deadlock), the inbox row was already gone and nothing was scheduled. The message then existed only in the receiver's in-memory RetryBlock. If the node died, or shut down, before that retry succeeded, the message was lost with no trace in the inbox, the queue, the scheduled table or the dead letters.

How it was found

The out-of-process durability chaos rig ran with EF Core on SQL Server, the SQL Server queue transport, 5 second command timeouts, and random SIGKILLs, SIGTERMs and process pauses. SqlServerQueueSender.ScheduleRetryAsync timed out repeatedly, and 3 of 18,030 messages were lost, all of them messages that had gone through a scheduled retry. A PostgreSQL run with the same faults stayed clean.

The fix

Each sender now runs the delete and the scheduled write in one ADO.NET transaction, the same way as the GH-5107 outbox moves in the same classes. On SQL Server this is a client-side SqlTransaction, not BEGIN TRANSACTION in the batch: a command timeout's attention is not caught by T-SQL TRY/CATCH, and would leave a server-side transaction open on a pooled connection.

Tests

There is one scheduled_retry_is_atomic class per store, in SqlServerTests, MySqlTests, OracleTests and SqliteTests:

• A scheduled retry moves the inbox row to the scheduled table.
• A failed scheduled write leaves the inbox row in place, and the retry completes the move once the write can succeed.
• The failure is forced with a trigger on the scheduled table. On Oracle it is forced instead with a message type longer than the column, because the CI Oracle user cannot create triggers.

The failed-write test of each store fails without that store's fix and passes with it.

Run locally on net9.0:

• the transport namespaces of SqlServerTests (132), SqliteTests (78), MySqlTests (62) and OracleTests (62), with no failures
• dotnet build wolverine.slnx -c Release -f net9.0

A SQL Server chaos rerun on this branch lost nothing. The timeout did not recur in that run, so the deterministic tests are the evidence for the fix.

Not in this PR

The same rerun found a separate problem. With EF Core, when a handler's commit is applied on the server but the client sees an error after a timeout, the envelope is moved to the dead letters even though its work committed, so a replay runs it twice. The commit's cascaded local-queue inbox rows also stay Incoming, owned by the live node, and nothing runs them. That needs its own design decision.

🤖 Generated with Claude Code