Skip to content

GH-5395 A dead letter marked replayable mid-move is no longer deleted without reaching the inbox - #5397

Merged
jeremydmiller merged 1 commit into
mainfrom
gh-5395-replay-move-race
Nov 6, 2026
Merged

jeremydmiller merged 1 commit into
mainfrom
gh-5395-replay-move-race

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Fixes #5395.

The bug

MoveReplayableErrorMessagesToIncomingOperation moves replayable dead letters with four statements in one transaction, and each of them selected replayable = true on its own:

  1. insert into the inbox from the replayable dead letters
  2. delete their replay history
  3. insert their new replay history
  4. delete the replayable dead letters

At READ COMMITTED every statement takes its own snapshot, so a dead letter marked replayable (by IDeadLetters.ReplayAsync / ReplayMatchingAsync or the documented update ... set replayable = true) after statement 1 started and before statement 4 started was deleted by statement 4 without ever being copied. The message ended up in neither table and never ran, and nothing was logged.

Reproduced

A new test in RelationalDeadLetterFailureGroupCompliance, a_dead_letter_marked_replayable_while_the_move_is_running_is_not_lost, holds the move between its statements deterministically: a second connection locks the dead letter history table, the move copies (nothing yet) and blocks on the history statement, the dead letter is marked replayable, and the lock is released. Each relational store supplies its own lock:

Store Lock Without the fix
PostgreSQL lock table ... in share mode lost
SQL Server select count(*) ... with (tablock, holdlock) lost
Oracle lock table ... in share mode lost
MySQL lock tables ... read safe (REPEATABLE READ: the insert ... select locks the dead letter rows, so the mark waits for the move)
SQLite none, skipped safe (one writer)

This matches the analysis in the issue.

The fix

Statements 2 to 4 now act only on the dead letters statement 1 actually copied: replayable = true and a matching inbox row exists (same id, null-safe same received_at, so it is right under MessageIdentity.IdAndDestination too). That inbox row is the transaction's own uncommitted insert, so no other transaction can add or remove it mid-batch. A dead letter marked during the move is left alone and picked up, with its history, on the next pass. It is one portable change to the shared operation, so every relational store gets it, with no isolation level change and no provider specific SQL.

Verified locally (net9.0)

  • PostgresqlTests dead letter / message store / partitioning classes
  • SqlServerTests, MySqlTests, OracleTests, SqliteTests: the same classes, plus oracle_durability_command_translation (still four commands per move)
  • dotnet build wolverine.slnx -c Release -f net9.0

🤖 Generated with Claude Code

… without reaching the inbox

MoveReplayableErrorMessagesToIncomingOperation's four statements each selected
replayable = true on their own. At READ COMMITTED (PostgreSQL, SQL Server, Oracle)
a dead letter marked after the inbox insert ran was deleted by the final statement
without ever being copied. The history and delete statements now act only on dead
letters with a matching (uncommitted) inbox row from the insert.

New compliance test holds the move on a history table lock to open the window
deterministically; red before the fix on PostgreSQL, SQL Server and Oracle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jeremydmiller
jeremydmiller merged commit a89e663 into main Nov 6, 2026
49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A dead letter marked replayable while the durability agent is moving can be deleted without reaching the inbox

1 participant