Repository navigation
GH-5395 A dead letter marked replayable mid-move is no longer deleted without reaching the inbox - #5397
Merged
Conversation
… without reaching the inbox MoveReplayableErrorMessagesToIncomingOperation's four statements each selected replayable = true on their own. At READ COMMITTED (PostgreSQL, SQL Server, Oracle) a dead letter marked after the inbox insert ran was deleted by the final statement without ever being copied. The history and delete statements now act only on dead letters with a matching (uncommitted) inbox row from the insert. New compliance test holds the move on a history table lock to open the window deterministically; red before the fix on PostgreSQL, SQL Server and Oracle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5395.
The bug
MoveReplayableErrorMessagesToIncomingOperationmoves replayable dead letters with four statements in one transaction, and each of them selectedreplayable = trueon its own:At READ COMMITTED every statement takes its own snapshot, so a dead letter marked replayable (by
IDeadLetters.ReplayAsync/ReplayMatchingAsyncor the documentedupdate ... set replayable = true) after statement 1 started and before statement 4 started was deleted by statement 4 without ever being copied. The message ended up in neither table and never ran, and nothing was logged.Reproduced
A new test in
RelationalDeadLetterFailureGroupCompliance,a_dead_letter_marked_replayable_while_the_move_is_running_is_not_lost, holds the move between its statements deterministically: a second connection locks the dead letter history table, the move copies (nothing yet) and blocks on the history statement, the dead letter is marked replayable, and the lock is released. Each relational store supplies its own lock:lock table ... in share modeselect count(*) ... with (tablock, holdlock)lock table ... in share modelock tables ... readThis matches the analysis in the issue.
The fix
Statements 2 to 4 now act only on the dead letters statement 1 actually copied:
replayable = trueand a matching inbox row exists (sameid, null-safe samereceived_at, so it is right underMessageIdentity.IdAndDestinationtoo). That inbox row is the transaction's own uncommitted insert, so no other transaction can add or remove it mid-batch. A dead letter marked during the move is left alone and picked up, with its history, on the next pass. It is one portable change to the shared operation, so every relational store gets it, with no isolation level change and no provider specific SQL.Verified locally (net9.0)
PostgresqlTestsdead letter / message store / partitioning classesSqlServerTests,MySqlTests,OracleTests,SqliteTests: the same classes, plusoracle_durability_command_translation(still four commands per move)dotnet build wolverine.slnx -c Release -f net9.0🤖 Generated with Claude Code