Deadlock: MemoryReadSafe hangs after a second-chance ACCESS_VIOLATION stop

[linsmod]

Bug

MemoryReadSafe hangs forever when a debugger pauses in the UE_CH_ACCESSVIOLATION / UE_CH_PAGEGUARD callback (second-chance exception) and then reads the target's memory.

Repro

With aidbg (a TitanEngine-based debugger) debugging a 64-bit program that raises a write-AV:

aidbg --batch -x "file test_exception.exe" -ex "set args av" -ex "run" -ex "dump 0x140000000 16"

The session stops at 0xc0000005, then dump (which calls MemoryReadSafe) never returns and the debugger hangs. bt still works (it does not use MemoryReadSafe). The same happens on a 32-bit (WoW64) target calling through a null function pointer.

Root cause

DebugLoop() holds LockBreakPointBuffer (via breakpointLock) for the whole STATUS_GUARD_PAGE_VIOLATION / STATUS_ACCESS_VIOLATION case. For a debuggee-generated AV (no memory breakpoint) DBGCode stays DBG_EXCEPTION_NOT_HANDLED and chAccessViolation is invoked inside that lock scope. A debugger that blocks in the callback leaves the lock held, so the next MemoryReadSafe (→ BreakPointPostReadFilter, same lock) deadlocks.

Suggested fix

Release breakpointLock before invoking the AV/GUARD_PAGE handler, mirroring the existing unlock() before the memory-breakpoint callback.

[linsmod]

For a reproducible reference, the scenario (plus a permanent regression test) lives in our debugger repo:

• Commit 443643b — "Fix WoW64 crash stops: real 32-bit context for rip/registers/bt, filter thunk frames, fix post-crash memory-read deadlock"
  linsmod/aidbg@443643b
• The included regression case 4.33 drives a 32-bit target (test_wow64.exe) that calls through a null function pointer, then asserts dump 0x400000 16 completes after the crash stop (i.e. no deadlock). The crash target (wow_crash) was added to tests/src/test_wow64.c in the same commit.
• Reproduce by hand against that target:

  aidbg --batch -x "file test_wow64.exe" -ex "set args crash" -ex "run" -ex "dump 0x400000 16" -ex "bt"
  

  Before the fix, the run stops at 0xc0000005 and dump (MemoryReadSafe) hangs; after the fix it prints the image bytes and the full 32-bit backtrace.

[linsmod]

Run comparison on a 64-bit target raising a write-AV, using aidbg (a TitanEngine-based debugger):

aidbg --batch -x "file test_exception.exe" -ex "set args av" -ex "run" -ex "dump 0x140000000 16" -ex "bt"

Before the fix — the session stops at the exception, then dump (a MemoryReadSafe) never returns and the debugger hangs (killed by timeout):

target = test_exception.exe
args = av
Target started. pid=22720 base=0x0000000140000000
Stopped: exception  [thread 24896]
  exception 0xc0000005 at 0x0000000140007620
  rip = 0x0000000140007620  (cause_access_violation+0x10 (test_exception.c:23))
<- `dump 0x140000000 16` hangs here (MemoryReadSafe never returns)

After the fix — the same command completes: dump prints the image bytes and bt walks the stack:

target = test_exception.exe
args = av
Target started. pid=22176 base=0x0000000140000000
Stopped: exception  [thread 16748]
  exception 0xc0000005 at 0x0000000140007620
  rip = 0x0000000140007620  (cause_access_violation+0x10 (test_exception.c:23))

0x0000000140000000: 4d 5a 90 00 03 00 00 00 04 00 00 00 ff ff 00 00  |MZ..............|

#0  0x0000000140007620 in test_exception.exe!cause_access_violation+0x10 (test_exception.c:23)
#1  0x00000001400076ac in test_exception.exe!main+0x6c (test_exception.c:31)
#2  0x0000000140007be9 in test_exception.exe!invoke_main+0x39 (exe_common.inl:79)
#3  0x0000000140007a92 in test_exception.exe!__scrt_common_main_seh+0x132 (exe_common.inl:288)
#4  0x000000014000794e in test_exception.exe!__scrt_common_main+0xe (exe_common.inl:331)
#5  0x0000000140007c7e in test_exception.exe!mainCRTStartup+0xe (exe_main.cpp:17)
#6  0x00007ffea2b6ccb7 in KERNEL32.DLL!BaseThreadInitThunk+0x17
#7  0x00007ffea462ad6c in ntdll.dll!RtlUserThreadStart+0x2c

Same hang and fix on a 32-bit (WoW64) null-pointer-call target.

[linsmod]

Exact code location (upstream x64dbg branch, file TitanEngine/TitanEngine.Debugger.DebugLoop.cpp, in DebugLoop()):

• The STATUS_GUARD_PAGE_VIOLATION / STATUS_ACCESS_VIOLATION case begins at line 965.
• Line 971 acquires the transaction lock: CriticalSectionLocker breakpointLock(LockBreakPointBuffer);
• Lines 1194-1209: for a debuggee-generated AV, if(DBGCode == DBG_EXCEPTION_NOT_HANDLED) invokes chAccessViolation (line 1198) / chPageGuard (line 1206) — still inside the lock scope (the case block, and with it the lock, only ends a few lines later).

Fix: release the lock before that handler call (see PR #40).

[mrexodia]

This needs to be addressed differently. The lock exists for a reason, removing it fixes this hang but reopens the actual bug fixed by the lock.

[linsmod]

You're right - the lock protects the memory-breakpoint transaction, so I'm dropping the engine-side approach (PR #40 closed).

Fixed it on the debugger side instead, no engine change needed: aidbg no longer calls MemoryReadSafe / MemoryWriteSafe / DisassembleEx / LengthDisassembleEx from its REPL thread (they take LockBreakPointBuffer, which is what deadlocked while paused in the AV handler). Reads/writes are now self-contained, and disassembly uses distorm on a local buffer.

\dump\ / \x\ / \disas\ / \�t\ all work after a second-chance AV stop, verified by the WoW64 crash-stop regression case (4.33) plus all 40 tests, against stock TitanEngine.

Leaving this open in case you still want to address the underlying design (a callback that blocks while the engine holds an internal lock).