Server (Windows) 38435: memory corruption in the parallel per-client sync tick (5 crashes in 22 h: double free, corrupted object_pool free lists, corrupted control block) #4384

What happened?

Our production server crashed 4 times in 16.5 hours on 38435 (2026-11-07 03:39 to 20:03 UTC), within the first 35 hours after we deployed it. All four crashes are inside the same function: the body of the per-client lambda passed to tbb::parallel_for_each in ServerGameState::Tick (ServerGameState.cpp L1409), which is citizen-server-state-fivesv.dll+0x1a240 (the function that references playerEnteredScope / playerLeftScope / onPlayerDropped). Two crashed on a TBB worker and two on UV loop: svSync. There are no script frames on any stack.

I don't have PDBs. I matched functions by disassembly against the source at 5c3065c. The struct offsets below line up with object_pool.h, shared_reference.h and detached_queue.h. Please confirm with symbols.

A: double free under frameStates.erase

ntdll!RtlFreeHeap (fail-fast, HEAP_FAILURE_INFORMATION FailureType 8 = block not busy)
ucrtbase!free+0x208b
citizen-server-state-fivesv.dll+0x35fe0
citizen-server-state-fivesv.dll+0x4971c
citizen-server-state-fivesv.dll+0x1c79b   <- per-client lambda
citizen-server-state-fivesv.dll+0x4eaf0   <- TBB task

I matched the free to clientDataUnlocked->frameStates.erase(firstFrameState); (L1993-L2012). The freed block was already free.

B and C: corrupted m_tail in a thread-local bucket_entry::free_queue

Both crashes fault on the same instruction, mov rax,[rdx], which is m_tail->next in detached_mpsc_queue::pop_until_empty (L112-L113). They are two instantiations of the same code: +0xed70 and +0xee50. Each one is reached from the thread-local bucket_proxy → bucket_entry::allocate() (TLS slots +0x238 and +0x250).

• B (+0xed87 ← +0x41c40 ← lambda +0x1a322): the pool of shared_reference<sync::SyncCommandList, &syncPool>::Construct (L1440-L1441). m_tail pointed to unmapped memory.

• C (+0xee67 ← +0x48c7e ← +0x1cb3b ← lambda +0x1c605). The function at +0x1cac0 builds a FixedFunction<…,128> (invoker pointers at +0x80/+0x88) and pushes the key at +0x90. That is SyncCommandList::EnqueueCommand → syncPool.construct → commands.push (ServerGameState.h L1181-L1184), so this is the SyncCommand pool.

  From the full dump, the bucket_entry at 0x21b692dab80 holds:

  • m_head (+0x00) = 0x21c4d3f7440
  • m_tail (+0x80) = 0x00d0021c4d827f68, non-canonical
  • m_stub.next (+0x88) = 0x21c4d1ea1a0

  The low 48 bits of m_tail do not point to a free_queue_key. Entries in that pool instance are 0xA8 bytes with the key at +0xA0, and 0x21c4d827f68 falls inside the callback storage of a SyncCommand entry. So a node's next was overwritten while the node was on the free queue: a write after free, or the same entry freed twice.

D: shared_reference<SyncCommandList>::reset() with a corrupted control block

citizen-server-state-fivesv.dll+0x3288b   cmp byte ptr [rdx+110h],0   ; rdx = bucket = nullptr
citizen-server-state-fivesv.dll+0x1c95b   per-client lambda, end of scope (destructor of `scl`)

+0x32810 is shared_reference::reset() (L97-L113) for SyncCommandList; the value's object_entry::pool is at value+0x18 = sizeof(SyncCommandList).

1. Both counters reached zero, and Pool->destruct(value) succeeded.
2. control_blocks.destruct(block) then read block's object_entry::pool → pool_instance::bucket → bucket->expired (object_pool.h L284, expired at bucket+0x110). The bucket pointer was null.

The full dump shows what went wrong. Control-block entries are 24 bytes (control_block 8, pool 8, free_queue_key 8):

block-0x18: 00000000'00000000  pool=00000184'b9e73048  key=00000184'b9e7c4e8
block     : 00000000'00000000  pool=00000183'b9e73048  key=00000184'b9ec5400   <- ours
block+0x18: 00000000'00000000  pool=00000184'b9e73048  key=00000000'00000000

0x184b9e73048 is a real pool_instance: bucket 0x183c4b30580, object_count 0xaaaa = 1 MiB / 24. Our block's pool is exactly 0x1'0000'0000 lower. Something did a 32-bit decrement on the upper half of block->pool (block+0xC). That matches dec_weak() called through a control-block pointer of block+8, or dec_ref() through block+0xC. In other words, a reference was released through a stale or corrupted control-block pointer. That last part is my inference.

Common pattern (inference, not confirmed)

All four crashes are use-after-free or double-free signatures in memory that the per-client lambda allocates or frees: frameStates, SyncCommandList, SyncCommand and control_blocks. The lambda carries the upstream comment at L1415:

// #TODO: imagine if this mutates state alongside but after OnDrop clears it. WHAT COULD GO WRONG?

At this population players join and drop every few seconds. A lifetime race between the per-client tick and client drop/cleanup is my best guess.

Expected result

FXServer keeps running; no memory corruption in the OneSync per-client tick.

Reproduction steps

Observed 4 times on production in 16.5 hours. I cannot trigger it on demand.

1. Windows FXServer 38435, OneSync with sv_maxclients 2000 (big mode), onesync_population false, sv_enforceGameBuild 2944, default NAK sync (onesync_automaticResend off).
2. 165-560 players connected and playing normally, with constant joins and drops.
3. The crash comes at a random time: after 28 minutes to 3h10m of uptime.

Our anticheat logs of game events and state bag writes in the 60 s before each crash show normal traffic: no floods, no oversized values, nothing unusual on entity bags. In the minute before each crash, scripts did only ordinary things: joins, quits, timeouts, and one DropPlayer ban 16 s before A.

Importancy

Crash

Area(s)

FXServer, OneSync

Specific version(s)

Server 38435 windows (master 5c3065c), game build 2944

Additional information

• Environment: Windows Server 2022 (20348), 32 logical CPUs, 128 GB RAM, txAdmin 8.1.2. Memory was healthy before each crash: working set under 4 GB, system commit far below the 143 GB limit, and no hitch warnings beforehand.

• Previous artifact: feature/network-improvements-clean 37264. Its first crash came after ~9 days (a different heap corruption, in the ENet free path on svNetwork). On 38435 we had these 4 crashes in 16.5 hours.

• Profiler: a resource ran profiler record / stop on svMain every ~20 s during A and B, but it was not running at all during C and D. No saveJSON thread was alive in any of these dumps.

• Minidumps: fxserver-38435-onesync-crashes-sanitized-minidumps.zip (all 4) is attached in the first comment below. Sanitized because they are public:

  • Kept intact: thread contexts (GPRs, rip/rsp), thread stacks (return addresses), the module list with CodeView records, and the exception records.
  • Removed: all captured memory outside thread stacks is zeroed, and printable strings of 6 or more characters are zeroed inside stacks, vector/x87 register areas and the function-table stream.

  The raw minidumps (4-5 MB) and full dumps (3-8 GB) contain server secrets and player data. I can share them privately with Cfx staff.

• Possibly related: [Urgent] No Viable Production Artifact After b25770 Degradation - Sync Network Hitches / Mass Connection Timeouts #4244.