Server (Windows) 38435: uncaught std::bad_alloc in profiler saveJSON thread kills the server, and terminateStub calls a null TerminateForException #4385

What happened?

Our production server (about 920 players) crashed on 2026-11-07 15:52 UTC while running profiler saveJSON. Two separate bugs combined.

1. Profiler off-thread commands have no exception handling

ExecuteOffThread, Profiler.cpp L33-L54 runs the command on a detached std::thread with no try/catch. In the dump, the faulting thread is the saveJSON worker (L634-L652). ConvertToJSON builds the whole trace as an nlohmann DOM, then copies all of it again at L362:

obj["traceEvents"] = traceEvents;

During that copy operator new(0x50) failed and threw std::bad_alloc (I decoded the C++ exception record from the dump). Nothing catches it, so it leaves the thread function and the process goes to std::terminate.

Heap state in the dump:

• Process heap CommitFailures = 1, and the thread's last NTSTATUS is 0xC0000017 (STATUS_NO_MEMORY).
• System commit was only ~13 GB of a 143 GB limit, and there is no job object limit. I do not know why the commit failed.
• Whatever the cause, a failed allocation in a diagnostic command should not take down a live server.

Context: a server resource ran profiler record + profiler saveJSON about once per minute while hitches were present. Each file was 12-16 MB of JSON, and there were ~110 saves that day before the crash.

2. On FXServer, terminateStub calls a null function pointer

SEHTableHandler.Win32.cpp L464 looks the handler up in the main executable:

_TerminateForException = (decltype(_TerminateForException))GetProcAddress(GetModuleHandle(NULL), "TerminateForException");

That export is defined in code/client/launcher/MiniDump.Shared.cpp L57. I parsed the export table of FXServer.exe from server 38435: it has no exports at all, so on FXServer _TerminateForException is nullptr.

For a C++ exception (0xE06D7363), terminateStub L446-L460 calls it without a null check:

_TerminateForException(&ptrs);   // L456, nullptr on FXServer
...
FatalError("UCRT terminate() called");   // never reached

In our dump this is an access violation at rip = 0, called from terminateStub in CoreRT.dll. This affects every uncaught C++ exception on Windows FXServer, not only the profiler: the crash report becomes a null call instead of the real exception and FatalError message.

Expected result

profiler saveJSON failing to allocate prints an error and aborts the save, and the server keeps running. Uncaught exceptions on FXServer reach FatalError with the exception info instead of crashing at address 0.

Reproduction steps

Observed once on production. Part 1 needs an allocation failure, so I can't trigger it on demand.

1. Windows FXServer 38435 on a busy server (hundreds of players), so recordings are large.
2. Repeatedly run profiler record <frames>, then profiler saveJSON <file>.json.
3. If an allocation fails inside ConvertToJSON or json.dump, the detached thread throws, std::terminate runs, and the process faults at address 0 in CoreRT's terminateStub.

Part 2 needs no profiler: any uncaught C++ exception on Windows FXServer goes through terminateStub with _TerminateForException == nullptr.

Importancy

Crash

Area(s)

FXServer

Specific version(s)

Server 38435 windows (master 5c3065c)

Additional information

Possible fixes:

• Wrap the lambda body in ExecuteOffThread in a try { ... } catch (const std::exception& e) { ... } that prints the error. This also covers save, dump and view; only view <path> has its own try/catch today.
• Use obj["traceEvents"] = std::move(traceEvents); in ConvertToJSON. It avoids the second full copy of the DOM and roughly halves peak memory for large recordings.
• In terminateStub, check _TerminateForException for null before calling it, or export TerminateForException from FXServer.exe.

Environment: Windows Server 2022 (20348), 32 logical CPUs, 128 GB RAM, txAdmin 8.1.2.

Dump id 9f27cb1a-350b-4b20-9f6a-1065bd7784a8. fxserver-38435-profiler-savejson-crash-sanitized-minidump.zip (attached in the first comment below) is a sanitized copy of the minidump. Thread contexts, stacks, modules and the exception record are intact. All other captured memory is zeroed, and printable strings are zeroed inside stacks and vector registers. The raw minidump (4.6 MB) and full dump (8.2 GB) contain server secrets and player data; I can share them privately with Cfx staff.

Workaround on our side: we stopped running profiler saveJSON automatically on production.

The OneSync crashes we had on the same build that day are reported in #4384.