spawnSync: restore the VM's event loop handle as soon as the wait is over by robobun · Pull Request #41762 · oven-sh/bun

robobun · 2026-09-06T21:27:48Z

Closed in favor of #40078, which fixes the same root cause. This branch keeps a smaller alternative and a deterministic fixture, described in #40078 (comment).

Problem

test/js/bun/perf/linker-order.test.ts (function tracer > records exact entries, and keeps them across an exec'd child) timed out after 90 s in the bun test --parallel batch on the aarch64 glibc lanes (builds 110996, 111195, 111496, 111536) and passed alone.
The test's spawnSync(llvm-nm) spun: Bun.spawnSync points vm.event_loop_handle at a private loop until the call returns, and it built its result (JS allocations) before restoring it. A GC finalizer that ran there and released a main-loop poll (FilePoll::deinit resolves vm.event_loop_handle) debited the private loop. With the private loop at -2, the next spawnSync read num_polls == 0 with its pidfd and stdout still armed, and us_loop_run_bun_tick returned before epoll_wait on every turn.
The finalized objects were the process.stdout/process.stderr sinks that every file of an --isolate worker leaves behind (FileSinks polling a dup of the worker's stdio socket).

Fix

Restore the handle right after the wait loop, before any JS value is created; SpawnSyncEventLoop::cleanup is idempotent so the scope guard still covers early returns.
Verified: test/js/bun/spawn/spawnsync-isolated-event-loop.test.ts (two new cases, both fail on an unfixed build and pass with the change), plus the spawn, spawnSync and child_process suites.

robobun · 2026-09-06T21:28:03Z

^{Updated 2:22 AM PT - Sep 8th, 2026}

@autofix-ci[bot], your commit 82af3a8 is building: #112790

github-actions · 2026-09-08T03:24:50Z

+/// Diagnostic: the loop `Bun.spawnSync` runs on (set by `SpawnSyncEventLoop`).
+/// With `BUN_DEBUG_SPAWNSYNC_DRIFT` set, every counter adjustment that lands on
+/// it is printed with a backtrace, so an inc/dec that belongs to the main loop
+/// but hits this one can be attributed.


If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

github-actions · 2026-09-08T03:24:51Z

+    // Raw return addresses as offsets into the executable, for a symbolizer,
+    // in case the names above come out as `<unknown>`.


If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

github-actions · 2026-09-08T03:24:51Z

+        // bun links as ET_EXEC (no PIE), so the address itself is what a
+        // symbolizer wants; tag the ones inside the executable.


If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

robobun · 2026-09-08T03:26:29Z

The three comment-cop findings are on src/uws_sys/Loop.rs tracing that exists only for this investigation: it prints a backtrace for every counter adjustment that lands on the spawnSync loop, gated behind BUN_DEBUG_SPAWNSYNC_DRIFT. It is a probe, not a fix, and it does not ship. It comes out, together with the probe tests, once the CI run on this commit names the code path that decrements the wrong loop. The PR stays a draft until then.

github-actions · 2026-09-08T04:11:23Z

+    // Return addresses via the frame-pointer walk the crash handler uses. bun
+    // links as ET_EXEC, so an address is what a symbolizer wants as-is.


If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

robobun · 2026-09-08T04:22:26Z

Same as above for the new comment-cop finding on src/uws_sys/Loop.rs:35: investigation-only tracing, removed before this leaves draft.

github-actions · 2026-09-08T04:39:35Z

+            // PROBE: the loop can iterate without a wake source when the isolated
+            // loop has no polls registered (`us_loop_run_bun_tick` returns at
+            // once on `num_polls == 0`) while some stdio still reports pending
+            // activity. Name what is pending the first time that happens.


If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

robobun · 2026-09-08T04:52:35Z

The comment-cop finding on src/runtime/api/bun/js_bun_spawn_bindings.rs:1888 is on the same investigation-only instrumentation. Current state of the hunt, for the record: the hang is Bun.spawnSync spinning in its wait loop, not a blocked syscall. The isolated spawnSync loop reaches num_polls == 0 while compute_has_pending_activity() is still true, and us_loop_run_bun_tick returns immediately at num_polls == 0, so the loop turns without a wake source until the test deadline. That matches the captured kernel state: main thread in state R with seconds of CPU, the child a zombie, the pidfd still in the isolated epoll. The probe now prints which stdio still reports pending activity in that state. Once that is named, the fix and a regression test replace all of this instrumentation.

…over While Bun.spawnSync() waits for its child it points vm.event_loop_handle at a private uSockets loop. It restored the handle only when the call returned, after it had built the result object. Building the result allocates on the JS heap (the stdout and stderr Buffers, the resourceUsage object), and an allocation can run finalizers of dead cells. FilePoll::deinit resolves its loop through vm.event_loop_handle, so a finalizer that released a main-loop poll in that window debited the private loop instead: its num_polls went negative and the main loop kept a poll count nothing would ever release. A private loop left at -2 makes the next spawnSync spin: with its three polls registered num_polls is 1, the first EOF takes it to 0, and us_loop_run_bun_tick returns before epoll_wait whenever num_polls is 0. The wait loop then turns at full speed and never collects the child's exit. That is the hang behind linker-order.test.ts timing out in the aarch64 parallel test batch: every file of an --isolate worker leaves the previous global's process.stdout/stderr FileSinks behind, each holding a poll on a dup() of the worker's stdio socket, and the test's spawnSync(llvm-nm) calls finalized them. Restore the handle right after the wait loop, before any JS value is created. cleanup() becomes idempotent so the existing scope guard still covers the early-return paths.

github-actions · 2026-09-08T09:20:20Z

+//! - Temporarily overrides vm.event_loop_handle to point to the isolated loop.
+//!   Poll and keep-alive accounting resolves its loop through that handle, so
+//!   the override must end before anything that can run GC finalizers of
+//!   main-loop objects (any JS allocation); `spawnSync` calls `cleanup` as soon
+//!   as its wait is over.


If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

github-actions · 2026-09-08T09:20:21Z

+    /// Restore the original event loop handle. Idempotent: `spawnSync` calls it
+    /// as soon as its wait is over, and again on every exit path.


If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

github-actions · 2026-09-08T09:20:22Z

+        // Early returns before the wait; after the wait `cleanup` already ran
+        // and this is a no-op.


If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

github-actions · 2026-09-08T09:20:23Z

+        // Restore the VM's loop before the JS allocations below: they can run
+        // finalizers, and a finalizer that releases a main-loop poll resolves
+        // `vm.event_loop_handle`. It must not debit this private loop.


If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code

robobun · 2026-09-08T09:24:18Z

Closing this in favor of #40078. The root cause is the one that PR fixes: a GC finalizer that runs while Bun.spawnSync has vm.event_loop_handle swapped releases a main-loop poll against spawnSync's private loop, the private loop's num_polls goes negative, and a later spawnSync spins once the count reads 0 with polls still armed. For linker-order.test.ts the finalized objects were the process.stdout/process.stderr sinks left behind by earlier files of the --isolate worker, and the spinning call was the test's spawnSync(llvm-nm).

The branch now holds only a smaller alternative fix (restore the handle right after the wait loop, +25/-3 under src/) and a fixture that reproduces the drift deterministically on x64; both are described in #40078 (comment) for whoever lands that PR. The probe commits and instrumentation from the investigation are gone from the branch, so the review notes above about comment length refer to code that no longer ships from here.

github-actions Bot added the claude label Sep 6, 2026

github-actions Bot reviewed Sep 8, 2026

View reviewed changes

robobun force-pushed the robobun/83a74dd3/linker-order-aarch64-hang branch from f1a16f0 to 4f7e569 Compare September 8, 2026 09:20

github-actions Bot reviewed Sep 8, 2026

View reviewed changes

[autofix.ci] apply automated fixes

82af3a8

robobun mentioned this pull request Sep 8, 2026

spawnSync: release polls and keep-alives on the loop they counted on #40078

Closed

robobun changed the title ~~test: find and fix the linux-aarch64 hang of the function tracer test~~ spawnSync: restore the VM's event loop handle as soon as the wait is over Sep 8, 2026

robobun closed this Sep 8, 2026

		// Raw return addresses as offsets into the executable, for a symbolizer,
		// in case the names above come out as `<unknown>`.

		// bun links as ET_EXEC (no PIE), so the address itself is what a
		// symbolizer wants; tag the ones inside the executable.

		// Return addresses via the frame-pointer walk the crash handler uses. bun
		// links as ET_EXEC, so an address is what a symbolizer wants as-is.

		/// Restore the original event loop handle. Idempotent: `spawnSync` calls it
		/// as soon as its wait is over, and again on every exit path.

		// Early returns before the wait; after the wait `cleanup` already ran
		// and this is a no-op.

Conversation

robobun commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Uh oh!

robobun commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

robobun commented Sep 8, 2026

Uh oh!

github-actions Bot Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

robobun commented Sep 8, 2026

Uh oh!

github-actions Bot Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

robobun commented Sep 8, 2026

Uh oh!

github-actions Bot Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

robobun commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

robobun commented Sep 6, 2026 •

edited

Loading

robobun commented Sep 6, 2026 •

edited

Loading