Skip to content

fs.promises.watch: bound the event queue when maxQueue or overflow is passed - #46937

Draft
robobun wants to merge 1 commit into
claude/e3fb6844/fs-promises-watch-async-generatorfrom
robobun/8f06fa9a/fs-promises-watch-maxqueue
Draft

robobun wants to merge 1 commit into
claude/e3fb6844/fs-promises-watch-async-generatorfrom
robobun/8f06fa9a/fs-promises-watch-maxqueue

Conversation

@robobun

@robobun robobun commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • fs.promises.watch() reads neither maxQueue nor overflow: overflow: 'error' never rejects with ERR_FS_WATCH_QUEUE_OVERFLOW, and bad values pass.
  • The listener pushes every event with no size check (src/js/node/fs.promises.ts:90 on the base).

Fix

  • Stacked on fs.promises.watch: implement as a real async generator #34718 (one commit on 1164ab606a).
  • With either option the listener keeps at most maxQueue unread events (default 2048). After that, 'ignore' drops events with a warning and 'error' rejects next(). The first next() validates both.
  • With neither option the queue stays unbounded. Node.js's default drops events that callers get today, so a maintainer decides it.
  • Verified: test/js/node/watch/fs.watch.test.ts (12 new tests, 9 fail on the base), test/js/node/watch/, 38 Node.js watch tests. Self-reviewed: 15 concerns raised, 14 addressed.

Background

Downsides

  • A maxQueue below 8 can overflow on POSIX when the consumer reads every event (10 of 10 bursts at 1, 4, 7). The native watcher calls the listener up to 8 times per task (fs.watch: run nextTicks and microtasks between the events of one batch #43715).
  • Bad values now reject, also overflow: 'throw' from @types/node. With recursive: true off macOS and Windows, Node.js reads neither option. Bun applies both.
  • Callers with neither option pay 2 property reads per watch() call. Release binary: +1,031 B.
Notes

Repro (Linux x64)

import fs from "node:fs"; import fsp from "node:fs/promises"; import path from "node:path"; import os from "node:os";
const d = fs.mkdtempSync(path.join(os.tmpdir(), "pwmq-"));
const it = fsp.watch(d, { maxQueue: 16, overflow: "error" })[Symbol.asyncIterator]();
const armed = it.next(); await new Promise(r => setTimeout(r, 300));
for (let i = 0; i < 200; i++) fs.writeFileSync(path.join(d, "f" + i), "");
await new Promise(r => setTimeout(r, 2000)); // nothing reads: the queue must overflow
const got = new Set(); let thrown = null;
try { const r1 = await armed; if (r1.value) got.add(r1.value.filename);
  for (;;) { const nx = await Promise.race([it.next(), new Promise(r => setTimeout(() => r("Q"), 800))]);
    if (nx === "Q" || nx.done) break; got.add(nx.value.filename); }
} catch (e) { thrown = e.code || e.name; }
console.log(JSON.stringify({ distinctDelivered: got.size, thrown })); process.exit(0);
output
main 179664ea89, #34718 1164ab606a {"distinctDelivered":200,"thrown":null}
this PR {"distinctDelivered":1,"thrown":"ERR_FS_WATCH_QUEUE_OVERFLOW"}
Node.js v26.3.0 {"distinctDelivered":1,"thrown":"ERR_FS_WATCH_QUEUE_OVERFLOW"}

Where this PR equals Node.js v26.3.0

  • Validation: 17 rows of maxQueue and overflow values give the same code and message, in the same order, at the first next().
  • The six fixtures of the new tests that do not say "Bun differs" print the same text under node.
  • After 10,000 unread events with maxQueue: 16: 16 kept (plus the one that the waiting next() took), 9,983 warnings.
  • test/js/node/test/parallel/test-fs-promises-watch.js is now the upstream file with no change. It does not end on main or on fs.promises.watch: implement as a real async generator #34718. Only its three validation cases need this PR. Node.js has no test for the bound.

Where this PR differs from Node.js

Case Node.js v26.3.0 this PR
Neither option, 10,000 unread events keeps 2,048, warns 7,951 times keeps 10,000, no warning (as main)
recursive: true on Linux with a bad maxQueue or overflow no error rejects the first next()
recursive: true on Linux at the bound no bound: its JS watcher has no queue the bound applies
maxQueue 1, 2, 4, 7, a consumer that reads every event, a burst of 50 mkdirSync, overflow: 'error' 50 of 50 events, 10 of 10 runs overflows in 10, 7, 10 and 10 of 10 runs
maxQueue 8, 16, 2048, the same burst 50 of 50 50 of 50
{ persistent: "x", maxQueue: "y" } reports options.persistent reports options.maxQueue (test.todo)
A queued overflow error, then an abort, then next() rejects with the overflow error rejects with AbortError

Why there is no microtask checkpoint at the bound

Costs, measured

Release builds (ThinLTO) of the base 1164ab606a and of this PR, Linux x64. perf, strace, valgrind and bloaty are not installed. The numbers come from a perf_event_open counter (instructions:u, main thread), size -A and bun:jsc heapStats.

base this PR
size -A .bun_builtins +1,031 B, no other section changes
Host functions 0 added
Instructions per delivered event, neither option, JIT (slope 100,000 to 200,000 events, 10 paired runs) 9,571 9,654 (delta median +68, min -134, max +371)
The same with BUN_JSC_useJIT=0 (8 runs) 16,583 16,934 (delta median +170, min -895, max +1,539)
Instructions per delivered event with maxQueue passed, JIT (8 runs, this PR against its own default listener) delta median -111, min -305, max +521
Instructions per watch() call, neither option, JIT (10 runs) 8,852 8,980 (delta median +128, min -191, max +439)
The same with BUN_JSC_useJIT=0 (6 runs) 20,143 21,128 (delta median +985)
Instructions per watch() call with both options, JIT 8,889 9,415 (delta median +515)
GC cells per delivered event 12.5 12.4
GC cells per watch() call, neither option 18.56 18.58
GC cells per watch() call, both options 18.57 19.58 (the array for validateOneOf)
  • The listener for callers that pass neither option is the listener of fs.promises.watch: implement as a real async generator #34718, with no change. The per-event deltas in that row are run-to-run noise.
  • The native queue between the watcher thread and the JS thread has no bound (FSWatchTaskPosix::enqueue). This PR bounds the JS queue only.
  • At the bound, Bun's record for events that the kernel lost ('change' with a null filename) is dropped like every other event.

Tests

  • Each clause of the change has a test that fails without it: 20 mutants of the built module, 20 killed.
  • 9 of the 12 new tests fail on the base. The 3 that pass there check what must not change: a watcher error is not counted (2 tests), and neither option keeps every event.
  • A fixture that waits for something that does not come prints hung and ends after 4 s.
  • Not run: macOS and Windows. the queue keeps the first maxQueue unread events and the watcher-error tests are Linux only.
  • test/js/bun/module-graph/module-graph-isolation.test.ts uses fs.promises.watch. Its 5 fsPromisesWatch tests pass alone. In a full debug run of that file, 3 or 4 tests time out on this machine, with main's fs.promises.ts too.

Self-review

Related


no test proof · iteration 4 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/node/watch/fs.watch.test.ts

… passed

The iterator queued every event and read neither option. A consumer
that did not read kept every event, and overflow: 'error' never fired.

When one of the two options is passed, the listener keeps at most
maxQueue unread events (2048 when only overflow is passed). At the
bound, overflow 'ignore' drops the event and emits the warning
'fs.watch maxQueue exceeded'. Overflow 'error' replaces the queue with
ERR_FS_WATCH_QUEUE_OVERFLOW, and the next next() rejects with it. An
error from the watcher then closes it. The first next() validates both
options before the signal.

The values and the messages are those of Node.js v26.3.0. Two things
differ from Node.js. With neither option the queue has no bound, as
before. A recursive watch applies both options on every platform.

test-fs-promises-watch.js is now the upstream file with no change.
@robobun

robobun commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 6:09 PM PT - Oct 25th, 2026

✅ @robobun, your commit 35a0f2ec26cd5d8e01a264620e276373f7e73c90 passed in Build #132836! 🎉


🧪   To try this PR locally:

bunx bun-pr 46937

That installs a local version of the PR into your bun-46937 executable, so you can run:

bun-46937 --bun

@robobun

robobun commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Status: reproduced with the script under Notes in the PR body. main 179664ea89 and the base #34718 (1164ab606a) print {"distinctDelivered":200,"thrown":null}. This PR and Node.js v26.3.0 print {"distinctDelivered":1,"thrown":"ERR_FS_WATCH_QUEUE_OVERFLOW"}.

PR: #46937

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants