mac CI: ServeConnectionTests is timing-flaky under load (two different tests in two days)

[ozymandiashh]

Two unrelated PRs, two different tests in the same suite, both pure timing failures on macos-latest, both branches untouched ServeConnection:

date	PR	run	test	failure
2026-09-11	#1290 (menubar second row)	34626755506	a request queued during cancelled hydration completes on the same child (ServeConnectionTests.swift:263)	expected events [..., "second-replied"], got the array without the last event
2026-09-12	#1289 (zh-Hans localization)	34725149445	cancelling a hung request returns promptly (ServeConnectionTests.swift:182, :185)	error → ServeUnavailable() where CancellationError was expected, and elapsed → 1.08 s against a 500 ms budget

Both re-runs of the same head went green, and neither branch touches mac/Sources/CodeBurnMenubar/Data/ServeConnection*. The second failure is the informative one: a 500 ms wall-clock budget on a shared runner is not a property of the code, and the cancellation arriving as ServeUnavailable suggests the child was torn down by the timeout path before the cancellation propagated, i.e. the two races the suite is supposed to distinguish collapse under load.

Suggested fix, for whoever owns ServeConnection: drive these tests with an injected clock / continuation instead of real sleeps, or at least widen the wall-clock assertions to something a busy runner can meet (the neighbouring cli-status-menubar vitest flake at the 30 s boundary is the same class, see #1216's history). Filing so the next red run on this suite is recognised as this rather than chased.

[ozymandiashh]

Third data point, same head as the second row above, re-run of the failed job only (job 103638897740): cancelling a hung request returns promptly again, this time only the wall-clock assertion, elapsed → 0.516 s against .milliseconds(500). A 16 ms overshoot on a shared runner is the whole story; the code path is fine.

One aggravating factor worth knowing when reading these: swift-testing runs suites concurrently in one process, so any CPU-heavy suite that happens to be scheduled alongside this one eats into its 500 ms. #1289 adds a source-tree scan test that was measured at several seconds; it is being made cheap on that branch, but the assertion will stay fragile until it is driven by an injected clock rather than real time.

[ozymandiashh]

Proposed mitigation up as #1340. The localization coverage suite that runs concurrently with ServeConnectionTests was spending 17 to 22 s of CPU per run in the debug build CI uses: 26 passes per file, an array allocation per character, and a quadratic enclosing-type lookup, repeated across four tests. It now does one byte-level pass per file with a cached result: about 0.2 s of CPU, detections identical to the old scanner on every source file plus mutated copies (313 findings, 1,406 keys), and a scanStaysCheap test that fails above 2 s of scanning-thread CPU. That removes the heaviest neighbour of the 500 ms assertion; it does not change ServeConnection itself, so this issue stays open for an injected clock there.

[ozymandiashh]

Confirmed on main itself, with no PR involved: the push run for 5498818 (merge of #1330), macOS Menubar CI run 34725673173 at 2026-09-12T23:35Z, failed a request queued during cancelled hydration completes on the same child (ServeConnectionTests.swift:263) with the same shape as the 2026-09-11 row above: events → ["first-read", "late-first", "second-read"], missing "second-replied". So both tests in this suite reproduce on main; the PR runs were never the cause. Main's mac badge is red for this reason alone.

[ozymandiashh]

It is getting more frequent, and a third test in the suite now fails the same way. Between 2026-09-13T00:05Z and 00:21Z, four of the macOS Menubar CI runs on open PRs failed only in ServeConnectionTests, including a re-run of an already failed attempt:

run	attempt	failures
34727083896 (#1327)	1	cancelling a hung request returns promptly, elapsed 1.50 s vs 500 ms
34727225482 (#1341)	1	same test, elapsed 2.16 s
34727225482 (#1341)	2	same test, ServeUnavailable instead of CancellationError plus elapsed 1.10 s; new: external cancellations keep one child and safely discard late replies (ServeConnectionTests.swift:559, Bool(false))
#1343 macOS run, created 00:16Z	1	cancelling a hung request returns promptly (1.73 s) and a request queued during cancelled hydration completes on the same child (:263) in the same run

None of those branches touches ServeConnection. Three distinct tests, all asserting orderings or wall-clock bounds across a cancellation race, all passing on re-runs, is the signature of real-time sleeps and timeouts inside the tests rather than a regression. Until they are driven by an injected clock or continuations, every macOS PR needs one to two re-runs to go green.

[ozymandiashh]

Fix up as #1345, test file only, no production change. The three tests now enforce ordering through continuations and explicit gates instead of real sleeps and wall-clock budgets; the ServeUnavailable-instead-of-CancellationError symptom was the test's own timeout racing the cancellation, not the production timeout path.

Local stress, 200 iterations per test per variant, two load profiles (A: 16 CPU-burning neighbours; B: unloaded):

test	original A	original B	fixed A	fixed B
cancelling a hung request returns promptly	200/200 failed (:185)	0	0	0
a request queued during cancelled hydration completes on the same child	0	3/200 failed (:263)	0	0
external cancellations keep one child and safely discard late replies	43/200 failed (:559)	0	0	0

All three CI failures reproduce on the original under one of the profiles, which matches what CI showed (the budget and late-reply races under a starved scheduler, the queued-hydration ordering race even without load); the fixed suite had 0 failures in 1,200 runs. swift test cannot run on the machine that produced these, so the PR's own macOS CI run is the final gate.