Repository navigation
fix(ci): Windows test legs take 8–15 min since #250 — aqe-store-merge.test.mjs is the critical path #262
Description
Problem
The Windows legs of ci.yml take 8–15 minutes. Ubuntu and macOS legs take 1–3 minutes. Every PR waits on Windows, so feedback is 4–5× slower than it needs to be.
This is not a Node-version effect: Node 22, 24 and 26 are all slow, and which one finishes first changes from run to run. It is a step change that arrived with #250 (the AQE store-integrity PR, merged 2026-09-28). Before #250, every Windows leg took about 3 minutes.
Evidence
When it started. The last 40 ci.yml runs, Windows job minutes as node 22 / 24 / 26:
| Run | Commit | Windows minutes (22 / 24 / 26) |
|---|---|---|
| 17 runs up to 36358405196 | up to 88e26999 (main, 2026-09-27 23:21) |
2.3–4.4 on every leg |
| 36379126905 | fc7624f6 (#250's branch, 2026-09-28 04:46) |
12.7 / 10.7 / 10.6 |
| every run since | #250 onward | 7.5–26.8, no consistent order by Node version |
The only change between the last fast run and the first slow one is #250. The time is all in the "Unit + statusline tests" step. Checkout, setup-node and pnpm setup take under a minute together.
Where it goes. Per-test durations from one run, 36498460373 (node 22), parsed from the TAP duration_ms lines:
| Source file | Tests | Windows | Ubuntu | Ratio |
|---|---|---|---|---|
tests/kit/aqe-store-merge.test.mjs |
34 | 10.9 min | 1.7 min | 6.3× |
tests/kit/provider-cli.test.mjs |
26 | 0.8 min | 0.1 min | 6.2× |
tests/kit/usage-opencode.test.mjs |
26 | 0.3 min | <0.1 min | 12× |
tests/kit/sync-command.test.mjs |
53 | 0.3 min | <0.1 min | 8.4× |
tests/kit/daemons.test.mjs |
12 | 0.2 min | <0.1 min | 47× |
| all tests | ~5,660 | 17.1 min | 3.7 min | 4.6× |
Single aqe-store-merge tests take 20–56 s each on Windows against 2–4 s on Ubuntu; the slowest is "AQE starter patterns are counted per stray…" at 55.9 s against 2.9 s. The whole file runs in 7 s on a local macOS machine.
node --test runs test files concurrently, up to CPUs − 1, which is 3 on the 4-vCPU windows-latest. The tests inside one file run one after another. So on Windows this one file is the job's critical path, and the rest of the suite finishes long before it does.
Why this file is slow on Windows (hypotheses, strongest first)
- One transaction per statement, which means one disk flush each. Each test's
project()fixture builds three stores withbuildStore(). That runs the captured AQE 3.14.4 schema (146 statements: 47 tables, 87 indexes, 11 triggers, 1 FTS5 table) one statement at a time, then oneINSERTper row, all auto-committed. On Windows every commit is aFlushFileBuffers, which is slow on the runner's disk. The code under test adds aVACUUM INTObackup,copyFileSyncrehearsal copies,integrity_check/foreign_key_checkpasses and recursivecpSyncarchives. - Microsoft Defender scans every new file. Each test creates dozens of new
memory.db,-wal,-shmand journal files under the temp folder, and real-time protection scans each on create and close. This is a well-known cause of slow Node and SQLite test suites onwindows-latest. - Coverage runs on every leg.
scripts/run-tests.mjspasses--experimental-test-coverageon all 9 matrix legs. V8 coverage writes per-process profiles, and that I/O is slower on Windows. One leg would be enough for the coverage threshold. - The rest is child-process spawning.
provider-cli,usage-opencode,sync-commandanddaemonsshow a 6–47× Windows penalty. Spawning a process costs far more on Windows. This is secondary: about 2 minutes of test time, and it runs alongside other files.
Proposed fixes, as experiments (measure each on a Windows leg, then keep what pays)
- Build the fixture schema once per file. Build one empty 3.14.4 store in a
beforehook, thencopyFileSyncit per store. Wrap eachbuildStoreinsert batch inBEGIN/COMMIT. Test-only change; no production code changes. - Split
aqe-store-merge.test.mjsinto 3–4 files by concern: preview/dry-run, holders and refusals, apply and archive, starter patterns. Alternatively, run independent tests concurrently (test(..., { concurrency })or adescribewithconcurrency), since each builds its own temp project. Either lets the runner's file-level parallelism cut the critical path. - Exclude the runner's temp and workspace folders from Defender on Windows legs only. For example, a first step:
Add-MpPreference -ExclusionPath $env:RUNNER_TEMP, $env:GITHUB_WORKSPACE, guarded byif: runner.os == 'Windows'. This is CI-only and speeds every Windows test. - Collect coverage on one leg (for example ubuntu / node 24) and run the other legs without
--experimental-test-coverage. Keep the 70/70/70 threshold on that one leg. - Optional, if the above isn't enough: on Windows, run the full suite on one Node version and a smoke or Windows-focused subset on the other two.
Acceptance criteria
- The Windows legs' median time is back under 5 minutes (it was about 3 before fix(aqe): one AQE store per project, safe stray-store merge, honest verification #250). Measure over 10 consecutive runs, and post the before/after table on this issue.
- No Windows-relevant coverage is lost.
aqe-store-merge's Windows cases, such as the EBUSY rename, still run on Windows. tests/kit/aqe-store-merge.test.mjs's assertions are unchanged, apart from being moved between files.node scripts/run-tests.mjs unitstill passes the real-state tripwire.- The 30-minute job limit (ci: stop every job after 30 minutes #261) stays as the backstop and is not raised.
Impact
Every PR waits 8–15 minutes on Windows instead of about 3, and a slow Windows run (26.8 minutes on 2026-09-28) comes close to the 30-minute limit. Faster Windows legs shorten every review cycle, and they also stop the limit being reached by a job that is only slow, not stuck.
Activity
docs(plans): remediation program v2 — everything left, decisions first (
pacphi commentedon Sep 29, 2026
V1's fixture/coverage changes landed in #267. The fixed ten-before/ten-after cohort below retains all 60 Windows observations, including failures and cancellations. All-job medians for Node 22/24/26 changed from 649/705.5/875 seconds to 221.5/225/210 seconds. The after cohort ends at run 36582013896; it includes three failed workflows and a 359-second leg. These are observational medians across evolving source and runners, not a controlled causal benchmark.
The separate three-consecutive-PR timing gate remains OPEN. Its fresh, unfiltered newest-three capture at 2026-09-29T20:27:20.685348+00:00 is:
ebce0758b3845bd3cd63087b16ef974c27d9928a09b198b6960c2dae14819d08729caa8b3794129bfa0cb7b788791a90577a2dfd9f9ca61e07d0591ePR #284 passed all 16 applicable checks, including all nine OS/Node legs, but Windows Node 24 took 311 seconds. That misses the strict under-300-second rule. No CI rerun was requested to replace this observation. This issue remains open for timing consistency; no timing waiver or universal under-five-minute claim is made.
V1's split preserved all 36 original test titles and 191 normalized assertion calls. Windows EBUSY coverage, the full matrix, the Ubuntu/Node 24 70/70/70 coverage gate, real-state tripwire and 30-minute job limit remain. Final local V7 unit/legacy/UI checks and the complete PR matrix passed. The new ESLint-based source-label guard runs as a mandatory installed-dependency quality step; it does not reduce the runtime matrix.
Cells retain seconds and conclusion for each Windows Node job. Full heads bind source;
job links retain timestamps and runner metadata. Cancelled values are partial elapsed times.
ec7497179a2a6321ce2906b4346d5720ba50cba2bd8335748fbf117147a930322d3cc032009dfce27936ca48942e4ed1b629641eb61dc0dfbcf129514d654d81dbaf59b475ff86d77e7a7ed3d10d4caa4f4f0201527952a99e4d5c165dd37c6a94cbc334174e5032e2dc36b8280941bc98593da18280e234b3761ba62f8b6253b3908a25c677a4ebbb5656632992c93c92ce1d8ea84f26283d6ccac2d146c8eee7cfe9ca4d49cb5e94fba698cd0e8376e0c829b7c959db36a9c01e0a6454fb3386747bbe908971eabb2e7efe88abd3cb2d73baf26530386549e3dc2ef19566adf2d0f0d7027bc4ea45f9f990e2b0355e989c5e563c8aa0527b1497fb64643f8277664c0ae8751224857f1301a350c8fa6e51dfc999c812068430757488d1c154611d0f857d6c32db73e6fcca00a6faaa2622396309a231ef794628c856fc1071f440c1ec2e055f633438ea329ce629156439b3ba087a67154dccba79399900545574c72c86815041886b565dd5304f485af4c079c0c7c06aa83a13cded8f4cb96836e1911aae9a89bbb3f015f033e115Source-bound Windows receipt retains cohort definitions and limits; its 19:20 snapshot is historical and the newer table above is the current closeout observation. Confirmed execution plan keeps missing performance evidence visible while the final develop → main PR remains human-reviewed.
pacphi commentedon Oct 1, 2026
Card
A61-06: v4 GA programThis issue joined the v4.0.0 board. Its original text is unchanged.
Why
Windows legs reach 311–359 s in single runs (#262).
In scope
Measure the schema-batch candidate (PR #292); apply the rule approved on 2026-09-29; close #262.
Out of scope
New test-speed work beyond the candidate.
Acceptance criteria
Dependencies
Blocked by: None
Blocks: #320
Plan
4.0.0-alpha.61size:M· Workstream:ws:remediationA61-06in the card inventoryVerification
Generated by Claude Code
test(aqe): batch schema writes in the migration oracle (#292)
pacphi commentedon Oct 2, 2026
Closing on the rule approved on 2026-09-29, with no waiver needed. The maintainer chose option A on 2026-10-02: merge #292, then count the next PR runs.
The rule, and where it stands
ciruns, the median of each run's slowest Windows leg is 265.5 s (4.4 min), and the median over all 30 legs is 237.5 s.f42a681, 2026-10-02 18:19Z) the next three PR runs were 37056115649, 37056562309 and 37057900414, with slowest legs of 233, 241 and 265 s. The push-to-mainruns in between were also all under 300 s (the slowest leg was 266 s).Run table. The newest 14 successful
ciruns, in seconds per Windows leg (job start to finish), newest first. All 14 are first attempts. Runs marked * are over 300 s.What this does and does not show
Generated by Claude Code