Skip to content

fix(process): Stop actually kills the dev server (#90) - #136

Merged
alamb-hex merged 2 commits into
mainfrom
fix/stop-kills-process-group
Aug 11, 2026
Merged

alamb-hex merged 2 commits into
mainfrom
fix/stop-kills-process-group

Conversation

@alamb-hex

Copy link
Copy Markdown
Collaborator

Closes #90 — "Stop returns success while next-server keeps running."

The bug, in four layers

  1. startProject spawned with { shell: true, detached: false }, so child.pid was the sh -c wrapper, not the real next-server.
  2. stopProject sent SIGTERM to that wrapper. POSIX sh doesn't forward signals, so the server survived and kept its port.
  3. It then returned { success: true } having verified nothing.
  4. The ss/kill -9 fallback only ran if the tracked kill threw, which it never did — dead code in the common case.

Observed live yesterday: two orphaned sh -c node / node server.js pairs held port 3000, a plain SIGTERM couldn't reach them, and a fresh start failed with EADDRINUSE.

The fix

  • detached: true so the child is a process-group leader.
  • Group kill — process.kill(-pid, …) reaches the whole tree. Windows degrades to a single-process kill.
  • Verified success — poll checkPort after signalling, escalate SIGTERM → SIGKILL, and return success: false with a clear error if the port is still bound. Reporting success while a server lives was the entire bug.
  • The port-based fallback is now a real path, including when there's no tracked entry at all — an orphan from a crash, or a server started before HexOps launched.
  • stopProject is now async; its caller and runWithDevServerGuard updated.

Also fixed along the way: stoppingProjects leaked permanently for untracked ids, suppressing the next genuine crash notification for that project.

Review found five blockers, all fixed

Two were serious enough to call out:

  • A shipped test executed process.kill(-6666, 'SIGKILL') for real. It asserted only success === false, which held whether the signal hit nothing or destroyed an unrelated process tree — so it could never fail because of it. On a box running ~35 dev servers that's a live hazard. process.kill is now mocked structurally in a shared beforeEach; verified with a tripwire config that traps the real syscall — 31 tests, zero hits.
  • A deliberate Stop was misreported as a crash. stoppingProjects was gated on "is the child live" rather than "does a tracked child exist". With shell: true the wrapper can exit while a grandchild holds the pipes and the port — the P1: Stop returns success while next-server keeps running — orphans accumulate #90 shape itself — so 'close' fired with intentional: false, and with restartOnCrash enabled it relaunched the server the user just stopped. Verified by reverting the guard in a temp copy and watching the new test fail with exactly that notification.

Plus: stopByPort now refuses process.pid/process.ppid (a project misconfigured to :3000 would have made HexOps kill its own server); runWithDevServerGuard aborts rather than running pnpm install against a still-live server (newly reachable, since a failed stop is now a real outcome); and a SIGINT/SIGTERM handler reaps tracked children, because setsid() severed the foreground group that used to do it.

Verification

399 tests / 46 files, tsc --noEmit clean, pnpm build exit 0.

Real processes, not mocks — SIGKILL escalation against a SIGTERM-ignoring server on a scratch port: stopProject returned success after 4.2s, port free, all 8 group pids dead.

Known follow-ups (issues to file)

  • M3 is ineffective in the real shape. The process-exit confirmation checks the tracked child, which under shell: true is the sh wrapper — killed instantly by the group SIGTERM, so the guard always passes. A server that closes its listener then hangs returns success: true with the port free and the process still alive. Only affects servers that trap SIGTERM (a stock Next dev server doesn't), but it's a narrower instance of P1: Stop returns success while next-server keeps running — orphans accumulate #90.
  • The SIGINT/SIGTERM handler leaks listeners across module re-evaluation. Registration is module-scoped, so an HMR reload adds another; the oldest wins and holds a stale process map, silently reaping nothing. SIGHUP is unhandled.
  • override-remove doesn't check blocked, so a failed guard-stop surfaces as success there.
  • A failed guard-stop skips the restart in finally.
  • shell: false projects are force-killed 500ms after releasing their port.

…rting success (#90)

Stop signalled only the tracked pid, which under shell:true is the sh -c
wrapper, not the real dev server. POSIX sh doesn't forward SIGTERM to its
child, so the real process (e.g. next-server) survived and kept the port
bound while stopProject returned { success: true } unconditionally. The
ss/kill -9 fallback only ran on a thrown error, which never happened, so it
was dead code.

- startProject now spawns with detached: true so the child is its own
  process-group leader; stopProject signals the whole group via
  process.kill(-pid, ...) instead of just the wrapper. Windows has no
  negative-pid group signalling, so it degrades to single-process kill there.
  ESRCH (group already gone) is treated as already-stopped, not an error.
- stopProject is now async: SIGTERM -> poll checkPort -> escalate to SIGKILL
  -> poll again -> only then decide. It returns success: false with an error
  if the port is still bound after both escalations, instead of lying.
- The ss/kill -9 port-based fallback is now a real path that always runs
  when the tracked route didn't verifiably free the port, including when
  there's no tracked entry at all (an orphan from before hexops started, or
  from an earlier crash -- today's incident).
- Fixed a latent stoppingProjects leak: it was being marked unconditionally
  even when there's no tracked ChildProcess to ever fire the 'close' event
  that clears it, which would permanently disable crash notifications for
  that projectId after an orphan stop. Now only marked right before
  signalling a live tracked entry.
- Updated the one production caller (stop route) and DevServerGuardDeps.stop
  to await the new async signature; runWithDevServerGuard was already
  async/await throughout so this was a one-line change.

Extended process-manager.test.ts with 9 new tests (mocked spawn/execFileSync/
checkPort, fake timers for the escalation windows -- no real process spawned
or port bound). Verified separately against a real process: full report in
stop-fix-report.md, including a real-process transcript showing the tracked
sh+node pair on a scratch port is fully gone and the port released after
stopProject, with hexops' own dev server on port 3000 left untouched.

382 -> 391 tests, 46 files, all passing. tsc --noEmit clean.
…C2/I1-3/M1-4)

Independent review of 4e8c550 confirmed the core group-kill + port-verified
success shape is correct, but found two critical issues and three important
ones, plus four minors. All addressed:

- C1: one test in process-manager.test.ts had no mock on process.kill, so
  after the previous test's afterEach restored all mocks, its
  process.kill(-pid, ...) calls hit the real syscall against a fabricated
  pid. Fixed structurally: process.kill is now spied in the stopProject
  describe block's own beforeEach with a safe no-op default, so no test in
  the file can reach the real syscall even if it forgets to stub it itself.

- C2: a real regression in the previous commit. stoppingProjects was only
  marked when the tracked entry was "live" (exitCode === null), but with
  shell: true a grandchild holding the stdio pipes (and the real port) can
  keep 'close' pending well after the direct child looks exited -- exactly
  the #90 shape. That case fell through unmarked, producing a false
  "crashed" notification and, with restartOnCrash on, relaunching a server
  the user had just stopped. Now marked whenever a tracked ChildProcess
  exists at all, not only when it's live.

- I1: stopByPort now refuses to kill process.pid or process.ppid (logs and
  skips instead), since it's newly reachable from a tracked stop whose group
  died but whose port is held by something else, not just the pre-existing
  fully-untracked-orphan case.

- I2: runWithDevServerGuard now aborts (blocked: true) before running the
  operation if the stop failed, instead of silently running an install
  against a server that may still be live -- unreachable before #90 since
  stop effectively always "succeeded", real now.

- I3: detached: true (the #90 fix) severs tracked children from hexops's own
  process group and session, so Ctrl-C/terminal-close no longer reaps them
  the way they did when children stayed attached. Added
  shutdownTrackedProcesses(), wired to real SIGINT/SIGTERM outside test runs
  (guarded so importing this module under vitest never installs a handler
  that could process.exit() a test worker).

- M1: close handler now guards activeProcesses.delete on process identity,
  so a late close from a superseded child can't untrack its replacement.
- M2: reverted checkPort's timeout back to its 1000ms default -- a shortened
  per-attempt timeout was a false-success path in the function meant to
  eliminate them (checkPort resolves false on its own connect timeout too).
- M3: the tracked-success path now also confirms the process itself exited
  (short grace window + a defensive final SIGKILL if not), not just the
  port, since we're already holding a live ChildProcess handle.
- M4: exported signalProcessGroup for direct testing and added ESRCH-vs-EPERM
  unit tests plus an integration-level EPERM test -- the prior ESRCH test
  only proved the overall stop succeeded via mocked checkPort, which passed
  identically whether ESRCH was actually swallowed or just caught upstream.

391 -> 399 tests, 46 files, all passing. tsc --noEmit clean. Re-verified
against a real process (same scratch-port harness as the original commit):
stopProject succeeds, port released, tracked pid and all real child pids
confirmed dead, hexops' own port-3000 server untouched throughout. Full
writeup appended to stop-fix-report.md.
@alamb-hex
alamb-hex merged commit 438118d into main Aug 11, 2026
1 check passed
@alamb-hex
alamb-hex deleted the fix/stop-kills-process-group branch August 11, 2026 13:56
alamb-hex added a commit that referenced this pull request Aug 11, 2026
A 24KB implementation working-note was committed alongside the #90 fix. It is
scratch material — review correspondence and verification transcripts — not a
repo artifact, and docs/ is where anything durable belongs.

Removing rather than relocating: the content it preserves (review findings and
their reproductions) is captured in PR #136's description and in the commit
messages themselves.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

P1: Stop returns success while next-server keeps running — orphans accumulate

1 participant