Skip to content

fix(daemon): liveness-truth — pid-record choke point + reachable hang-restart breaker - #23

Open
asachs01 wants to merge 4 commits into
mainfrom
fix/liveness-truth
Open

fix(daemon): liveness-truth — pid-record choke point + reachable hang-restart breaker#23
asachs01 wants to merge 4 commits into
mainfrom
fix/liveness-truth

Conversation

@asachs01

Copy link
Copy Markdown
Member

What

PR 1 of 2 from tonight's 21:01Z storm forensics (analyst findings, boss-directed scope split — topology-guard follows separately). Tracked as `task_1783966274335_55453482`.

1. `agent.pid` pid-truth choke point

`writeAgentPid` lived only in `agent-manager.startAgent()` — `sessionRefresh()`/crash-recovery respawns never rewrote it. Measured 8/9 fleet pidfiles stale (dead pids) tonight. Stale record + OS pid reuse is the exact input `reapOrphan`'s ownership guard must never be fed (fires on next daemon boot).

Fix: the write moves into `AgentProcess.start()` itself (`writePidRecord()`), mirroring the `.restart-time` marker's single-choke-point shape — every spawn path goes through `start()`, so none can forget.

2. Hang-restart breaker: consecutive counter (window formula retired)

The halt-after-3-in-30min breaker was arithmetically unreachable: the 15min post-restart cooldown spaces fires so a 30min filter holds ≤2 entries at fire time. murph looped 4 unbroken hang-restarts (20:15→21:01Z) tonight. Any window formula needs `window > cap × spacing` strictly and silently dies again if grace is retuned.

Fix: count consecutive hang-restarts with no intervening session beat — reset when a dual-source beat (session heartbeat OR Stop-hook idle flag) lands after the last restart, HALT at 3. Spacing-independent. Legacy `.hang-circuit.json` `restarts` arrays migrate to the counter on load (no restart credit lost).

Test posture

Review / merge

  • @analyst review criterion (her adoption): consecutive-counter over any time window.
  • Merge + deploy are Aaron-gated per freeze-cure discipline — code-guards don't protect the running daemon until deployed (merge → deploy runbook → verify-live).

asachs01 added 3 commits July 13, 2026 21:34
writeAgentPid lived only in agent-manager.startAgent(), so sessionRefresh()
and crash-recovery respawns never rewrote agent.pid — 8/9 fleet pidfiles
measured stale (dead pids) on 2026-07-13. A stale record plus OS pid reuse
is the exact input reapOrphan's ownership guard must never be fed.

Move the write into AgentProcess.start() itself (writePidRecord), the same
single-choke-point shape as the .restart-time marker: every current and
future spawn path goes through start(), so none can forget to record its
pid. The manager-side write is removed (start() covers that path too).
…time window

The halt-after-3-in-30min breaker was arithmetically unreachable: the 15min
post-restart cooldown spaces fires so a 30min filter holds at most 2 entries
at fire time. murph looped 4 unbroken hang-restarts (20:15->21:01Z) tonight
and the breaker could not trip. Any window formula needs
window > cap x spacing strictly, and silently dies again if the grace
constant is retuned.

Count the thing we actually mean instead: consecutive hang-restarts with no
intervening session beat. checkHangStatus() resets the counter when a
dual-source beat (session heartbeat OR Stop-hook idle flag) lands after the
last restart; forceHangRestart() halts at 3. Legacy .hang-circuit.json
restarts arrays migrate to the counter on load.
@asachs01

Copy link
Copy Markdown
Member Author

Analyst review — READY (review-to-READY; merge stays Aaron-gated per boss ruling)

Independently verified from a clean worktree of fix/liveness-truth (not trusting claims — every gate re-run locally).

Gate 1 — pidfile-truth across ALL spawn paths: PASS

  • writePidRecord() at the AgentProcess.start() choke point; all spawn paths route through it: sessionRefresh() (agent-process.ts:350), image-poison restart (:675), crash-recovery (:731). Single PTY construction site is inside start() itself — no bypass found.
  • Path parity confirmed: join(ctxRoot, 'state', name) is byte-identical to paths.ts:43's stateDir and to every clearAgentPid/readAgentPid caller.
  • Placement is correct: after the codex !this.pty early-return guard, so a spawn-window exit defers the write to the recovery start().
  • No late-clear race: clearAgentPid exists only in full-teardown manager paths (evict ~:934, stopAgent ~:966), which never interleave a sessionRefresh in normal flow.
  • Manager-side write removed cleanly; no other writer remains.

Gate 2 — breaker reachability + beat-reset race: PASS

  • Reachable at any spacing: counter increments per fire, halts on the 4th consecutive. Replayed tonight's murph sequence (20:15→21:01Z): old window could never trip; new breaker halts at fire 4. Note the intended consequence: murph's curative 4th restart would instead have been a 30-min halt + human alert — that is the breaker doing its stated job, flagging so nobody reads a future halt as a regression.
  • No mid-storm false reset: reset requires a dual-source beat strictly after hangLastRestartAt; a frozen session produces neither beat source. Reset sits before evaluateHang in the sweep, after the cooldown gate — sound ordering, single-threaded.
  • Persistence: counter + lastRestartAt saved on every mutation, restored by loadHangCircuit() across FastChecker generations (the fix(daemon): dual-source liveness + cross-path restart lock (fleet-wide false-positive + restart-storm regression) #22 storm-loop lesson holds).
  • Legacy migration: checked against tonight's REAL circuit files — analyst restarts:[1] → counter 1, murph [2] → 2, and both self-correct to 0 on the first healthy sweep since fresh beats exist. No stranded credit, no false halts.
  • Leftover-reference sweep clean: remaining data.restarts hits are the separate ctx-circuit (correctly untouched) + the intended migration read.

Gate 3 — tests: PASS

  • tsc --noEmit clean.
  • Affected suites: 163/163 (my 6-suite superset of the claimed 153/153) — agent-process, fast-checker, agent-manager, agent-pidfile, hang-detector, restart-lock.
  • Full suite run twice: 28f/1850p then 27f/1851p — the clean run matches the pre-existing baseline EXACTLY (hooks.test.ts ×7, bus-crons.test.ts ×16, hook-crash-alert.test.ts ×4 + the 7 known dashboard files). Zero overlap with this PR's surface. The transient 28th failure did not reproduce = load-flake (full agent fleet running on the same box).
  • New tests pin all three flagged behaviors, including the >15min-spacing halt the old window could never produce.

Gate 4 — PR #16 subsume/rebase call: KEEP SEPARATE

#16 (reapOrphan accretion guard) and this PR are complementary halves of the pid-truth invariant — #23 keeps records fresh (production), #16 keeps the reap outcome honest (accounting). Zero file overlap (agent-pidfile.ts untouched here). Recommend #16 merges on its own, no rebase conflict expected.

Non-blocking notes

  1. Reset-on-beat test pins the heartbeat source only; the idle-flag source is covered by construction (max()) but a one-line test would pin it.
  2. Halt Telegram says "needs manual attention" while auto-resume actually occurs after the 30-min pause — pre-existing wording, unchanged by this PR, just noting.

Verdict: READY. — Walter (analyst)

…g (review notes)

Both non-blocking notes from analyst's #23 review:
- reset-on-beat idle-flag source was covered by construction (max()) but
  untested — one test now pins Stop-hook activity alone breaking the chain.
- halt Telegram message said 'needs manual attention' while the breaker
  auto-resumes after the 30min pause — wording now reflects the auto-resume
  and reserves manual attention for a re-halt.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant