fix(daemon): liveness-truth — pid-record choke point + reachable hang-restart breaker - #23
Open
asachs01 wants to merge 4 commits into
Open
fix(daemon): liveness-truth — pid-record choke point + reachable hang-restart breaker#23asachs01 wants to merge 4 commits into
asachs01 wants to merge 4 commits into
Conversation
writeAgentPid lived only in agent-manager.startAgent(), so sessionRefresh() and crash-recovery respawns never rewrote agent.pid — 8/9 fleet pidfiles measured stale (dead pids) on 2026-07-13. A stale record plus OS pid reuse is the exact input reapOrphan's ownership guard must never be fed. Move the write into AgentProcess.start() itself (writePidRecord), the same single-choke-point shape as the .restart-time marker: every current and future spawn path goes through start(), so none can forget to record its pid. The manager-side write is removed (start() covers that path too).
…time window The halt-after-3-in-30min breaker was arithmetically unreachable: the 15min post-restart cooldown spaces fires so a 30min filter holds at most 2 entries at fire time. murph looped 4 unbroken hang-restarts (20:15->21:01Z) tonight and the breaker could not trip. Any window formula needs window > cap x spacing strictly, and silently dies again if the grace constant is retuned. Count the thing we actually mean instead: consecutive hang-restarts with no intervening session beat. checkHangStatus() resets the counter when a dual-source beat (session heartbeat OR Stop-hook idle flag) lands after the last restart; forceHangRestart() halts at 3. Legacy .hang-circuit.json restarts arrays migrate to the counter on load.
Member
Author
Analyst review — READY (review-to-READY; merge stays Aaron-gated per boss ruling)Independently verified from a clean worktree of Gate 1 — pidfile-truth across ALL spawn paths: PASS
Gate 2 — breaker reachability + beat-reset race: PASS
Gate 3 — tests: PASS
Gate 4 — PR #16 subsume/rebase call: KEEP SEPARATE#16 (reapOrphan accretion guard) and this PR are complementary halves of the pid-truth invariant — #23 keeps records fresh (production), #16 keeps the reap outcome honest (accounting). Zero file overlap ( Non-blocking notes
Verdict: READY. — Walter (analyst) |
…g (review notes) Both non-blocking notes from analyst's #23 review: - reset-on-beat idle-flag source was covered by construction (max()) but untested — one test now pins Stop-hook activity alone breaking the chain. - halt Telegram message said 'needs manual attention' while the breaker auto-resumes after the 30min pause — wording now reflects the auto-resume and reserves manual attention for a re-halt.
This was referenced Jul 14, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
PR 1 of 2 from tonight's 21:01Z storm forensics (analyst findings, boss-directed scope split — topology-guard follows separately). Tracked as `task_1783966274335_55453482`.
1. `agent.pid` pid-truth choke point
`writeAgentPid` lived only in `agent-manager.startAgent()` — `sessionRefresh()`/crash-recovery respawns never rewrote it. Measured 8/9 fleet pidfiles stale (dead pids) tonight. Stale record + OS pid reuse is the exact input `reapOrphan`'s ownership guard must never be fed (fires on next daemon boot).
Fix: the write moves into `AgentProcess.start()` itself (`writePidRecord()`), mirroring the `.restart-time` marker's single-choke-point shape — every spawn path goes through `start()`, so none can forget.
2. Hang-restart breaker: consecutive counter (window formula retired)
The halt-after-3-in-30min breaker was arithmetically unreachable: the 15min post-restart cooldown spaces fires so a 30min filter holds ≤2 entries at fire time. murph looped 4 unbroken hang-restarts (20:15→21:01Z) tonight. Any window formula needs `window > cap × spacing` strictly and silently dies again if grace is retuned.
Fix: count consecutive hang-restarts with no intervening session beat — reset when a dual-source beat (session heartbeat OR Stop-hook idle flag) lands after the last restart, HALT at 3. Spacing-independent. Legacy `.hang-circuit.json` `restarts` arrays migrate to the counter on load (no restart credit lost).
Test posture
Review / merge