Repository navigation
fix(core): recover confirmed subagent calls after provider stream close - #437
Conversation
|
🔍 Automated Cursor review is underway. Please hold off on merging until the review comment lands here (usually 10–20 minutes). |
|
Review complete: one medium finding and one low finding. Prior review fixes that do hold are noted at the bottom. Tests were not re-run here (bun is not installed and dependencies are not present). MediumForeground child sessions with an in-flight generic tool are resumed, so the model can retry an ambiguous effect. On the real runner, that resume hits The process-death fixture replaces Suggested fix: before Low
Suggested fix: pass Prior fixes that hold
|
…onfirmed subagent recovery after process death > reconciles the exhausted boundary…` in packages/core/test/subagent-recovery-process.test.ts. The recover fixture hit its own 5s `Effect.timeout`, logged no recovery error, and the test's stderr assertion caught the TimeoutError. **Root cause (a race in the test harness, not in production code):** - The test SIGKILLs the seed process as soon as the `ready` file exists. - The recover stage then read its phase from that file's contents. - `Bun.write` creates and truncates the file first, then writes the content. Under CI load the kill can land between those two steps, leaving `ready` empty. - With an empty phase, the recover stage took the "wait for completed" path instead of the "wait for failed" path, and timed out. - `exhausted` was just the test that got hit. The earlier unit failure in this run was a different flaky test (PowerShell ShellTool), so this one is intermittent. **Reproduced:** with the HEAD fixture, I seeded `exhausted`, emptied `ready` and ran recover. It produced the same TimeoutError as CI. **Fix (test-only, 3 lines):** - `test/fixture/subagent-recovery-process.ts`: both stages now take the phase from argv, and `ready` is only a presence signal. A short comment explains why. - `test/subagent-recovery-process.test.ts`: passes `phase` to the recover command. **Verified:** - All 17 tests in subagent-recovery-process.test.ts pass. - `bun typecheck` in packages/core is clean. - `git diff --check` and Prettier are clean
…ailing test is not one this PR adds or touches. **What failed:** one test in `unit`: `ShellTool ordinary shell syntax > PowerShell legacy: ordinary syntax reuses approvals: Write-Output "$(Write-Output hello)"` in `packages/core/test/tool-shell.test.ts`. It died at 10087ms with an Effect `TimeoutError`. This PowerShell test also failed in an earlier CI run on this PR. Recent `test` runs on `integration-v2` pass. Core tests run one at a time, so this PR's process-death tests do not run alongside it. **Root cause:** a timeout conflict in the test. - The PowerShell tests were given a 15s per-test timeout because PowerShell starts slowly on CI. - But the shared helper `runPermissionCommand` wraps every call in its own 10s `Effect.timeout`, so that 10s limit always hit first. - The first PowerShell test in the run pays the cold-start cost, and on a loaded runner that took just over 10s. **Fix (3 lines, test only):** - Removed the helper's own 10s timeout, so the per-test timeout decides. For the `sh` tests this changes nothing, because Bun's 5s default already fired first. - Raised the PowerShell tests' timeout from 15s to 30s and updated the comment. **Verified locally:** - All 108 tests in `tool-shell.test.ts` pass with `pwsh` installed, including the PowerShell tests. - `bun typecheck` in `packages/core` is clean. - `git diff --check` is clean. - Prettier `--check` flags this file, but it flags the unchanged HEAD version too, so that is not from this change. I could not reproduce the CI timeout locally, because PowerShell starts fast here. So the 30s figure is a judgment about CI load, not a measured value
|
Cursor Medium in #437 (comment) is fixed by |
|
Cursor Low in #437 (comment) is fixed by |
Intent
Recover interrupted streaming tool calls after provider WebSocket closes in shuvcode issue #407. Durable identity design and fault injection must precede automatic replay: never execute unfinished or half-received calls, and prove no duplicate child or effect. Recover only provider-confirmed subagent calls with a stable durable operation identity and one child, admission, and canonical result; rejoin admitted, running, or completed children for the same operation without another prompt or child. Preserve cancellation and terminal distinctions and bound restart attempts across process death. Never automatically replay ambiguous generic local side effects. Apply the new path only to WebSocket read/close codes 1006 and 1012, leaving 1009 fallback, other errors, permissions, and user cancellation behavior unchanged. Use isolated real Session/tool-registry fixtures and process-death fault injection around identity, admission, and result boundaries; no live service or broad replay promise. The approved follow-up corrections require foreground prepared children and nested subagents deeper than one to stop ambiguous generic effects before any real runner drain, and require execution registration before prepareResume and recoverPreparedCall so a wake in that gap cannot drop recovery. Cover both with behavioral real-runner regressions. Keep prior corrections and PR #437; the main lead alone owns merge.
What Changed
1006or1012, tool calls that were still streaming now fail withrecovery: "unconfirmed"metadata. They stay in the durable transcript but are left out of regenerated model requests. Other close codes and other errors keep their existing behavior.session.tool.subagent.preparedevent and asubagent_operationtable (new migration). Together they record a stable identity for each provider-confirmed subagent call: child Session ID, inbox ID, input digest, agent, and model. The subagent tool reuses this identity, so a recovered call rejoins or reads back the existing child (admitted, running, or completed) instead of sending another prompt or creating a second child.session/execution/restart.tsnow recovers prepared subagent calls under the parent's execution ownership.SessionExecution.resumeand the run coordinator take apreparestep that runs after the execution is registered and before the first drain, so a wake during recovery can't drop it. Ambiguous generic tool calls, including those in foreground and nested children, fail and stop the claimed execution before any drain instead of being replayed. Cancelled and failed child outcomes stay distinct. The session spec and the generated client types are updated to match.Risk Assessment
Testing
I drove 13 live runs, covering all 7 intent scenarios, against realshuvcode serveprocesses from the candidate worktree. Each run had an isolated HOME/XDG/DB, used a local fake OpenAI Responses provider over HTTP SSE and WebSocket, and was killed with SIGKILL. Persisted SQLite state confirmed every scenario: - Foreground and background subagent calls recover with exactly one child, one admission and one result. A cancelled child stays cancelled. - Ambiguous shell effects and half-received calls run once and fail closed. - Restarts are capped at 10 attempts, then the parent fails. - At depth 2, the background child and its grandchild stop before any drain. - A prompt sent the moment the restarted server answers produces no dangling request. - After a 1006 or 1012 close, the model request is regenerated without the draft and keeps the completed sibling. A 1009 close keeps its old fallback and error behaviour. The targeted process-death and runner regression tests also pass. The only evidence is JSON/Markdown transcripts and persisted-state dumps; there are no screenshots because this change has no UI.Evidence: Live run summary (candidate 61651d7)
Evidence: Live harness (real shuvcode serve + fake provider + SIGKILL)
Evidence: Live scenarios script
~/.no-mistakes/evidence/01M47YT458YZH24QDN1TT5N808/live/run2/batch1.out)~/.no-mistakes/evidence/01M47YT458YZH24QDN1TT5N808/live/run2/batch2.out)~/.no-mistakes/evidence/01M47YT458YZH24QDN1TT5N808/live/README.md)Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 1 issue found → auto-fixed ✅
packages/core/src/session/execution/restart.ts:456- A recovered background child can still drain over a nested subagent whose generic effect is ambiguous. The newstopStaleChildonly runs fromrecoverPreparedCall, which only sees top-level parents. A background child with a pending job takes the firstsuspendedloop instead. Example, withexperimental.subagent_depth> 1: parent P starts background child B. B calls a foreground subagent G (a prepared call). G is mid-echo/shell when the process dies. On restart, B's only running call is prepared, so B is not marked ambiguous. G goes intopreparedChildren(its claim is kept, but nothing ever recovers it). ThenrecoverSubagentcallsprepareResume(B)andexecution.resume(B). B's drain runssettleStaleToolCalls, which fails B's subagent call as a plainTool execution interrupted. B's model can then call subagent again and create a second grandchild that repeats G's possibly completed effect. G's claim also stays held forever. This breaks the intent requirement that "nested subagents deeper than one … stop ambiguous generic effects before any real runner drain", and the requirement to prove no duplicate child or effect. Fix: in the first loop, for each prepared call of a recovered background child, callstopStaleChild(operation.childSessionID)(orstopStaleChild(sessionID)on the child). If anything was stopped, fail the call and add the child toambiguousbeforerecoverSubagentruns. This reuses the helper the commit already adds.🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 **Test** - 1 issue found → auto-fixed ✅
bun test test/subagent-recovery-process.test.ts(packages/core): 17 SIGKILL process-death boundary phases, all passbun test test/session-runner.test.ts -t "recovered background child|restart preparation owned|without replaying an unconfirmed|settled sibling|prepared|recover|nested": 22 real-SessionRunnerLLM recovery scenarios, all passbun test test/session-model-transport.test.ts test/tool-subagent.test.ts: 50 pass, covering close codes 1006/1012, the unchanged 1009 HTTP fallback, and reuse of a reserved subagent operationA/B check: restoredrestart.tsfrom HEAD~1 and ran-t "stops nested ambiguous effects before a recovered background child drains". It fails with 2 LLM requests where 1 is expected. Then restored HEADManual run: for the admitted, prompted, background-running, ambiguous and exhausted phases, ran the fixture's seed stage, SIGKILLed it at the ready boundary, then ran the recover stage and recorded its JSON counts🔧 Fix applied.
✅ Re-checked - no issues remain.
bun run scenarios.ts fg-running fg-result bg-running bg-cancelled ambiguous unprepared(live harness: realshuvcode serve --servicefrom worktree, isolated HOME/XDG, fake local Responses provider, SIGKILL)bun run scenarios.ts nested wake-gap ws-1012 ws-1006 ws-1009 ws-reject1009 exhausted(same live harness)Inspected persisted session_v2 / event / session_message / subagent_operation rows after each kill and restartbun test test/subagent-recovery-process.test.ts test/tool-subagent.test.ts test/session-model-transport.test.ts(packages/core, 68 pass)bun test test/session-runner.test.ts -t "recover|prepared|nested|background child|wake|unconfirmed|1006|1012"(packages/core, 22 pass)✅ **Document** - passed
✅ No issues found.
packages/core/test/tool-shell.test.ts:1011- Prettier flags this file, but the base commit 28350a1 fails the same check, and none of the flagged hunks (lines ~1011, ~1377, ~1406) are lines this change touched. I left them alone to keep the change scoped and to avoid editing tests. A follow-up formatting-only commit can fix them.ast-grepbinary is not installed in this environment, solint:effect-patternscould not run. oxlint found 0 issues on the changed TS files,bun typecheckin packages/core is clean, andgit diff --checkis clean.✅ **Push** - passed
✅ No issues found.