Skip to content

fix(core): recover confirmed subagent calls after provider stream close - #437

Merged
shuv1337 merged 8 commits into
integration-v2from
fm/sc-i407-toolcall-recovery
Oct 6, 2026
Merged

shuv1337 merged 8 commits into
integration-v2from
fm/sc-i407-toolcall-recovery

Conversation

@shuv1337

@shuv1337 shuv1337 commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

Intent

Recover interrupted streaming tool calls after provider WebSocket closes in shuvcode issue #407. Durable identity design and fault injection must precede automatic replay: never execute unfinished or half-received calls, and prove no duplicate child or effect. Recover only provider-confirmed subagent calls with a stable durable operation identity and one child, admission, and canonical result; rejoin admitted, running, or completed children for the same operation without another prompt or child. Preserve cancellation and terminal distinctions and bound restart attempts across process death. Never automatically replay ambiguous generic local side effects. Apply the new path only to WebSocket read/close codes 1006 and 1012, leaving 1009 fallback, other errors, permissions, and user cancellation behavior unchanged. Use isolated real Session/tool-registry fixtures and process-death fault injection around identity, admission, and result boundaries; no live service or broad replay promise. The approved follow-up corrections require foreground prepared children and nested subagents deeper than one to stop ambiguous generic effects before any real runner drain, and require execution registration before prepareResume and recoverPreparedCall so a wake in that gap cannot drop recovery. Cover both with behavioral real-runner regressions. Keep prior corrections and PR #437; the main lead alone owns merge.

What Changed

  • When the provider WebSocket closes on read with code 1006 or 1012, tool calls that were still streaming now fail with recovery: "unconfirmed" metadata. They stay in the durable transcript but are left out of regenerated model requests. Other close codes and other errors keep their existing behavior.
  • Adds a durable session.tool.subagent.prepared event and a subagent_operation table (new migration). Together they record a stable identity for each provider-confirmed subagent call: child Session ID, inbox ID, input digest, agent, and model. The subagent tool reuses this identity, so a recovered call rejoins or reads back the existing child (admitted, running, or completed) instead of sending another prompt or creating a second child.
  • Restart recovery in session/execution/restart.ts now recovers prepared subagent calls under the parent's execution ownership. SessionExecution.resume and the run coordinator take a prepare step that runs after the execution is registered and before the first drain, so a wake during recovery can't drop it. Ambiguous generic tool calls, including those in foreground and nested children, fail and stop the claimed execution before any drain instead of being replayed. Cancelled and failed child outcomes stay distinct. The session spec and the generated client types are updated to match.

Risk Assessment

⚠️ Medium: The fix-round change is small and correct. Before a recovered background child drains, it now runs stopStaleChild over that child's prepared nested children. It fails the child and its parent operation fail-closed and releases the stopped subtree's claims, and releaseChildClaims clears any remaining orphaned claims. Risk stays medium only because the overall branch adds intricate restart-recovery state machinery.

Testing

I drove 13 live runs, covering all 7 intent scenarios, against real shuvcode serve processes from the candidate worktree. Each run had an isolated HOME/XDG/DB, used a local fake OpenAI Responses provider over HTTP SSE and WebSocket, and was killed with SIGKILL. Persisted SQLite state confirmed every scenario: - Foreground and background subagent calls recover with exactly one child, one admission and one result. A cancelled child stays cancelled. - Ambiguous shell effects and half-received calls run once and fail closed. - Restarts are capped at 10 attempts, then the parent fails. - At depth 2, the background child and its grandchild stop before any drain. - A prompt sent the moment the restarted server answers produces no dangling request. - After a 1006 or 1012 close, the model request is regenerated without the draft and keeps the completed sibling. A 1009 close keeps its old fallback and error behaviour. The targeted process-death and runner regression tests also pass. The only evidence is JSON/Markdown transcripts and persisted-state dumps; there are no screenshots because this change has no UI.

  • Live validation: ✅ go - 7 of 7 scenarios driven live against the product
Scenario Result Live Evidence
Confirmed foreground subagent call: the server is SIGKILLed while the child runs, or while the result goes back to the parent; after restart there is exactly one child, one admission and one result ✅ pass live run2/result-fg-running.json, run2/result-fg-result.json: 1 child, 1 admission, operation prepared->completed, tool completed, parent succeeded; the child was not re-run at the result boundary
Background subagent: the server is SIGKILLed while the child runs (it rejoins the same child and sends exactly one completion notice) or after the child was cancelled (it stays interrupted) ✅ pass live run2/result-bg-running.json (same child, parent admissions=2); run2/result-bg-cancelled.json (child interrupted, 0 child requests after restart)
Adversarial: an ambiguous shell effect, or a half-received subagent call, is never replayed after process death ✅ pass live run2/result-ambiguous.json (effect.txt 1 line, 'Tool outcome is ambiguous after restart', 0 replays); run2/result-unprepared.json (recovery=unconfirmed, draft excluded from the resumed request, 0 chil…
Adversarial: repeated process death is bounded; the parent fails after the restart counter is used up ✅ pass live run2/result-exhausted.json: resume_attempts 0..10 across 11 kills, then the parent fails with 'interrupted repeatedly'; 1 child, operation failed
Background child at depth 2 with an in-flight ambiguous grandchild shell effect is stopped before it drains ✅ pass live run2/result-nested.json: grand-effect 1 line, 0 child or grandchild LLM requests after restart, B failed 'Nested subagent recovery stopped', G failed ambiguous
A prompt sent the moment the restarted server answers cannot drop or split recovery ownership ✅ pass live run2/result-wake-gap.json: 0 dangling parent requests, exactly one parent request carrying both the subagent result and the wake prompt, 1 child
After a WebSocket 1006 or 1012 close, the request is regenerated without the unconfirmed draft and keeps the completed sibling; 1009 behaviour is unchanged (HTTP fallback before acceptance, generic er… ✅ pass live run2/result-ws-1012.json, result-ws-1006.json (regenerated over WS, sibling ran once, draft excluded, recovery=unconfirmed); result-ws-reject1009.json (ws -> http); result-ws-1009.json (no unconfirmed…
Evidence: Live run summary (candidate 61651d7)
# Live re-run against committed candidate 61651d75f8dfd7da2a52d81e57dc43c9740f0ed9 (2026-10-06 PDT)

Launch: `bun run packages/cli/src/index.ts serve --service --hostname 127.0.0.1 --port <48000-53000>`
from the worktree. Each scenario gets its own HOME/XDG/OPENCODE_DB under /tmp/nm-live/<scenario>-XXXX,
with every inherited OPENCODE_* variable removed and OPENCODE_CHANNEL=latest. The provider is a local fake
OpenAI Responses server on 127.0.0.1:<random>/v1 (HTTP SSE + WS). Faults are injected by SIGKILL of the
server process. State is read from the real persisted SQLite DB (through a copy). Harness: ../harness.ts, ../scenarios.ts.
Nothing touched :4096, the installed binary, the real config or auth, or production sessions.

| scenario | observed |
|---|---|
| fg-running | 1 child, 1 admission, operation prepared->completed, 1 tool success, parent succeeded |
| fg-result | kill on the parent follow-up; the child was not re-run, 1 child, 1 admission, parent succeeded |
| bg-running | rejoined the same child (1 child), the parent got exactly 1 completion notice (admissions 2) |
| bg-cancelled | child stays `interrupted`, 0 child requests after restart |
| ambiguous | effect.txt has 1 line, shell failed "ambiguous after restart", 0 replays, no child |
| unprepared | half-received subagent call tagged recovery=unconfirmed, excluded from the resumed request, 0 children |
| exhausted | resume_attempts 0..10 across 11 kills; the 12th fails the parent "interrupted repeatedly"; 1 child, operation failed |
| nested (depth 2) | grandchild effect 1 line, 0 child or grandchild requests after restart, B and G fail closed |
| wake-gap | prompt fired the moment the restarted server answered: 0 dangling requests, 1 parent request carrying the result and the wake prompt |
| ws-1012 / ws-1006 | regenerated over WS, settled shell sibling kept (ran once), draft excluded, tagged unconfirmed, 0 children |
| ws-1009 (after acceptance) | generic error path unchanged (no unconfirmed tag) |
| ws-reject1009 | WS 1009 before acceptance -> same request retried over HTTP, succeeded |
Evidence: Live harness (real shuvcode serve + fake provider + SIGKILL)
// Disposable live harness: real `shuvcode serve` from the candidate worktree, isolated HOME/XDG,
// a local fake OpenAI Responses provider (HTTP SSE + WebSocket), SIGKILL fault injection,
// and inspection of the real persisted SQLite state.
import { Database } from "bun:sqlite"
import fs from "node:fs/promises"
import path from "node:path"

const WORKTREE = "~/.no-mistakes/worktrees/284027b67ce8/01M47YT458YZH24QDN1TT5N808"
const CLI = path.join(WORKTREE, "packages/cli/src/index.ts")

export type Req = { transport: "http" | "ws"; body: any; firstUser: string; kind: string; outputs: number; text: string }
export type Action =
  | { type: "text"; text: string }
  | { type: "call"; calls: { name: string; args: any }[]; partial?: { name: string; args: string }; end?: "complete" | "hang" | "close1012" | "terminate" | "close1009" }
  | { type: "hang" }
  | { type: "reject1009" }

let counter = 0
const id = (p: string) => `${p}_${++counter}_${Date.now().toString(36)}`

function userTexts(input: any[]): string[] {
  return (input ?? [])
    .filter((item) => item.role === "user")
    .map((item) =>
      typeof item.content === "string" ? item.content : (item.content ?? []).map((c: any) => c.text ?? "").join(""),
    )
}

function events(action: Action): { frames: any[]; end: string } {
  const rid = id("resp")
  const created = { type: "response.created", response: { id: rid } }
  const usage = { input_tokens: 10, output_tokens: 5, total_tokens: 15 }
  if (action.type === "text") {
    const mid = id("msg")
    const item = { type: "message", id: mid, role: "assistant", status: "completed", content: [{ type: "output_text", text: action.text, annotations: [] }] }
    return {
      end: "complete",
      frames: [
        created,
        { type: "response.output_item.added", output_index: 0, item: { ...item, status: "in_progress", content: [] } },
        { type: "response.output_text.delta", output_index: 0, item_id: mid, content_index: 0, delta: action.text },
        { type: "response.output_item.done", output_index: 0, item },
        { type: "response.completed", response: { id: rid, status: "completed", output: [item], usage } },
      ],
    }
  }
  if (action.type === "hang") return { frames: [created], end: "hang" }
  const frames: any[] = [created]
  const output: any[] = []
  action.calls.forEach((call, index) => {
    const fid = id("fc")
    const args = JSON.stringify(call.args)
    const item = { type: "function_call", id: fid, call_id: id("call"), name: call.name, arguments: args, status: "completed" }
    frames.push(
      { type: "response.output_item.added", output_index: index, item: { ...item, arguments: "", status: "in_progress" } },
      { type: "response.function_call_arguments.delta", output_index: index, item_id: fid, delta: args },
      { type: "response.function_call_arguments.done", output_index: index, item_id: fid, arguments: args },
      { type: "response.output_item.done", output_index: index, item },
    )
    output.push(item)
  })
  if (action.partial) {
    const fid = id("fc")
    const index = action.calls.length
    frames.push(
      { type: "response.output_item.added", output_index: index, item: { type: "function_call", id: fid, call_id: id("call"), name: action.partial.name, arguments: "", status: "in_progress" } },
      { type: "response.function_call_arguments.delta", output_index: index, item_id: fid, delta: action.partial.args },
    )
  }
  const end = action.end ?? "complete"
  if (end === "complete") frames.push({ type: "response.completed", response: { id: rid, status: "completed", output, usage } })
  return { frames, end }
}

export function startProvider(script: (req: Req) => Action) {
  const requests: Req[] = []
  const parse = (body: any, transport: "http" | "ws"): Req => {
    const input = body.input ?? []
    const texts = userTexts(input)
    return {
      transport,
      body,
      firstUser: texts[0] ?? "",
      kind: (body.tools ?? []).length === 0 ? "untooled" : "tooled",
      outputs: input.filter((i: any) => i.type === "function_call_output").length,
      text: JSON.stringify(input),
    }
  }
  const server = Bun.serve({
    port: 0,
    hostname: "127.0.0.1",
    async fetch(request, srv) {
      if (request.headers.get("upgrade")?.toLowerCase() === "websocket") {
        if (srv.upgrade(request, { data: {} })) return undefined
        return new Response("upgrade failed", { status: 400 })
      }
      const url = new URL(request.url)
      if (request.method !== "POST" || !url.pathname.endsWith("/responses")) return new Response("not found", { status: 404 })
      const req = parse(await request.json(), "http")
      requests.push(req)
      const action = script(req)
      const plan = events(action.type === "reject1009" ? { type: "text", text: "unexpected" } : action)
      const enc = new TextEncoder()
      const stream = new ReadableStream({
        start(controller) {
          for (const frame of plan.frames) controller.enqueue(enc.encode(`event: ${frame.type}\ndata: ${JSON.stringify(frame)}\n\n`))
          if (plan.end === "complete") controller.close()
          if (plan.end === "terminate" || plan.end.startsWith("close")) controller.error(new Error("drop"))
        },
      })
      return new Response(stream, { headers: { "content-type": "text/event-stream" } })
    },
    websocket: {
      message(ws, message) {
        const body = JSON.parse(String(message))
        const req = parse(body, "ws")
        requests.push(req)
        const action = script(req)
        // A request the provider refuses as too large closes 1009 before it is accepted.
        if (action.type === "reject1009") return ws.close(1009, "message too big")
        const plan = events(action)
        for (const frame of plan.frames) ws.send(JSON.stringify(frame))
        if (plan.end === "close1012") setTimeout(() => ws.close(1012, "service restart"), 50)
        if (plan.end === "close1009") setTimeout(() => ws.close(1009, "too big"), 50)
        if (plan.end === "terminate") setTimeout(() => ws.terminate(), 50)
      },
    },
  })
  return { url: `http://127.0.0.1:${server.port}/v1`, requests, stop: () => server.stop(true) }
}

export async function makeRoot(name: string, providerURL: string, transport: "http" | "websocket", extra: any = {}) {
  const root = await fs.mkdtemp(`/tmp/nm-live/${name}-`)
  const project = path.join(root, "project")
  await fs.mkdir(project, { recursive: true })
  await Bun.write(
    path.join(project, "opencode.json"),
    JSON.stringify(
      {
        model: "openai/gpt-5.2",
        small_model: undefined,
        permissions: [{ action: "*", resource: "*", effect: "allow" }],
        snapshots: false,
        providers: {
          openai: { settings: { baseURL: providerURL, apiKey: "test-key", transport }, models: { "gpt-5.2": {} } },
        },
        ...extra,
      },
      null,
      2,
    ),
  )
  return { root, project }
}

export function env(root: string) {
  return {
    // Drop every inherited OPENCODE_* variable (e.g. OPENCODE_CONFIG_DIR) so nothing real leaks in.
    ...Object.fromEntries(Object.entries(process.env).filter(([key]) => !key.startsWith("OPENCODE_"))),
    HOME: root,
    USERPROFILE: root,
    OPENCODE_TEST_HOME: root,
    OPENCODE_DB: path.join(root, "opencode.db"),
    XDG_CACHE_HOME: path.join(root, "cache"),
    XDG_CONFIG_HOME: path.join(root, "config"),
    XDG_DATA_HOME: path.join(root, "data"),
    XDG_STATE_HOME: path.join(root, "state"),
    OPENCODE_CHANNEL: "latest",
    OPENCODE_PASSWORD: "pw",
    OPENAI_API_KEY: "test-key",
  }
}

export const AUTH: Record<string, string> = { authorization: "Basic " + btoa("opencode:pw") }
let nextPort = 48000 + Math.floor(Math.random() * 5000)
const live = new Set<Bun.Subprocess>()
process.on("exit", () => live.forEach((proc) => proc.kill("SIGKILL")))

export async function startServer(root: string, log: string[]) {
  const port = nextPort++
  const cmd = [process.execPath, "run", CLI, "serve", "--service", "--hostname", "127.0.0.1", "--port", String(port)]
  log.push(`$ (cwd=${root}/project, HOME/XDG under ${root}) ${cmd.join(" ")}`)
  const proc = Bun.spawn(cmd, { cwd: path.join(root, "project"), env: env(root), stdout: "ignore", stderr: "pipe" })
  live.add(proc)
  proc.exited.then(() => live.delete(proc))
  const stderr = new Response(proc.stderr).text()
  const base = `http://127.0.0.1:${port}`
  const deadline = Date.now() + 60_000
  while (Date.now() < deadline) {
    const registration = Bun.file(path.join(root, "state", "shuvcode", "service-local.json"))
    if (await registration.exists()) {
      const info = await registration.json().catch(() => undefined)
      if (info?.password) AUTH.authorization = "Basic " + btoa(`opencode:${info.password}`)
    }
    const res = await fetch(`${base}/api/info`, { headers: AUTH, signal: AbortSignal.timeout(1000) }).catch(() => undefined)
    if (res?.ok) return { proc, base, port, stderr }
    if (proc.exitCode !== null) break
    await Bun.sleep(100)
  }
  proc.kill("SIGKILL")
  throw new Error(`server did not start: ${await stderr}`)
}

export async function kill(server: { proc: Bun.Subprocess }) {
  server.proc.kill("SIGKILL")
  await server.proc.exited
}

export async function api(base: string, method: string, route: string, body?: unknown) {
  const res = await fetch(base + route, {
    method,
    headers: { ...AUTH, "content-type": "application/json" },
    body: body === undefined ? undefined : JSON.stringify(body),
    signal: AbortSignal.timeout(15_000),
  })
  const text = await res.text()
  if (!res.ok) throw new Error(`${method} ${route} -> ${res.status} ${text}`)
  return text ? JSON.parse(text) : undefined
}

export async function createAndPrompt(base: string, project: string, text: string) {
  const session = (await api(base, "POST", "/api/session", { location: { directory: project } })).data
  await api(base, "POST", `/api/session/${session.id}/prompt?location[directory]=${encodeURIComponent(project)}`, { text })
  return session.id as string
}

export async function until(label: string, fn: () => boolean | Promise<boolean>, ms = 60_000) {
  const deadline = Date.now() + ms
  while (Date.now() < deadline) {
    if (await fn()) return
    await Bun.sleep(100)
  }
  throw new Error(`timed out waiting for ${label}`)
}

export function inspect(root: string, parentID: string) {
  // Inspect a copy so a SIGKILLed server's WAL is recovered without touching the real state.
  const copy = path.join(root, `inspect-${Date.now()}`)
  require("node:fs").mkdirSync(copy)
  for (const suffix of ["", "-wal", "-shm"])
    if (require("node:fs").existsSync(path.join(root, "opencode.db" + suffix)))
      require("node:fs").copyFileSync(path.join(root, "opencode.db" + suffix), path.join(copy, "opencode.db" + suffix))
  const db = new Database(path.join(copy, "opencode.db"))
  try {
    const sessions = db.query("select id, parent_id, idle_outcome, time_suspended, resume_attempts from session_v2").all() as any[]
    const descendants = (id: string): any[] =>
      sessions.filter((s) => s.parent_id === id).flatMap((s) => [s, ...descendants(s.id)])
    const tree = [sessions.find((s) => s.id === parentID), ...descendants(parentID)].filter(Boolean)
    const eventCounts = (aggregate: string) =>
      Object.fromEntries(
        (db.query("select type, count(*) n from event where aggregate_id = ? group by type").all(aggregate) as any[]).map((r) => [r.type, r.n]),
      )
    const tools = (session: string) =>
      (db.query("select data from session_message where session_id = ? and type = 'assistant' order by seq").all(session) as any[])
        .flatMap((r) => JSON.parse(r.data).content ?? [])
        .filter((c: any) => c.type === "tool")
        .map((c: any) => ({
          name: c.name,
          status: c.state?.status,
          error: c.state?.error?.message,
          recovery: c.state?.metadata?.recovery,
          childStatus: c.state?.metadata?.status,
        }))
    const operations = db.query("select session_id, child_session_id, status from subagent_operation").all()
    return {
      sessions: tree.map((s: any) => ({ ...s, events: eventCounts(s.id), tools: tools(s.id) })),
      operations,
    }
  } finally {
    db.close()
    require("node:fs").rmSync(copy, { recursive: true, force: true })
  }
}
Evidence: Live scenarios script
import fs from "node:fs/promises"
import path from "node:path"
import { api, createAndPrompt, inspect, kill, makeRoot, startProvider, startServer, until, type Action, type Req } from "./harness"

const role = (req: Req) =>
  req.kind === "untooled" ? "title" : req.firstUser.includes("GRAND") ? "grand" : req.firstUser.includes("CHILD") ? "child" : "parent"
const sub = (prompt: string, background = false) => ({
  name: "subagent",
  args: { agent: "general", description: "live check", prompt, ...(background ? { background: true } : {}) },
})
const shell = (command: string) => ({ name: "shell", args: { command, timeout: 0 } })
const count = (reqs: Req[], r: string) => reqs.filter((req) => role(req) === r).length
const summary = (reqs: Req[]) =>
  reqs.filter((r) => role(r) !== "title").map((r) => `${role(r)}:${r.transport}:outputs=${r.outputs}`)
const lines = async (file: string) => (await fs.readFile(file, "utf8").catch(() => "")).split("\n").filter(Boolean).length

type Ctx = { log: string[] }

async function scenario(name: string, transport: "http" | "websocket", script: (req: Req) => Action, body: (x: any) => Promise<any>, extra?: any) {
  const provider = startProvider(script)
  const { root, project } = await makeRoot(name, provider.url, transport, extra)
  const log: string[] = [`provider: ${provider.url} (fake local OpenAI Responses, HTTP SSE + WS)`, `root: ${root}`]
  try {
    const result = await body({ provider, root, project, log })
    return { name, ok: true, log, ...result }
  } catch (error) {
    return { name, ok: false, log, error: String(error), requests: summary(provider.requests) }
  } finally {
    provider.stop()
  }
}

// 1. Confirmed foreground subagent killed at the child-running boundary and at the result boundary.
async function foreground(boundary: "running" | "result") {
  let phase = "seed"
  return scenario(`fg-${boundary}`, "http", (req) => {
    const r = role(req)
    if (r === "parent" && req.outputs === 0) return { type: "call", calls: [sub("CHILD-TASK fg")] }
    if (r === "child") return boundary === "running" && phase === "seed" ? { type: "hang" } : { type: "text", text: "CHILD-RESULT fg" }
    if (r === "parent" && boundary === "result" && phase === "seed") return { type: "hang" }
    return { type: "text", text: "PARENT-DONE" }
  }, async ({ provider, root, project, log }) => {
    let server = await startServer(root, log)
    const sid = await createAndPrompt(server.base, project, "PARENT-TASK fg")
    if (boundary === "running") await until("child request in flight", () => count(provider.requests, "child") >= 1)
    if (boundary === "result")
      await until("parent result request in flight", () => provider.requests.some((r: Req) => role(r) === "parent" && r.outputs === 1))
    await Bun.sleep(500)
    await kill(server)
    log.push(`SIGKILL server at ${boundary} boundary`)
    const before = inspect(root, sid)
    const beforeReqs = summary(provider.requests)
    phase = "recover"
    server = await startServer(root, log)
    await until("parent completes", () => {
      const s = inspect(root, sid).sessions[0]
      return s?.idle_outcome === "succeeded" || s?.idle_outcome === "failed"
    }, 90_000)
    await Bun.sleep(1000)
    await kill(server)
    const after = inspect(root, sid)
    return { before: brief(before), beforeRequests: beforeReqs, after: brief(after), requests: summary(provider.requests) }
  })
}

// 2. Background subagent killed while its child runs (rejoin) and after cancellation (stays cancelled).
async function background(mode: "running" | "cancelled") {
  let phase = "seed"
  return scenario(`bg-${mode}`, "http", (req) => {
    const r = role(req)
    if (r === "parent" && req.outputs === 0) return { type: "call", calls: [sub("CHILD-TASK bg", true)] }
    if (r === "child") return phase === "seed" ? { type: "hang" } : { type: "text", text: "CHILD-RESULT bg" }
    return { type: "text", text: "PARENT-ACK" }
  }, async ({ provider, root, project, log }) => {
    let server = await startServer(root, log)
    const sid = await createAndPrompt(server.base, project, "PARENT-TASK bg")
    await until("child request in flight", () => count(provider.requests, "child") >= 1)
    await until("parent acked", () => provider.requests.some((r: Req) => role(r) === "parent" && r.outputs === 1))
    await Bun.sleep(1000)
    if (mode === "cancelled") {
      const child = inspect(root, sid).sessions.find((s: any) => s.parent_id === sid)
      await api(server.base, "POST", `/api/session/${child.id}/interrupt?location[directory]=${encodeURIComponent(project)}`, {})
      log.push(`POST /api/session/${child.id}/interrupt`)
      await Bun.sleep(1500)
    }
    await kill(server)
    log.push("SIGKILL server")
    const before = inspect(root, sid)
    const childBefore = count(provider.requests, "child")
    phase = "recover"
    server = await startServer(root, log)
    await Bun.sleep(mode === "cancelled" ? 8000 : 2000)
    if (mode === "running")
      await until("parent received one completion notice", () => count(provider.requests, "parent") >= 3, 60_000).catch(() => {})
    await Bun.sleep(1500)
    await kill(server)
    const after = inspect(root, sid)
    return {
      before: brief(before),
      after: brief(after),
      childRequestsBeforeRestart: childBefore,
      childRequestsAfterRestart: count(provider.requests, "child") - childBefore,
      requests: summary(provider.requests),
    }
  })
}

// 3a. Ambiguous generic local effect: shell writes then blocks; never replayed.
async function ambiguous() {
  return scenario("ambiguous", "http", (req) => {
    const r = role(req)
    if (r === "parent" && req.outputs === 0 && !req.text.includes("restart"))
      return { type: "call", calls: [shell("echo effect >> effect.txt; sleep 600")] }
    return { type: "text", text: "PARENT-DONE" }
  }, async ({ provider, root, project, log }) => {
    let server = await startServer(root, log)
    const sid = await createAndPrompt(server.base, project, "PARENT-TASK ambiguous")
    const effect = path.join(project, "effect.txt")
    await until("effect ran", async () => (await lines(effect)) >= 1)
    await kill(server)
    log.push("SIGKILL server while shell effect in flight")
    const parentBefore = count(provider.requests, "parent")
    server = await startServer(root, log)
    await until("parent settles", () => !!inspect(root, sid).sessions[0]?.idle_outcome, 60_000).catch(() => {})
    await Bun.sleep(2000)
    await kill(server)
    return {
      effectLines: await lines(effect),
      parentRequestsAfterRestart: count(provider.requests, "parent") - parentBefore,
      shellReplayedInRequests: provider.requests.filter((r: Req) => r.text.includes("sleep 600")).length,
      after: brief(inspect(root, sid)),
      requests: summary(provider.requests),
    }
  })
}

// 3b. Unprepared (half-received) subagent call: process dies mid-arguments; never executed.
async function unprepared() {
  let phase = "seed"
  return scenario("unprepared", "http", (req) => {
    const r = role(req)
    if (r === "parent" && phase === "seed")
      return { type: "call", calls: [], partial: { name: "subagent", args: '{"agent":"general","description":"x","prompt":"CHILD-TASK half' }, end: "hang" }
    return { type: "text", text: "PARENT-DONE" }
  }, async ({ provider, root, project, log }) => {
    let server = await startServer(root, log)
    const sid = await createAndPrompt(server.base, project, "PARENT-TASK unprepared")
    await until("partial call streamed", () => count(provider.requests, "parent") >= 1)
    await Bun.sleep(1500)
    await kill(server)
    log.push("SIGKILL server while subagent arguments half-received")
    phase = "recover"
    const parentBefore = count(provider.requests, "parent")
    server = await startServer(root, log)
    await until("parent settles", () => !!inspect(root, sid).sessions[0]?.idle_outcome, 60_000).catch(() => {})
    await Bun.sleep(1500)
    await kill(server)
    const resumed = provider.requests.filter((r: Req) => role(r) === "parent").slice(parentBefore)
    return {
      after: brief(inspect(root, sid)),
      childRequests: count(provider.requests

... [943 bytes truncated] ...

ver)
      const s = inspect(root, sid).sessions[0]
      attempts.push({ kill: i + 1, resume_attempts: s.resume_attempts, outcome: s.idle_outcome, childRequests: count(provider.requests, "child") })
      if (s.idle_outcome) break
      const seen = count(provider.requests, "child")
      server = await startServer(root, log)
      await until("child re-request or terminal", () => count(provider.requests, "child") > seen || !!inspect(root, sid).sessions[0]?.idle_outcome, 30_000).catch(() => {})
    }
    if (server.proc.exitCode === null) await kill(server)
    const after = inspect(root, sid)
    return { attempts, after: brief(after), childSessions: after.sessions.filter((s: any) => s.parent_id === sid).length }
  })
}

// 5. Background child at depth 2 calls a foreground grandchild whose shell effect is in flight.
async function nested() {
  let phase = "seed"
  return scenario("nested", "http", (req) => {
    const r = role(req)
    if (r === "parent" && req.outputs === 0) return { type: "call", calls: [sub("CHILD-TASK nested", true)] }
    if (r === "child" && req.outputs === 0) return { type: "call", calls: [sub("GRAND-TASK nested")] }
    if (r === "grand" && req.outputs === 0 && phase === "seed") return { type: "call", calls: [shell("echo grand >> grand-effect.txt; sleep 600")] }
    return { type: "text", text: `${r.toUpperCase()}-DONE` }
  }, async ({ provider, root, project, log }) => {
    let server = await startServer(root, log)
    const sid = await createAndPrompt(server.base, project, "PARENT-TASK nested")
    const effect = path.join(project, "grand-effect.txt")
    await until("grandchild effect ran", async () => (await lines(effect)) >= 1)
    await kill(server)
    log.push("SIGKILL server while grandchild shell effect in flight")
    phase = "recover"
    const childBefore = count(provider.requests, "child")
    const grandBefore = count(provider.requests, "grand")
    server = await startServer(root, log)
    await Bun.sleep(10_000)
    await kill(server)
    const after = inspect(root, sid)
    return {
      effectLines: await lines(effect),
      childRequestsAfterRestart: count(provider.requests, "child") - childBefore,
      grandRequestsAfterRestart: count(provider.requests, "grand") - grandBefore,
      grandchildSessions: after.sessions.filter((s: any) => s.parent_id && s.parent_id !== sid).length,
      after: brief(after),
      requests: summary(provider.requests),
    }
  }, { experimental: { subagent_depth: 2 } })
}

// 6. A prompt to the parent arriving immediately at restart while recovery of a prepared call is pending.
async function wakeGap() {
  let phase = "seed"
  return scenario("wake-gap", "http", (req) => {
    const r = role(req)
    if (r === "parent" && req.outputs === 0 && !req.text.includes("WAKE")) return { type: "call", calls: [sub("CHILD-TASK wake")] }
    if (r === "child") return phase === "seed" ? { type: "hang" } : { type: "text", text: "CHILD-RESULT wake" }
    return { type: "text", text: "PARENT-DONE" }
  }, async ({ provider, root, project, log }) => {
    let server = await startServer(root, log)
    const sid = await createAndPrompt(server.base, project, "PARENT-TASK wake")
    await until("child request", () => count(provider.requests, "child") >= 1)
    await Bun.sleep(500)
    await kill(server)
    log.push("SIGKILL server with prepared foreground call; prompt fired the moment the restarted server answers")
    phase = "recover"
    const parentBefore = count(provider.requests, "parent")
    server = await startServer(root, log)
    await api(server.base, "POST", `/api/session/${sid}/prompt?location[directory]=${encodeURIComponent(project)}`, { text: "WAKE-PROMPT" })
    await until("parent settles", () => inspect(root, sid).sessions[0]?.idle_outcome === "succeeded", 60_000).catch(() => {})
    await Bun.sleep(2000)
    await kill(server)
    const resumed = provider.requests.filter((r: Req) => role(r) === "parent").slice(parentBefore)
    return {
      // A dangling request would carry the subagent function_call without its function_call_output.
      danglingParentRequests: resumed.filter((r: Req) => {
        const input = r.body.input ?? []
        const calls = input.filter((i: any) => i.type === "function_call").map((i: any) => i.call_id)
        const outs = new Set(input.filter((i: any) => i.type === "function_call_output").map((i: any) => i.call_id))
        return calls.some((c: string) => !outs.has(c))
      }).length,
      parentRequestsAfterRestart: resumed.map((r: Req) => `outputs=${r.outputs} wake=${r.text.includes("WAKE-PROMPT")}`),
      after: brief(inspect(root, sid)),
      requests: summary(provider.requests),
    }
  })
}

// 7b. 1009 rejection before acceptance falls back to HTTP for the same request.
async function wsReject1009() {
  return scenario("ws-reject1009", "websocket", (req) => {
    if (role(req) === "parent" && req.transport === "ws") return { type: "reject1009" }
    return { type: "text", text: "PARENT-DONE" }
  }, async ({ provider, root, project, log }) => {
    const server = await startServer(root, log)
    const sid = await createAndPrompt(server.base, project, "PARENT-TASK reject1009")
    await until("parent settles", () => !!inspect(root, sid).sessions[0]?.idle_outcome, 60_000).catch(() => {})
    await Bun.sleep(1500)
    await kill(server)
    return {
      parentRequests: provider.requests.filter((r: Req) => role(r) === "parent").map((r: Req) => r.transport),
      after: brief(inspect(root, sid)),
    }
  })
}

// 7. WebSocket close 1012 / 1006 mid-stream with a settled sibling; 1009 falls back to HTTP.
async function wsClose(end: "close1012" | "terminate" | "close1009") {
  let closed = false
  return scenario(`ws-${end}`, "websocket", (req) => {
    const r = role(req)
    if (r === "parent" && !closed) {
      closed = true
      return {
        type: "call",
        calls: [shell("echo sibling >> sibling.txt")],
        partial: { name: "subagent", args: '{"agent":"general","description":"x","prompt":"CHILD-TASK unconfirmed' },
        end,
      }
    }
    return { type: "text", text: "PARENT-DONE" }
  }, async ({ provider, root, project, log }) => {
    const server = await startServer(root, log)
    const sid = await createAndPrompt(server.base, project, "PARENT-TASK ws")
    await until("parent settles", () => !!inspect(root, sid).sessions[0]?.idle_outcome, 60_000).catch(() => {})
    await Bun.sleep(1500)
    await kill(server)
    const parents = provider.requests.filter((r: Req) => role(r) === "parent")
    return {
      siblingEffectLines: await lines(path.join(project, "sibling.txt")),
      parentRequests: parents.map((r: Req) => ({
        transport: r.transport,
        outputs: r.outputs,
        containsSettledSibling: r.text.includes("echo sibling"),
        containsUnconfirmedDraft: r.text.includes("CHILD-TASK unconfirmed"),
      })),
      childRequests: count(provider.requests, "child"),
      after: brief(inspect(root, sid)),
    }
  })
}

function brief(state: any) {
  return state.sessions.map((s: any) => ({
    id: s.id,
    parent: s.parent_id,
    outcome: s.idle_outcome,
    resume_attempts: s.resume_attempts,
    admissions: s.events["session.inbox.enqueued.1"] ?? 0,
    tools: s.tools,
    subagentEvents: Object.fromEntries(Object.entries(s.events).filter(([k]) => /subagent|tool\.(success|failed)|execution\.(failed|succeeded|interrupted)/.test(k))),
  })).concat([{ operations: state.operations }])
}

const all: Record<string, () => Promise<any>> = {
  "fg-running": () => foreground("running"),
  "fg-result": () => foreground("result"),
  "bg-running": () => background("running"),
  "bg-cancelled": () => background("cancelled"),
  ambiguous,
  unprepared,
  exhausted,
  nested,
  "wake-gap": wakeGap,
  "ws-1012": () => wsClose("close1012"),
  "ws-1006": () => wsClose("terminate"),
  "ws-1009": () => wsClose("close1009"),
  "ws-reject1009": wsReject1009,
}
for (const name of process.argv.slice(2)) {
  const result = await all[name]!()
  console.log(JSON.stringify(result, null, 1))
  await Bun.write(`/tmp/nm-live/result-${name}.json`, JSON.stringify(result, null, 1))
}
process.exit(0)
  • Evidence: Batch 1 transcript (fg/bg/ambiguous/unprepared) (local file: ~/.no-mistakes/evidence/01M47YT458YZH24QDN1TT5N808/live/run2/batch1.out)
  • Evidence: Batch 2 transcript (nested/wake-gap/ws/exhausted) (local file: ~/.no-mistakes/evidence/01M47YT458YZH24QDN1TT5N808/live/run2/batch2.out)
  • Evidence: Pre-fix finding (model:undefined metadata) and first live round (local file: ~/.no-mistakes/evidence/01M47YT458YZH24QDN1TT5N808/live/README.md)
  • Outcome: 🔧 1 issue found → auto-fixed ✅ across 2 runs (20m11s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed ✅
  • ⚠️ packages/core/src/session/execution/restart.ts:456 - A recovered background child can still drain over a nested subagent whose generic effect is ambiguous. The new stopStaleChild only runs from recoverPreparedCall, which only sees top-level parents. A background child with a pending job takes the first suspended loop instead. Example, with experimental.subagent_depth > 1: parent P starts background child B. B calls a foreground subagent G (a prepared call). G is mid-echo/shell when the process dies. On restart, B's only running call is prepared, so B is not marked ambiguous. G goes into preparedChildren (its claim is kept, but nothing ever recovers it). Then recoverSubagent calls prepareResume(B) and execution.resume(B). B's drain runs settleStaleToolCalls, which fails B's subagent call as a plain Tool execution interrupted. B's model can then call subagent again and create a second grandchild that repeats G's possibly completed effect. G's claim also stays held forever. This breaks the intent requirement that "nested subagents deeper than one … stop ambiguous generic effects before any real runner drain", and the requirement to prove no duplicate child or effect. Fix: in the first loop, for each prepared call of a recovered background child, call stopStaleChild(operation.childSessionID) (or stopStaleChild(sessionID) on the child). If anything was stopped, fail the call and add the child to ambiguous before recoverSubagent runs. This reuses the helper the commit already adds.

🔧 Fix applied.
✅ Re-checked - no issues remain.

🔧 **Test** - 1 issue found → auto-fixed ✅
  • ⚠️ live validation verdict: inconclusive (0 of 7 scenarios were driven live against the product); untested: Confirmed foreground subagent call killed at prepared/created/admitted/running/prompted/settled/result boundaries is recovered after restart with exactly one child, one admission and one result, Background subagent killed at prepared/admitted/running/settled/cancelled rejoins its existing child with one completion notice and keeps the cancelled state, Adversarial: an ambiguous generic local effect or an unprepared call is never replayed after process death (effect runs once, call fails closed, no child), Adversarial: restart attempts are bounded across process death (the exhausted counter fails the parent instead of retrying forever), Recovered background child at subagent depth >1 with an ambiguous grandchild effect is stopped before it drains (no second LLM request or second grandchild), A wake that arrives during restart preparation cannot drop or split recovery ownership, After close code 1006 or 1012, the session regenerates without replaying an unconfirmed tool call and keeps settled siblings; close code 1009 still falls back to HTTP
  • Live validation: ⚠️ inconclusive - 0 of 7 scenarios driven live against the product
Scenario Result Live Evidence
Confirmed foreground subagent call killed at prepared/created/admitted/running/prompted/settled/result boundaries is recovered after restart with exactly one child, one admission and one result ⏸️ untested no The prior payload established this only with isolated Session fixtures and SIGKILL between Bun test processes, not a live run. A live check needs a running shuvcode service and a live provider whose s…
Background subagent killed at prepared/admitted/running/settled/cancelled rejoins its existing child with one completion notice and keeps the cancelled state ⏸️ untested no The prior payload established this only with fixture-based process death, not a live run. A live check needs a running shuvcode service and a live provider with a background subagent in flight.
Adversarial: an ambiguous generic local effect or an unprepared call is never replayed after process death (effect runs once, call fails closed, no child) ⏸️ untested no The prior payload established this only with fixture-based process death, not a live run. A live check needs a running shuvcode service and a provider session interrupted during an ambiguous tool effe…
Adversarial: restart attempts are bounded across process death (the exhausted counter fails the parent instead of retrying forever) ⏸️ untested no The prior payload established this only with fixture-based process death, not a live run. A live check needs a running shuvcode service that is restarted repeatedly with a live session.
Recovered background child at subagent depth >1 with an ambiguous grandchild effect is stopped before it drains (no second LLM request or second grandchild) ⏸️ untested no The prior payload established this only with the real SessionRunnerLLM and the TestLLM provider, not a live run. A live nested-subagent run needs a real provider and a running service.
A wake that arrives during restart preparation cannot drop or split recovery ownership ⏸️ untested no The prior payload established this only in a real-runner fixture, not a live run. The live timing of this race cannot be driven deterministically against a running service.
After close code 1006 or 1012, the session regenerates without replaying an unconfirmed tool call and keeps settled siblings; close code 1009 still falls back to HTTP ⏸️ untested no The prior payload established this only by injecting closes through the real transport in tests, not a live run. A live check needs a provider WebSocket that can be forced to close with a given code a…
  • bun test test/subagent-recovery-process.test.ts (packages/core): 17 SIGKILL process-death boundary phases, all pass
  • bun test test/session-runner.test.ts -t &#34;recovered background child|restart preparation owned|without replaying an unconfirmed|settled sibling|prepared|recover|nested&#34;: 22 real-SessionRunnerLLM recovery scenarios, all pass
  • bun test test/session-model-transport.test.ts test/tool-subagent.test.ts: 50 pass, covering close codes 1006/1012, the unchanged 1009 HTTP fallback, and reuse of a reserved subagent operation
  • A/B check: restored restart.ts from HEAD~1 and ran -t &#34;stops nested ambiguous effects before a recovered background child drains&#34;. It fails with 2 LLM requests where 1 is expected. Then restored HEAD
  • Manual run: for the admitted, prompted, background-running, ambiguous and exhausted phases, ran the fixture's seed stage, SIGKILLed it at the ready boundary, then ran the recover stage and recorded its JSON counts

🔧 Fix applied.
✅ Re-checked - no issues remain.

  • Live validation: ✅ go - 7 of 7 scenarios driven live against the product
Scenario Result Live Evidence
Confirmed foreground subagent call: the server is SIGKILLed while the child runs, or while the result goes back to the parent; after restart there is exactly one child, one admission and one result ✅ pass live run2/result-fg-running.json, run2/result-fg-result.json: 1 child, 1 admission, operation prepared->completed, tool completed, parent succeeded; the child was not re-run at the result boundary
Background subagent: the server is SIGKILLed while the child runs (it rejoins the same child and sends exactly one completion notice) or after the child was cancelled (it stays interrupted) ✅ pass live run2/result-bg-running.json (same child, parent admissions=2); run2/result-bg-cancelled.json (child interrupted, 0 child requests after restart)
Adversarial: an ambiguous shell effect, or a half-received subagent call, is never replayed after process death ✅ pass live run2/result-ambiguous.json (effect.txt 1 line, 'Tool outcome is ambiguous after restart', 0 replays); run2/result-unprepared.json (recovery=unconfirmed, draft excluded from the resumed request, 0 chil…
Adversarial: repeated process death is bounded; the parent fails after the restart counter is used up ✅ pass live run2/result-exhausted.json: resume_attempts 0..10 across 11 kills, then the parent fails with 'interrupted repeatedly'; 1 child, operation failed
Background child at depth 2 with an in-flight ambiguous grandchild shell effect is stopped before it drains ✅ pass live run2/result-nested.json: grand-effect 1 line, 0 child or grandchild LLM requests after restart, B failed 'Nested subagent recovery stopped', G failed ambiguous
A prompt sent the moment the restarted server answers cannot drop or split recovery ownership ✅ pass live run2/result-wake-gap.json: 0 dangling parent requests, exactly one parent request carrying both the subagent result and the wake prompt, 1 child
After a WebSocket 1006 or 1012 close, the request is regenerated without the unconfirmed draft and keeps the completed sibling; 1009 behaviour is unchanged (HTTP fallback before acceptance, generic er… ✅ pass live run2/result-ws-1012.json, result-ws-1006.json (regenerated over WS, sibling ran once, draft excluded, recovery=unconfirmed); result-ws-reject1009.json (ws -> http); result-ws-1009.json (no unconfirmed…
  • bun run scenarios.ts fg-running fg-result bg-running bg-cancelled ambiguous unprepared (live harness: real shuvcode serve --service from worktree, isolated HOME/XDG, fake local Responses provider, SIGKILL)
  • bun run scenarios.ts nested wake-gap ws-1012 ws-1006 ws-1009 ws-reject1009 exhausted (same live harness)
  • Inspected persisted session_v2 / event / session_message / subagent_operation rows after each kill and restart
  • bun test test/subagent-recovery-process.test.ts test/tool-subagent.test.ts test/session-model-transport.test.ts (packages/core, 68 pass)
  • bun test test/session-runner.test.ts -t &#34;recover|prepared|nested|background child|wake|unconfirmed|1006|1012&#34; (packages/core, 22 pass)
✅ **Document** - passed

✅ No issues found.

⚠️ **Lint** - 2 infos
  • ℹ️ packages/core/test/tool-shell.test.ts:1011 - Prettier flags this file, but the base commit 28350a1 fails the same check, and none of the flagged hunks (lines ~1011, ~1377, ~1406) are lines this change touched. I left them alone to keep the change scoped and to avoid editing tests. A follow-up formatting-only commit can fix them.
  • ℹ️ The ast-grep binary is not installed in this environment, so lint:effect-patterns could not run. oxlint found 0 issues on the changed TS files, bun typecheck in packages/core is clean, and git diff --check is clean.
✅ **Push** - passed

✅ No issues found.

shuv1337 commented Oct 6, 2026

Copy link
Copy Markdown
Owner Author

🔍 Automated Cursor review is underway. Please hold off on merging until the review comment lands here (usually 10–20 minutes).

@cursor

cursor Bot commented Oct 6, 2026

Copy link
Copy Markdown

Review complete: one medium finding and one low finding. Prior review fixes that do hold are noted at the bottom. Tests were not re-run here (bun is not installed and dependencies are not present).

Medium

Foreground child sessions with an in-flight generic tool are resumed, so the model can retry an ambiguous effect. resumeSuspendedSessions builds suspended from store.listSuspended() plus running background-job children (packages/core/src/session/execution/restart.ts:360-362). listSuspended only returns rows with parent_id null (packages/core/src/session/store.ts:194-198), so a foreground subagent child is never scanned. When the parent’s running calls are all prepared, the child id is only added to preparedChildren (restart.ts:375-379) and its claim is kept. recoverPreparedCall then starts that child whenever it is not already settled (restart.ts:262-266) with no ambiguous-tool check. recoverSubagent does consult ambiguous (restart.ts:170-172); this path does not.

On the real runner, that resume hits settleStaleToolCalls (packages/core/src/session/runner/llm.ts:68-69 and 329-354), which records Tool execution interrupted and then lets the drain continue. The child model sees a normal tool error and can call the same shell/write/edit again. The parent-level ambiguous path fails every running call and publishes Execution.Failed (restart.ts:382-410). A generic tool inside the recovered child does not get that stop. The same scan also means a prepared grandchild (experimental.subagent_depth > 1) is failed by settleStaleToolCalls on the child’s drain instead of being recovered.

The process-death fixture replaces SessionExecution with a stub drain, so the 17 SIGKILL cases never run this path.

Suggested fix: before subagents.start, treat a prepared child’s running local non-subagent calls the same way as the parent ambiguous path (fail them, do not resume, fail the parent call). Run prepared-subagent recovery for that child before its first drain, or keep failing nested calls closed and cover it with a test that uses the real runner.

Low

execution.resume(sessionID, prepare) still drops prepare when it joins an execution that is already active. Confirmed in the current code, not only in the earlier note. run awaits the active execution and never runs prepare (packages/core/src/session/run-coordinator.ts:131-135). Recovery still calls prepareResume (attempt increment and the restart synthetic) and only then execution.resume (restart.ts:441-453), and that fiber is forked before suspended.clear() (restart.ts:482-502). A wake in that gap — a prompt accepted at boot, or a background completion after suspended.clear() — starts a drain with no prepare step. settleStaleToolCalls no longer skips prepared calls, so it fails the call as interrupted, the joiner’s prepare never runs, and the model can launch a new subagent. Outcome is fail-closed (no second admission of the same inbox id). The prompted process-death case waits until child-runs exists, which is after ownership is taken, so it does not cover this window.

Suggested fix: pass prepareResume and recoverPreparedCall as the prepare effect so executions.set happens before the first yield. Wakes then set pendingWake and wait for the settled call.

Prior fixes that hold

  • Digest sorts keys before JSON.stringify (packages/core/src/session/subagent-recovery.ts:14-24). Subagent input is a flat struct, so top-level sort is sufficient.
  • A defect or false result from recoverPreparedCall fails the remaining prepared calls and raises StepFailedError (restart.ts:456-478), which the coordinator settles as Execution.Failed and releases (packages/core/src/session/execution.ts:149-156). The removed child prepareResume branch is gone; only the parent counter bounds restarts.
  • Streaming drafts left by process death are failed with metadata.recovery: "unconfirmed" (llm.ts:347-348), and to-llm-message.ts omits those calls. In-process 1006/1012 uses failUnconfirmedTools only for uncalled tools (publish-llm-event.ts:434-438, step.ts:186-216). Close 1009 does not match that condition.

…onfirmed subagent recovery after process death > reconciles the exhausted boundary…` in packages/core/test/subagent-recovery-process.test.ts. The recover fixture hit its own 5s `Effect.timeout`, logged no recovery error, and the test's stderr assertion caught the TimeoutError. **Root cause (a race in the test harness, not in production code):** - The test SIGKILLs the seed process as soon as the `ready` file exists. - The recover stage then read its phase from that file's contents. - `Bun.write` creates and truncates the file first, then writes the content. Under CI load the kill can land between those two steps, leaving `ready` empty. - With an empty phase, the recover stage took the "wait for completed" path instead of the "wait for failed" path, and timed out. - `exhausted` was just the test that got hit. The earlier unit failure in this run was a different flaky test (PowerShell ShellTool), so this one is intermittent. **Reproduced:** with the HEAD fixture, I seeded `exhausted`, emptied `ready` and ran recover. It produced the same TimeoutError as CI. **Fix (test-only, 3 lines):** - `test/fixture/subagent-recovery-process.ts`: both stages now take the phase from argv, and `ready` is only a presence signal. A short comment explains why. - `test/subagent-recovery-process.test.ts`: passes `phase` to the recover command. **Verified:** - All 17 tests in subagent-recovery-process.test.ts pass. - `bun typecheck` in packages/core is clean. - `git diff --check` and Prettier are clean
…ailing test is not one this PR adds or touches. **What failed:** one test in `unit`: `ShellTool ordinary shell syntax > PowerShell legacy: ordinary syntax reuses approvals: Write-Output "$(Write-Output hello)"` in `packages/core/test/tool-shell.test.ts`. It died at 10087ms with an Effect `TimeoutError`. This PowerShell test also failed in an earlier CI run on this PR. Recent `test` runs on `integration-v2` pass. Core tests run one at a time, so this PR's process-death tests do not run alongside it. **Root cause:** a timeout conflict in the test. - The PowerShell tests were given a 15s per-test timeout because PowerShell starts slowly on CI. - But the shared helper `runPermissionCommand` wraps every call in its own 10s `Effect.timeout`, so that 10s limit always hit first. - The first PowerShell test in the run pays the cold-start cost, and on a loaded runner that took just over 10s. **Fix (3 lines, test only):** - Removed the helper's own 10s timeout, so the per-test timeout decides. For the `sh` tests this changes nothing, because Bun's 5s default already fired first. - Raised the PowerShell tests' timeout from 15s to 30s and updated the comment. **Verified locally:** - All 108 tests in `tool-shell.test.ts` pass with `pwsh` installed, including the PowerShell tests. - `bun typecheck` in `packages/core` is clean. - `git diff --check` is clean. - Prettier `--check` flags this file, but it flags the unchanged HEAD version too, so that is not from this change. I could not reproduce the CI timeout locally, because PowerShell starts fast here. So the 30s figure is a judgment about CI load, not a measured value
@shuv1337

shuv1337 commented Oct 6, 2026

Copy link
Copy Markdown
Owner Author

Cursor Medium in #437 (comment) is fixed by 87f2a43ffc and 76019d2a1b. The first stops a foreground prepared child with an in-flight generic effect before any real runner drain and fails nested running calls closed; the second covers the background-child path before it can drain over an ambiguous grandchild. Real SessionRunnerLLM regressions in packages/core/test/session-runner.test.ts cover the direct child, depth-two foreground, and depth-two background cases. The background regression fails against the prior implementation with two model requests, then passes with the fix. The disposable product run also observed one shell effect and no replay for the nested background case; the 17 SIGKILL boundary fixtures remain passing. Nested in-flight calls are intentionally stopped rather than auto-replayed.

@shuv1337

shuv1337 commented Oct 6, 2026

Copy link
Copy Markdown
Owner Author

Cursor Low in #437 (comment) is fixed by 87f2a43ffc. prepareResume and recoverPreparedCall now run together as the parent execution prepare effect, with immediate fork start, so the coordinator registers ownership before the restart notice can trigger a wake. The real-runner regression keeps restart preparation owned when a wake arrives during its synthetic notice in packages/core/test/session-runner.test.ts triggers the wake inside that notice and verifies active ownership, a completed prepared call, one child, and one child admission. The existing prompted SIGKILL recovery case remains passing. No ambiguous or unconfirmed tool is made retryable by this change.

@shuv1337
shuv1337 merged commit 456a579 into integration-v2 Oct 6, 2026
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant