Skip to content

Add a wisp dispatch circuit breaker for runaway re-dispatch loops#463

Merged
rockfordlhotka merged 1 commit into
mainfrom
claude/wisp-dispatch-circuit-breaker
Jun 5, 2026
Merged

Add a wisp dispatch circuit breaker for runaway re-dispatch loops#463
rockfordlhotka merged 1 commit into
mainfrom
claude/wisp-dispatch-circuit-breaker

Conversation

@rockfordlhotka

Copy link
Copy Markdown
Member

Why

Follow-up to #462 (the trim-loop runaway fix). That PR stopped the specific wedge; this one adds the guard that was missing one layer up: nothing counts "I've already run this exact wisp too many times — stop."

The only existing guard against repeated identical tool calls is the per-agent-loop RepetitiveToolCallDetector. It has two gaps:

  1. It's rebuilt fresh per AgentLoopRunner.RunAsync and dies with that loop — so it never spans completion-eval reprompts, scheduled re-fires, or message/A2A re-triggers, which is exactly how a wisp gets re-dispatched across many short loops.
  2. spawn_wisps results embed fresh batch-{Guid}/wisp-{Guid} IDs, so two byte-identical dispatches never look identical to a detector keyed on tool+args+result.

SpawnWispsExecutor is the single choke point every wisp flows through, and a process-wide singleton there is the only guard that survives across loops, reprompts, and message-driven re-triggers.

What

WispDispatchCircuitBreaker keeps a fixed-window dispatch count keyed by the exact definition hash. Once a definition exceeds DispatchCircuitBreakerMaxPerWindow (default 30) within DispatchCircuitBreakerWindow (default 5m), further dispatches of that definition are refused until the window rolls over.

Design choices:

  • Exact definition hash, not the value-stripped shape hash — wisps that legitimately vary by date/id never trip it; only a truly identical re-dispatch loop does.
  • Fixed window keeps memory at O(distinct definitions) regardless of dispatch rate — which matters precisely in the runaway case the breaker exists to contain. (A sliding window with per-attempt timestamps would balloon under a 400k/min loop.)
  • A tripped dispatch does no work (no LLM, no tool calls) and is not written to the execution log, so a runaway collapses into a cheap, self-limiting error instead of an unbounded spin — and doesn't re-bloat the JSONL log.
  • The refusal message is stable (no counts/ids), so identical refusals also look identical to the agent-loop's repetitive-result detector, which then nudges the model off the loop.
  • Emits a rockbot.wisp.circuit_breaker.trips counter for alerting on runaways.

Conservative by default and fully configurable via WispOptions; the disabled flag and a non-positive limit both short-circuit to always-allow.

Tests

7 new tests (fake TimeProvider): below/at threshold, sustained trip stays tripped, window reset re-allows, per-definition independence, disabled / zero-limit / empty-hash always-allow. Full RockBot.Wisp.Tests suite green (142/142). Agent composition root builds clean.

Not included (still open from the incident write-up)

The RepetitiveToolCallDetector blind spot to GUID-bearing results, and the O(n²) FileWispExecutionLog.ReadAllAsync full re-parse on every successful wisp, are left for separate PRs.

🤖 Generated with Claude Code

Follow-up to the trim-loop fix (#462). The per-agent-loop
RepetitiveToolCallDetector is the only existing guard against repeated
identical tool calls, but it is rebuilt fresh per AgentLoopRunner.RunAsync and
dies with that loop — so it never spans completion-eval reprompts, scheduled
re-fires, or message/A2A re-triggers. And because spawn_wisps results embed
fresh batch/wisp GUIDs, two byte-identical dispatches never look identical to
it anyway. Nothing counts "I have already run this exact wisp too many times —
stop."

WispDispatchCircuitBreaker is a process-wide singleton checked at the single
choke point every wisp flows through (SpawnWispsExecutor). It keeps a
fixed-window dispatch count keyed by the exact definition hash; once a
definition exceeds WispOptions.DispatchCircuitBreakerMaxPerWindow (default 30)
within DispatchCircuitBreakerWindow (default 5m), further dispatches of that
definition are refused until the window rolls over.

- Keyed on the exact definition hash, not the value-stripped shape hash, so
  wisps that vary by date/id never trip it — only a truly identical loop does.
- A fixed window keeps memory at O(distinct definitions) regardless of dispatch
  rate, which matters precisely in the runaway case.
- A tripped dispatch does NO work (no LLM, no tool calls) and is NOT written to
  the execution log, so a runaway collapses into a cheap, self-limiting error
  instead of an unbounded spin (and avoids re-bloating the JSONL log).
- The refusal message is intentionally stable (no counts/ids), so identical
  refusals also look identical to the agent-loop's repetitive-result detector,
  which then nudges the model off the loop.
- Emits a rockbot.wisp.circuit_breaker.trips metric for alerting.

Conservative by default and fully configurable; disabled-path and zero-limit
both short-circuit to always-allow. 7 new tests cover threshold, sustained
trip, window reset, per-definition independence, and the disabled paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant