Add a wisp dispatch circuit breaker for runaway re-dispatch loops#463
Merged
Conversation
Follow-up to the trim-loop fix (#462). The per-agent-loop RepetitiveToolCallDetector is the only existing guard against repeated identical tool calls, but it is rebuilt fresh per AgentLoopRunner.RunAsync and dies with that loop — so it never spans completion-eval reprompts, scheduled re-fires, or message/A2A re-triggers. And because spawn_wisps results embed fresh batch/wisp GUIDs, two byte-identical dispatches never look identical to it anyway. Nothing counts "I have already run this exact wisp too many times — stop." WispDispatchCircuitBreaker is a process-wide singleton checked at the single choke point every wisp flows through (SpawnWispsExecutor). It keeps a fixed-window dispatch count keyed by the exact definition hash; once a definition exceeds WispOptions.DispatchCircuitBreakerMaxPerWindow (default 30) within DispatchCircuitBreakerWindow (default 5m), further dispatches of that definition are refused until the window rolls over. - Keyed on the exact definition hash, not the value-stripped shape hash, so wisps that vary by date/id never trip it — only a truly identical loop does. - A fixed window keeps memory at O(distinct definitions) regardless of dispatch rate, which matters precisely in the runaway case. - A tripped dispatch does NO work (no LLM, no tool calls) and is NOT written to the execution log, so a runaway collapses into a cheap, self-limiting error instead of an unbounded spin (and avoids re-bloating the JSONL log). - The refusal message is intentionally stable (no counts/ids), so identical refusals also look identical to the agent-loop's repetitive-result detector, which then nudges the model off the loop. - Emits a rockbot.wisp.circuit_breaker.trips metric for alerting. Conservative by default and fully configurable; disabled-path and zero-limit both short-circuit to always-allow. 7 new tests cover threshold, sustained trip, window reset, per-definition independence, and the disabled paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Follow-up to #462 (the trim-loop runaway fix). That PR stopped the specific wedge; this one adds the guard that was missing one layer up: nothing counts "I've already run this exact wisp too many times — stop."
The only existing guard against repeated identical tool calls is the per-agent-loop
RepetitiveToolCallDetector. It has two gaps:AgentLoopRunner.RunAsyncand dies with that loop — so it never spans completion-eval reprompts, scheduled re-fires, or message/A2A re-triggers, which is exactly how a wisp gets re-dispatched across many short loops.spawn_wispsresults embed freshbatch-{Guid}/wisp-{Guid}IDs, so two byte-identical dispatches never look identical to a detector keyed ontool+args+result.SpawnWispsExecutoris the single choke point every wisp flows through, and a process-wide singleton there is the only guard that survives across loops, reprompts, and message-driven re-triggers.What
WispDispatchCircuitBreakerkeeps a fixed-window dispatch count keyed by the exact definition hash. Once a definition exceedsDispatchCircuitBreakerMaxPerWindow(default 30) withinDispatchCircuitBreakerWindow(default 5m), further dispatches of that definition are refused until the window rolls over.Design choices:
O(distinct definitions)regardless of dispatch rate — which matters precisely in the runaway case the breaker exists to contain. (A sliding window with per-attempt timestamps would balloon under a 400k/min loop.)rockbot.wisp.circuit_breaker.tripscounter for alerting on runaways.Conservative by default and fully configurable via
WispOptions; the disabled flag and a non-positive limit both short-circuit to always-allow.Tests
7 new tests (fake
TimeProvider): below/at threshold, sustained trip stays tripped, window reset re-allows, per-definition independence, disabled / zero-limit / empty-hash always-allow. FullRockBot.Wisp.Testssuite green (142/142). Agent composition root builds clean.Not included (still open from the incident write-up)
The
RepetitiveToolCallDetectorblind spot to GUID-bearing results, and the O(n²)FileWispExecutionLog.ReadAllAsyncfull re-parse on every successful wisp, are left for separate PRs.🤖 Generated with Claude Code