feat(testkit): mocking, tracing, and live debugging, surfaced as agent tool calls

[senamakel]

Why

A workflow that runs is not a workflow that works. The engine will happily execute a graph whose every binding resolved to null, whose agent node dispatched with an empty prompt, and whose failure was swallowed by an on_error policy — and report all of it as success, because each of those is a legal value rather than an error. The output of a broken run looks exactly like the output of a correct one.

What was missing was not execution. It was the means to interrogate an execution. Today the whole story is a RunObserver whose callbacks return (), a caps::mock suite of fixed echoes, and a Diagnosis stranded behind the store feature — so every test that needed more wrote its own AtomicUsize-counting double, and there are 15+ such one-offs in this repo's own suite.

Workflows here are written by agents as often as by people, and an agent that cannot debug what it wrote can only guess at why it failed.

What this adds

The engine seam — tinyflows::interception (always compiled, inert unless used)

A RunObserver can watch a run and never change one. A StepInterceptor returns a StepAction the engine obeys — which is what makes breakpoints and output overrides expressible at all. New entry point engine::run_intercepted.

The action vocabulary is deliberately small, and each variant lands the activation back on a path the engine already has: an injected failure enters the node's own on_error policy, a replaced output routes through the same port and lane logic real output does, and a substituted activation is still recorded as a step and reported to the observer. Nothing invents control flow beside the engine.

Placed at the one point a node executor runs, and nowhere else: after the cancellation check and approval gate, after input resolution, outside the retry loop (so a breakpoint fires once per activation, not once per attempt) and outside the per-attempt timeout race (so time parked at a breakpoint is never charged against node_timeout_secs).

Four layers on top — tinyflows::testkit (default-off testkit feature, no new dependencies)

• mocks — programmable, recording capability doubles. * globs, per-call sequences, injected failures, delays, schema-synthesized answers, per-node scoping. Every call lands in one log across all capabilities and is attributed to the node that made it.
• trace — each activation's input and output, plus every =-binding with the value it resolved to and, when it resolved to nothing, the upstream node it was reading from. That last field turns "it produced null" into a pointer at the node that should have produced the value.
• harness — TestHarness and named assertions, notably assert_no_null_bindings.
• debug — real breakpoints: pause before or after a node, inspect what it was about to receive, override its output, skip it, fail it, patch the run state, or single-step. Conditions cover on-error, the nth activation of a loop, and arbitrary =-expressions.
• tools — all of the above as 10 named tools with real JSON Schemas and a JSON-in/JSON-out TestkitRegistry::dispatch. tinyflows registers nothing and talks to no model — the same division catalog already draws for the node-kind contracts.

Supporting moves

diagnostics and evidence lifted out from behind the store feature (both are pure functions of engine records; a trace needs them as much as a durable record does), and caps::sample_for_schema out from behind host-caps (the auto-mock shouldn't have to pull in a process runner and an HTTP client to reach 35 dependency-free lines). Both re-exported from their old homes, so no downstream caller breaks.

build_graph/build_and_run now take a private RunConfig params struct instead of 8–9 positional arguments. This removes four #[allow(clippy::too_many_arguments)] and leaves every public signature untouched — a net reduction, and the reason adding the interceptor cost one field rather than fourteen signature edits.

A bug fixed on the way

A node that failed once and then succeeded on retry was reported as failed to observers of an activation's settled state. The engine's retry loop keeps the last failed attempt's error even after a later attempt succeeds; surfacing it unconditionally showed a recovered node as a failed one — and would have fired every on-error breakpoint on it. Fixed, with a regression test.

Design note: why breakpoints are in-process

The engine already knows how to pause — a requires_approval gate raises a real interrupt, checkpoints, and waits. That mechanism is built for waiting on a person, so it ends the run and resumes later by re-running the interrupted node from the top.

A breakpoint needs three things that path structurally cannot do: break after a node (resuming re-runs it, firing side effects twice), override what a node produced (needs the activation still on the stack), and be driven from another task (so an agent can inspect and step across separate tool calls).

So a breakpoint parks the activation in place and a DebugSession owns the run. The honest cost, stated in the docs: a session lives in one process and dies with it. PauseMode::Durable covers the one case where surviving a restart matters more — a break before a node, where nothing has run and the re-run is free — and is refused at registration for after-breakpoints rather than silently doubling side effects.

A paused run cannot wedge. Four independent releases, any one of which frees it: a pause timeout (5 min default), detach(), a dropped release channel, and dropping the session (which detaches → cancels → aborts, in that order). All four are tested. The controller uses a std::sync::Mutex deliberately: its guard is !Send, so holding the lock across the pause await fails to compile rather than being caught in review.

Verification

• 1312 tests pass, 0 failures (102 new). Clippy clean on --all-targets --all-features, cargo fmt --check clean, cargo publish --dry-run clean.
• All 8 feature combinations compile: default, store, host-caps, mock, testkit, graph-debug, chrome-extension, --all-features, --no-default-features.
• The no-cost property is a property test, not a claim. tests/fuzz_interception.rs asserts over generated graphs that a plain run and a run with an inert interceptor produce identical output — and that inspecting every frame, and tracing, are equally free of side effects.
• tests/interception_e2e.rs asserts an inert interceptor leaves both the outcome and the full observer step record byte-identical, then covers each action.
• A full debug session is driven entirely over JSON in registry_tests, asserting only on JSON — which is exactly what an agent sees.

Not included

• recording.rs — record-against-real-capabilities then replay. Respond::Passthrough is not implemented, so the fixture/pinning story is missing.
• protocol/testkit-v1.schema.json — deliberately skipped. all_tools() already returns real JSON Schemas generated in Rust and tested for closedness; a hand-maintained parallel file would duplicate that and drift. Happy to add one if cross-language consumers need it.
• Item-level breakpoints inside map_items — node-level covers the debugging need, and pausing one item holds a buffer_unordered slot with a subtle interaction with ItemErrorPolicy::FailFast's lowest-index cancellation. Left out rather than rushed.

The ADR in local/docs/11-decisions.md could not be written — local/ is gitignored and not present in this worktree.

[coderabbitai]

Warning

Review limit reached

@senamakel, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 21 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 7deb864a-a4ae-4f74-9efa-1586f88b82e0

📥 Commits

Reviewing files that changed from the base of the PR and between cd39220 and e38ec7e.

📒 Files selected for processing (48)

• CHANGELOG.md
• CLAUDE.md
• Cargo.toml
• README.md
• src/caps/host/mocks.rs
• src/caps/mod.rs
• src/caps/schema.rs
• src/diagnostics.rs
• src/diagnostics_tests.rs
• src/engine.rs
• src/engine/api.rs
• src/engine/build.rs
• src/engine/build/activation.rs
• src/engine/build/handlers.rs
• src/engine/resumable.rs
• src/engine/run_config.rs
• src/engine/run_state.rs
• src/evidence.rs
• src/interception.rs
• src/lib.rs
• src/store/types/mod.rs
• src/store/types/proposal.rs
• src/store/types/run.rs
• src/store/types/types_tests.rs
• src/testkit/debug/breakpoint.rs
• src/testkit/debug/breakpoint_tests.rs
• src/testkit/debug/controller.rs
• src/testkit/debug/controller_tests.rs
• src/testkit/debug/mod.rs
• src/testkit/debug/session.rs
• src/testkit/debug/session_tests.rs
• src/testkit/harness.rs
• src/testkit/harness_tests.rs
• src/testkit/mocks.rs
• src/testkit/mocks_tests.rs
• src/testkit/mod.rs
• src/testkit/tools/contracts.rs
• src/testkit/tools/contracts_tests.rs
• src/testkit/tools/error.rs
• src/testkit/tools/mod.rs
• src/testkit/tools/registry.rs
• src/testkit/tools/registry_tests.rs
• src/testkit/trace.rs
• src/testkit/trace_tests.rs
• tests/fuzz_interception.rs
• tests/interception_e2e.rs
• wiki/Testing-and-Debugging.md
• wiki/_Sidebar.md

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}