feat(testkit): mocking, tracing, and live debugging, surfaced as agent tool calls

CHANGELOG.md

@@ -7,6 +7,67 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- **`tinyflows::testkit` — testing, mocking, and live debugging for workflows.**
+  Behind the default-off `testkit` feature; adds no dependencies.
+
+  The problem it addresses: the engine will happily run a graph whose every
+  binding resolved to `null`, whose agent node dispatched with an empty prompt,
+  and whose failure was swallowed by an `on_error` policy — and report all of it
+  as success, because each of those is a legal value rather than an error. What
+  was missing was not execution but the means to interrogate an execution.
+
+  - `testkit::mocks` — programmable, recording capability doubles.
+    `MockCaps::new().on_tool("slack.send", Respond::value(…))`, with `*` globs,
+    per-call sequences (`Respond::sequence`), injected failures, delays,
+    schema-synthesized answers, and per-node scoping (`only_from`). Every call
+    is logged in one sequence across all capabilities and attributed to the node
+    that made it.
+  - `testkit::trace` — a structured run record: each activation's input *and*
+    output, every `=`-binding with the value it resolved to, and — when it
+    resolved to nothing — the upstream node it was reading from. That last part
+    turns "it produced null" into a pointer at the node that should have
+    produced it.
+  - `testkit::harness` — `TestHarness` plus named assertions, including
+    `assert_no_null_bindings`, which catches the failure a green run hides.
+  - `testkit::debug` — real breakpoints. Pause before or after a node, inspect
+    what it was about to receive, override its output, skip it, fail it, patch
+    the run state, or single-step. Conditions cover `on_error`, the nth
+    activation of a loop, and arbitrary `=`-expressions. A `DebugSession` owns
+    the spawned run, so it can be driven from another task.
+  - `testkit::tools` — every one of the above as a named tool with a real JSON
+    Schema and a JSON-in/JSON-out `TestkitRegistry::dispatch`, so a host can
+    hand the whole module to an agent without writing an adapter. tinyflows
+    registers nothing and talks to no model; these are descriptors and handlers,
+    the same division `catalog` already draws for node kinds.
+
+- **`tinyflows::interception` — the engine's one execution-*gating* hook.**
+  Always compiled, and inert unless used. A `RunObserver`'s callbacks return
+  `()`, so it can watch a run and never change one; a `StepInterceptor` returns
+  a `StepAction` the engine obeys, which is what makes breakpoints and output
+  overrides expressible at all. New entry point `engine::run_intercepted`. With
+  no interceptor attached the engine builds no `StepFrame` and makes no call —
+  asserted by a property test over generated graphs, not only by inspection.
+
+- **`tinyflows::diagnostics` and `tinyflows::evidence`** — `Diagnosis`,
+  `diagnose`, and the evidence-bounding helpers moved out from behind the
+  `store` feature. Both are pure functions of engine records, and a trace or a
+  tool reply needs them as much as a durable record does. Re-exported from
+  `store::types`, so no downstream caller breaks.
+
+- **`caps::sample_for_schema`** is no longer behind `host-caps`; the auto-mock
+  needs it and should not have to pull in a process runner and an HTTP client to
+  get it. Still re-exported from `caps::host::mocks`.
+
+### Fixed
+
+- A node that failed once and then **succeeded on retry** is no longer reported
+  as failed to observers of an activation's settled state. The engine's retry
+  loop keeps the last failed attempt's error even after a later attempt
+  succeeds; surfacing it unconditionally showed a recovered node as a failed one
+  and would have fired every on-error breakpoint on it.
+
 ### Changed
 
 - **Breaking: the Chrome companion moved behind the `chrome-extension`

CLAUDE.md

@@ -52,6 +52,21 @@ model::WorkflowGraph  →  validate  →  compiler::compile  →  engine::run
   superstep executor, channels, checkpointing, interrupts, event journal).
   Vendored out of `tinyagents` and trimmed: agents themselves are a host concern
   reached through `caps`, so the crate carries no agent-harness dependency.
+- `interception.rs` — the engine's one execution-*gating* hook. A `RunObserver`
+  watches a run; a `StepInterceptor` can change one (substitute a node's output,
+  inject a failure, patch the state it reads, park the activation). Always
+  compiled, inert unless a run is started with `engine::run_intercepted`.
+- `testkit/` — testing, mocking, and live debugging, behind the default-off
+  `testkit` feature: programmable capability doubles with a call log (`mocks`),
+  a structured run trace that names the upstream node behind every null binding
+  (`trace`), `TestHarness` and its assertions (`harness`), breakpoints and debug
+  sessions (`debug`), and all of it as agent-callable tools with JSON Schemas
+  (`tools`). Built entirely on `interception` — nothing in it is special-cased
+  inside the engine.
+- `diagnostics.rs` / `evidence.rs` — reading a run's steps for what a green
+  outcome hides, and bounding what gets handed back. Pure functions of engine
+  records, so neither sits behind a feature; both are re-exported from
+  `store::types` for callers that always reached them there.
 - `error.rs` — shared error types across validate/compile/execute (thiserror).
 - `lib.rs` — crate surface + module declarations; `main.rs` — thin binary stub.
 

Cargo.toml

@@ -77,6 +77,14 @@ host-caps = ["dep:reqwest", "dep:sha2", "dep:tempfile", "reqwest/rustls", "tokio
 store = ["dep:sha2", "dep:fs2"]
 # A file-writing graph visualizer intended for development and diagnostics.
 graph-debug = ["dep:image", "dep:font8x8"]
+# Off by default; enables `testkit` — programmable capability mocks, a
+# structured run trace, and live step-debugging (breakpoints, inspection,
+# output override) driven from another task. Adds no dependencies: tokio,
+# futures-timer, async-trait and jaq are already required by the engine. The
+# engine's interception seam itself is always compiled and inert; this feature
+# only switches on the implementation that uses it. Always available inside
+# this crate's own tests, like `store` and `mock`.
+testkit = []
 
 [dev-dependencies]
 proptest = "1"

README.md

@@ -198,6 +198,101 @@ Omitting `--features mock` is harmless: the demo body is
 `#[cfg(feature = "mock")]`-gated, so a default build stays green and the example
 just prints a hint to re-run with the feature enabled.
 
+### Testing and debugging workflows
+
+Enable `testkit` for programmable mocks, a structured run trace, and real
+breakpoints:
+
+```toml
+tinyflows = { version = "0.8", features = ["testkit"] }
+```
+
+The problem it exists for: a workflow that runs *green* and does nothing. Every
+node ran, nothing errored, the output is an object — and a binding read from a
+field no node produces, resolved to `null`, and sent an empty value onward. Null
+is a legal value, so the engine has no complaint.
+
+```rust,no_run
+# async fn example(graph: tinyflows::model::WorkflowGraph) -> tinyflows::error::Result<()> {
+use tinyflows::testkit::{Respond, TestHarness};
+use serde_json::json;
+
+let run = TestHarness::new(&graph)
+    .trigger(json!({ "repo": "acme/api" }))
+    .mock_tool("slack.send", Respond::value(json!({ "ok": true })))
+    // First call rate-limits, the retry succeeds — a flaky dependency
+    // without a flaky test.
+    .mock_tool("gh.issues.*", Respond::sequence([
+        Respond::error("429 rate limited"),
+        Respond::value(json!({ "number": 7 })),
+    ]))
+    .run()
+    .await?;
+
+run.assert_completed();
+run.assert_node_ran("send_email");
+run.assert_no_null_bindings();   // the check a green run hides
+run.assert_call_count("tools", Some("slack.send"), 1);
+# Ok(())
+# }
+```
+
+A failing `assert_no_null_bindings` names the binding *and* the upstream node it
+was reading from, so it points at the node that should have produced the value
+rather than only at the one that went without it.
+
+Breakpoints pause a live run so another task can look at it and change it:
+
+```rust,no_run
+# async fn example(compiled: tinyflows::compiler::CompiledWorkflow) -> tinyflows::error::Result<()> {
+use std::time::Duration;
+use tinyflows::caps::mock::mock_capabilities;
+use tinyflows::testkit::debug::{BreakpointSpec, DebugCommand, DebugSession};
+use serde_json::json;
+
+let mut session = DebugSession::start_quiet(compiled, json!({}), mock_capabilities())?;
+session.controller().set_breakpoint(BreakpointSpec::before("send_email"))?;
+
+if let Some(pause) = session.next_pause(Duration::from_secs(5)).await {
+    println!("about to run with: {:?}", pause.input);
+    println!("empty bindings: {:?}", pause.null_bindings);
+    session.controller().release(pause.pause_id, DebugCommand::Continue)?;
+}
+session.finish().await?;
+# Ok(())
+# }
+```
+
+A paused run cannot wedge: a pause times out, detaching releases it, and
+dropping the session winds it down.
+
+#### For agents
+
+Workflows here are written by agents as often as by people, and an agent that
+cannot debug what it wrote can only guess at why it failed. `testkit::tools`
+exposes all of the above as named tools with real JSON Schemas and a
+JSON-in/JSON-out dispatcher, so a host can hand the whole module to an agent
+without writing an adapter:
+
+```rust,no_run
+# async fn example() -> Result<(), tinyflows::testkit::tools::ToolError> {
+use tinyflows::testkit::tools::{TestkitRegistry, all_tools};
+use serde_json::json;
+
+for tool in all_tools() {
+    println!("{} — {}", tool.name, tool.summary);
+}
+
+let registry = TestkitRegistry::new();
+let result = registry.dispatch("flow_test.run", json!({ "graph": { /* … */ } })).await?;
+println!("{}", result["nullBindings"]);
+# Ok(())
+# }
+```
+
+tinyflows registers nothing and talks to no model — the host owns registration,
+the same division `catalog` already draws for the node-kind contracts.
+
 ### Visual graph debugging
 
 Enable `graph-debug` to render a workflow's nodes, port-labelled edges, branch

src/caps/host/mocks.rs

@@ -15,47 +15,11 @@ use crate::error::Result;
 use async_trait::async_trait;
 use serde_json::{Value, json};
 
-/// Synthesize a value satisfying a JSON Schema well enough to pass validation.
-///
-/// Deliberately shallow — it honours `type`, `properties`, `required`, and
-/// `enum`, which is what node schemas in practice use. Anything it does not
-/// understand becomes null, and a schema strict enough to reject that is a
-/// schema whose graph deserves a real run before being trusted.
-pub fn sample_for_schema(schema: &Value) -> Value {
-    let Some(object) = schema.as_object() else {
-        return Value::Null;
-    };
-    if let Some(first) = object
-        .get("enum")
-        .and_then(Value::as_array)
-        .and_then(|v| v.first())
-    {
-        return first.clone();
-    }
-    match object.get("type").and_then(Value::as_str) {
-        Some("object") => {
-            let mut out = serde_json::Map::new();
-            if let Some(properties) = object.get("properties").and_then(Value::as_object) {
-                // Every declared property, not only the required ones: a graph
-                // binding `=item.json.optional_field` should still resolve.
-                for (name, property) in properties {
-                    out.insert(name.clone(), sample_for_schema(property));
-                }
-            }
-            Value::Object(out)
-        }
-        Some("array") => match object.get("items") {
-            // One element, so a downstream `per_item` node has something to map
-            // over and a `[0]` expression resolves.
-            Some(items) => json!([sample_for_schema(items)]),
-            None => json!([]),
-        },
-        Some("string") => json!("sample"),
-        Some("integer") | Some("number") => json!(0),
-        Some("boolean") => json!(false),
-        _ => Value::Null,
-    }
-}
+// The sample synthesizer moved to [`crate::caps::schema`]: the `testkit`
+// auto-mock needs the same function and should not have to enable `host-caps`
+// — and so pull in a process runner and an HTTP client — to reach it.
+// Re-exported here because this is where callers have always found it.
+pub use crate::caps::sample_for_schema;
 
 /// The `output_parser.schema` a request declares, if any.
 fn declared_schema(request: &Value) -> Option<&Value> {

src/caps/mod.rs

@@ -10,6 +10,7 @@ pub mod agent;
 pub mod host;
 #[cfg(any(test, feature = "mock"))]
 pub mod mock;
+pub mod schema;
 pub mod shell;
 pub mod tasks;
 
@@ -24,6 +25,7 @@ pub use self::agent::{
     AgentInput, AgentModelSelection, AgentRunIdentity, AgentRunOutcome, AgentRunRequest,
     AgentRunner, AgentUsage, ContextBlock, StopReason, ToolDescriptor,
 };
+pub use self::schema::sample_for_schema;
 pub use self::shell::{ShellInterpreter, ShellOutcome, ShellRequest, ShellRunner, ShellScript};
 pub use self::tasks::{TaskRunner, TaskSpec, TaskState, TokioTaskRunner};
 

src/caps/schema.rs

@@ -0,0 +1,67 @@
+//! Synthesizing a value that satisfies a declared output schema.
+//!
+//! A node that declares an `output_parser.schema` says what its result must
+//! look like. Anything standing in for that node's real capability — a dry-run
+//! mock, a test double, an auto-mocked `testkit` run — has to produce something
+//! that shape, or the graph fails validation for a reason that has nothing to
+//! do with the graph.
+//!
+//! This lives in [`crate::caps`] rather than beside any one set of mocks
+//! because more than one of them needs it and none of them should have to pull
+//! in a process runner or an HTTP client to get it.
+
+use serde_json::{Value, json};
+
+/// Synthesize a value satisfying a JSON Schema well enough to pass validation.
+///
+/// Deliberately shallow — it honours `type`, `properties`, `required`, and
+/// `enum`, which is what node schemas in practice use. Anything it does not
+/// understand becomes null, and a schema strict enough to reject that is a
+/// schema whose graph deserves a real run before being trusted.
+///
+/// ```
+/// use tinyflows::caps::sample_for_schema;
+/// use serde_json::json;
+///
+/// let sample = sample_for_schema(&json!({
+///     "type": "object",
+///     "properties": { "name": { "type": "string" }, "count": { "type": "integer" } }
+/// }));
+/// assert_eq!(sample, json!({ "name": "sample", "count": 0 }));
+/// ```
+#[must_use]
+pub fn sample_for_schema(schema: &Value) -> Value {
+    let Some(object) = schema.as_object() else {
+        return Value::Null;
+    };
+    if let Some(first) = object
+        .get("enum")
+        .and_then(Value::as_array)
+        .and_then(|v| v.first())
+    {
+        return first.clone();
+    }
+    match object.get("type").and_then(Value::as_str) {
+        Some("object") => {
+            let mut out = serde_json::Map::new();
+            if let Some(properties) = object.get("properties").and_then(Value::as_object) {
+                // Every declared property, not only the required ones: a graph
+                // binding `=item.json.optional_field` should still resolve.
+                for (name, property) in properties {
+                    out.insert(name.clone(), sample_for_schema(property));
+                }
+            }
+            Value::Object(out)
+        }
+        Some("array") => match object.get("items") {
+            // One element, so a downstream `per_item` node has something to map
+            // over and a `[0]` expression resolves.
+            Some(items) => json!([sample_for_schema(items)]),
+            None => json!([]),
+        },
+        Some("string") => json!("sample"),
+        Some("integer") | Some("number") => json!(0),
+        Some("boolean") => json!(false),
+        _ => Value::Null,
+    }
+}

src/store/types/diagnosis.rs → src/diagnostics.rs

@@ -300,5 +300,5 @@ pub struct DryRun {
 }
 
 #[cfg(test)]
-#[path = "diagnosis_tests.rs"]
+#[path = "diagnostics_tests.rs"]
 mod tests;

src/engine.rs

@@ -58,6 +58,7 @@ use crate::caps::Capabilities;
 use crate::compiler::CompiledWorkflow;
 use crate::data::Item;
 use crate::error::{EngineError, Result, ValidationError};
+use crate::interception::{StepAction, StepFrame, StepInterceptor, StepPhase};
 use crate::model::NodeKind;
 use crate::nodes::{NodeContext, executor_for};
 use crate::observability::{ExecutionStep, Run, RunObserver, RunStatus, StepStatus};
@@ -250,6 +251,9 @@ pub use api::{
 mod build;
 use build::build_graph;
 
+mod run_config;
+use run_config::RunConfig;
+
 mod run_state;
 pub use run_state::resume;
 use run_state::{build_and_run, default_thread_id, merge_approvals};