Skip to content

feat(engine): scatter/gather lanes, async spawn/gate, stateful loops - #47

Merged
senamakel merged 177 commits into
mainfrom
parallel-gates-loops
Aug 13, 2026
Merged

feat(engine): scatter/gather lanes, async spawn/gate, stateful loops#47
senamakel merged 177 commits into
mainfrom
parallel-gates-loops

Conversation

@senamakel

Copy link
Copy Markdown
Member

Adds massive parallelization, async gates, and stateful loops — plus four fixes to the graph runtime that this work uncovered, two of which are latent bugs on main today.

1161 tests pass; clippy, fmt, all four feature configs, and cargo publish --dry-run clean.

The new capability

scatter / gather — fans out the downstream path, not just immediate successors. Drawing two edges from one port already runs both successors concurrently; what that cannot express is running the same pipeline several times over different data. scatter → enrich → score → gather over 8 items becomes 8 concurrent three-node pipelines.

Lowered onto RouteTarget::Send, which is the only mechanism that can do it — plain activations are deduplicated by node id (graph/compiled/routing.rs:63), so repeating a target collapses back to one. Lane state lives in lane-keyed sub-slots (nodes.<id>.lanes.<lane>) written by a single constructor; a lane activation never touches the node's top-level slot, which is what stops N concurrent activations of one node clobbering each other under a key-by-key reducer.

spawn / gate — start work without blocking, collect it later on a release policy (all / any / first_n / quorum / timeout_partial). Backed by a new TaskRunner capability with a tokio-backed default, so overlap works out of the box; with no runner injected spawn runs inline and the graph still computes the right answer, just without the concurrency.

Loop accumulatorstate.init / state.update make a loop a fold, until tests the post-fold accumulator, and exit_reason finally lets downstream tell a loop that converged from one that ran out of tries.

Concurrency dialstrigger.max_concurrency and trigger.max_item_concurrency, plus per-node bounds. Documented as admission control, not backpressure: a super-step engine cannot block a producer mid-step.

Cross-boundary approval — a sub-workflow child paused at a gate now pauses the parent instead of failing it (the sub_workflow.rs TODO).

Runtime fixes (src/graph/)

Two are latent bugs on main, independent of this feature work:

  1. Barrier relief could clear a barrier before its predecessors ran. reaches_deterministically walked a single-successor edge map, so a fan-out stopped the walk and relief concluded "branch untaken" — firing a phantom arrival. A merge downstream of condition → fan-out silently dropped both branches' data. The existing doc called stopping there "conservative"; it is the opposite, since reporting unreachable is what fires relief. Regression test verified to fail with the fix disabled.
  2. Resume re-ran branches that had already completed. Under parallel execution the whole active set runs before anything is folded, so "after the interrupt" ≠ "did not run". A sibling of an interrupted gate fired its side effects twice; the regression test counts capability invocations and shows 2 without the fix.

Plus: a barrier now ignores arrivals from predecessors outside its required set (this legalizes a loop head that is also a fan-in), and per-node concurrency bounds.

Notable decisions

  • Two validator refusals lifted, one deliberately kept. "Loop head is a fan-in" and "merge on a cycle" are now legal. A fully-untaken barrier is still not relieved — it would otherwise fire on empty input and, inside a loop, ping-pong forever.
  • The loop fold is at-least-once and documented as such, not hidden. A replayed activation folds twice. This is not new — iteration + 1 has always behaved this way — the accumulator just makes it visible.
  • $replace sentinel in the reducer, with the soundness argument as a test: merge never walks into an items array, so user data containing a $replace key cannot trigger it.
  • v1 scatter refuses nested scatter, a loop head, or requires_approval inside a lane — each with its own message. The approval case matters: a resume is addressed by node id, so every lane would share one approval.

Testing

New property-test layer (tests/support/graphgen.rs) generating graphs from a shape grammar rather than random edges — random edges are almost all rejected by the validator and never reach the engine. Both fuzz files carry a non-vacuity guard pinning generator yield, because the first version passed in 0.18s while doing almost nothing.

Tests assert the thing that actually matters rather than just output: overlap is measured as observed concurrency, ordering via RunObserver activation traces, and double-execution via capability invocation counts — none of which show up in final state.

Remaining test work is tracked in #46.

senamakel and others added 30 commits August 13, 2026 22:05
Removed the unused `std::collections::HashMap` import from the module file to keep the codebase clean and avoid compiler warnings.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Removed the unused `std::collections::HashMap` import from the module file to keep the codebase clean and avoid compiler warnings.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Removed the unused `std::collections::HashMap` import from the module file to keep the codebase clean and avoid compiler warnings.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `use std::collections::HashMap;` import in `src/engine.rs` was not referenced anywhere in the code, so it has been removed to keep the source clean and avoid compiler warnings.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `use std::collections::HashMap;` import in `src/engine.rs` is no longer needed after recent refactoring, so it has been removed to keep the codebase clean and avoid compiler warnings.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `use std::collections::HashMap;` import in `src/engine.rs` is no longer needed, as the code no longer references `HashMap` anywhere in the file. Removing it cleans up the module and avoids compiler warnings about unused imports.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `use std::collections::HashMap;` import in `src/engine.rs` is no longer needed after recent refactoring, so it has been removed to keep the codebase clean and avoid compiler warnings.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `use std::collections::HashMap;` import in `src/engine.rs` is no longer needed after recent refactoring, so it has been removed to keep the codebase clean and avoid compiler warnings.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The execution context struct gained new `lane` and `step` fields, so all test helpers across the node modules now populate these fields when constructing contexts. This keeps the test code compiling against the updated context type and ensures tests exercise the new fields with default values.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `use std::collections::HashMap;` import in `src/engine.rs` is no longer needed after the recent refactoring, so it has been removed to keep the codebase clean and avoid compiler warnings.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new support module in the test suite to house common helper functions and setup logic, reducing duplication across test files and providing a single place for future shared test infrastructure.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
This change introduces a new support module for tests that provides shared utilities for generating test data. The module centralizes common generation logic to reduce duplication across test files and make future test additions more straightforward.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
This change introduces a new support module for tests that provides shared utilities for generating test data. The module centralizes common generation logic to reduce duplication across test files and make future test additions more straightforward.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
This change introduces a new fuzz test targeting the graph module to improve robustness against unexpected inputs. The test exercises the module with randomized data to uncover potential panics or logic errors.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Renamed the test support module from `gen` to `graphgen` to better reflect its purpose of generating graph structures for fuzz testing, and updated the corresponding import in the fuzz test file.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
This change introduces a new test file for diagnostic generation, providing coverage for the diagnostic output functionality. The test file is currently untracked and contains no diff content, so this commit simply adds it to the repository.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Added the missing `ValueTree` import from `proptest::strategy` to the diagnostic generation test file, enabling the test to properly use proptest's value tree API for generating and shrinking test cases.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The mock-gated diagnostic generator test was removed because it relied on the `mock` feature and the `graphgen` support module, which are no longer part of the codebase. The test's purpose of validating compiler diagnostics is now covered by other integration tests, making this file redundant.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Adds a new fuzz target for the graph module to exercise the code with random inputs, helping to uncover edge cases and potential panics that might not be hit by regular unit tests.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new support module that provides graph generation utilities for use in tests. This adds a reusable helper to create graph structures, simplifying test setup and improving consistency across test cases.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new support module that provides graph generation utilities for use in tests. This adds a reusable helper to create graph structures, making it easier to set up test scenarios without duplicating setup code.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new support module for generating graphs in tests, providing reusable utilities to create graph structures for test scenarios. This simplifies test setup and reduces duplication across test cases.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new support module that provides graph generation utilities for use in tests. This adds a reusable helper to create graph structures, enabling more comprehensive and consistent test coverage across the test suite.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a new support module for generating graphs in tests, providing a reusable utility to create graph structures for test scenarios. This simplifies test setup and reduces duplication across test cases.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new support module that provides graph generation utilities for use in tests. This adds a reusable helper to create graph structures, simplifying test setup and improving consistency across test cases.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a fuzz test targeting the resume feature to exercise it with arbitrary inputs, helping to uncover edge cases and potential panics that might not be covered by regular unit tests.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Adds a regression test case discovered by fuzzing to the resume test suite, ensuring the previously failing input is covered and the fix remains stable.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a fuzz test targeting the resume feature to exercise it with arbitrary inputs, helping to uncover edge cases and potential panics that might not be covered by existing unit tests.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a new fuzz test targeting the resume feature to exercise it with random inputs, improving coverage and robustness of the resume code path.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a new fuzz test targeting the resume feature to exercise it with unexpected inputs. This helps uncover edge cases and potential panics in the resume code path that might not be hit by regular unit tests.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
senamakel and others added 25 commits August 13, 2026 23:31
Add a new `NodeControl::Scatter` variant and a corresponding `NodeOutput::scatter` constructor that fans the entire downstream path out into parallel copies, one per lane, each carrying its own slice of work. This differs from a regular fan-out by duplicating the whole downstream pipeline per lane rather than running each successor once, enabling reproducible parallel execution with ordered emission by lane index.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce three helper functions that construct and read lane-specific state updates and envelopes. These helpers centralize the structural rule that lane activations write only under their own lane key, never touching the slot's top-level items or port, which keeps concurrent activations collision-free without runtime checks.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Lane activations were incorrectly reading predecessor slots, causing all N lanes in a super-step to receive identical input items. The change now passes the lane's own send argument directly to `lane_input` when a lane activation is detected, ensuring each branch processes its distinct work instead of the shared committed snapshot.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Inside a lane, routing now carries the lane identity onward by re-scheduling every successor as a `Send` that holds the activation's output and the same lane, so the downstream path runs once per lane. Gather nodes are the exception: they are scheduled as plain activations, which dedupe by node, so multiple lanes converge on a single gather activation instead of triggering it repeatedly.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The lane envelope for routed sends now receives the full routed item list instead of relying on a precomputed lane-specific value. This ensures the envelope accurately reflects the actual items being routed through the lane, fixing cases where the previous approach produced incorrect or incomplete envelope data.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The emit calls in the error handling paths now include the items being emitted, so downstream handlers can access the actual data rather than only the update metadata. This ensures error routing and continue policies receive the full item context they need to operate correctly.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The build_graph function now precomputes the list of plain routing targets and the set of gather nodes for each node. This metadata is needed inside lanes, where plain routing must explicitly re-schedule every successor as a Send, and where gather nodes serve as convergence points for lane activations. The values are cloned into the closure so they are available during graph construction.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Added the missing RouteTarget type to the import list from the graph module, which is now required for the engine's routing logic.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new `route` command that can target a mixed set of plain activations and send packets, enabling fan-out to successors of different kinds in a single node emission. Add `Scatter` and `Gather` node kinds to support parallel downstream pipelines, where a scatter fans the path into concurrent lanes that converge on a gather.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The engine now handles the `Scatter` node control by opening a separate lane for each entry, scheduling every successor once per lane with its own envelope. This is necessary because plain activations dedupe by node id, so repeating a target would collapse back to one; the scatter's slot records the lane count so a gather can count arrivals accurately.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The lane envelope lookup can now fail without aborting the entire graph build. Instead of propagating the error, a null value is used as a fallback so routing can continue for targets that lack an envelope.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The scatter node previously emitted outputs in a non-deterministic order due to hash map iteration. This change restores the original deterministic ordering based on the scatter dimension, ensuring consistent output sequences across runs.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The gather node previously emitted its outputs in a non-deterministic order, which could cause subtle inconsistencies in downstream consumers. This change restores the original, stable ordering of gathered outputs to ensure predictable behavior across executions.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Adds the gather and scatter node modules to the control flow module's public declarations, making them available for use elsewhere in the crate.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Expose the GatherNode and ScatterNode types from the control_flow module and register their executors in the node dispatch table, making these control-flow primitives available for use in node graphs.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The conditional-edge router now passes a slice created with `std::slice::from_ref` instead of an array reference when emitting an error update. This avoids an unnecessary temporary array allocation and aligns the call with the slice type expected by `items_update`, simplifying the code without changing behavior.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
This change introduces a new end-to-end test file for the scatter/gather functionality, verifying that data is correctly distributed and collected across multiple operations. The test exercises the full workflow to ensure the feature behaves as expected in a realistic scenario.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Lane activations were incorrectly stamping the node's top-level slot, which could cause a loop head to mistake a stale arm for its own re-entry. Only non-lane activations now stamp the slot, and lane activations write to their own lane slot instead of the node's top-level one.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The lane update and envelope functions now return `crate::graph::Result<Value>` instead of the engine-specific error type, and the explicit `map_err` calls for serialization failures are removed since the graph result type already handles capability errors. This simplifies the error handling and aligns the return types with the graph module's conventions.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add validation rules that ensure every scatter region is properly formed: each gather must be downstream of a scatter, every scatter must have a gather to collect its lanes, and lane bodies cannot contain nested scatters, loops, or approval gates. The checks also verify that no edge leaves a lane region without passing through its gather, since a stranded lane envelope would silently produce incorrect results rather than a runtime failure.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add end-to-end tests covering the validation rules that guard the scatter/gather lane region: a lane escaping the region, a scatter without a gather, a gather without a scatter, and unsupported members like nested scatters and approval gates. These tests ensure each invalid configuration is refused with a specific reason rather than producing a subtly incorrect run.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The validation previously flagged only direct edges leaving a lane region, but a member could still strand its output by routing through intermediate nodes before exiting. It now checks whether any path from the member reaches a gather, catching all cases where lane results would be invisible and refusing them as errors.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Adds catalog contracts for the new scatter and gather node kinds, which enable parallel lane-based fan-out and collection in the DSL. Scatter runs the downstream pipeline once per lane over an array path or input items, while gather collects lane results with configurable release policies and error handling. Also updates the node kind registry and related tests to include the new kinds.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The scatter and gather nodes are only meaningful when used together, so the smoke test now exercises the full scatter-gather workflow rather than testing either node in isolation. This change also includes minor formatting adjustments across the affected source files and tests.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The scatter node now stores the lane count in its own output slot rather than duplicating it in the scatter metadata. This ensures the gather node counts arrivals against the correct total, preventing it from releasing prematurely when only the first lane has reported.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@senamakel, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 48 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 89175ae5-1d55-4c39-a4b6-c5756366fdbd

📥 Commits

Reviewing files that changed from the base of the PR and between edfe89f and 4611964.

📒 Files selected for processing (53)
  • src/caps/mock.rs
  • src/caps/mod.rs
  • src/caps/tasks.rs
  • src/catalog.rs
  • src/engine.rs
  • src/graph/builder/mod.rs
  • src/graph/builder/types.rs
  • src/graph/command/mod.rs
  • src/graph/compiled/executor.rs
  • src/graph/compiled/mod.rs
  • src/graph/compiled/routing.rs
  • src/graph/compiled/test.rs
  • src/graph/compiled/types.rs
  • src/main.rs
  • src/model/node_kind.rs
  • src/nodes/control_flow/condition.rs
  • src/nodes/control_flow/dedup.rs
  • src/nodes/control_flow/gather.rs
  • src/nodes/control_flow/loop_node.rs
  • src/nodes/control_flow/merge.rs
  • src/nodes/control_flow/mod.rs
  • src/nodes/control_flow/scatter.rs
  • src/nodes/control_flow/split_out.rs
  • src/nodes/control_flow/switch.rs
  • src/nodes/control_flow/transform.rs
  • src/nodes/integration/agent.rs
  • src/nodes/integration/code.rs
  • src/nodes/integration/gate.rs
  • src/nodes/integration/http_request.rs
  • src/nodes/integration/memory.rs
  • src/nodes/integration/mod.rs
  • src/nodes/integration/output_parser.rs
  • src/nodes/integration/shell_tests.rs
  • src/nodes/integration/spawn.rs
  • src/nodes/integration/sub_workflow.rs
  • src/nodes/integration/tool_call.rs
  • src/nodes/map.rs
  • src/nodes/mod.rs
  • src/nodes/release.rs
  • src/validate.rs
  • src/visualization.rs
  • tests/async_gates_e2e.rs
  • tests/eng5_barrier_relief_e2e.rs
  • tests/fuzz_async.rs
  • tests/fuzz_graph.rs
  • tests/fuzz_resume.rs
  • tests/hitl_e2e.rs
  • tests/loop_e2e.rs
  • tests/parallel_e2e.rs
  • tests/scatter_gather_e2e.rs
  • tests/smoke_all_nodes.rs
  • tests/support/graphgen.rs
  • tests/support/mod.rs
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

senamakel and others added 2 commits August 13, 2026 23:47
Removed the unused `std::collections::HashMap` import from the visualization module to keep the codebase clean and avoid compiler warnings.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
@senamakel
senamakel merged commit 2ca36c4 into main Aug 13, 2026
5 checks passed
@senamakel
senamakel deleted the parallel-gates-loops branch August 13, 2026 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant