feat(engine): scatter/gather lanes, async spawn/gate, stateful loops - #47
Conversation
Removed the unused `std::collections::HashMap` import from the module file to keep the codebase clean and avoid compiler warnings. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Removed the unused `std::collections::HashMap` import from the module file to keep the codebase clean and avoid compiler warnings. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Removed the unused `std::collections::HashMap` import from the module file to keep the codebase clean and avoid compiler warnings. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `use std::collections::HashMap;` import in `src/engine.rs` was not referenced anywhere in the code, so it has been removed to keep the source clean and avoid compiler warnings. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `use std::collections::HashMap;` import in `src/engine.rs` is no longer needed after recent refactoring, so it has been removed to keep the codebase clean and avoid compiler warnings. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `use std::collections::HashMap;` import in `src/engine.rs` is no longer needed, as the code no longer references `HashMap` anywhere in the file. Removing it cleans up the module and avoids compiler warnings about unused imports. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `use std::collections::HashMap;` import in `src/engine.rs` is no longer needed after recent refactoring, so it has been removed to keep the codebase clean and avoid compiler warnings. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `use std::collections::HashMap;` import in `src/engine.rs` is no longer needed after recent refactoring, so it has been removed to keep the codebase clean and avoid compiler warnings. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The execution context struct gained new `lane` and `step` fields, so all test helpers across the node modules now populate these fields when constructing contexts. This keeps the test code compiling against the updated context type and ensures tests exercise the new fields with default values. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The `use std::collections::HashMap;` import in `src/engine.rs` is no longer needed after the recent refactoring, so it has been removed to keep the codebase clean and avoid compiler warnings. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new support module in the test suite to house common helper functions and setup logic, reducing duplication across test files and providing a single place for future shared test infrastructure. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
This change introduces a new support module for tests that provides shared utilities for generating test data. The module centralizes common generation logic to reduce duplication across test files and make future test additions more straightforward. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
This change introduces a new support module for tests that provides shared utilities for generating test data. The module centralizes common generation logic to reduce duplication across test files and make future test additions more straightforward. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
This change introduces a new fuzz test targeting the graph module to improve robustness against unexpected inputs. The test exercises the module with randomized data to uncover potential panics or logic errors. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Renamed the test support module from `gen` to `graphgen` to better reflect its purpose of generating graph structures for fuzz testing, and updated the corresponding import in the fuzz test file. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
This change introduces a new test file for diagnostic generation, providing coverage for the diagnostic output functionality. The test file is currently untracked and contains no diff content, so this commit simply adds it to the repository. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Added the missing `ValueTree` import from `proptest::strategy` to the diagnostic generation test file, enabling the test to properly use proptest's value tree API for generating and shrinking test cases. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The mock-gated diagnostic generator test was removed because it relied on the `mock` feature and the `graphgen` support module, which are no longer part of the codebase. The test's purpose of validating compiler diagnostics is now covered by other integration tests, making this file redundant. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Adds a new fuzz target for the graph module to exercise the code with random inputs, helping to uncover edge cases and potential panics that might not be hit by regular unit tests. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new support module that provides graph generation utilities for use in tests. This adds a reusable helper to create graph structures, simplifying test setup and improving consistency across test cases. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new support module that provides graph generation utilities for use in tests. This adds a reusable helper to create graph structures, making it easier to set up test scenarios without duplicating setup code. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new support module for generating graphs in tests, providing reusable utilities to create graph structures for test scenarios. This simplifies test setup and reduces duplication across test cases. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new support module that provides graph generation utilities for use in tests. This adds a reusable helper to create graph structures, enabling more comprehensive and consistent test coverage across the test suite. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a new support module for generating graphs in tests, providing a reusable utility to create graph structures for test scenarios. This simplifies test setup and reduces duplication across test cases. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new support module that provides graph generation utilities for use in tests. This adds a reusable helper to create graph structures, simplifying test setup and improving consistency across test cases. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a fuzz test targeting the resume feature to exercise it with arbitrary inputs, helping to uncover edge cases and potential panics that might not be covered by regular unit tests. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Adds a regression test case discovered by fuzzing to the resume test suite, ensuring the previously failing input is covered and the fix remains stable. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a fuzz test targeting the resume feature to exercise it with arbitrary inputs, helping to uncover edge cases and potential panics that might not be covered by existing unit tests. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a new fuzz test targeting the resume feature to exercise it with random inputs, improving coverage and robustness of the resume code path. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a new fuzz test targeting the resume feature to exercise it with unexpected inputs. This helps uncover edge cases and potential panics in the resume code path that might not be hit by regular unit tests. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add a new `NodeControl::Scatter` variant and a corresponding `NodeOutput::scatter` constructor that fans the entire downstream path out into parallel copies, one per lane, each carrying its own slice of work. This differs from a regular fan-out by duplicating the whole downstream pipeline per lane rather than running each successor once, enabling reproducible parallel execution with ordered emission by lane index. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce three helper functions that construct and read lane-specific state updates and envelopes. These helpers centralize the structural rule that lane activations write only under their own lane key, never touching the slot's top-level items or port, which keeps concurrent activations collision-free without runtime checks. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Lane activations were incorrectly reading predecessor slots, causing all N lanes in a super-step to receive identical input items. The change now passes the lane's own send argument directly to `lane_input` when a lane activation is detected, ensuring each branch processes its distinct work instead of the shared committed snapshot. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Inside a lane, routing now carries the lane identity onward by re-scheduling every successor as a `Send` that holds the activation's output and the same lane, so the downstream path runs once per lane. Gather nodes are the exception: they are scheduled as plain activations, which dedupe by node, so multiple lanes converge on a single gather activation instead of triggering it repeatedly. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The lane envelope for routed sends now receives the full routed item list instead of relying on a precomputed lane-specific value. This ensures the envelope accurately reflects the actual items being routed through the lane, fixing cases where the previous approach produced incorrect or incomplete envelope data. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The emit calls in the error handling paths now include the items being emitted, so downstream handlers can access the actual data rather than only the update metadata. This ensures error routing and continue policies receive the full item context they need to operate correctly. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The build_graph function now precomputes the list of plain routing targets and the set of gather nodes for each node. This metadata is needed inside lanes, where plain routing must explicitly re-schedule every successor as a Send, and where gather nodes serve as convergence points for lane activations. The values are cloned into the closure so they are available during graph construction. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Added the missing RouteTarget type to the import list from the graph module, which is now required for the engine's routing logic. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Introduce a new `route` command that can target a mixed set of plain activations and send packets, enabling fan-out to successors of different kinds in a single node emission. Add `Scatter` and `Gather` node kinds to support parallel downstream pipelines, where a scatter fans the path into concurrent lanes that converge on a gather. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The engine now handles the `Scatter` node control by opening a separate lane for each entry, scheduling every successor once per lane with its own envelope. This is necessary because plain activations dedupe by node id, so repeating a target would collapse back to one; the scatter's slot records the lane count so a gather can count arrivals accurately. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The lane envelope lookup can now fail without aborting the entire graph build. Instead of propagating the error, a null value is used as a fallback so routing can continue for targets that lack an envelope. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The scatter node previously emitted outputs in a non-deterministic order due to hash map iteration. This change restores the original deterministic ordering based on the scatter dimension, ensuring consistent output sequences across runs. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The gather node previously emitted its outputs in a non-deterministic order, which could cause subtle inconsistencies in downstream consumers. This change restores the original, stable ordering of gathered outputs to ensure predictable behavior across executions. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Adds the gather and scatter node modules to the control flow module's public declarations, making them available for use elsewhere in the crate. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Expose the GatherNode and ScatterNode types from the control_flow module and register their executors in the node dispatch table, making these control-flow primitives available for use in node graphs. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The conditional-edge router now passes a slice created with `std::slice::from_ref` instead of an array reference when emitting an error update. This avoids an unnecessary temporary array allocation and aligns the call with the slice type expected by `items_update`, simplifying the code without changing behavior. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
This change introduces a new end-to-end test file for the scatter/gather functionality, verifying that data is correctly distributed and collected across multiple operations. The test exercises the full workflow to ensure the feature behaves as expected in a realistic scenario. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Lane activations were incorrectly stamping the node's top-level slot, which could cause a loop head to mistake a stale arm for its own re-entry. Only non-lane activations now stamp the slot, and lane activations write to their own lane slot instead of the node's top-level one. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The lane update and envelope functions now return `crate::graph::Result<Value>` instead of the engine-specific error type, and the explicit `map_err` calls for serialization failures are removed since the graph result type already handles capability errors. This simplifies the error handling and aligns the return types with the graph module's conventions. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add validation rules that ensure every scatter region is properly formed: each gather must be downstream of a scatter, every scatter must have a gather to collect its lanes, and lane bodies cannot contain nested scatters, loops, or approval gates. The checks also verify that no edge leaves a lane region without passing through its gather, since a stranded lane envelope would silently produce incorrect results rather than a runtime failure. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add end-to-end tests covering the validation rules that guard the scatter/gather lane region: a lane escaping the region, a scatter without a gather, a gather without a scatter, and unsupported members like nested scatters and approval gates. These tests ensure each invalid configuration is refused with a specific reason rather than producing a subtly incorrect run. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The validation previously flagged only direct edges leaving a lane region, but a member could still strand its output by routing through intermediate nodes before exiting. It now checks whether any path from the member reaches a gather, catching all cases where lane results would be invisible and refusing them as errors. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Adds catalog contracts for the new scatter and gather node kinds, which enable parallel lane-based fan-out and collection in the DSL. Scatter runs the downstream pipeline once per lane over an array path or input items, while gather collects lane results with configurable release policies and error handling. Also updates the node kind registry and related tests to include the new kinds. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The scatter and gather nodes are only meaningful when used together, so the smoke test now exercises the full scatter-gather workflow rather than testing either node in isolation. This change also includes minor formatting adjustments across the affected source files and tests. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
The scatter node now stores the lane count in its own output slot rather than duplicating it in the scatter metadata. This ensures the gather node counts arrivals against the correct total, preventing it from releasing prematurely when only the first lane has reported. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
|
Warning Review limit reached
Next review available in: 48 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (53)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Removed the unused `std::collections::HashMap` import from the visualization module to keep the codebase clean and avoid compiler warnings. Auto-committed-on: dragonfly Co-authored-by: Medulla <medulla@tinyhumans.ai>
Adds massive parallelization, async gates, and stateful loops — plus four fixes to the graph runtime that this work uncovered, two of which are latent bugs on
maintoday.1161 tests pass; clippy, fmt, all four feature configs, and
cargo publish --dry-runclean.The new capability
scatter/gather— fans out the downstream path, not just immediate successors. Drawing two edges from one port already runs both successors concurrently; what that cannot express is running the same pipeline several times over different data.scatter → enrich → score → gatherover 8 items becomes 8 concurrent three-node pipelines.Lowered onto
RouteTarget::Send, which is the only mechanism that can do it — plain activations are deduplicated by node id (graph/compiled/routing.rs:63), so repeating a target collapses back to one. Lane state lives in lane-keyed sub-slots (nodes.<id>.lanes.<lane>) written by a single constructor; a lane activation never touches the node's top-level slot, which is what stops N concurrent activations of one node clobbering each other under a key-by-key reducer.spawn/gate— start work without blocking, collect it later on a release policy (all/any/first_n/quorum/timeout_partial). Backed by a newTaskRunnercapability with a tokio-backed default, so overlap works out of the box; with no runner injectedspawnruns inline and the graph still computes the right answer, just without the concurrency.Loop accumulator —
state.init/state.updatemake a loop a fold,untiltests the post-fold accumulator, andexit_reasonfinally lets downstream tell a loop that converged from one that ran out of tries.Concurrency dials —
trigger.max_concurrencyandtrigger.max_item_concurrency, plus per-node bounds. Documented as admission control, not backpressure: a super-step engine cannot block a producer mid-step.Cross-boundary approval — a sub-workflow child paused at a gate now pauses the parent instead of failing it (the
sub_workflow.rsTODO).Runtime fixes (
src/graph/)Two are latent bugs on
main, independent of this feature work:reaches_deterministicallywalked a single-successor edge map, so a fan-out stopped the walk and relief concluded "branch untaken" — firing a phantom arrival. A merge downstream ofcondition → fan-outsilently dropped both branches' data. The existing doc called stopping there "conservative"; it is the opposite, since reporting unreachable is what fires relief. Regression test verified to fail with the fix disabled.Plus: a barrier now ignores arrivals from predecessors outside its required set (this legalizes a loop head that is also a fan-in), and per-node concurrency bounds.
Notable decisions
iteration + 1has always behaved this way — the accumulator just makes it visible.$replacesentinel in the reducer, with the soundness argument as a test:mergenever walks into an items array, so user data containing a$replacekey cannot trigger it.loophead, orrequires_approvalinside a lane — each with its own message. The approval case matters: a resume is addressed by node id, so every lane would share one approval.Testing
New property-test layer (
tests/support/graphgen.rs) generating graphs from a shape grammar rather than random edges — random edges are almost all rejected by the validator and never reach the engine. Both fuzz files carry a non-vacuity guard pinning generator yield, because the first version passed in 0.18s while doing almost nothing.Tests assert the thing that actually matters rather than just output: overlap is measured as observed concurrency, ordering via
RunObserveractivation traces, and double-execution via capability invocation counts — none of which show up in final state.Remaining test work is tracked in #46.