Skip to content

Test coverage: remaining fuzz targets and complex graph fixtures #46

Description

@senamakel

Context

The parallelization/async-gates/loops work added a property-test layer under tests/:

  • tests/support/graphgen.rs — a shape-grammar graph generator (Linear, Branch, Fanout, Loop, Gate, Spawned, Nested) rather than random edges, because random edges are almost all rejected by the validator and never reach the engine.
  • tests/fuzz_graph.rs — validate ⇒ terminate, determinism, slot coverage.
  • tests/fuzz_resume.rs — resume equivalence, resume determinism.
  • tests/fuzz_async.rs — gate ordering, no-under-delivery.

Both fuzz files carry a non-vacuity guard pinning generator yield (~85% of generated graphs actually run), because the first version passed in 0.18s and was doing almost nothing.

This issue tracks the targets and fixtures that are still missing.

Fuzz targets not yet written

  • Lane isolation — after a scatter run, every nodes.<id>.lanes.<lane> is written exactly once and no lane activation wrote the top-level slot. Needs an instrumented reducer in the harness. Blocked on scatter/gather.
  • Cancellation — cancel at an arbitrary point ⇒ the run settles promptly with cancelled: true and no orphaned tickets (assert the mock TaskRunner saw a cancel for every un-collected ticket). tests/fuzz_async.rs already has the Staggered runner recording cancellations; the property is not written.
  • Validator soundness — take a graph the validator refuses, force it past validation, and assert it actually misbehaves. This keeps refusals honest rather than superstitious. Directly relevant: the merge-on-a-cycle refusal was narrowed during this work, and the "loop head is a fan-in" refusal was lifted entirely — both on the strength of hand-written reasoning that a property test would have checked.
  • Reducer associativityengine::merge over generated update sequences, plus the $replace sentinel round-tripping at arbitrary nesting. Blocked on the loop accumulator, which introduces the sentinel.
  • Timing schedules — the generator produces graphs but not a Schedule (per-node latency, injected failures, a cancellation point). Determinism is currently checked across repeated runs, not across differing timings, which is the stronger property.

Complex hand-built fixtures (tests/complex_graphs_e2e.rs, not yet created)

Generated graphs are shallow by construction. These are the compositions worth naming, each asserting on final state and on the observed activation sequence via RunObserver — ordering bugs do not show up in final state:

  • scatter → per-lane sub-workflow → gather(quorum) → loop(accumulator) → second scatter (blocked on scatter/gather + accumulator)
  • a refinement loop whose body scatters: agent fan-out per candidate, gather best-of, accumulate, until the score passes — the headline use case (same blockers)
  • nested spawn/gate across a sub-workflow boundary where the child pauses at an approval gate — writable now, exercises the cross-boundary approval fix
  • a lane containing a switch with an on_error: route arm, plus a lane that fails while others succeed, under each on_lane_error policy (blocked on scatter)
  • diamond-in-a-loop — already covered by a_diamond_inside_the_loop_body_iterates in tests/loop_e2e.rs, which asserts one merge activation per pass
  • a wide scatter (256 lanes) with max_concurrency set low, asserting the admission bound holds and the run still completes (blocked on scatter; the non-scatter half is covered by max_concurrency_bounds_how_many_branches_run_at_once in tests/parallel_e2e.rs)

Also worth doing

  • Pin shrunk counterexamples by checking in proptest-regressions/, so a found failure stays a permanent test.
  • Extend tests/reference_workflows.rs with a JSON reference workflow per new node kind, so the published wire format is covered. spawn/gate are in smoke_all_nodes.rs but not in the reference workflows.

Explicitly not planned

cargo-fuzz/libFuzzer. The interesting inputs here are structured graphs, which proptest generates far better than a byte mutator, and it would add a nightly toolchain requirement. The one place a byte fuzzer would earn its keep is WorkflowGraph deserialization — worth a separate small target if untrusted graph JSON ever becomes a real threat model.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions