Round 3 ablation: real LLM-in-loop arm vs plain libFuzzer, + blog

[quinn-dougherty]

Summary

Executing the tree-sitter uplift ablation end to end surfaced a chain of defects that between them made the experiment unable to answer its own question. This fixes them and adds the run scripts for both arms.

libFuzzer stopped at the first crash. LibFuzzerRunner never passed -fork/-ignore_crashes, so -max_total_time never bound — every 300s window ended after 2–3 executions with cov: never printed. Measured on the real nushell-nu binary at an identical 30s budget:

	execs	exec/s	cov	crashes
before	3	0.0	0	1
after	2,127	70.9	1,516	243

This was asymmetric: baseline.py already passed fork flags via extra_flags, so the control fuzzed properly while the treatment stopped at bug #1. Any uplift number from that pairing would have measured libFuzzer flags, not the LLM. Fork mode now lives on LibFuzzerRunner (fork_workers) so both arms are configured identically, and --no-fork works again instead of being silently overridden.

Fork mode's follow-on failures. The TimeoutExpired branch read only exc.stdout, but libFuzzer writes to stderr — a timed-out window produced an empty log and looked like a finding (cov=0, crashes=N) rather than a failure. The safety margin was also duration+60s, which fork teardown outlasts. 6 of 75 treatment windows were lost before these landed.

Budget matching was silently broken for --reps > 1. targets_from_results globbed only the top level, but llm-fuzz --reps N writes rep1/..repN/. The control found zero targets and would have fuzzed nothing while appearing to succeed. The same bug in load_sweep was worse — an empty baseline reads as 100% uplift. Both now glob rep*/, and the CLI errors instead of reporting zero.

Equal fuzz-seconds, honestly. Adds TSFuzzResult.elapsed_seconds (duration_seconds is only the configured cap and overstated every early-exiting window — the exact mechanism that would have hidden the bug above from the budget matcher). The control now replays the treatment's window cadence (5×300s on a shared corpus) rather than one long run, since libFuzzer restarts re-seed from the corpus and would otherwise confound "the LLM rewrote the harness" with "restarted 5×".

Three scoring corrections.

• Harness bugs counted as uplift — only the LLM arm writes a harness, so its own crashes land wholly in hybrid_only. reaches_target() now requires a grammar-source or runtime frame.
• …but handler-only stacks are not harness bugs. typst's c6f014c21c9d has one and was found by both arms — and the control cannot produce harness bugs, which settles it. Now kept and tracked as unattributed. Replay with ASAN_OPTIONS=fast_unwind_on_fatal=0 gives the true stack: an assertion failure in ts_parser__external_scanner_serialize (parser.c:409, length <= 1024) — a latent overflow the debug assert catches.
• One function reached from two of its own call sites hashed as two bugs. function_key() keys on symbols, not file:line.

Net effect on the headline: 6 vs 4 with 2 hybrid-only → 6 vs 5 with 1.

Registry. Adds the five ablation grammars. They were survey slugs resolvable only from a registry.jsonl that has never been committed, so llm-fuzz -g nushell-nu,... died on KeyError. Static entries avoid ~1,960 GitHub API calls (490 wiki grammars × 4 endpoints) — ~33h at the unauthenticated 60/hr limit, which is a quota, not a burst limit, so backoff does not help.

Run scripts. homelab-treatment.sh / homelab-control.sh, nix-shell shebang, exporting SSL_CERT_FILE (NixOS ships no default CA path for Python's OpenSSL, and survey/sources.py swallows the resulting CERTIFICATE_VERIFY_FAILED into an empty result). Control pins -j 1 deliberately — the treatment fuzzed serially on one core, so concurrent control processes would bias the comparison toward the LLM arm.

Results (150 windows, ~12.5h, equal fuzz-seconds)

6 memory-safety bugs for the LLM arm vs 5 for plain libFuzzer, 1 hybrid-only.

The replicated finding is reliability, not capability: on foam, gren and nushell-nu the LLM arm found the known bug in 3/3 reps, the control in 1/3. The single genuine capability instance is sql's scan_dollar_string_tag leak — LLM 3/3, control 0/3 across ~68k executions; reaching it needs structurally valid $tag$...$tag$ input.

Coverage is mixed, not a clean win: the LLM arm leads on 4 of 5 grammars, but the fixed template beats it on nushell-nu in both valid reps — where it also finds the bug least often.

Caveats: five grammars, one model, one target class. nushell-nu rep2 lost all 5 windows to the timeout bug (valid for bug counts, not coverage). Sweep outputs are not included — scaffold/results/* is gitignored with no negation for the new dirs.

Test plan

• cd scaffold && uv run ruff check && uv run ruff format --check && uv run pytest — 1038 pass, ruff clean. ty reports 200 diagnostics, unchanged from the pre-existing baseline on a clean tree, none in touched files.
• Fork fix: build any grammar and run LibFuzzerRunner(..., fork_workers=1).run(30) vs fork_workers=0; the latter reproduces the 2–3 exec / cov=0 pathology.
• Rep discovery: targets_from_results on a dir laid out as rep1/<g>.jsonl returns targets (previously 0); baseline-compare on a dir with no JSONL at any level exits 1 instead of reporting zero bugs.
• Filter: reaches_target() / is_handler_only() against the stored frames — gren's deep_let drops, nushell's scanner.c:116 keeps, typst's handler-only stack is unattributed rather than dropped.
• Hash collapsing: function_key() returns the same key for scan_dollar_string_tag at :67 and :72.
• End to end: ./homelab-treatment.sh then ./homelab-control.sh, then treesitter baseline-compare --hybrid results/treesitter-llm --baseline results/treesitter-baseline.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QnSQHg4FhSxEXRyM6Pf2Eo