Plain-libFuzzer baseline ablation (no-LLM) for tree-sitter sweep

[quinn-dougherty]

Summary

Adds the discovery-side ablation the neurosymbolic hypothesis needs, plus the funder-facing writeup it feeds.

Context. The hybrid fuzz→fix loop swept 118 tree-sitter grammars and found genuine memory-safety bugs in 9 of them (10 distinct crashes), all in hand-written scanner.c — unlike the OSS-Fuzz/ARVO sweep, which found ~zero because those targets are already fuzzed to fixpoint. But for tree-sitter the harness is fixed (bytes → parser), so the LLM writes no harness; its only contribution is the fix phase. That makes the load-bearing question: does vanilla libFuzzer — no LLM anywhere — find the same bugs in the same walltime?

treesitter/baseline.py — plain-libFuzzer sweep, no fixer, no telemetry/keys in the path (imports only runtime/triage/models):

• targets_from_results reconstructs the identical grammar set and each grammar's fuzzing walltime straight from the hybrid's JSONL outputs → apples-to-apples (LLM/rebuild time excluded, generous to the baseline).
• Emits the same TSLoopIteration JSONL schema with fix=None, so existing tooling reads both sweeps identically.
• libFuzzer fork mode (-fork=1 -ignore_crashes=1) so a crash doesn't halt the campaign — keeps finding distinct crashes (the fair analogue of the hybrid patching bug Implement ARVO dataset ingestion #1 to reach bug Implement oss-fuzz data ingestion and target import #2). Crashes deduped by stack hash; hang/OOM classified from filename without replay; MAX_ARTIFACTS_TRIAGED caps flappy targets and records the cap in notes (no silent truncation).

Capped parallelism (--jobs/-j, ThreadPoolExecutor):

• Each fuzz process is ~1 core (fork=1). The hybrid ran serially, one core per grammar, so keeping jobs ≤ physical cores preserves per-grammar throughput parity; oversubscription starves each proc of CPU (fewer execs/sec, spurious wall-clock timeouts) and RAM (jobs × rss_limit), skewing the result. CLI estimates physical cores (logical/2 for SMT), suggests -j <phys>, warns past it. Shared runtime cloned once up front to avoid a concurrent-clone race.
• Reference box (24 threads / 12 physical cores): ~24h serial → ~2–3h at -j 12.

treesitter/baseline_compare.py — diffs hybrid vs baseline on memory-safety stack hashes (overflows, SEGV, UAF, scanner leaks), discounting timeout/OOM resource exhaustion. Prints the headline baseline reproduced N/M overlap plus per-grammar shared / hybrid-only / baseline-only. Self-compare of the real hybrid data checks at 10/10.

CLI: treesitter baseline --hybrid DIR [-j N] and treesitter baseline-compare --hybrid DIR --baseline DIR.

Blog (comms/blog/main.md) — negative-result draft with the real numbers and a clearly-marked ⚠️ PLACEHOLDER — baseline not yet run block where the overlap figure lands. Honest scope: on hardened targets the LLM adds nothing; on soft targets the fuzzer alone plausibly finds the same bugs (pending this baseline); the LLM's defensible value is patching, not discovery.

Note: the baseline has not been executed yet — this PR is code + tooling only. It reads results/treesitter/*.jsonl (gitignored; synced from the bucket) as input.

Test plan

• uv run pytest src/tests/test_treesitter_baseline.py — 15 hermetic unit tests (walltime summing, build-failure skipping, hang/OOM classification without replay, stack-hash dedup, triage cap, JSONL emission). Authored via the antigravity (agy/Gemini 3) executor, verified here.
• uv run ruff check / ruff format / uv run ty check — clean on all new/changed files.
• Existing treesitter tests still pass.
• treesitter baseline --help / baseline-compare render; self-compare against real hybrid data reports 10/10.
• Run the ablation (needs nix toolchain): nix-shell scaffold/treesitter-shell.nix --run 'cd scaffold && uv run parser-security-eval treesitter baseline --hybrid results/treesitter -j 12 && uv run parser-security-eval treesitter baseline-compare --hybrid results/treesitter --baseline results/treesitter-baseline' → fills the blog placeholder.

🤖 Generated with Claude Code