mutate: refuse a file the binary under test is not built from

[martinus]

The worst way this tool can be wrong is in the flattering direction, and this was the last way left open.

A file that no test binary includes scores every mutant survived, because the mutation never reaches the binary the tests run against. The report then says:

nothing noticed these - whatever covers them is decoration:
  src/oans.c:2172  == -> !=

…which reads as an indictment of the tests when nothing was ever measured.

Not hypothetical — it is how this was found

oans's src/tests.c #includes most of src/*.c into a single translation unit, but deliberately not oans.c or run_dedupe.c, which belong to the shipped binary alone — and make test-build compiles only tests.c. Sweeping src/oans.c planned 1,394 mutants and would have scored all of them survived: a 0% kill rate over 2,177 lines, presented as a fact about the test suite.

The probe

Append an #error to the target and build. If the build still succeeds, nothing the test binary is made of ever included that file.

That is deterministic and needs to know nothing about the build system, which is what makes it the right shape for a core shared by make, cmake and meson. A compilation database can say which files are compiled, but not which are #included into something that is — and oans's case is exactly the latter, so the cheaper backend-specific check would have missed it.

Two details that are deliberate:

• It runs before the baseline build, not after, so a refusal leaves the tree with a good build rather than a failed one.
• A timeout is inconclusive, not a refusal. A slow machine must not become a wrong answer about the tests.

It sits directly beside the check that already refuses a filter matching no test cases, which exists for the same reason and is worded almost the same way:

A filter that matches nothing is green, and every mutant under it survives. Refusing here is the difference between "your filter names nothing" and a report claiming the suite covers nothing.

Verification

scripts/test_mutate.py gains six cases, checked the way this repository asks — disabling the refusal makes test_a_file_nothing_includes_is_refused fail and everything else pass:

FAILED (failures=1)
  test_a_file_nothing_includes_is_refused
  AssertionError: unexpectedly None

They cover the refusal, the acceptance, that the probe really puts an #error in the file (without which the check would pass for the wrong reason on any tree whose build happens to fail), that the file is restored on all three outcomes, and that a timeout does not refuse.

257 tests, green.

End to end on two of the three backends, since the change was developed against make:

backend	repo	result
make	oans	refuses src/oans.c and src/run_dedupe.c; still sweeps src/util.c normally and returns real verdicts; make mutation-replay unchanged at 65/65
cmake	nanobench	accepts src/include/nanobench.h, baseline builds, mutants score caught as before
meson	this repo	not verified end to end — meson is not installed on this machine. Covered by the hermetic suite only

I would rather state that gap than imply otherwise. The mechanism touches no backend-specific code beyond Backend.build_argv, which the hermetic suite does cover for meson, but that is an argument rather than a measurement.

Sync

Copied into nanobench and oans with mutate_core.sha256 updated in all three; lint-mutate-core.py passes in each.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GeJoYWWvf2ewg87ih8DpGJ

---

Generated by Claude Code