Skip to content

mutate: refuse oans.c and run_dedupe.c instead of reporting 0% killed on them - #240

Merged
martinus merged 1 commit into
masterfrom
claude/oans-testing-improvements-wslk93
Aug 22, 2026
Merged

mutate: refuse oans.c and run_dedupe.c instead of reporting 0% killed on them#240
martinus merged 1 commit into
masterfrom
claude/oans-testing-improvements-wslk93

Conversation

@martinus

Copy link
Copy Markdown
Owner

Vendored sync of martinus/unordered_dense#205, where the change was made and where the hermetic suite covers it. This is the repository the bug was found in, and the only one of the three that can hit it.

The problem

src/tests.c #includes most of src/*.c into a single translation unit, but deliberately not oans.c or run_dedupe.c — those belong to the shipped binary alone, and make test-build compiles only tests.c.

So a mutant in either file never reaches ./test, and the tool scored it survived:

$ scripts/mutate/mutate.py --file src/oans.c --lines 2172

nothing noticed these - whatever covers them is decoration:
  src/oans.c:2172  == -> !=
      if (ret == ENOMEM || debug)

A dry-run over the whole file planned 1,394 mutants, and every one of them would have come back the same way — a 0% kill rate over 2,177 lines, reported as a fact about the tests rather than about the invocation. That is the flattering direction, and CLAUDE.md's standing rule for this tool is that every way it has been wrong so far flattered the tests.

After

checking the target reaches the binary under test

src/oans.c is not compiled into test, so every mutant in it would come back
`survived` however good the tests are - the build succeeded with an #error in
the file. Sweep a file the test binary is actually built from, or add this one
to it.

The message names the consequence rather than the condition, so the reader is not left to work out why "not compiled in" matters.

How it works

An #error is appended to the target and the build is run. If it still succeeds, nothing the test binary is made of ever included the file. Backend-agnostic on purpose: a compilation database can say which files are compiled, but not which are #included into something that is — and this repository's case is exactly the latter.

Verified here, end to end

  • src/oans.c and src/run_dedupe.c — refused.
  • src/util.c — still sweeps, still returns real verdicts (delete: clock_gettime(...) survived, as it did before).
  • make mutation-replay — unchanged at 65 across eight files, nothing surviving.
  • make lintlint-mutate-core: vendored at c71e0f7a106d, adapter fits.

This is also the make backend, which is one of the two the upstream PR verified end to end (the other being cmake via nanobench; the meson leg is covered by the hermetic suite only, as meson is not installed on this machine).

Note

make lint will fail until martinus/unordered_dense#205 lands, if it is checking the hash against upstream — the mutate_core.sha256 here is updated to match the new copy, which is the convention CLAUDE.md describes (change it there, run that suite, re-copy into all three, update each hash).

🤖 Generated with Claude Code

https://claude.ai/code/session_01GeJoYWWvf2ewg87ih8DpGJ


Generated by Claude Code

Re-copied from unordered_dense, where the change was made and where the
hermetic suite covers it. This is the repository the bug was found in and
the only one of the three that can hit it: src/tests.c #includes most of
src/*.c into one translation unit but deliberately not oans.c or
run_dedupe.c, and `make test-build` compiles only tests.c.

Sweeping src/oans.c planned 1,394 mutants and would have scored every one
`survived` - a 0% kill rate over 2,177 lines, reported as "whatever covers
them is decoration". That reads as an indictment of the tests when nothing
was ever measured, which is the flattering direction.

Both files are now refused with a message that names the consequence rather
than the condition. Verified here end to end: src/oans.c and
src/run_dedupe.c refuse, src/util.c still sweeps and still returns real
verdicts, and make mutation-replay is unchanged at 65 across eight files
with nothing surviving.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GeJoYWWvf2ewg87ih8DpGJ
@martinus
martinus merged commit 5e22dde into master Aug 22, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants