A set of agent skills that reproduce a machine learning paper's results from its own codebase — scoping the experiments out of the paper, fetching the assets, building the environment, getting the code to run, launching the full runs, comparing the output against the paper, and auditing the whole thing at the end.
Each stage is a separate skill with a hard boundary around what it may do, and each one leaves a markdown report behind. The reports are the pipeline's memory: a later stage reads the earlier stages' reports rather than the conversation, so the work survives context loss, resumes cleanly, and can be checked by a human who wasn't watching.
| # | Skill | Does | Writes |
|---|---|---|---|
| — | paper-reproduction |
drives stages 1-7 in order, one at a time | (nothing — the stages write) |
| 1 | experiment-scoping |
reads the paper and the code together; picks the experiments; writes a mock command and a full command per experiment | experiment-scoping.md |
| 2 | resource-download |
checks the hardware can support each experiment, then fetches the datasets and checkpoints | resource-download.md, assets/ |
| 3 | environment-setup |
builds the conda env from lockfiles → README → paper → imports; also repairs it on demand | environment-setup.md, environment.lock.txt |
| 4 | run-validation |
runs the mock commands and fixes the code in an error-driven loop, one git commit per fix | run-validation.md, runs/validation/ |
| 5 | run-experiment |
scales the validated commands back to paper-faithful length and runs them sequentially | run-experiment.md, runs/<run_id>/ |
| 5.5 | baseline-reproduction |
clones each baseline method's repo and drives stages 1-5 inside it, scoped to just the datapoints the comparison needs | baselines.md |
| 6 | result-analysis |
extracts the metrics with auditable scripts and rebuilds the paper's table or figure from them, side by side | result-analysis.md, analysis/ |
| 7 | reproduction-audit |
reads every stage's report, checks the chain of evidence against what's on disk, explains the paper-vs-ours gaps, and gives the verdict | reproduction-audit.md |
| — | run-visible |
helper: launches a run in a terminal the human can watch, and monitors it without flooding the agent's context | runs/<RUN_ID>/ |
Skills are directories under skills/. Make them visible to your agent (e.g. Claude Code) by linking them
into a skills directory — user-wide:
git clone https://github.com/<you>/PaperReproduction-Skills.git
ln -s "$PWD/PaperReproduction-Skills/skills/"* ~/.claude/skills/or per-project, into .claude/skills/ of the repo you're reproducing.
Clone the paper's codebase, put the paper's PDF inside it, and work from its root:
> reproduce this paper
That triggers paper-reproduction, which runs the stages in order and pauses for your
go-ahead before the two expensive ones (the full runs, and the baseline sub-pipelines).
Every stage also stands alone — ask for one by name when you want just that piece:
> set up the environment for this repo # environment-setup, standalone
> the mock runs pass, launch the full runs # run-experiment
> audit what we've got so far # reproduction-audit
The pipeline resumes rather than restarts: a stage whose report already exists is skipped, so an interrupted reproduction picks up where it left off.
Everything lands in .paper-reproduction/ inside the target repo — one directory holding
the entire reproduction:
.paper-reproduction/
├── experiment-scoping.md # stage 1
├── resource-download.md # stage 2
├── environment-setup.md # stage 3
├── environment.lock.txt
├── run-validation.md # stage 4
├── run-experiment.md # stage 5
├── baselines.md # stage 5.5
├── result-analysis.md # stage 6
├── reproduction-audit.md # stage 7 ← read this one
├── assets/{datasets,checkpoints}/
├── runs/
│ ├── validation/exp<N>-attempt<K>.log
│ └── <run_id>/{console.log, exit_code, run.json, status.json}
└── analysis/{scripts,data,figures}/
reproduction-audit.md is the human-facing summary; the rest is the evidence behind it.
A few rules shape every skill in here:
- One report per stage, named after the stage. State lives in files, not in context.
- Verdicts flow downstream and are never resurrected. Stage 2 says
FEASIBLE/SKIPPED/BLOCKED, stage 4 saysVALIDATED/FAILED/SKIPPED, stage 5 saysCOMPLETE/FAILED/NOT RUN, stage 6 analyzes only what stage 5 completed. No stage quietly upgrades an upstream refusal or works around it. - Every stage has a boundary it won't cross. Scoping doesn't download. Downloading
doesn't install torch. Environment setup doesn't fix code. Validation doesn't touch the
env — it sends errors back to
environment-setupin repair mode, so the lockfile stays truthful. The full runs don't fix anything; failures route back instead. - Numbers must be auditable. Stage 6 extracts with saved scripts into saved data files; a number that exists only in prose is a bug. Paper values come from table cells and stated text — never digitized off a curve.
- Judgment is quarantined in stage 7. Stage 6 presents the comparison and refuses to
score it. Only
reproduction-audit, which has read the whole chain, gives a verdict — and it may never give one stronger than the evidence supports.
Per-repo conventions: the conda env is pr-<repo-dirname>, code fixes live on a
run-validation git branch, and run ids look like exp1-run2-syn4-t5.
skills/
├── paper-reproduction/ # the wrapper
├── experiment-scoping/ # stages 1-7
├── resource-download/
├── environment-setup/
├── run-validation/
├── run-experiment/
├── baseline-reproduction/
├── result-analysis/
├── reproduction-audit/
└── run-visible/ # helper + scripts/spawn_visible.sh
Six of the stage skills carry an evals/evals.json describing scenario tests (the fixture
repos they reference are not committed here). workflow_outline.md is the original design
sketch from before the pipeline was split into skills, kept for history — this README is
the current description.
MIT — see LICENSE.