Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

16 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

PaperReproduction-Skills

A set of agent skills that reproduce a machine learning paper's results from its own codebase — scoping the experiments out of the paper, fetching the assets, building the environment, getting the code to run, launching the full runs, comparing the output against the paper, and auditing the whole thing at the end.

Each stage is a separate skill with a hard boundary around what it may do, and each one leaves a markdown report behind. The reports are the pipeline's memory: a later stage reads the earlier stages' reports rather than the conversation, so the work survives context loss, resumes cleanly, and can be checked by a human who wasn't watching.

The pipeline

# Skill Does Writes
— paper-reproduction drives stages 1-7 in order, one at a time (nothing — the stages write)
1 experiment-scoping reads the paper and the code together; picks the experiments; writes a mock command and a full command per experiment experiment-scoping.md
2 resource-download checks the hardware can support each experiment, then fetches the datasets and checkpoints resource-download.md, assets/
3 environment-setup builds the conda env from lockfiles → README → paper → imports; also repairs it on demand environment-setup.md, environment.lock.txt
4 run-validation runs the mock commands and fixes the code in an error-driven loop, one git commit per fix run-validation.md, runs/validation/
5 run-experiment scales the validated commands back to paper-faithful length and runs them sequentially run-experiment.md, runs/<run_id>/
5.5 baseline-reproduction clones each baseline method's repo and drives stages 1-5 inside it, scoped to just the datapoints the comparison needs baselines.md
6 result-analysis extracts the metrics with auditable scripts and rebuilds the paper's table or figure from them, side by side result-analysis.md, analysis/
7 reproduction-audit reads every stage's report, checks the chain of evidence against what's on disk, explains the paper-vs-ours gaps, and gives the verdict reproduction-audit.md
— run-visible helper: launches a run in a terminal the human can watch, and monitors it without flooding the agent's context runs/<RUN_ID>/

Install

Skills are directories under skills/. Make them visible to your agent (e.g. Claude Code) by linking them into a skills directory — user-wide:

git clone https://github.com/<you>/PaperReproduction-Skills.git
ln -s "$PWD/PaperReproduction-Skills/skills/"* ~/.claude/skills/

or per-project, into .claude/skills/ of the repo you're reproducing.

Use

Clone the paper's codebase, put the paper's PDF inside it, and work from its root:

> reproduce this paper

That triggers paper-reproduction, which runs the stages in order and pauses for your go-ahead before the two expensive ones (the full runs, and the baseline sub-pipelines).

Every stage also stands alone — ask for one by name when you want just that piece:

> set up the environment for this repo          # environment-setup, standalone
> the mock runs pass, launch the full runs       # run-experiment
> audit what we've got so far                    # reproduction-audit

The pipeline resumes rather than restarts: a stage whose report already exists is skipped, so an interrupted reproduction picks up where it left off.

What it produces

Everything lands in .paper-reproduction/ inside the target repo — one directory holding the entire reproduction:

.paper-reproduction/
├── experiment-scoping.md       # stage 1
├── resource-download.md        # stage 2
├── environment-setup.md        # stage 3
├── environment.lock.txt
├── run-validation.md           # stage 4
├── run-experiment.md           # stage 5
├── baselines.md                # stage 5.5
├── result-analysis.md          # stage 6
├── reproduction-audit.md       # stage 7  ← read this one
├── assets/{datasets,checkpoints}/
├── runs/
│   ├── validation/exp<N>-attempt<K>.log
│   └── <run_id>/{console.log, exit_code, run.json, status.json}
└── analysis/{scripts,data,figures}/

reproduction-audit.md is the human-facing summary; the rest is the evidence behind it.

How it's built

A few rules shape every skill in here:

  • One report per stage, named after the stage. State lives in files, not in context.
  • Verdicts flow downstream and are never resurrected. Stage 2 says FEASIBLE / SKIPPED / BLOCKED, stage 4 says VALIDATED / FAILED / SKIPPED, stage 5 says COMPLETE / FAILED / NOT RUN, stage 6 analyzes only what stage 5 completed. No stage quietly upgrades an upstream refusal or works around it.
  • Every stage has a boundary it won't cross. Scoping doesn't download. Downloading doesn't install torch. Environment setup doesn't fix code. Validation doesn't touch the env — it sends errors back to environment-setup in repair mode, so the lockfile stays truthful. The full runs don't fix anything; failures route back instead.
  • Numbers must be auditable. Stage 6 extracts with saved scripts into saved data files; a number that exists only in prose is a bug. Paper values come from table cells and stated text — never digitized off a curve.
  • Judgment is quarantined in stage 7. Stage 6 presents the comparison and refuses to score it. Only reproduction-audit, which has read the whole chain, gives a verdict — and it may never give one stronger than the evidence supports.

Per-repo conventions: the conda env is pr-<repo-dirname>, code fixes live on a run-validation git branch, and run ids look like exp1-run2-syn4-t5.

Repo layout

skills/
├── paper-reproduction/       # the wrapper
├── experiment-scoping/       # stages 1-7
├── resource-download/
├── environment-setup/
├── run-validation/
├── run-experiment/
├── baseline-reproduction/
├── result-analysis/
├── reproduction-audit/
└── run-visible/              # helper + scripts/spawn_visible.sh

Six of the stage skills carry an evals/evals.json describing scenario tests (the fixture repos they reference are not committed here). workflow_outline.md is the original design sketch from before the pipeline was split into skills, kept for history — this README is the current description.

License

MIT — see LICENSE.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages