fix(campaign): compare optimization methods on untouched test data

.claude/skills/agent-eval/SKILL.md

@@ -702,6 +702,10 @@ If you're skipping any of the four for a reason that isn't "this is a unit test,
 
 15. **Hook errors are swallowed and logged by default.** `TraceEmitterOptions.onRunComplete` hooks that throw don't crash the run — that's intentional, auto-orchestration must not fail the underlying flow. If a hook is *load-bearing* for the run's correctness (e.g. a gate that must pass before declaring success), set `hookErrors: 'throw'` or wire the gate as an explicit assertion outside the hook.
 
+16. **`compareOptimizationMethods` requires three disjoint scenario sets.** Training data is used to create candidates, selection data may accept or stop candidates, and test data alone determines the final ranking.
+    Each method receives only training and selection data through `OptimizationMethodInput`.
+    The old `holdoutScenarios` input is rejected because it does not distinguish selection from final testing.
+
 ---
 
 ## Regression tests worth writing

.github/workflows/empirical-gate.yml

@@ -1,11 +1,9 @@
 name: Empirical Lift (scheduled)
 
-# The LIVE companion to the deterministic mechanism gate in ci.yml. ci.yml
-# proves the optimizer harness RANKS correctly (compareProposers/skill-opt/presets
-# unit tests, every PR, no LLM). This job proves the optimizers move a REAL
-# held-out number on a real backend and records which proposer wins — a periodic,
-# durable artifact, NOT a per-PR gate (live LLM runs are nondeterministic +
-# cost money; gating PRs on them would make CI flaky and expensive).
+# The live companion to deterministic tests in ci.yml. Unit tests check ranking
+# behavior without a model. This job periodically measures the methods with a
+# real model and stores the result. It does not block pull requests because live
+# model results vary and cost money.
 #
 # Permissive: never blocks a deploy. Neutral-skips when no LLM key secret is
 # provisioned (forks, pre-secret), so it is never a spurious red.
@@ -40,7 +38,7 @@ jobs:
       - name: Install JS deps
         run: pnpm install --frozen-lockfile
 
-      # Fail-safe guard: skip (neutral, not red) when no LLM key is configured.
+      # Skip cleanly when no model key is configured.
       - name: Check for an LLM key
         id: guard
         env:
@@ -49,11 +47,11 @@ jobs:
           if [ -n "$KEY" ]; then
             echo "run=true" >> "$GITHUB_OUTPUT"
           else
-            echo "::notice::No LLM_API_KEY / TANGLE_API_KEY secret set — skipping the live empirical run. The deterministic mechanism gate (compareProposers/skill-opt/presets) still runs in ci.yml on every PR."
+            echo "::notice::No LLM_API_KEY / TANGLE_API_KEY secret set. Skipping the live method comparison; deterministic tests still run in ci.yml."
             echo "run=false" >> "$GITHUB_OUTPUT"
           fi
 
-      - name: Run compareProposers (gepa-reflection vs gepa-pareto vs skill-opt)
+      - name: Compare GEPA reflection, GEPA Pareto, and SkillOpt
         if: steps.guard.outputs.run == 'true'
         env:
           # Defaults to the Tangle router; override the base/model via repo vars
@@ -66,12 +64,12 @@ jobs:
           POPULATION: ${{ github.event.inputs.population || '2' }}
           GENERATIONS: ${{ github.event.inputs.generations || '2' }}
           EPOCHS: ${{ github.event.inputs.epochs || '3' }}
-        run: pnpm tsx examples/compare-proposers-canonical/index.ts
+        run: pnpm tsx examples/compare-optimization-methods/index.ts
 
-      - name: Upload lift artifact
+      - name: Upload method comparison artifact
         if: steps.guard.outputs.run == 'true'
         uses: actions/upload-artifact@v4
         with:
           name: empirical-lift-${{ github.run_number }}
-          path: .evolve/compare-proposers-canonical/**/lift-proposers.json
+          path: .evolve/compare-optimization-methods/**/comparison.json
           if-no-files-found: warn

CHANGELOG.md

@@ -21,12 +21,34 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ### Fixed
 
+- `compareOptimizationMethods()` owns three non-empty, pairwise-disjoint scenario sets.
+  Methods receive independent copies of train and selection data, every method finishes before final test scoring starts, and final test data is absent from `OptimizationMethodInput`.
+  Built-in GEPA, SkillOpt, and FAPO methods read the shared baseline, runner, judges, directories, and execution settings from the comparison call instead of duplicating them in each method config.
+  Final test scoring uses one shared spend limit.
+  Reused cost ledgers isolate comparison receipts from unrelated calls, and final test scenarios are copied for each measured surface.
+  Results report optimization, test, and total cost with accounting completeness and reasons for unknown charges.
+  Per-method results include the paired scenario values used to compute lift.
+  Cost orders a tied-lift group only when every method in that group has complete accounting.
+  Bootstrap intervals adjust across all method contrasts, and the default resample count increases when needed to represent the adjusted tails.
+  `optimizationConcurrency` can run independent methods in parallel.
+- `runSkillOpt()` validates every numeric control before scoring, enforces `patchesPerEpoch` for custom proposers, and no longer mutates the caller's `runDir`.
 - Include the complete required response shape in `llmPolicyEditProposer` model instructions, and add `LlmClientOptions.jsonSchemaTransport: 'json-object'` for providers that do not implement native JSON Schema enforcement.
 - `compositeProposer` restores each member's original labels when replaying history, so stateful members do not repeat candidates whose labels were decorated for provenance.
 - `InsightReport.interRater.kappa` now reports quadratic weighted kappa instead of Pearson correlation.
   Read `interRater.pearson` for the previous correlation measure; `icc` and `spearman` are now reported separately.
 - Compute contextual-bandit doubly robust estimates with separate logged-action and target-policy value terms, expose how many rows use DR versus IPS or the deprecated scalar path, and carry both values through belief-state records.
 
+### Breaking
+
+- `compareProposers()` is replaced by `compareOptimizationMethods()`.
+  Rename `proposers` to `methods`, `ProposerEntry` to `OptimizationMethod`, and the built-in `*Entry` factories to `*Method`.
+  Pass `trainScenarios`, `selectionScenarios`, and `testScenarios`; the ambiguous `holdoutScenarios` option is rejected.
+  `OptimizerEntryConfig` is replaced by `BuiltinOptimizationMethodConfig`, which contains method-specific settings only.
+  Move shared method execution settings to `optimizationRunOptions` on the comparison call.
+  Read `optimizationCost`, `testCost`, and `totalCost`; each includes `totalCostUsd`, `accountingComplete`, and `incompleteReasons`.
+- `runSkillOpt({ holdoutScenarios })` fails closed because those rows are adaptively reused.
+  Pass `selectionScenarios`; selection result fields now use `Selection` instead of `Holdout`, and `lift` is now `selectionLift`.
+
 ## [0.122.2] — 2026-07-17 — premeasured optimization continuation
 
 ### Added

README.md

@@ -8,7 +8,7 @@ A TypeScript library that measures whether your AI agent got better or worse, us
 [![license: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE)
 
 You give it agent runs: outputs, traces, scores, and production feedback.
-It gives you numbers you can act on: did the new prompt beat the old one, is the difference statistically real, what failed and why, and whether the change should ship.
+It gives you numbers you can act on: how much the new prompt changed outcomes, how uncertain that estimate is, what failed and why, and whether the change meets your release rule.
 
 Use it when you need to:
 
@@ -132,18 +132,61 @@ const judge = llmJudge<string, SupportScenario>(
 Pass `judge` to `defineAgentEval()` in place of the offline judge.
 The model receives the scenario, artifact, scoring prompt, and dimension descriptions.
 
+### Compare Optimization Methods
+
+Use `compareOptimizationMethods()` when you need to compare complete search procedures rather than individual prompts.
+The function gives every method the same baseline, runner, judges, train data, and selection data, then ranks their selected surfaces on separate final test data.
+
+```ts
+import {
+  type BuiltinOptimizationMethodConfig,
+  compareOptimizationMethods,
+  gepaParetoMethod,
+  gepaReflectionMethod,
+  skillOptMethod,
+} from '@tangle-network/agent-eval/campaign'
+
+const methodConfig: BuiltinOptimizationMethodConfig<MyScenario, MyArtifact> = {
+  llm,
+  model,
+  target: 'the complete prompt being improved',
+}
+
+const result = await compareOptimizationMethods<MyScenario, MyArtifact>({
+  methods: [
+    gepaReflectionMethod(methodConfig),
+    gepaParetoMethod(methodConfig),
+    skillOptMethod(methodConfig),
+  ],
+  baselineSurface,
+  trainScenarios,
+  selectionScenarios,
+  testScenarios,
+  dispatchWithSurface,
+  judges,
+  runDir,
+})
+```
+
+Read `result.scores` for lift and intervals.
+Read `result.totalCost` for dollars plus whether every charge was known.
+Ranks follow estimated lift; use the intervals and pairwise results to determine whether the observed difference excludes zero.
+See the [method-comparison guide](./docs/campaign-proposers.md) and [runnable example](./examples/compare-optimization-methods/).
+
 ---
 
-## What's in the box
+## Core APIs
 
-One-line tour of the primitives. All of these are plain functions and interfaces you compose; start from `/contract` and pull in more only when you need it.
+Start from `/contract` for the common path.
+Use `/campaign` when you need direct control over runs, candidate generation, or method comparison.
 
 | Primitive | What it does |
 |---|---|
 | **Evaluation** (`runEval`, `runCampaign`) | Run agent × scenarios × repetitions, score every run, and record the result. |
 | **Scoring** (`JudgeConfig`, `llmJudge`, calibration) | Score one output on weighted dimensions with code or a model, then compare model scores against human ratings. |
 | **Release rules** (`heldOutGate`, `paretoSignificanceGate`, `composeGate`, …) | Decide whether a candidate ships, such as requiring an improvement on scenarios that candidate generation never saw. |
 | **Candidate generation** (`gepaProposer`, `evolutionaryProposer`, …) | Generate candidate prompts or configs from prior failures. |
+| **Method comparison** (`compareOptimizationMethods`) | Run complete optimization methods on shared train and selection data, then rank them on separate final test data. |
 | **Run analysis** (`analyzeRuns`, `diffRuns`) | Turn any set of `RunRecord`s into a report: score distributions, baseline-vs-candidate lift with confidence intervals, failure clusters, cost breakdown, recommendations. |
 | **Intake adapters** (`fromFeedbackTable`, `fromOtelSpans`) | Convert data you already have, such as human ratings tables and OpenTelemetry spans, into `RunRecord`s. |
 | **Cost tracking** | Attribute every model call's tokens and dollars to the run, phase, and judge that spent them, including interrupted calls. |
@@ -159,6 +202,7 @@ Our own experiments with these primitives live in [`examples/`](./examples/READM
 | [`examples/selfimprove-quickstart/`](./examples/selfimprove-quickstart/) | The closed improve-and-verify loop, fully offline |
 | [`examples/customer-feedback-loop/`](./examples/customer-feedback-loop/) | Multi-rater human feedback (CSV/Sheets/Obsidian) → per-rater judges → report |
 | [`examples/customer-otel-traces/`](./examples/customer-otel-traces/) | Production OpenTelemetry traces → report, no closed loop required |
+| [`examples/compare-optimization-methods/`](./examples/compare-optimization-methods/) | Compare complete optimization methods with separate train, selection, and test data |
 
 Each is a single `index.ts` you run with `pnpm tsx`.
 
@@ -192,7 +236,7 @@ The root export (`@tangle-network/agent-eval`) remains broad for compatibility;
 - [`docs/concepts.md`](./docs/concepts.md): the mental model for runs, judges, verifiers, traces, and the top-level functions (5-minute read)
 - [`docs/customer-journeys.md`](./docs/customer-journeys.md): three complete adoption paths with code
 - [`docs/insight-report.md`](./docs/insight-report.md): annotated walkthrough of every section of the `analyzeRuns()` report
-- [`docs/campaign-proposers.md`](./docs/campaign-proposers.md): which proposer to use and when
+- [`docs/campaign-proposers.md`](./docs/campaign-proposers.md): candidate generation and fair comparison of complete optimization methods
 - [`docs/adapters-observability.md`](./docs/adapters-observability.md): composing with LangSmith, Langfuse, Phoenix, and OpenLLMetry
 - [`docs/wire-protocol.md`](./docs/wire-protocol.md): the HTTP/RPC contract for other languages
 - [`docs/design.md`](./docs/design.md): how this package relates to the rest of the Tangle agent stack, and the dependency rules that keep it reusable