## Context Six nights of SkillOpt-Sleep against real task sets on two of my own skills (an essay skill; a router with 24 candidate targets). The engine never crashed and the gate never behaved contrary to its own spec — but **the headline verdict was never self-sufficient**. Every night the real story only emerged from the per-task diff in `evidence.jsonl`. Two structural properties explain that, and both look addressable. ## Problem 1 — the gate has no "no task regressed" condition `consolidate.py` accepts on `cand_score > base_score`, where both are slice means. Nothing checks whether an individual val task got worse. Night 1, real numbers (`gate_metric="mixed"`, val = 4 tasks): | task | baseline | candidate | |---|---|---| | `essay_w3` | soft 0.55 | **1.00** | | `essay_wl2` | soft **0.91** | **0.82** ← regressed | | two others | 1.00 | 1.00 | Aggregate `0.682 → 0.852`, verdict `accept_new_best`. The accepted rule had made the model drop a required scriptural citation from `essay_wl2` — a genuine regression, invisible in the mean. Worse, the +0.17 was not a quality gain at all: it came from the rule satisfying a check of mine that was too strict (a heading-format artifact). So the gate accepted an edit that **damaged one task and improved none**, and its report said `accept_new_best`. This is the gate-side sibling of #154, and a counterexample to the assumption in #67 that "the held-out gate rejected that candidate, which is exactly the right outcome" — here it accepted one. **Suggestion:** an opt-in `gate_no_regression: true` that rejects a candidate if any val task's score drops, regardless of the mean. For small val slices this is the difference between a usable and a misleading signal. ## Problem 2 — improvements are never attributed to the rule that caused them The gate checks *that* the mean moved, never *which* task moved or whether it belongs to the class the rule addresses. Night 4: the optimizer proposed a genuinely correct rule (prashna vs muhurta disambiguation). Verdict `accept_new_best`, `0.900 → 1.000`. But the entire delta came from `tajaka_sixty_hour`, an unrelated task the rule says nothing about — and re-running it under the accepted skill gave 1/5, i.e. that night's pass was luck. Meanwhile the task the rule actually fixes was in `train`, so the gate could not see the real effect at all (independently: 0/1 → 5/5 with the rule). Night 6 repeated the pattern with a different rule. So in 3 of 3 accepts across six nights, the reported reason for acceptance was not the actual reason. The rule was right twice and wrong once — but the report gave no way to tell those cases apart. **Suggestion:** record, per accepted candidate, which val tasks changed state, and surface that in `report.md` next to the delta. Even without changing the accept criterion, `accepted because: essay_w3 0→1; regressed: essay_wl2 0.91→0.82` would have made all three nights immediately readable. Tasks already carry `tags`, so a coarse "did any task sharing the rule's class improve?" signal is cheap. ## Why this matters beyond my setup The safety story of the project rests on the gate ("only changes that help are kept"). That holds for the *mean*, which is a weaker guarantee than it reads as — and the two failure modes above are exactly the ones a user cannot detect without reading raw evidence. I ended up writing an external harness that re-runs targeted tasks with and without a candidate rule, plus guards for neighbouring routes, before trusting any accept. Happy to prototype either piece if there is interest — the no-regression flag looks like a small, well-contained change to `_gate_apply`. ## Environment `main` @ fdeebaf, Python 3.11, `azure_openai` backend in `openai_compatible` mode against a local proxy; targets `gemini-3.1-pro` and `gemini-3.5-flash`. Task sets were real (mined from my own consultation artifacts), 12–72 tasks, val 4–30.