Skip to content

Treat only an explicit refusal as not merged in the merge skills - #148

Merged
aoshimash merged 4 commits into
mainfrom
fix/merge-outcome-timeout-fail-closed
Oct 25, 2026
Merged

aoshimash merged 4 commits into
mainfrom
fix/merge-outcome-timeout-fail-closed

Conversation

@aoshimash

@aoshimash aoshimash commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #147 (base feat/merge-skills-platform-access). Retarget to main once #147 merges.

Decisions

  • Only an explicit refusal means "not merged". The merge call's own result is classified first: an explicit refusal is an errors payload whose message gives a reason the merge is not allowed, or the access method declining to submit because it read the PR as unmergeable. Everything else — timeout, transport error, an HTTP status error of any code, a killed command, a payload reporting a timeout/internal failure, a success exit — is not a refusal, and a not-merged read after it is the existing merge outcome unconfirmed escalation (stop the line, flip-blocking in merge-issue-prs).
  • Uncertain → not a refusal. A refusal misread that way costs a stopped line, never a merge.
  • The head re-read still discriminates among refusals (merge-issue-prs). Error text decides only whether there was a refusal at all. A changed head without an explicit refusal is not read as the guard firing.
  • Bounded settle re-read: not taken. No window length proves a merge did not land, so it would make escalations rarer without removing the fail-open.
  • SKILL.md "Platform Access" in both skills gets one sentence ("an open PR proves nothing either"), since the always-loaded text is where the fail-open wording lived.

Risk areas

  • The CLI mapping is read from source, not observed on a live failing merge; no merge was submitted. On the mutation path, GraphQL: … is an errors payload in a 200 response and non-200 OK status code: … is any other status. gh's pre-submit is not mergeable: (plus its --auto/--admin hints) means nothing was sent. On a merge-queue branch gh exits 0 after the auto-merge mutation. Sources: gh v2.103.2 pkg/cmd/pr/merge/merge.go, http.go, internal/ghcmd/cmd.go; go-gh v2.17.0 pkg/api/graphql_client.go, errors.go; cli/shurcooL-graphql v0.0.4 graphql.go.
  • Classifying an errors payload as a refusal is a judgment over its message text. Reading a timeout or internal-error payload as a refusal would still fail open; the only guard is the "uncertain → not a refusal" instruction. This is recorded as a known limit in both evaluation logs.
  • merge-renovate-prs's merge carries no head guard, so it has no head-mismatch refusal. Adding one is out of scope (see Separately).

Acceptance criteria → evidence

Criterion Evidence
A timeout or transport error followed by an OPEN, head-unchanged read no longer defers merge-issue-prs workflow.md 2-3 table + 2-5 rows; merge-renovate-prs workflow.md 2-6 table
Only an explicit refusal (payload, or the head guard in merge-issue-prs) counts as not merged same, plus both platform-github.md Merge sections
Eval case per skill merge-issue-prs Case 42 (+ Case 23 setup now states a refusal payload); merge-renovate-prs Case 15; both evals/evals.json
Evaluation-log entry per skill both references/eval-cases.md Evaluation Logs
Commands run as written the confirmation reads were extracted from the files and run against #147 / #144 and aoshimash/homelab-k8s#389
Desk-check by a separate agent instance Three rounds, each with a fresh instance. Round 3: merge-issue-prs Cases 23 and 42 Pass, merge-renovate-prs Case 15 Pass, no fail-open. Rounds 1–2 were Conditional, and every finding was fixed (both Evaluation Logs list them).

Mechanical changes

  • merge-renovate-prs platform-github.md gained the confirmation read (gh pr view {pr} --json state,mergeCommit …) it lacked.
  • Explicit refusals in merge-renovate-prs now defer explicitly (the old text named no outcome for a refused merge); SKILL.md step 6, 2-5 and 3-1 list it, and 3-1 / Phase 3 report merge outcome unconfirmed.
  • merge-issue-prs: 2-3 gained a row for "merged but merge commit not shown reachable" (unconfirmed), mirrored in 2-5; milestone-pr.md M2's State column gained escalated — merge outcome unconfirmed.

Separately

  • merge-renovate-prs merges without --match-head-commit, so a Renovate push between the pre-check and the merge is not guarded on the platform side.
  • Pre-existing: a merge outcome unconfirmed escalation is not persisted. A later (scheduled) run cannot see it and may merge on top of a late-landing, unverified merge. This change makes that escalation more frequent, so it is worth a follow-up issue (durable marker + Phase 0 check).

🤖 Generated with Claude Code

aoshimash and others added 3 commits October 25, 2026 14:52
A merge call that timed out or hit a transport error was followed by a
confirmation read, and an open PR with an unchanged head was treated as a
refusal: the PR deferred and the loop continued. A timed-out merge can still
complete on the server after that read, breaking one-merge-in-flight and
leaving an unverified merge on the integration branch (merge-issue-prs) or
the default branch (merge-renovate-prs).

Both skills now classify the merge call's own result first. Only an explicit
refusal (an errors payload giving a reason, or the access method declining
to submit) can conclude "not merged"; every other not-merged outcome,
including a success exit with the PR still open, is a merge outcome
unconfirmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gaps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aoshimash
aoshimash marked this pull request as ready for review October 25, 2026 06:18
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from feat/merge-skills-platform-access to main October 25, 2026 06:48
@aoshimash
aoshimash merged commit 686ddf9 into main Oct 25, 2026
1 check passed
@aoshimash
aoshimash deleted the fix/merge-outcome-timeout-fail-closed branch October 25, 2026 06:48
github-actions Bot added a commit that referenced this pull request Oct 25, 2026
Bump level: minor — The diff changes merge-gate decision rules in merge-issue-prs SKILL.md and its references: a merge call that ends without an explicit refusal and does not read as merged now stops the line and escalates, where before it deferred. (PR #148)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant