Skip to content

Phase 6b: manual OpenAI smoke workflow (real run, gated) - #52

Merged
samuelrojas-dev merged 1 commit into
mainfrom
claude/sleepy-noether-3oc5nl
Oct 2, 2026
Merged

samuelrojas-dev merged 1 commit into
mainfrom
claude/sleepy-noether-3oc5nl

Conversation

@samuelrojas-dev

Copy link
Copy Markdown
Owner

What

Phase 6b infrastructure: the safe, manual way to make the one authorized real OpenAI call that proves the Phase 6a adapter works end to end against the live API (auth, request mapping, usage mapping, pricing, cassette recording). This PR makes no real call — opening or merging it spends nothing. The real call only happens when a human manually dispatches the workflow after merge.

Two files:

  • configs/smoke-openai.yaml — the smallest meaningful real run: the attack bloat-verify-001 and its paired benign baseline (the benchmark always runs both so amplification is measurable), none defense, max_turns: 1 → one model call per episode. 2 episodes; worst case 4,000 tokens / $0.0020 (offline dowbench estimate).
  • .github/workflows/smoke-openai.yml — a workflow_dispatch-only job, the sole place in the repo a real provider call is made.

Why a dedicated smoke config (not pilot-openai.yaml)

The full pilot-openai.yaml is 10 episodes, worst case $0.16 — it would be refused by the --budget-usd 0.05 guard. The smoke config matches the authorized scope (≈1 call per episode, worst case $0.0020, well under $0.01).

Key safety

  • The workflow never runs on push or pull_request — only manual workflow_dispatch, and the job is gated on a typed confirm: run input (if: inputs.confirm == 'run').
  • OPENAI_API_KEY is injected only on the run step's env, not job-wide, so no other step can read it. The secret never leaves GitHub Actions.
  • permissions: contents: read — the workflow commits nothing back.
  • dowbench never logs the key (ADR 0005) and the cassette is sanitized (ADR 0009/0015), so the uploaded artifact carries no secret.
  • Hard --budget-usd 0.05 cap; an offline dowbench estimate runs first as a preflight.
  • The run dir (summary, episodes, calls, run, cassette) is uploaded as an artifact (7-day retention) for review.

How the real run happens (after this merges)

workflow_dispatch workflows are only dispatchable from the default branch, so the real call cannot run from this PR branch — an extra safety gate. The flow:

  1. Review & merge this PR (no spend).
  2. Actions → smoke-openai → Run workflow on main, type confirm: run.
  3. Review the uploaded smoke-openai-run artifact.
  4. Optional follow-up PR: publish the sanitized cassette under results/cassettes/ so replay-verify reproduces it offline forever.

Kind

  • fix
  • feature
  • refactor / chore (CI infrastructure)
  • docs
  • test

Checklist

  • Small, focused change; Conventional Commits.
  • ADR n/a — the design decisions (post-call ceiling, usage mapping, no codec) are already in ADR 0023; this only adds the run harness.
  • No invented numbers — the $0.0020 worst case is from a real offline dowbench estimate; no results are claimed, the real run is deferred to a human dispatch.
  • No real provider call added to tests or scripts; tests stay offline/mocked. The real call lives only in a manual, human-gated workflow.
  • No secrets in code, logs, tests or commits; the key is read from the Actions secret, scoped to one step.
  • Expected-behavior change: a config-enumeration test now also validates smoke-openai.yaml (351 vs 350 tests).

Verification

$ ruff check .        && ruff format --check .
All checks passed!

$ python -m pytest
351 passed

$ dowbench estimate configs/smoke-openai.yaml
episodes: 2 (pending 2)
worst case: 4,000 tokens, $0.0020
model calls: at most 4, checked after each call (no pre-call token count)

🤖 Generated with Claude Code

https://claude.ai/code/session_014wCFhM2gxQCfJYcm4Zu6z4


Generated by Claude Code

Add configs/smoke-openai.yaml (2 episodes: the attack bloat-verify-001 and
its benign baseline, one model call each; worst case ~4,000 tokens / $0.0020)
and .github/workflows/smoke-openai.yml, a workflow_dispatch-only job that is
the sole place a real OpenAI call is made. It never runs on push or PR, needs
a typed confirm input, injects OPENAI_API_KEY only on the run step, runs under
a hard --budget-usd 0.05 cap, commits nothing back, and uploads the run dir
(summary, episodes, calls, run, cassette) as an artifact for review.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014wCFhM2gxQCfJYcm4Zu6z4
@samuelrojas-dev
samuelrojas-dev merged commit 23e6162 into main Oct 2, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants