Repository navigation
Phase 6b: manual OpenAI smoke workflow (real run, gated) - #52
Merged
Merged
Conversation
Add configs/smoke-openai.yaml (2 episodes: the attack bloat-verify-001 and its benign baseline, one model call each; worst case ~4,000 tokens / $0.0020) and .github/workflows/smoke-openai.yml, a workflow_dispatch-only job that is the sole place a real OpenAI call is made. It never runs on push or PR, needs a typed confirm input, injects OPENAI_API_KEY only on the run step, runs under a hard --budget-usd 0.05 cap, commits nothing back, and uploads the run dir (summary, episodes, calls, run, cassette) as an artifact for review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014wCFhM2gxQCfJYcm4Zu6z4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Phase 6b infrastructure: the safe, manual way to make the one authorized real OpenAI call that proves the Phase 6a adapter works end to end against the live API (auth, request mapping, usage mapping, pricing, cassette recording). This PR makes no real call — opening or merging it spends nothing. The real call only happens when a human manually dispatches the workflow after merge.
Two files:
configs/smoke-openai.yaml— the smallest meaningful real run: the attackbloat-verify-001and its paired benign baseline (the benchmark always runs both so amplification is measurable),nonedefense,max_turns: 1→ one model call per episode. 2 episodes; worst case 4,000 tokens / $0.0020 (offlinedowbench estimate)..github/workflows/smoke-openai.yml— aworkflow_dispatch-only job, the sole place in the repo a real provider call is made.Why a dedicated smoke config (not
pilot-openai.yaml)The full
pilot-openai.yamlis 10 episodes, worst case $0.16 — it would be refused by the--budget-usd 0.05guard. The smoke config matches the authorized scope (≈1 call per episode, worst case $0.0020, well under $0.01).Key safety
pushorpull_request— only manualworkflow_dispatch, and the job is gated on a typedconfirm: runinput (if: inputs.confirm == 'run').OPENAI_API_KEYis injected only on the run step'senv, not job-wide, so no other step can read it. The secret never leaves GitHub Actions.permissions: contents: read— the workflow commits nothing back.dowbenchnever logs the key (ADR 0005) and the cassette is sanitized (ADR 0009/0015), so the uploaded artifact carries no secret.--budget-usd 0.05cap; an offlinedowbench estimateruns first as a preflight.How the real run happens (after this merges)
workflow_dispatchworkflows are only dispatchable from the default branch, so the real call cannot run from this PR branch — an extra safety gate. The flow:main, typeconfirm: run.smoke-openai-runartifact.results/cassettes/soreplay-verifyreproduces it offline forever.Kind
Checklist
dowbench estimate; no results are claimed, the real run is deferred to a human dispatch.smoke-openai.yaml(351 vs 350 tests).Verification
🤖 Generated with Claude Code
https://claude.ai/code/session_014wCFhM2gxQCfJYcm4Zu6z4
Generated by Claude Code