Repository navigation
feat(engine): verdicts say what they looked at and what would settle a claim - #182
Merged
Merged
Conversation
…a claim (verdicts prompt v6) (#173) Each checked verdict now carries `looked`: the ranges of the head copy the agent says it read for the claim. The engine re-checks every range as it re-checks a cited line and drops one naming a file or a line the copy does not have, saying so in `lookedRecheck`. Each unverifiable verdict carries `settles`, what could settle the claim, from a fixed list (code, library, ci-log, sandboxed-run, outside-repository, author), with a one-line note; a verdict the engine drops to unverifiable gets one too, and a claim judged again in a fetched library's source keeps what the agent read in the head copy. The result version goes to 22, and the extension's validator requires both fields, so a result of an older version is refused rather than misread. The verdicts prompt goes to version 6, asking for both fields, refuting a claim true only in part, and keeping a library claim unverifiable even when memory would refute it. Two cases recorded from public interview-coach pull requests label measured numbers and live runs as settled by the author, a claim that the checks pass as settled by a CI log, and claims about ranx as settled by the library. The evaluation scores settles-accuracy and verdict-looked, and the baseline records the Claude Code run of the new prompt.
lbildzinkas
force-pushed
the
fm/sl-verdicts-v6
branch
from
October 12, 2026 15:38
daf3025 to
87eeb7f
Compare
lbildzinkas
changed the title
Master took result version 23 for the results a review carries over from an earlier commit's saved review, so the looked and settles fields move to version 24, recorded after it. A carried-over claim keeps its verdict's looked and settles unchanged, and a saved review of an older result version is still refused rather than carried.
… in evaluation README
lbildzinkas
changed the title
…a claim the reviewer picks...' (extension.test.ts:1201). Root cause: the PR's protocol bump made `looked` a required array on every checked verdict (isClaimVerdict in packages/extension/src/protocol.ts), and all fixtures were updated except the inline refuted verdict in the integration test's verifiedAnswer helper, so the fake engine's ask answer failed isAskAnswer validation and never rendered. Fix: added `looked: [{ path: 'src/retry.py', line: 1, endLine: 8 }]` to that verdict (packages/extension/test/integration/extension.test.ts:1164), matching the results.ts fixture convention. Verified: the failing test passes, the full integration suite passes 81/81, and npm run check (build, typecheck, lint, 1945 unit tests) passes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
An unverifiable verdict carried only a one-sentence reason and whatever it cited, so a reviewer could not tell a claim the code cannot settle from one the agent never looked into. This change makes every checked verdict list the lines of the head copy the agent read for its claim, re-checked by the engine like a cited line, and makes every unverifiable verdict say what could settle it, from a fixed list: the code, a library's source, a CI log, a sandboxed run, something outside the repository, or the author.
The review result's version goes up and the extension accepts the new fields, so a result of an older version is refused rather than misread. The verdicts prompt moves to version 6, with evaluation cases recorded from two public pull requests whose descriptions report runs, measurements and passing checks, and its baseline is recorded in the tested-models page. Validation never launches, downloads or installs VS Code or any other tool: extension tests that need VS Code run only in CI, and live validation goes through the engine protocol, the unit tests and rendered output.
Closes #173
What Changed
looked: the head-copy files and line ranges the agent read for the claim, each re-checked by the engine like a citation, with entries naming a missing file or line dropped and noted inlookedRecheck(capped at 20 ranges). Every unverifiable verdict also carriessettles: what could settle the claim from a fixed list (code,library,ci-log,sandboxed-run,outside-repository,author) with a one-line note — taken from the agent when it says, and derived by the engine for verdicts it drops itself. The result version goes up to 24, the extension's validators accept the new fields, and a verdict checked behind a library fetch keeps itslookedwhile itssettlesbecomes the library's source.settles-accuracyandverdict-lookedscores, allows expected claims with no evidence, and records the new baseline indocs/tested-models.md.CONTEXT.md, the reviewing walkthrough, and the tested-models baseline rows.Risk Assessment
✅ Low: The change is well-bounded and internally consistent: every acceptance criterion is source-verified (looked on every checked verdict, settles on exactly the unverifiable ones with drop-path fallbacks, dropped-range recheck notes, version-24 results refused rather than misread when older), the recorded fix rounds match their prescribed narrow remedies with genuine regression tests, and the master merge coherently re-versions the carry-over result format.
Testing
Drove the change live through the engine protocol exactly as the ticket's standing boundary allows: the real verdicts pass over the recorded interview-coach#98 case (prompt v6 delivered, looked re-checked with drop notes, settles from the fixed list on every unverifiable verdict, engine-dropped verdicts refusing the agent's settles), and the real engine server against the live GitHub pull request (version-24 result, saved-at-once control, older-version saved record ignored and replaced). The offline evaluation gate passes over the new cases against the recorded v6 baseline, and all targeted engine/evaluation/extension unit tests pass. No rendered-UI evidence was produced because the change touches no rendered surface — the extension edits are protocol validators, VS Code is never launched on this machine, and the settles UI is deferred to #107.
npm run evalexit 0 over both new interview-coach cases + baseline.json claude-code v6 rows…Evidence: Live verdicts drive: engine judgeClaims over recorded interview-coach#98, scripted agent via the adapter seam (15/15 checks)
Adversarial excerpt — claim 'A str seed is hashed with SHA-512…' answered verified citing search_eval.py:999 with settles {kind:'author'}: {"kind":"unverifiable","looked":[{"path":"src/interview_coach/search_eval.py","line":50,"endLine":60}],"recheck":"the citation src/interview_coach/search_eval.py:999 names a line src/interview_coach/search_eval.py does not have","settles":{"kind":"code","note":"the agent judged it verified, so the code should settle it: check it"}}. Dropped-range note on the live-run claim: 'dropped what the agent read at src/interview_coach/search_eval.py:1-9999 names a line … does not have; src/interview_coach/missing.py:1-5 names a file the head copy does not have; src/interview_coach/cli.py:9 names a line src/interview_coach/cli.py does not have'Evidence: Live engine-server drive against the real interview-coach#98: version 24, saved-at-once, older-version record ignored (7/7 checks)
fresh review result is version 24 (9 parts); next visit served from the saved result at once; after rewriting the saved record to version 23 with a TAMPERED marker: 'older-version saved result is ignored, not served' PASS, 'review after an older-version record is a fresh version 24 result' PASS, 'the ignored record was replaced by a fresh current-version save' PASS (disk record 24/24)Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 1 issue found → auto-fixed ✅
packages/evaluation/README.md:206- The "prompts so far" registry paragraph still ends "the verdicts prompt's version 5 names a reviewer's selection as where its claim is made", but this change bumps the verdicts prompt to version 6 everywhere else in the same paragraph's verdicts description, in packages/evaluation/prompts.json and in docs/tested-models.md. The paragraph describes the current prompts (it closes with "Bump a prompt's version there and here together"), so the reference is now stale; say version 6, or "since version 5" if the historical reading is meant. Doc accuracy only — no code behavior depends on it.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
npm run evalexit 0 over both new interview-coach cases + baseline.json claude-code v6 rows…npx vitest run packages/engine/test/verdicts.test.ts packages/engine/test/review.test.ts packages/engine/test/verify.test.ts packages/engine/test/carry-over.test.ts packages/engine/test/library-verdicts.test.ts packages/engine/test/library-fetch.test.ts packages/engine/test/cli.test.ts packages/engine/test/draft-comment.test.ts— 164 tests passnpx vitest run packages/engine/test/server.test.ts packages/engine/test/saved-review.test.ts— 59 tests passnpx vitest run packages/evaluation/test— 149 tests pass (score, case, run, claims, cli)npx vitest run packages/extension/test/review-result.test.ts packages/extension/test/engine-client.test.ts packages/extension/test/findings.test.ts packages/extension/test/overview.test.ts packages/extension/test/asks.test.ts packages/extension/test/draft-comment.test.ts packages/extension/test/side-bar-page.test.ts packages/extension/test/side-bar.test.ts packages/extension/test/tree.test.ts— 296 tests passnpm run build— compiles cleannpm run eval— exit 0; 'baseline: 0 dropped, 0 missing, 0 gained, 105 unchanged', both new interview-coach cases covered; baseline.json inspected: claude-code/claude-sonnet-5-5/high rows stamped verdicts v6 with verdict-accuracy 1, false-verified 0, settles-accuracy 1, verdict-looked on the new casesLive drive 1:node .live-tmp/live-verdicts.mjs(scratch, deleted after) — real enginejudgeClaimsover the recorded lbildzinkas-interview-coach-98 case via the product's AgentAdapter seam with a scripted agent answering the hand-labelled claims; 15/15 checks pass (evidence: live-verdicts-drive.json)Live drive 2:GITHUB_TOKEN="$(gh auth token)" node .live-tmp/live-serve.mjs(scratch, deleted after) — real engine JSON-RPC server (initialize + review) against the real feat: add hand-written search metrics and the coach eval qasper random baseline interview-coach#98 with a disposable cache dir; 7/7 checks pass, including tampering the saved review-result.json to version 23 and proving it is ignored (evidence: live-serve-drive.json)✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.