feat(engine): verdicts say what they looked at and what would settle a claim

CONTEXT.md

@@ -47,7 +47,7 @@ A statement about how the code or a library behaves, made by the pipeline, the P
 _Avoid_: Assertion
 
 **Verdict**:
-The outcome of checking a claim: **verified**, **refuted**, **unverifiable**, or **not checked**, always with its evidence.
+The outcome of checking a claim: **verified**, **refuted**, **unverifiable**, or **not checked**, always with its evidence and the lines the agent read for it; an unverifiable one also says what could settle it: the code, a library's source, a CI log, a sandboxed run, something outside the repository, or the author.
 _Avoid_: Result, status, score
 
 **Evidence source**:

README.md

@@ -114,7 +114,7 @@ Then the same agent lists the claims the change makes: statements about how code
 
 ### The verdicts
 
-Then the same agent judges each listed claim against the change itself — the diff and the read-only head copy — and gives it a verdict: **verified**, **refuted** or **unverifiable**, with its evidence source (the change itself, a CI log when a check failed, or the model's memory), a one-line reason, and the lines it cites as evidence, each a file, a line and a quote. The engine re-reads every citation in the head copy: when the file is not there, the line is not one of its lines, or the quote does not start on that line as written, the verdict drops to **unverifiable** and says why. The model's memory never yields **verified**, and a verdict from memory cites no line. A claim that turns on how a third-party library behaves names the library whose source it needs, and is never verified without it; fetching that source is the reviewer's choice, never the companion's (ADR 0003). An answer that leaves a claim out is retried once, and then every claim stays **not checked** and the result says why. Each claim in the result's `claims` carries its verdict, and `claims.judging` holds the verdicts prompt's version, the outcome and the run's stamp. When a check failed, the agent also reads each failed job's log, trimmed to its failing step, and may cite its lines with **a CI log** as the evidence source; the engine re-reads each such citation in that log, and the citation names the check run and is labelled as a CI log's line. Refuted and unverifiable claims are findings. The verdicts prompt is evaluated over its own cases, scored by verdict accuracy and false-verified rate against hand verdicts; see the [evaluation](packages/evaluation/README.md).
+Then the same agent judges each listed claim against the change itself — the diff and the read-only head copy — and gives it a verdict: **verified**, **refuted** or **unverifiable**, with its evidence source (the change itself, a CI log when a check failed, or the model's memory), a one-line reason, and the lines it cites as evidence, each a file, a line and a quote. The engine re-reads every citation in the head copy: when the file is not there, the line is not one of its lines, or the quote does not start on that line as written, the verdict drops to **unverifiable** and says why. The model's memory never yields **verified**, and a verdict from memory cites no line. Every verdict also lists the lines of the head copy the agent says it read for the claim (`looked`), each re-checked like a citation and dropped, with a note, when the copy does not have it, so a reviewer can tell a claim the code cannot settle from one nobody looked into; and every **unverifiable** verdict says what could settle it (`settles`) — the code itself, which makes it a finding to check, a library's source, a CI log, a sandboxed run, something outside the repository, or only the author — with a one-line note. A claim that turns on how a third-party library behaves names the library whose source it needs, and is never verified without it; fetching that source is the reviewer's choice, never the companion's (ADR 0003). An answer that leaves a claim out is retried once, and then every claim stays **not checked** and the result says why. Each claim in the result's `claims` carries its verdict, and `claims.judging` holds the verdicts prompt's version, the outcome and the run's stamp. When a check failed, the agent also reads each failed job's log, trimmed to its failing step, and may cite its lines with **a CI log** as the evidence source; the engine re-reads each such citation in that log, and the citation names the check run and is labelled as a CI log's line. Refuted and unverifiable claims are findings. The verdicts prompt is evaluated over its own cases, scored by verdict accuracy, false-verified rate and what settles each unverifiable claim against hand verdicts; see the [evaluation](packages/evaluation/README.md).
 
 ### The acceptance criteria
 

docs/reviewing.md

@@ -64,7 +64,7 @@ A **claim** is a statement about how the code or a library behaves, quoted as wr
 
 - **verified** — the change itself, or a source the engine re-read, proves it;
 - **refuted** — the evidence contradicts it;
-- **unverifiable** — nothing readable settles it;
+- **unverifiable** — the review could not settle it, and the verdict says what could settle it;
 - **not checked** — not judged (yet).
 
 Every verdict names its **evidence source**, from the [glossary](../CONTEXT.md)'s list — and the model's memory alone never yields **verified**. The engine re-reads every line a verdict cites before showing it: a citation that does not match drops the verdict to unverifiable and says why.

docs/tested-models.md

@@ -9,7 +9,7 @@ The list is the `TESTED_MODELS` constant in [`packages/engine/src/tested-models.
 | Agent | Version | Model | Effort | Run date |
 | --- | --- | --- | --- | --- |
 | Pi | 0.86.1 | `zai-coding-cn/glm-5.3` | default | 2026-10-07T15:08:38.849Z |
-| Claude Code | 2.1.296 | `claude-sonnet-5-5` | high | 2026-10-10T14:26:15.320Z |
+| Claude Code | 2.1.296 | `claude-sonnet-5-5` | high | 2026-10-12T10:06:12.771Z |
 
 One row per tested combination. The run date is the latest run recorded behind it; the scores below keep each prompt's own run date, because the recorded baseline merges the runs taken as each prompt landed.
 
@@ -23,7 +23,7 @@ The agent ranking is the default for this combination (`TESTED_RANKINGS` in [`pa
 | ranking | 2026-10-04T06:32:35.720Z | rank-median 1, rank-top-3 0.9 |
 | story | 2026-10-04T08:38:20.193Z | story-must-review 1, story-order 1, story-names 0.9898 |
 | claims | 2026-10-04T12:45:41.622Z | claims-recall 1, claims-precision 0.9545 |
-| verdicts | 2026-10-07T11:06:40.494Z | verdict-accuracy 1, false-verified 0, verify-accuracy 0.75, verify-false-verified 0, verify-fetch-offered 1 |
+| verdicts (version 5) | 2026-10-07T11:06:40.494Z | verdict-accuracy 1, false-verified 0, verify-accuracy 0.75, verify-false-verified 0, verify-fetch-offered 1 |
 | unexplained changes | 2026-10-05T17:21:41.222Z | unexplained-recall 1, unexplained-precision 1, described-recall 1, described-precision 1 |
 | criteria mapping | 2026-10-06T08:48:12.610Z | criteria-accuracy 1, criteria-false-met 0, criteria-code-recall 0.8889, criteria-tests-recall 1, criteria-manual-recall 1 |
 | draft comments | 2026-10-06T11:33:18.523Z | draft-cites-evidence 1, draft-no-new-claim 1, draft-under-cap 1 |
@@ -43,15 +43,15 @@ The agent ranking is the default for this combination (`TESTED_RANKINGS` in [`pa
 | ranking | 2026-10-10T14:26:15.320Z | rank-median 1, rank-top-3 1 |
 | story | 2026-10-10T14:26:15.320Z | story-must-review 1, story-order 0.4444, story-names 1 |
 | claims | 2026-10-10T14:26:15.320Z | claims-recall 1, claims-precision 0.95 |
-| verdicts | 2026-10-10T14:26:15.320Z | verdict-accuracy 1, false-verified 0, verify-accuracy 1, verify-false-verified 0, verify-fetch-offered 1 |
+| verdicts (version 6) | 2026-10-12T10:06:12.771Z | verdict-accuracy 1, false-verified 0, settles-accuracy 1, verdict-looked 0.96, verify-accuracy 1, verify-false-verified 0, verify-fetch-offered 1 |
 | unexplained changes | 2026-10-10T14:26:15.320Z | unexplained-recall 1, unexplained-precision 1, described-recall 1, described-precision 1 |
 | criteria mapping | 2026-10-10T14:26:15.320Z | criteria-accuracy 1, criteria-false-met 0, criteria-code-recall 1, criteria-tests-recall 1, criteria-manual-recall 1 |
 | draft comments | 2026-10-10T14:26:15.320Z | draft-cites-evidence 1, draft-no-new-claim 1, draft-under-cap 1 |
 | explain | 2026-10-10T14:26:15.320Z | explain-cites-part 1, explain-names-in-change 0.8333 |
 | cover | 2026-10-10T14:26:15.320Z | cover-cites-checked 1, cover-tests-recall 0.8, cover-tests-precision 1, cover-manual-recall 1, cover-none-found 1 |
 | documentation links | 2026-10-10T14:26:15.320Z | doc-links-on-site 1, doc-links-checked 1 |
 
-The library verdicts prompt's claim checks are recorded case by case in the [baseline](../packages/evaluation/baseline.json) for this combination too.
+The verdicts prompt's version 6, which asks for what the agent read for each claim and what could settle an unverifiable one, was run with this combination alone, on its eight cases, the two interview-coach pull requests among them; the verdicts row is that run's, and the library verdicts prompt's claim checks on the canaries were taken in the same run. Pi's verdicts row is still version 5's. The library verdicts prompt's claim checks are recorded case by case in the [baseline](../packages/evaluation/baseline.json) for this combination too.
 
 ## Adding a combination
 

packages/engine/src/library-verdicts.ts

@@ -4,7 +4,7 @@ import { decompileNuGetLibrary, type DecompileOptions } from './decompile.js';
 import type { JsonSchema } from './json-schema.js';
 import { fetchLibrary, findLibraryPin, type LibraryDownload } from './library-fetch.js';
 import { NoExactSourceError } from './nuget-fetch.js';
-import type { CheckedVerdictKind, Claim, ClaimVerdict, EvidenceSource, FetchedLibrary, LibraryFetchOffer, Part } from './protocol.js';
+import type { CheckedVerdictKind, Claim, ClaimVerdict, EvidenceSource, FetchedLibrary, LibraryFetchOffer, Part, Settles } from './protocol.js';
 import { fetchNamedRepository } from './repository-fetch.js';
 import { UNTRUSTED_INPUT_RULE, untrustedBlock } from './untrusted.js';
 import { claimPlace, claimText, copyReader, diffLines, recheckCitation, settleVerdict } from './verdicts.js';
@@ -163,7 +163,8 @@ export function holdToExactSource(verdict: ClaimVerdict, unproven: readonly stri
   const cited = [...new Set(verdict.evidence.map((each) => each.path).filter((path) => isUnprovenSource(path, unproven)))];
   if (cited.length === 0) return verdict;
   const files = cited.length === 1 ? `${cited[0]}, which is unproven` : `${cited.join(', ')}, which are unproven`;
-  return { ...verdict, kind: 'unverifiable', recheck: `the verdict cites ${files}: no hash its PDB records matches, so it may not be the source the library was built from` };
+  const settles: Settles = { kind: 'library', note: `exact source of ${cited.join(', ')} would settle it` };
+  return { ...verdict, kind: 'unverifiable', settles, recheck: `the verdict cites ${files}: no hash its PDB records matches, so it may not be the source the library was built from` };
 }
 
 /**
@@ -269,7 +270,14 @@ export async function pressLibraryFetch(parts: readonly Part[], claim: Claim, op
   const rechecked = await Promise.all(cited.map((each) => recheckCitation(read, each, where)));
   // A named repository's evidence is labelled as such, and decompiled code's as decompiled, never as pinned source.
   const source = answer.source !== LIBRARY_SOURCE ? answer.source : named !== undefined ? 'a named repository' : decompile ? 'decompiled library code' : answer.source;
-  const settled = holdToExactSource(settleVerdict({ id: '', ...answer, source, library: null }, rechecked, where), fetched.unproven ?? []);
+  // What the agent read in the head copy for the claim stays the verdicts pass's, and what still settles it is the library.
+  const { looked, lookedRecheck } = verdict;
+  const unsettled: Settles = { kind: 'library', note: `${where} did not settle it as the agent read it; the fetched source is there to read` };
+  const answered = { id: '', ...answer, source, looked: [], settles: null, library: null };
+  const settled = holdToExactSource(
+    settleVerdict(answered, rechecked, where, { looked, ...(lookedRecheck ? { lookedRecheck } : {}) }, unsettled),
+    fetched.unproven ?? [],
+  );
   if (settled.kind === 'not checked') return { outcome: 'fell back', claim, detail: 'the verdict could not be settled', stamp: result.stamp };
   const { file, sha256, archive, path, note, unproven } = fetched;
   const library: FetchedLibrary = {

packages/engine/src/protocol.ts

@@ -11,7 +11,7 @@ import type { AgentStamp } from './agent.js';
 import type { AskKind } from './asks.js';
 
 /** Version of the review result schema. */
-export const REVIEW_RESULT_VERSION = 23 as const;
+export const REVIEW_RESULT_VERSION = 24 as const;
 
 /**
  * Version 2 added the head commit's SHA and each part's noise assessment;
@@ -61,7 +61,8 @@ export const REVIEW_RESULT_VERSION = 23 as const;
  * it has new commits since; version 23 added the results a review carries
  * over from the saved review of an earlier commit: the commit and who
  * produced it, each part and claim carried over, and a stage carried over
- * whole.
+ * whole; version 24 added what each checked verdict's agent looked at in
+ * the head copy, and what could settle each unverifiable claim.
  */
 export type ReviewResultVersion = typeof REVIEW_RESULT_VERSION;
 
@@ -976,6 +977,16 @@ export type ClaimVerdict =
       libraryFetch?: LibraryFetchOffer;
       /** Why no library fetch is offered for the library the claim needs, said plainly. */
       noLibraryFetch?: string;
+      /**
+       * The lines of the head copy the agent read for this claim, each
+       * re-checked by the engine; an entry naming a file or a line the
+       * copy does not have is dropped, and said so in `lookedRecheck`.
+       */
+      looked: LookedRange[];
+      /** Why the engine dropped entries of `looked`, when it did. */
+      lookedRecheck?: string;
+      /** What could settle the claim; set on every unverifiable verdict, and only on one. */
+      settles?: Settles;
       /**
        * The library source the verdict was judged against, once the
        * reviewer pressed its library fetch; its citations are paths in
@@ -986,6 +997,34 @@ export type ClaimVerdict =
       recheck?: string;
     };
 
+/** A range of lines of one file of the head copy that the agent read for a claim. */
+export interface LookedRange {
+  /** The file, by its path in the head copy. */
+  path: string;
+  /** The 1-based first line read. */
+  line: number;
+  /** The 1-based last line read, at or after `line`. */
+  endLine: number;
+}
+
+/**
+ * What could settle an unverifiable claim: the code itself, which makes
+ * the verdict a finding to check; a library's source; a CI log; a
+ * sandboxed run; something outside the repository, such as a service or
+ * a dataset; or only the pull request's author.
+ */
+export type SettlesKind = 'code' | 'library' | 'ci-log' | 'sandboxed-run' | 'outside-repository' | 'author';
+
+/** What can settle an unverifiable claim, in the order a reviewer can act on it. */
+export const SETTLES_KINDS: readonly SettlesKind[] = ['code', 'library', 'ci-log', 'sandboxed-run', 'outside-repository', 'author'];
+
+/** What could settle an unverifiable claim, with a one-line note saying how. */
+export interface Settles {
+  kind: SettlesKind;
+  /** One plain line, such as which log or which measurement would settle it. */
+  note: string;
+}
+
 /**
  * A library fetch the companion offers with a verdict (the glossary's
  * library fetch): the download of one library's source at the version

packages/engine/src/tested-models.ts

@@ -89,7 +89,7 @@ export const TESTED_MODELS: readonly TestedModel[] = [
     agentVersion: '2.1.296',
     model: 'claude-sonnet-5-5',
     effort: 'high',
-    runDate: '2026-10-10T14:26:15.320Z',
+    runDate: '2026-10-12T10:06:12.771Z',
     scores: {
       coverage: 1,
       'grouping-agreement': 0.7946,
@@ -102,6 +102,8 @@ export const TESTED_MODELS: readonly TestedModel[] = [
       'claims-precision': 0.95,
       'verdict-accuracy': 1,
       'false-verified': 0,
+      'settles-accuracy': 1,
+      'verdict-looked': 0.96,
       'verify-accuracy': 1,
       'verify-false-verified': 0,
       'verify-fetch-offered': 1,