Skip to content

chore(ci): build the nightly preview even when Full Suite fails - #1404

Merged
apackeer merged 3 commits into
mainfrom
chore/nightly-build-despite-test-failures
Sep 25, 2026
Merged

apackeer merged 3 commits into
mainfrom
chore/nightly-build-despite-test-failures

Conversation

@apackeer

@apackeer apackeer commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The nightly Preview Release stopped at Release tests whenever any Full Suite job failed. A single failing test or live shard meant no preview that night (for example, runs 36060429097 and 35785443087 skipped every build job after Release tests failed).

Change

  • Release tests now requires only that Full Suite finished (success or failure). It downloads the full-suite-* evidence, lists this run's jobs, and renders scripts/ci-preview-test-report.ts into the run summary and a preview-test-report artifact. The report lists the failed legs, the failed jobs (grouped by leg, with links) and the failing test files and cases. Each artifact contributes only its own run, so the runner tests' deliberately failing fixture runs (t112 calibration, t05's NO_SUCH filter) never show up. Missing evidence or an unavailable job list narrows the report instead of blocking the build. Only this job gains actions: read.
  • Build chain: a job's implicit success() also covers its ancestors (Jobs skipped when NEEDS job ran successfully actions/runner#2205), so every job below Release tests would still have been skipped. native-smoke through release now use !cancelled() plus a success check on each direct need. That keeps the existing tests-then-build order. t332 fails if a job below Release tests is missing that condition.
  • Release: when Full Suite failed, the published preview's notes start with a warning line and end with the report. Passing previews keep their notes unchanged. ci-preview-test-report.ts --stage-notes keeps every planned line and the source footer. The report, then the warning, gives way to GitHub's 125,000-character release limit, so a failing suite never stops a preview whose planned notes fit.
  • Release result: a new final step fails the run when Full Suite failed, so the nightly stays red and the scheduled-failure alert still goes out, even though the preview published.

Unchanged: stable releases (they never consume Full Suite evidence), the native-smoke build checks (t238/t244 against the built binaries), the contract gate, and the unchanged-source publication skip.

Testing

  • Ran the report script against the real evidence and job list from run 36060429097. It named the 7 failing Windows test files and their cases, excluded the fixture failures, grouped 193 failed jobs into 4 lines, and came to 7.6 KB.
  • Rebased onto fix(ci): preserve full-suite evidence and verify completion #1369. The affected unit files (t-ci-preview-test-report, t332, t345, t244, t239, t-runner-production-guards) pass through tests/run-tests.ts: 206 cases. The smoke tier passes too.
  • The new test follows fix(ci): preserve full-suite evidence and verify completion #1369's test-budget convention (default timeout, budgeted spawn and cleanup timeouts).
  • With fix(ci): preserve full-suite evidence and verify completion #1369 uploading traces by default, recent trace-enabled Full Suite runs produced 59 to 130 MiB of full-suite-* evidence, which is fine for the report job to download.
  • bun run check and bun tests/gen-coverage-registry.ts --check pass.
  • Mutation check: removing build's condition makes t332 fail with "build must not inherit the Full Suite result".
  • actionlint 1.7.7 and markdownlint report no new findings compared with main.
  • Not exercised end to end: preview-release.yml only runs on main, so the first real proof is the next nightly (or a manual dispatch) after merge.

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.

The nightly Preview Release stopped at "Release tests" whenever any
Full Suite job failed, so a single failing test or live shard meant no
preview at all. The build now continues and the failures travel with it.

Release tests: instead of requiring a passing Full Suite, the job
requires only that the suite finished (success or failure). It then
downloads the full-suite-* evidence, lists this run's jobs, and renders
scripts/ci-preview-test-report.ts into the run summary and a
preview-test-report artifact. The report names the failed legs, the
failed jobs (grouped by leg, linked), and the failing test files and
cases. Each artifact contributes only its own run: the stamp its job
recorded, else the earliest, so the runner tests' deliberately failing
fixture runs never appear. Missing evidence or an unavailable job list
narrows the report instead of blocking the build. The job alone gains
actions: read, to list the jobs.

Build chain: a job's implicit success() also covers its ancestors
(actions/runner#2205), so with Full Suite failed every job below
Release tests would still be skipped. native-smoke through release now
state `!cancelled()` plus a success check of each direct need, which
keeps the existing tests-then-build order. t332 fails if a job below
Release tests lacks that explicit condition.

Release: a preview built from a failing suite opens its notes with a
warning and ends them with the report. Passing previews keep their
notes unchanged. The report is capped at 30,000 characters so the notes
stay under GitHub's body limit.

Release result: a new final step fails the run when Full Suite failed,
so the scheduled run stays red and still alerts, even though the preview
published. An unchanged source still skips publication, and the run
still fails when its tests fail.

Stable releases are unaffected; they never consume Full Suite evidence.
The native-smoke build checks (t238/t244 on the built binaries) remain
gates.

t332 now pins the non-blocking path, the notes staging, and the red
result. t-ci-preview-test-report covers the report script. Docs
(DEVELOPERS.md, the testing and supply-chain references, the install
guide's release channels, tests/README.md) and the ci.yml header no
longer say that Full Suite failures block the preview.
@github-actions github-actions Bot added documentation Improvements or additions to documentation github labels Sep 25, 2026
@github-actions github-actions Bot added aida:review-error AIDA could not produce a reliable review for the latest PR state and removed aida:review-error AIDA could not produce a reliable review for the latest PR state labels Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

AIDA findings ledger

ID Sev Status Title Decided by
F1 P1 ✅ resolved Near-limit planned notes still abort failed-suite publication AIDA · 2dbcf24
F2 P3 ✅ resolved Setup failures are inaccurately described as failing tests AIDA · 458a248
F3 P3 ✅ resolved Contributor reference still says Full Suite gates preview publication AIDA · 458a248
F4 P3 🔴 open Job-level result still misstates report availability —
F5 P2 🔴 open Near-limit failed previews can lose every failure marker —

Open blocking findings (P0/P1): 0. Accepted and rejected findings never count toward the next action.

Maintainer commands (repository write access) — put them on the first lines of a comment, one per line, several ids per line allowed:
/aida accept F# [F#…] <reason> · /aida reject F# [F#…] <reason> · /aida reopen F# [F#…] · /aida status · /aida full (next review covers the whole head)
P0 and P1 findings can be accepted (visible, risk owned by the maintainer) but not rejected. A comment is applied all-or-nothing.
Do not edit this comment: AIDA verifies its digest and refuses to run on an edited ledger. To start over, delete it.

ledger.json
{
  "version": 4,
  "pullRequest": 1404,
  "nextId": 6,
  "findings": [
    {
      "id": "F1",
      "priority": "P1",
      "category": "contracts",
      "title": "Near-limit planned notes still abort failed-suite publication",
      "anchors": [
        {
          "kind": "line",
          "path": "scripts/ci-preview-test-report.ts",
          "side": "RIGHT",
          "sha256": "e87dd06be2ca5b772e3955d2412bc9e2f066abb2466ec28f1cc978303daf809f"
        },
        {
          "kind": "line",
          "path": ".github/workflows/preview-release.yml",
          "side": "RIGHT",
          "sha256": "19f97a73d97c3336ecddae48186661311cdbcfc1294ea33890017a1dc87538f1"
        },
        {
          "kind": "line",
          "path": "scripts/ci-preview-test-report.ts",
          "side": "RIGHT",
          "sha256": "0017b33c2a4b4f9d30d49c86f9df9ec51df2fd8e973e31b96161ef5046564370"
        },
        {
          "kind": "line",
          "path": "docs/reference/09-testing.md",
          "side": "RIGHT",
          "sha256": "8fcbb119bb3f1e793806135a09062eff4c07c1407537f09841a4a25e2690ae49"
        }
      ],
      "status": "resolved",
      "firstSeen": {
        "head": "4df8f704600f34f3ba8042844a755e411481f4a3",
        "at": "2026-09-25T09:27:45.576Z"
      },
      "lastSeen": {
        "head": "2dbcf241bfc6c5e0adaad124b0512d80c0316d2b",
        "at": "2026-09-25T10:02:30.342Z"
      }
    },
    {
      "id": "F2",
      "priority": "P3",
      "category": "user-experience",
      "title": "Setup failures are inaccurately described as failing tests",
      "anchors": [
        {
          "kind": "line",
          "path": ".github/workflows/preview-release.yml",
          "side": "RIGHT",
          "sha256": "19f97a73d97c3336ecddae48186661311cdbcfc1294ea33890017a1dc87538f1"
        },
        {
          "kind": "line",
          "path": "scripts/ci-preview-test-report.ts",
          "side": "RIGHT",
          "sha256": "f901eac510a69f10b32b112ed295f60bb9239697b6a59175cf8bcee6be64debc"
        }
      ],
      "status": "resolved",
      "firstSeen": {
        "head": "4df8f704600f34f3ba8042844a755e411481f4a3",
        "at": "2026-09-25T09:27:45.576Z"
      },
      "lastSeen": {
        "head": "458a2488fd805c9cc82b5f92ec40adb57da3d4d2",
        "at": "2026-09-25T09:46:23.113Z"
      }
    },
    {
      "id": "F3",
      "priority": "P3",
      "category": "contracts",
      "title": "Contributor reference still says Full Suite gates preview publication",
      "anchors": [
        {
          "kind": "line",
          "path": ".github/workflows/preview-release.yml",
          "side": "RIGHT",
          "sha256": "55f9fa861ff6a48ca0d0738f7330aaa4b600f7d29dcdf9d6b889743f0b5b35bb"
        },
        {
          "kind": "line",
          "path": ".github/workflows/preview-release.yml",
          "side": "RIGHT",
          "sha256": "439647bfab762136d85af263456e76d52bae2ace63a93e07a9435cc3a624a5f8"
        }
      ],
      "status": "resolved",
      "firstSeen": {
        "head": "4df8f704600f34f3ba8042844a755e411481f4a3",
        "at": "2026-09-25T09:27:45.576Z"
      },
      "lastSeen": {
        "head": "458a2488fd805c9cc82b5f92ec40adb57da3d4d2",
        "at": "2026-09-25T09:46:23.113Z"
      }
    },
    {
      "id": "F4",
      "priority": "P3",
      "category": "user-experience",
      "title": "Job-level result still misstates report availability",
      "anchors": [
        {
          "kind": "line",
          "path": ".github/workflows/preview-release.yml",
          "side": "RIGHT",
          "sha256": "4853db26a78f92be2e74f14858d23b3674406493d51fa62585ed63350a50e613"
        },
        {
          "kind": "line",
          "path": ".github/workflows/preview-release.yml",
          "side": "RIGHT",
          "sha256": "b80097d28e476b0c2bd41078c752ea6c75b92df0121768787f6a99e01b945a21"
        }
      ],
      "status": "open",
      "firstSeen": {
        "head": "458a2488fd805c9cc82b5f92ec40adb57da3d4d2",
        "at": "2026-09-25T09:46:23.113Z"
      },
      "lastSeen": {
        "head": "2dbcf241bfc6c5e0adaad124b0512d80c0316d2b",
        "at": "2026-09-25T10:02:30.342Z"
      }
    },
    {
      "id": "F5",
      "priority": "P2",
      "category": "contracts",
      "title": "Near-limit failed previews can lose every failure marker",
      "anchors": [
        {
          "kind": "line",
          "path": "scripts/ci-preview-test-report.ts",
          "side": "RIGHT",
          "sha256": "783312ac35eb7b3da84ac8abe2ea173389c730ddf7ce839461c7a2b62b934c5e"
        },
        {
          "kind": "line",
          "path": "tests/unit/t-ci-preview-test-report.test.ts",
          "side": "RIGHT",
          "sha256": "d551ebefbdb0ccbce41aea6c5a8bdd3de84be9008160855e06ad7fa35e04367d"
        },
        {
          "kind": "line",
          "path": "docs/reference/09-testing.md",
          "side": "RIGHT",
          "sha256": "9f84c94f31e671c0a35f1a10accb5b821aec93e1e8d6d1b9b28163ac3b26c27f"
        }
      ],
      "status": "open",
      "firstSeen": {
        "head": "2dbcf241bfc6c5e0adaad124b0512d80c0316d2b",
        "at": "2026-09-25T10:02:30.342Z"
      },
      "lastSeen": {
        "head": "2dbcf241bfc6c5e0adaad124b0512d80c0316d2b",
        "at": "2026-09-25T10:02:30.342Z"
      }
    }
  ],
  "events": [
    {
      "at": "2026-09-25T09:27:45.576Z",
      "kind": "opened",
      "by": "aida",
      "id": "F1",
      "head": "4df8f704600f34f3ba8042844a755e411481f4a3"
    },
    {
      "at": "2026-09-25T09:27:45.576Z",
      "kind": "opened",
      "by": "aida",
      "id": "F2",
      "head": "4df8f704600f34f3ba8042844a755e411481f4a3"
    },
    {
      "at": "2026-09-25T09:27:45.576Z",
      "kind": "opened",
      "by": "aida",
      "id": "F3",
      "head": "4df8f704600f34f3ba8042844a755e411481f4a3"
    },
    {
      "at": "2026-09-25T09:46:23.113Z",
      "kind": "seen",
      "by": "aida",
      "id": "F1",
      "head": "458a2488fd805c9cc82b5f92ec40adb57da3d4d2"
    },
    {
      "at": "2026-09-25T09:46:23.113Z",
      "kind": "resolved",
      "by": "aida",
      "id": "F2",
      "head": "458a2488fd805c9cc82b5f92ec40adb57da3d4d2",
      "reason": "declared corrected by the judge; a cited line is gone"
    },
    {
      "at": "2026-09-25T09:46:23.113Z",
      "kind": "resolved",
      "by": "aida",
      "id": "F3",
      "head": "458a2488fd805c9cc82b5f92ec40adb57da3d4d2",
      "reason": "declared corrected by the judge; cited files changed since the last review"
    },
    {
      "at": "2026-09-25T09:46:23.113Z",
      "kind": "opened",
      "by": "aida",
      "id": "F4",
      "head": "458a2488fd805c9cc82b5f92ec40adb57da3d4d2"
    },
    {
      "at": "2026-09-25T10:02:30.342Z",
      "kind": "seen",
      "by": "aida",
      "id": "F4",
      "head": "2dbcf241bfc6c5e0adaad124b0512d80c0316d2b"
    },
    {
      "at": "2026-09-25T10:02:30.342Z",
      "kind": "resolved",
      "by": "aida",
      "id": "F1",
      "head": "2dbcf241bfc6c5e0adaad124b0512d80c0316d2b",
      "reason": "declared corrected by the judge; a cited line is gone"
    },
    {
      "at": "2026-09-25T10:02:30.342Z",
      "kind": "opened",
      "by": "aida",
      "id": "F5",
      "head": "2dbcf241bfc6c5e0adaad124b0512d80c0316d2b"
    }
  ],
  "review": {
    "head": "2dbcf241bfc6c5e0adaad124b0512d80c0316d2b",
    "readiness": 4,
    "risk": 2,
    "decision": "merge"
  }
}

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 4df8f704600f34f3ba8042844a755e411481f4a3 against 5975205f6aff222b9ab85d515d138f53abb2fcb4 and current repository behavior.

Inspection: 10 changed files. Scope: full head (first review of this pull request).

Final Assessment

Human decision aid only: Readiness 5/5 is best; Risk 1/5 is best. These scores inform the maintainer; the next action below follows finding severity (any open P0/P1 → author/change) and does not approve or merge the PR.

Readiness: 3/5 — The main DAG change is coherent and well covered by focused tests, but the failed-suite publication path still has a release-body boundary that can defeat its primary purpose. Two smaller reporting and documentation inconsistencies also remain.

Risk: 3/5 — The publication failure requires unusually large but valid planned notes, so blast radius is moderate. It blocks the nightly preview rather than corrupting releases or weakening asset verification.

Decision required: Author — make changes before this PR proceeds. The combined release-body limit defect can still block preview publication after a Full Suite failure, contradicting the primary behavior introduced by this PR.

Validation performed:

  • Inspected all ten changed paths, the full diff, review scope, metadata, discussion, empty findings ledger, and specialist candidates.
  • Traced the preview workflow DAG, release planner/publisher, update discovery, Full Suite artifacts, log sanitization, test runner failure format, and related documentation contracts.
  • Established statically that direct-need conditions allow a completed Full Suite failure through the build while preserving contract, build, lifecycle, publication, and final-result checks.
  • Verified immutable head snapshots against the stated head and found no diff whitespace errors.

Findings: 1 blocking, 2 advisory.

Ledger: 3 open, 0 retained blocking, 0 accepted, 0 suppressed as rejected by a maintainer. Maintainers act on findings with /aida commands in the ledger comment.

Contracts & Compatibility

P1 [F1]: Combined notes and report can exceed the release-body limit

Evidence: scripts/ci-preview-test-report.ts:15, .github/workflows/preview-release.yml:690.

Problem: When the uncapped planner produces an otherwise valid release body above roughly 95,000 characters, the failure path appends a report of up to 30,000 characters plus its warning. The combined body then exceeds the stated 125,000-character GitHub limit, so publish-release receives an API failure and no preview is published.

Impact: A failing Full Suite can still block the exact preview publication this change promises to preserve. This is a concrete release-path correctness failure, although it requires large accumulated changelog notes.

Required correction: Apply the limit to the combined warning, planned notes, and report. Dynamically budget or truncate the report while preserving required note content and the source footer, validate the final body before publication, and add a near-limit failed-suite test.

P3 [F3]: Contributor reference still says Full Suite gates preview publication

Evidence: .github/workflows/preview-release.yml:103, .github/workflows/preview-release.yml:165.

Problem: The unchanged authoritative contributor guide at docs/reference/11-contributing.md:97 still says preview builds are gated through the full deterministic/live suite, while this head explicitly allows a failed suite through publication.

Impact: Contributors and release maintainers receive conflicting guidance about why a red Full Suite run can still publish a preview.

Required correction: Update docs/reference/11-contributing.md to state that contract and release-asset checks gate publication while Full Suite failures are reported and keep the run red; extend the documentation contract test to guard this wording.

User Experience

User experience change: Preview users may now receive a prerelease after Full Suite failure, while release maintainers receive a red workflow, summary, artifact, and release-note report.

Before: A Full Suite failure stopped the preview build and publication chain.

After: A completed Full Suite failure permits publication with a warning and diagnostic report, then the final result keeps the workflow red.

Example: A nightly test failure can publish a preview whose notes begin with a warning and end with failed-job and test evidence.

Assessment: This improves access to testable preview artifacts and failure context, but setup-only failures are described inaccurately as failing tests, and large notes can still prevent publication.

P3 [F2]: Setup failures are inaccurately described as failing tests

Evidence: .github/workflows/preview-release.yml:690, scripts/ci-preview-test-report.ts:158.

Problem: If Full Suite fails during planning, setup, or provisioning before a test failure is recorded, the report correctly says no failing test files were found and lists the failed job, but the release warning and user documentation still promise that failing tests appear at the end.

Impact: Preview users receive contradictory recovery information and may look for nonexistent test failures instead of following the failed-job link.

Required correction: Describe the appended content as a Full Suite failure report rather than failing tests, distinguish missing test evidence from actual test failures, and align the guide and test documentation.

Residual risk: Repository code and tests were not executed under the read-only review contract. GitHub Actions expression and hosted-workflow behavior were assessed statically.

Reviewed by AIDA (AI-DLC Developer Agent).

[AI-PR-REVIEWED] 4df8f70

@github-actions github-actions Bot added action:change AIDA indicates changes are required before the PR proceeds aida:reviewed AIDA successfully reviewed the latest PR state next:author AIDA indicates the PR author needs to act next labels Sep 25, 2026
Addresses AIDA on #1404 (F1, F2, F3).

F1: the 30,000-character cap bounded only the report, so planned notes
above about 95,000 characters plus the warning and report could exceed
GitHub's 125,000-character release body and fail publication, the case
this change exists to allow. The release job now stages the notes with
`ci-preview-test-report.ts --stage-notes`, which keeps every planned
line and the source footer and trims only the report to the remaining
budget. It fails clearly if the planned notes alone leave no room.
t-ci-preview-test-report covers the exact, trimmed and near-limit
budgets and the command; t332 still runs the workflow step.

F2: a Full Suite that fails in setup records no failing tests, yet the
warning promised a list of them. The warning, the notes wording in the
docs, and the report now call it a Full Suite failure report, and the
report says when the failed jobs may have stopped before their tests.

F3: docs/reference/11-contributing.md still said Full Suite gates the
preview. It now names the contract and release-asset gates and states
that a failing suite is reported instead of blocking publication; t332
pins that wording.
@github-actions github-actions Bot removed aida:reviewed AIDA successfully reviewed the latest PR state next:author AIDA indicates the PR author needs to act next action:change AIDA indicates changes are required before the PR proceeds labels Sep 25, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 458a2488fd805c9cc82b5f92ec40adb57da3d4d2 against 5975205f6aff222b9ab85d515d138f53abb2fcb4 and current repository behavior.

Inspection: 11 changed files. Scope: incremental — 9 files with lines changed since the review at 4df8f704; the security lenses reviewed the full head. Findings on lines unchanged since that review are deferred, not decisive.

Final Assessment

Human decision aid only: Readiness 5/5 is best; Risk 1/5 is best. These scores inform the maintainer; the next action below follows finding severity (any open P0/P1 → author/change) and does not approve or merge the PR.

Readiness: 3/5 — The main workflow and prior documentation defects are substantially corrected, but a near-limit release body still breaks the feature's required failed-suite publication path.

Risk: 3/5 — The blocking condition requires unusually large but valid notes, limiting frequency, but it prevents nightly preview publication. The remaining diagnostic issue is low impact.

Decision required: Author — make changes before this PR proceeds. The remaining P1 release-body defect can still prevent the failed-suite preview publication this change requires.

Validation performed:

  • Inspected all 11 changed-file snapshots, the full diff, review scope, ledger, discussion, specialist candidates, AGENTS.md, and CONTRIBUTING.md.
  • Confirmed every snapshot matches the immutable head and the diff has no whitespace errors.
  • Traced the preview workflow DAG, report artifact, note staging, uncapped release-note planner, publisher, documentation, and focused tests.
  • Established statically that failed-suite publication still aborts when planned notes leave insufficient warning-marker capacity.

Findings: 1 blocking, 1 advisory.

Ledger: 2 open, 0 retained blocking, 0 accepted, 0 suppressed as rejected by a maintainer, resolved F2, F3 (F2, F3 declared corrected by the judge). Maintainers act on findings with /aida commands in the ledger comment.

Contracts & Compatibility

P1 [F1]: Near-limit planned notes still abort failed-suite publication

Evidence: scripts/ci-preview-test-report.ts:216, docs/reference/09-testing.md:1252.

Problem: When the uncapped planner produces notes for which the warning, body, newline, and minimum truncation marker exceed 125,000 characters, stagePreviewNotes throws even after dropping the entire report. Thus a body that could otherwise publish still aborts specifically when Full Suite fails, contradicting the documented publication contract.

Impact: A large but valid changelog or commit-subject body can still suppress the nightly preview after failed tests, breaking the feature's primary required behavior.

Required correction: Bound planner-generated notes while preserving the source footer and reserving failed-suite warning/report overhead; validate the final body and add a test where the original body fits but the minimum staged failure body does not. Reconcile the claim that planned notes always remain whole.

User Experience

User experience change: Preview maintainers and users can receive a preview after Full Suite fails, with a warning and failure report while the workflow remains red.

Before: A Full Suite failure stopped the build and publication chain.

After: A completed Full Suite failure normally permits publication and appends diagnostic evidence to the release notes.

Example: A nightly test failure publishes a warned prerelease, then Release result fails the workflow.

Assessment: This improves preview availability and diagnostics, but oversized planned notes can still prevent publication, and one final error can reference a report that was never generated.

P3 [F4]: Final error can point to a nonexistent failure report

Evidence: .github/workflows/preview-release.yml:753.

Problem: If Full Suite fails and Release tests then fails during checkout, Bun setup, or report generation, its summary has no failure report. Release result nevertheless emits an error claiming that report exists.

Impact: Maintainers investigating a failed nightly are directed to absent recovery evidence.

Required correction: Condition the message on needs.test.result: reference the summary only after successful report generation; otherwise state that Release tests failed and direct maintainers to that job's logs. Cover both paths in the workflow test.

Residual risk: Repository code and tests were not executed under the read-only review contract; GitHub Actions behavior was assessed statically.

Reviewed by AIDA (AI-DLC Developer Agent).

[AI-PR-REVIEWED] 458a248

@github-actions
github-actions Bot dismissed their stale review September 25, 2026 09:46

Superseded by AI review of 458a248

@github-actions github-actions Bot added action:change AIDA indicates changes are required before the PR proceeds aida:reviewed AIDA successfully reviewed the latest PR state next:author AIDA indicates the PR author needs to act next labels Sep 25, 2026
Addresses AIDA on #1404 (F1 round 2, F4).

F1: staging threw when the planned notes sat within the warning and
truncation marker of GitHub's 125,000-character limit, so a failing
suite could still stop a preview whose notes would publish. Staging now
never throws. The report is trimmed first, then the warning shrinks to
a one-line pointer to the run, and a planned body that fits only on its
own publishes unchanged. A failing suite therefore never causes the
overflow; notes the planner makes too long fail exactly as before.
t-ci-preview-test-report covers the pointer, the body-only fallback and
an already-oversized plan.

F4: Release result pointed at the failure report even when Release
tests failed before writing it. The message now depends on the Release
tests result, and t332 covers both paths.
@apackeer
apackeer deployed to ai-pr-review September 25, 2026 09:51 — with GitHub Actions Active
@apackeer
apackeer deployed to ai-pr-review September 25, 2026 09:51 — with GitHub Actions Active
@github-actions github-actions Bot removed aida:reviewed AIDA successfully reviewed the latest PR state next:author AIDA indicates the PR author needs to act next action:change AIDA indicates changes are required before the PR proceeds labels Sep 25, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2dbcf241bfc6c5e0adaad124b0512d80c0316d2b against 5975205f6aff222b9ab85d515d138f53abb2fcb4 and current repository behavior.

Inspection: 11 changed files. Scope: incremental — 5 files with lines changed since the review at 458a2488; the security lenses reviewed the full head. Findings on lines unchanged since that review are deferred, not decisive.

Final Assessment

Human decision aid only: Readiness 5/5 is best; Risk 1/5 is best. These scores inform the maintainer; the next action below follows finding severity (any open P0/P1 → author/change) and does not approve or merge the PR.

Readiness: 4/5 — The primary failed-suite publication path and prior blocking boundary failure are corrected and well covered. A near-limit disclosure gap and a smaller recovery-message defect remain.

Risk: 2/5 — The remaining defects affect rare near-limit notes and failure diagnostics for preview releases. Stable publication, asset verification, and workflow failure signaling remain intact.

Decision required: Maintainer — decide whether to merge this PR. No P0 or P1 finding survives; the remaining P2 and P3 findings are advisory under the repository decision contract.

Validation performed:

  • Inspected all 11 changed paths, the immutable diff and snapshots, incremental review scope, ledger, discussion, repository instructions, and specialist outputs.
  • Classified the change as a mixed workflow feature and recovery/reporting change and traced the preview DAG, Full Suite evidence, release planner, publisher, tests, and documentation contracts.
  • Statically established that completed Full Suite failures permit the build and publication chain while Release result keeps the workflow red.
  • Confirmed the previous near-limit publication failure is corrected when planned notes fit the release-body limit.
  • Confirmed the diff has no whitespace errors.

Findings: 0 blocking, 2 advisory.

Ledger: 2 open, 0 retained blocking, 0 accepted, 0 suppressed as rejected by a maintainer, resolved F1 (F1 declared corrected by the judge). Maintainers act on findings with /aida commands in the ledger comment.

Contracts & Compatibility

P2 [F5]: Near-limit failed previews can lose every failure marker

Evidence: scripts/ci-preview-test-report.ts:219, tests/unit/t-ci-preview-test-report.test.ts:252, docs/reference/09-testing.md:1252.

Problem: When planned notes fit the 125,000-character limit but leave insufficient room for FAILED_SUITE_POINTER, stagePreviewNotes returns the original body unchanged. The resulting preview was built after a failed Full Suite but contains neither a warning nor a report, contradicting the documented release-note contract.

Impact: Preview consumers viewing the release cannot distinguish this failed-validation artifact from a normally reported preview and lose the traceability intended to make publishing after failed tests safe.

Required correction: Always preserve a minimal failure marker linked to the workflow run, reserving its space during planning or truncating planned content while retaining the source footer. Update the boundary test and reconcile all release-note documentation.

User Experience

User experience change: Preview users can receive a prerelease after Full Suite fails, while maintainers receive a red workflow plus failure evidence in the Release tests summary, artifact, and normally the release notes.

Before: A Full Suite failure stopped the preview build and publication chain.

After: A completed Full Suite failure normally permits publication with a warning and report, then Release result fails the workflow.

Example: A nightly test failure can publish a warned preview while the Actions run remains red.

Assessment: This improves preview availability and diagnostics, but near-limit notes can omit every failure marker and one report-job failure path gives inaccurate recovery guidance.

P3 [F4]: Job-level result still misstates report availability

Evidence: .github/workflows/preview-release.yml:757.

Problem: The report step can append the report to the job summary before the subsequent artifact upload fails. That makes needs.test.result failure, so Release result claims no report was produced even though the summary contains it.

Impact: Maintainers investigating that failure are directed away from available diagnostic evidence.

Required correction: Track successful summary generation independently from the overall job result, or use conservative wording that directs maintainers to both the summary and logs. Cover a failure after report generation.

Residual risk: Repository code and tests were not executed under the read-only review contract; GitHub Actions expression and hosted-workflow behavior were assessed statically.

Reviewed by AIDA (AI-DLC Developer Agent).

[AI-PR-REVIEWED] 2dbcf24

@github-actions
github-actions Bot dismissed their stale review September 25, 2026 10:02

Superseded by AI review of 2dbcf24

@github-actions github-actions Bot added action:merge AIDA considers the PR ready for a maintainer merge decision aida:reviewed AIDA successfully reviewed the latest PR state next:maintainer AIDA indicates a maintainer needs to act next labels Sep 25, 2026
@apackeer
apackeer added this pull request to the merge queue Sep 25, 2026
Merged via the queue into main with commit 60214a0 Sep 25, 2026
33 of 38 checks passed
@apackeer
apackeer deleted the chore/nightly-build-despite-test-failures branch September 25, 2026 10:14
apackeer added a commit that referenced this pull request Sep 25, 2026
…longpaths

* origin/main:
  chore(ci): build the nightly preview even when Full Suite fails (#1404)
apackeer added a commit that referenced this pull request Sep 27, 2026
* origin/main: (93 commits)
  fix: tell reviewers the exact findings table so fewer reviews are refused (#1415)
  fix: improve Windows installation and preserve unowned files during uninstall (#1182)
  docs(install): say what a copy install asks you to trust (#1419)
  refactor(kiro-ide): target Kiro's unified agent harness (Kiro IDE 1.x and Kiro CLI v3) (#1416)
  feat(scopes): default express to Guard Policy off (#1397)
  fix(sensors): let reverse-engineering's declared sensors reach the CodeKB (#1403)
  fix(doctor): close the Kiro IDE ignore-sources follow-ups from #1391 (#1413)
  fix: name doctor through the harness skill prefix in state and registry refusals (#1410)
  fix(worktree): let Bolt worktrees pass MAX_PATH on Windows (#1399)
  chore(ci): build the nightly preview even when Full Suite fails (#1404)
  test: parse custom-harness SDK directive output on Windows (#1016)
  fix: normalize blank subagent types in completion audits (#1018)
  fix(ci): preserve full-suite evidence and verify completion (#1369)
  fix(doctor): probe every framework read and stop the repository search at filesystem boundaries (#1391)
  fix: keep open gates through scope-change and close the #1272 follow-ups (#1398)
  fix(windows): repair the cross-OS test failures on Windows (#1393)
  fix(doctor): compare the audit against the per-stage checkboxes (#1272)
  fix(sensor): drop a superseded detail file when the sensor passes (#1266)
  chore(ci): supersede Full Suite verification across branch heads (#1390)
  fix: never record an unreadable review findings table as no findings (#1163)
  ...

This branch was successfully deployed

1 active deployment
ai-pr-review — 2dbcf241 Deployed Sep 25, 2026 by apackeer via Review pull request #901
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

action:merge AIDA considers the PR ready for a maintainer merge decision aida:reviewed AIDA successfully reviewed the latest PR state documentation Improvements or additions to documentation github next:maintainer AIDA indicates a maintainer needs to act next

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant