Skip to content

ci: test that an invalid release BOM blocks basectl release check and publish #2169

Description

@codeforester

Goal

Add automated coverage proving that an invalid, malformed, or mismatched
release BOM blocks basectl release check and basectl release publish, and
that bom_finding maps every ReleaseBomError class to a blocking finding.

Background

From the 2026-09-09 deep review of the release-BOM feature (Base 1.9.0):

  • cli/python/base_release/tests/test_engine.py::test_publish_with_bom_uploads_and_verifies_stable_release_assets
    mocks base_release.engine.release_findings (returns READY_FINDINGS) and
    writes a non-BOM stub as the --bom input
    (bom_bytes = b'{"release":"base-1.9.0"}\n'). It therefore never exercises
    bom_finding / validate_bom_file, and it would pass even if BOM validation
    were removed entirely.
  • cli/python/base_release/release_readiness.py::bom_finding has no direct
    tests. There is no test_release_readiness.py at all, so
    inspect_release_provenance and the expected_commit=provenance.commit_sha
    wiring are only covered indirectly through heavily-mocked engine tests.
  • release_bom.validate_bom itself is well covered by
    cli/python/base_release/tests/test_release_bom.py. The gap is the wiring
    that turns a bad BOM into a failed release.

Net: the property the feature exists for -- "a bad BOM fails the release" -- is
unverified end to end.

Scope

  • cli/python/base_release/tests/test_release_readiness.py (new) or additions
    to test_engine.py: unit-test bom_finding for each branch --
    ctx.bom_path is None (ok), missing file (error), malformed JSON (error),
    schema-invalid document (error), repository / version / commit mismatch
    (error), happy path (ok).
  • Engine-level tests using a real BOM file on disk (no mock of
    release_findings):
    • release check --bom <invalid> --format json exits non-zero and emits a
      bom finding with status: error.
    • release publish --bom <invalid> is blocked and performs no git / gh
      mutations.
  • Replace the non-BOM stub in the existing publish-with-BOM test with a
    canonical, schema-conformant BOM fixture (or add a parallel test that uses
    one).
  • Add a small helper/fixture that builds a valid basefoundry/base-owned BOM;
    current fixtures (tests/fixtures/release-bom-valid.json) model a
    base-bash-libs release.

Acceptance Criteria

  • bom_finding has direct unit tests covering every error class and the ok
    path.
  • An engine test asserts release publish --bom <invalid> returns
    FAILURE and issues no git/gh commands.
  • An engine test asserts release check --bom <invalid> --format json
    reports status: error and a bom finding.
  • The existing publish-with-BOM engine test uses a valid, canonical,
    schema-conformant BOM.
  • A reusable valid-base-BOM fixture/builder exists for release tests.

Validation

python -m pytest cli/python/base_release/tests -q
bin/base-test
git diff --check

Non-Goals

  • No changes to validate_bom rules, the BOM schema, or the wire format
    (tracked separately in the schema-reconciliation and validator-tightening
    issues).
  • No new CLI flags or manifest fields (BOM enforcement is a separate issue).

Project Fields

  • Status: Backlog
  • Priority: P1
  • Size: S
  • Area: CI
  • Initiative: Contract Hardening
  • Milestone: v1.10.0

Agent Assignment

Human first. Straightforward test-authoring task once the fixture shape is
agreed; suitable for a Copilot candidate after triage.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

ciContinuous integration, tests, automation, or release workflows

Type

No type

Projects

  • Status
    Done

Relationships

None yet

Development

No branches or pull requests

Issue actions