Skip to content

Rollback drill image drifts from the release it is built on, with no signal #691

Closed

Description

@cameronsjo

Problem

The rollback drill image is built FROM one bosun release, pinned by digest, and does not track releases. DRILL_REF in scripts/rollback-drill.sh:85 is a hardcoded digest whose base is 0.43.1.

Nothing notices when it drifts, and the symptom is easy to misread. A drifted base makes the shadow render report RENDER-DIFFERS and list files, because the base and the running incumbent are different releases. That is expected noise, not a failure — answering y still cuts over and still exercises rollback. But an operator who did not build the image has no way to tell that from a real render difference, and the safe-looking answer, n, ends the run at exit 0 DECLINED at RENDER-DIFFERS with the rollback still unverified.

(Corrected: an earlier version of this issue said the symptom was exit 5 CANDIDATE-FAILED. It is not. That verdict means the drill image failed to render at all — a broken shim, not drift.)

Why this is low severity

A stale drill image is not dangerous. It is pinned by digest, lives in a private package, is never pulled by anything expecting a release, and only runs when an operator starts a drill. The cost is a judgement call the operator has no information to make.

Suggested Fix

Not a scheduled rebuild. A job that publishes a deliberately broken bosun image on a timer, unwatched, is a worse failure mode than a stale one, and the drill needs an operator at a prompt regardless.

Make the drill say what it knows. rollback-drill.sh can read the drill image's bosun --version and the running incumbent's before it touches anything, and print the comparison up front:

drill image is built on 0.43.1, incumbent is 0.45.0 — RENDER-DIFFERS below is expected. Answer y to drill anyway, or rebuild first: docs/rollback-drill-image.md

That converts the ambiguous prompt into an informed one without refusing anything. It costs one docker run and one docker exec, both of which the script already does elsewhere.

A CI freshness check comparing the base digest against the current release is possible but more machinery; only worth it if the above proves insufficient. Neither option should rebuild or publish automatically.

Context

Rebuild recipe: docs/rollback-drill-image.md. First and only drill to date: 2026-09-20, exit 1 ROLLED-BACK, against a base matching the then-current release.

Activity

  1. cameronsjo commented on Oct 14, 2026

    @cameronsjo
    OwnerAuthor

    Claimed by a local chief-of-staff sweep; a draft PR will reference this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions