Skip to content

fix: stop old pods seeding skills under a newer release's version - #49

Merged
dcmcand merged 1 commit into
mainfrom
fix/seed-version-guard-and-recreate
Aug 18, 2026
Merged

dcmcand merged 1 commit into
mainfrom
fix/seed-version-guard-and-recreate

Conversation

@dcmcand

@dcmcand dcmcand commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Fixes the root cause of a corrupted registry version, plus the deployment defect that triggered it.

What went wrong

seed.Run published whatever the running binary embedded, under whatever version APP_VERSION held. Nothing coupled those two. APP_VERSION comes from the chart's configmap, so during any window where two image generations coexist, a pod from the older image can read the newer release's configmap and publish stale content under the new version number. Skill versions are immutable (skill_versions PRIMARY KEY), so the wrong content is permanent.

This happened during the v0.2.1 rollout:

  1. The RWO volume deadlocked the default RollingUpdate. At replicaCount: 1, maxUnavailable computes to 0, so the controller held the old pod until the new one was Ready, while the new pod could not attach the claim until the old one released it. Multi-Attach error for volume ... Volume is already used by pod(s) skillsctl-5c854dddfd-w9x95.
  2. Deleting the old pod made its ReplicaSet immediately recreate one, still on image 0.2.0, but now reading the already-updated configmap with APP_VERSION=0.2.1.
  3. That pod won the volume race, seeded v0.2.0's skill text as version 0.2.1, then got scaled down.
  4. The correct 0.2.1 pod started, found 0.2.1 present, and returned via the ErrAlreadyExists branch, which logged nothing.

Verified by content: registry 0.2.1 is byte-identical to registry 0.2.0, and registry 0.2.0 is byte-identical to git show v0.2.0:skills/skillsctl-usage.md. Meanwhile v0.2.1's embedded file does contain the intended fixes, so the image was fine and only the seeding was wrong.

Changes

Seed under the binary's own version. main.buildVersion is stamped at build time. A stamped binary refuses to seed when APP_VERSION disagrees, because it cannot honestly publish content it did not build. Refusing logs a warning and the server keeps serving; it is not fatal. Unstamped local builds (dev) keep the existing APP_VERSION fallback so nothing changes for development.

strategy: Recreate unless the volume is ReadWriteMany. Fixes the deadlock and eliminates the two-generation overlap entirely. The condition is deliberately inverted (ne "ReadWriteMany") so an unset, empty, or ReadWriteOncePod access mode still gets the safe strategy rather than silently regressing. Trade-off: single-replica upgrades now have a brief gap instead of hanging forever.

Log the already-present skip. That silence is what made this hard to trace: the pod's entire startup log was two lines with no indication seeding had been skipped.

The invariant this introduces

Seeding now requires the stamped version and APP_VERSION to match exactly. Both derive from the release tag:

  • build-args: VERSION=${{ steps.meta.outputs.version }}. Per the metadata-action docs, type=semver has priority 900 against type=raw's 200 and "the first tag in the list (higher priority) will be used as the image version for generated OCI label and version output", so v0.2.2 yields 0.2.2, not latest.
  • release-chart rewrites appVersion to ${GITHUB_REF_NAME#v}, also 0.2.2.

If those ever diverge, seeding stops with a logged warning rather than corrupting a version. That is the intended failure direction, but it is worth knowing.

Verification

  • go test ./... -race passes; 8 new table-driven cases cover resolveSeedVersion including mismatches in both directions
  • golangci-lint run ./...: 0 issues
  • helm lint: passes. All five render paths checked: RWO, empty, ReadWriteOncePod, RWX, and persistence-disabled
  • End-to-end against scratch databases with real stamped binaries:
    • 0.2.0 binary with APP_VERSION=0.2.1 → seed: refusing to seed (APP_VERSION "0.2.1" does not match build version "0.2.0"...), reproducing the incident and blocking it
    • 0.2.2 binary with APP_VERSION=0.2.2 → seed: seeded skillsctl-usage@0.2.2, row is 8038 bytes and contains the corrected text
    • restart at the same version → seed: skillsctl-usage@0.2.2 already present, skipping

Follow-up not in this PR

registry 0.2.1 stays wrong and is currently what latest resolves to. Cutting v0.2.2 supersedes it. Also still open: proto/buf.gen.yaml uses unpinned remote plugins, so proto-check reddens on every upstream plugin release, and docs/site/content/cli/install.md claims GetSkillContent is unauthenticated when auth/interceptor.go requires a Bearer token on every RPC.

Seeding published whatever the running binary embedded, under whatever
version APP_VERSION happened to hold. Those are uncoupled, so any pod
from an older image that read a configmap a newer release had already
updated would publish stale content under the new version number. Skill
versions are immutable, so the wrong content sticks permanently.

That is not hypothetical. During the 0.2.1 rollout the RWO volume
deadlocked the default RollingUpdate: the new pod could not attach while
the old pod held the claim, and the old pod could not be released until
the new one was Ready. Deleting the old pod made its ReplicaSet recreate
a 0.2.0-image pod, which read APP_VERSION=0.2.1, won the volume race, and
seeded v0.2.0's skill text as 0.2.1. The correct pod then started, found
the version present, and skipped in silence.

Three changes:

- Seed under the binary's own build version, stamped via
  -ldflags -X main.buildVersion. A stamped binary refuses to seed when
  APP_VERSION disagrees, since it cannot honestly publish content it did
  not build. Unstamped local builds keep the APP_VERSION fallback.
- Use strategy: Recreate whenever the volume is not ReadWriteMany. This
  fixes the deadlock and closes the two-generation overlap window. The
  condition is inverted so an unset or unexpected accessMode still gets
  the safe strategy; upgrades now have a brief gap.
- Log the already-present skip. Its silence is what made the corrupted
  version hard to trace.

The stamped version and APP_VERSION must agree for seeding to run. Both
derive from the release tag: build-args VERSION uses the metadata-action
version output (semver priority 900 beats raw 200, so v0.2.2 yields
0.2.2), and release-chart rewrites appVersion to the same value.

registry 0.2.1 stays wrong; versions cannot be rewritten. A 0.2.2 release
supersedes it as latest.
@dcmcand
dcmcand merged commit bd09b3b into main Aug 18, 2026
10 checks passed
@dcmcand
dcmcand deleted the fix/seed-version-guard-and-recreate branch August 18, 2026 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant