Repository navigation
fix: stop old pods seeding skills under a newer release's version - #49
Merged
Merged
Conversation
Seeding published whatever the running binary embedded, under whatever version APP_VERSION happened to hold. Those are uncoupled, so any pod from an older image that read a configmap a newer release had already updated would publish stale content under the new version number. Skill versions are immutable, so the wrong content sticks permanently. That is not hypothetical. During the 0.2.1 rollout the RWO volume deadlocked the default RollingUpdate: the new pod could not attach while the old pod held the claim, and the old pod could not be released until the new one was Ready. Deleting the old pod made its ReplicaSet recreate a 0.2.0-image pod, which read APP_VERSION=0.2.1, won the volume race, and seeded v0.2.0's skill text as 0.2.1. The correct pod then started, found the version present, and skipped in silence. Three changes: - Seed under the binary's own build version, stamped via -ldflags -X main.buildVersion. A stamped binary refuses to seed when APP_VERSION disagrees, since it cannot honestly publish content it did not build. Unstamped local builds keep the APP_VERSION fallback. - Use strategy: Recreate whenever the volume is not ReadWriteMany. This fixes the deadlock and closes the two-generation overlap window. The condition is inverted so an unset or unexpected accessMode still gets the safe strategy; upgrades now have a brief gap. - Log the already-present skip. Its silence is what made the corrupted version hard to trace. The stamped version and APP_VERSION must agree for seeding to run. Both derive from the release tag: build-args VERSION uses the metadata-action version output (semver priority 900 beats raw 200, so v0.2.2 yields 0.2.2), and release-chart rewrites appVersion to the same value. registry 0.2.1 stays wrong; versions cannot be rewritten. A 0.2.2 release supersedes it as latest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the root cause of a corrupted registry version, plus the deployment defect that triggered it.
What went wrong
seed.Runpublished whatever the running binary embedded, under whatever versionAPP_VERSIONheld. Nothing coupled those two.APP_VERSIONcomes from the chart's configmap, so during any window where two image generations coexist, a pod from the older image can read the newer release's configmap and publish stale content under the new version number. Skill versions are immutable (skill_versionsPRIMARY KEY), so the wrong content is permanent.This happened during the v0.2.1 rollout:
replicaCount: 1,maxUnavailablecomputes to 0, so the controller held the old pod until the new one was Ready, while the new pod could not attach the claim until the old one released it.Multi-Attach error for volume ... Volume is already used by pod(s) skillsctl-5c854dddfd-w9x95.APP_VERSION=0.2.1.ErrAlreadyExistsbranch, which logged nothing.Verified by content: registry
0.2.1is byte-identical to registry0.2.0, and registry0.2.0is byte-identical togit show v0.2.0:skills/skillsctl-usage.md. Meanwhilev0.2.1's embedded file does contain the intended fixes, so the image was fine and only the seeding was wrong.Changes
Seed under the binary's own version.
main.buildVersionis stamped at build time. A stamped binary refuses to seed whenAPP_VERSIONdisagrees, because it cannot honestly publish content it did not build. Refusing logs a warning and the server keeps serving; it is not fatal. Unstamped local builds (dev) keep the existingAPP_VERSIONfallback so nothing changes for development.strategy: Recreateunless the volume is ReadWriteMany. Fixes the deadlock and eliminates the two-generation overlap entirely. The condition is deliberately inverted (ne "ReadWriteMany") so an unset, empty, orReadWriteOncePodaccess mode still gets the safe strategy rather than silently regressing. Trade-off: single-replica upgrades now have a brief gap instead of hanging forever.Log the already-present skip. That silence is what made this hard to trace: the pod's entire startup log was two lines with no indication seeding had been skipped.
The invariant this introduces
Seeding now requires the stamped version and
APP_VERSIONto match exactly. Both derive from the release tag:build-args: VERSION=${{ steps.meta.outputs.version }}. Per the metadata-action docs,type=semverhas priority 900 againsttype=raw's 200 and "the first tag in the list (higher priority) will be used as the image version for generated OCI label andversionoutput", sov0.2.2yields0.2.2, notlatest.release-chartrewritesappVersionto${GITHUB_REF_NAME#v}, also0.2.2.If those ever diverge, seeding stops with a logged warning rather than corrupting a version. That is the intended failure direction, but it is worth knowing.
Verification
go test ./... -racepasses; 8 new table-driven cases coverresolveSeedVersionincluding mismatches in both directionsgolangci-lint run ./...: 0 issueshelm lint: passes. All five render paths checked: RWO, empty,ReadWriteOncePod, RWX, and persistence-disabledAPP_VERSION=0.2.1→seed: refusing to seed (APP_VERSION "0.2.1" does not match build version "0.2.0"...), reproducing the incident and blocking itAPP_VERSION=0.2.2→seed: seeded skillsctl-usage@0.2.2, row is 8038 bytes and contains the corrected textseed: skillsctl-usage@0.2.2 already present, skippingFollow-up not in this PR
registry 0.2.1stays wrong and is currently whatlatestresolves to. Cutting v0.2.2 supersedes it. Also still open:proto/buf.gen.yamluses unpinned remote plugins, soproto-checkreddens on every upstream plugin release, anddocs/site/content/cli/install.mdclaimsGetSkillContentis unauthenticated whenauth/interceptor.gorequires a Bearer token on every RPC.