Skip to content

[Fix] Upgrade retries reject backups after a failed image pull - #1009

Merged
mrubens merged 1 commit into
developfrom
fix/upgrade-failed-pull-0w5e56tvly9zc
Aug 3, 2026
Merged

[Fix] Upgrade retries reject backups after a failed image pull#1009
mrubens merged 1 commit into
developfrom
fix/upgrade-failed-pull-0w5e56tvly9zc

Conversation

@roomote-community

Copy link
Copy Markdown
Contributor

Opened on behalf of tebieshi. View the task or mention @roomote for follow-up asks.

What changed

Failed roomote upgrade attempts now restore the deployment .env, Compose file, and Caddyfile from the pre-upgrade staging copy before exiting. If the rollout had already stopped the controller, the failure cleanup starts it again.

A focused host CLI regression test simulates a missing worker image and verifies that the prior release metadata is preserved, the controller is restarted, and temporary staging files are removed. CI now runs this test and includes it in deployment script syntax validation.

Why this change was made

Upgrade metadata was written before image pulls completed, but only migration failures restored it. A missing image could therefore leave an undeployed tag recorded as current, causing the next pre-upgrade backup to fail while inspecting an image that had never been deployed.

Impact

Operators can retry roomote upgrade normally after an image pull failure without using --skip-backup; backups continue to describe the release that is actually deployed. The focused regression test and pre-push lint, type, and dependency checks passed. Full deployment artifact validation could not complete locally because the sandbox has no Docker socket for Caddy adaptation.

@roomote-community

roomote-community Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

No code issues found. See task

Reviewed 99a15e2

@roomote-community
roomote-community Bot marked this pull request as ready for review August 3, 2026 16:47
@tebieshi

tebieshi commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Pull request overview

This PR improves the host upgrade path (deploy/host/roomote) so that if an upgrade fails during image pulls, the CLI restores the previously deployed release metadata/config files (notably .env, docker-compose.prod.yml, and caddy/Caddyfile) and restarts the controller when it had been stopped for the rollout.

Changes:

  • Added an EXIT trap-based rollback cleanup (cleanup_upgrade) to restore deployment configuration when roomote upgrade exits non-zero after staging the prior config.
  • Added a host-level regression test that simulates a failed docker pull and asserts the prior deployment metadata is preserved and staging artifacts are cleaned up.
  • Wired the new test into deployment artifact validation (bash syntax check) and CI execution.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
deploy/host/tests/upgrade-failed-pull.sh New regression test that forces a pull failure and verifies rollback restores deployed metadata and cleans staging dirs.
deploy/host/roomote Adds cleanup_upgrade EXIT trap to restore .env/compose/Caddyfile and restart controller on upgrade failure (including failed pulls).
deploy/ci/validate-deployment-artifacts.mjs Includes the new test script in the list of deployment scripts checked via bash -n.
.github/workflows/CI.yml Runs the new failed-pull regression test in CI.

@mrubens

mrubens commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

@tebieshi good to go?

@mrubens
mrubens merged commit 2187995 into develop Aug 3, 2026
19 checks passed
@mrubens
mrubens deleted the fix/upgrade-failed-pull-0w5e56tvly9zc branch August 3, 2026 17:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants