Skip to content

fix(updater): prune build workspaces no longer referenced by state - #1619

Merged
ilysenko merged 2 commits into
ilysenko:mainfrom
AbdelQuddous:fix/updater-prune-workspaces
Oct 24, 2026
Merged

ilysenko merged 2 commits into
ilysenko:mainfrom
AbdelQuddous:fix/updater-prune-workspaces

Conversation

@AbdelQuddous

@AbdelQuddous AbdelQuddous commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The updater leaks one full build workspace per upstream release. cache_cleanup::prune() only removes superseded upstream .deb files from packages/; it never touches workspaces/<version>/, and builder::build_update() only wipes the workspace of the version it is about to rebuild. Each workspace holds the extracted app (codex-app/, ~1.7 GB) plus the rebuilt package in dist/ (~0.5 GB), so the cache grows by roughly 2 GB for every release and never shrinks.

On one Fedora 44 install that had been auto-updating since August, ~/.cache/codex-update-manager reached 34 workspaces / 59 GB, while only 2–3 of them were still referenced by state.json.

This also contradicts docs/updater.md, which already states that cleanup "retains the installed source, current candidate, and immediately previous rollback artifact" — that was true for packages/, not for workspaces/.

Change

prune() now also calls a new prune_workspaces() that deletes a workspaces/<entry> directory only when no persisted path points into it. Retained references:

  • artifact_paths.workspace_dir (the active / most recent build)
  • artifact_paths.package_path (rebuilt candidate, or the installed package)
  • artifact_paths.rollback_package_path (immediately previous artifact)
  • install_transaction.package_path (an in-flight install)

Each reference is compared both as stored and canonicalized, so a missing file never causes its workspace to be treated as unreferenced by accident through path normalization. A symlinked workspaces/ root is never followed, and only real directories (not symlinks) inside it are considered.

No new call sites: prune() is already invoked after a successful upstream check and after a successful install, and both call sites run under the check.lock mutation lock introduced in #1141, so cleanup cannot race a build, install-ready, check-now, or rollback in another process.

Resulting behavior

Moment Workspaces on disk
Idle with N installed N (installed), N-1 (rollback)
Check finds N+1 and builds it N-1, N, N+1 (cleanup ran before the build started)
After installing N+1 N+1 (installed), N (rollback) — N-1 is removed

Steady state is two workspaces (~4 GB), three only while a new build is in progress, instead of unbounded growth. Rollback keeps working exactly as before, since the rollback package's workspace is always retained (and rollback::preserve_before_workspace_cleanup still copies it out before a same-version rebuild).

User-visible effect

Existing installs reclaim the leaked space automatically on the first check after upgrading. No config, state schema, or CLI changes; no new dependencies.

Scope

  • Formats: format-agnostic (deb, RPM, pacman share this code path). Not applicable to AppImage/Nix, which have no updater.
  • Architectures: amd64 and arm64 alike (pure filesystem logic).
  • Files: updater/src/cache_cleanup.rs, docs/updater.md, CHANGELOG.md.

Validation

  • cargo fmt --all -- --check
  • cargo clippy -p codex-update-manager --all-targets -- -D warnings
  • env -u XDG_CONFIG_HOME cargo test -p codex-update-manager -- --test-threads=1 — 125 passed, including 4 new tests:
    • keeps_current_and_rollback_workspaces_only — of 4 workspaces, only the active and rollback ones survive
    • keeps_active_build_and_install_transaction_workspaces — a workspace still being built (no dist/ yet) and one referenced only by an in-flight install transaction are kept
    • removes_workspaces_nothing_references — a dangling rollback path does not protect unrelated workspaces
    • ignores_symlinked_workspace_root — a symlinked workspaces/ root is left alone and its target untouched
  • git diff --check
  • Manually verified on Fedora 44 (RPM, amd64): reproduced the 59 GB leak, confirmed which workspaces state.json referenced, and confirmed the same retention set the new code computes; the subsequent 26.1021.22027 update built and installed successfully.
  • The diff was reviewed by a second coding agent; it flagged the symlinked-root case, which is fixed and covered by a test.

Not tested: a full end-to-end run of the patched daemon across two consecutive upstream releases (that requires waiting for the next signed release).

Checklist

  • This pull request is ready for review and is no longer a draft.
  • I followed CONTRIBUTING.md, kept the change focused, edited source files rather than generated output, and removed unrelated changes.
  • If this fixes upstream drift, it targets only the latest signed stable OpenAI Linux package and removes obsolete fallback code and tests. (Not drift-related.)
  • I added or updated relevant tests, ran the validation listed above, and confirmed that required CI checks pass. (Tests added and run locally; CI pending.)
  • I reviewed the final diff with my coding agent using maximum reasoning effort, addressed all findings, and reran the relevant tests.

AbdelQuddous and others added 2 commits October 24, 2026 12:06
Each update left a ~2 GB workspaces/<version> directory behind, so the
cache grew without bound (59 GB observed). prune() now also removes
workspaces that neither the active build, the rebuilt package, the
rollback package, nor an in-flight install transaction points into,
leaving the installed build, its rollback, and any build in progress.
A symlinked workspace root is never followed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@ilysenko ilysenko left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracking this down — the retention set matches how the builder and rollback paths populate state, both call sites already run under check.lock, and the symlinked-root guard plus tests look good. Approving.

The earlier red checks came from the upstream 26.1021.22027 drift, now repaired on main (#1621); after updating the branch, CI is fully green.

Non-blocking follow-ups, fine for a later PR:

  • prune_workspaces scans cache_dir/workspaces, while builder::build_update writes under config.workspace_root/workspaces. They are equal by default, but passing the configured workspace root would keep cleanup correct if it is overridden.
  • prune_workspaces(...)? runs before the .deb pruning, so one undeletable workspace now aborts the package cleanup that used to succeed. Consider best-effort per-entry removal or pruning packages first.
  • The new tests use std::env::temp_dir() with manual cleanup; the crate already depends on tempfile, which would avoid leftovers when an assertion fails.

@ilysenko
ilysenko merged commit 0aa82d3 into ilysenko:main Oct 24, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants