Skip to content

[DO NOT MERGE] Validate disk_size_gb fix for 2025.12.0 -> 2026.0.0 migration disk-space failures - #740

Closed
LukasAud wants to merge 6 commits into
PE-45867/main/release-2023.8.12-2025.12.0-2026.0.0from
cat-2538-disk-size-gb-validation
Closed

LukasAud wants to merge 6 commits into
PE-45867/main/release-2023.8.12-2025.12.0-2026.0.0from
cat-2538-disk-size-gb-validation

Conversation

@LukasAud

Copy link
Copy Markdown

Do not merge — scratch validation PR

Validates the fix for the disk-space failures PEADM reported on #718 (e.g. this job):

  • PE 2025.12.0 standard → 2026.0.0
  • PE 2025.12.0 large → 2026.0.0
  • PE 2025.12.0 extra-large → 2026.0.0

Confirmed root cause for standard/extra-large: tar: ... Cannot open/mkdir: No space left on device while extracting the PE 2026.0.0 tarball during peadm::pe_install, on the default-sized boot disk provision_service creates (the source image's own default disk size, never explicitly set). standard fails at the same point with the same short duration but its log didn't capture the detailed tar output in this attempt; worth double-checking this run's result for that one specifically.

Changes (temporary only)

  • .github/workflows/test-migration.yaml: matrix trimmed to just the 3 combos above (was ~16 jobs), to validate without burning CI time/cost on the unrelated rest of the matrix. Adds PROVISION_DISK_SIZE_GB (set to 100), scoped the same way as the existing PROVISION_MACHINE_TYPE override.
  • .fixtures.yml: provision pinned to the unreleased per-job-disk-size branch (puppetlabs/provision#303, not yet tagged) instead of v3.1.0, since that's where disk_size_gb support lives client-side.
  • No provision_service changes needed here — puppetlabs/provision_service#448 (the service-side disk_size_gb support) is already deployed to production.

Next steps

Once CI confirms the 3 jobs pass, close this PR without merging, then proceed with the real rollout: merge provision#303, cut a new provision tag, and bump PEADM's real .fixtures.yml pin in #718 (not this branch).

🤖 Generated with Claude Code

LukasAud and others added 2 commits October 28, 2026 16:19
…ations

Scratch validation only, not for merge. Trims test-migration.yaml's
matrix to the 3 combos PEADM reported as disk-space failures
(standard/large/extra-large -> 2026.0.0, which fail with
"tar: ... No space left on device" extracting the PE 2026.0.0
tarball), and pins provision to the unreleased per-job-disk-size
branch (puppetlabs/provision#303) with PROVISION_DISK_SIZE_GB set,
to confirm the fix before #303 is tagged. provision_service's
disk_size_gb support is already live in production.

Revert the matrix trim, the .fixtures.yml ref, and the
PROVISION_DISK_SIZE_GB env var once validated -- none of this should
reach main.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
puppetlabs_spec_helper's ref option is for tags/commit SHAs only;
branch names need the separate branch option, or
FixtureHelpers#revision's git reset --hard <ref> fails since a bare
branch name doesn't resolve against a fresh clone's
refs/remotes/origin/* (unlike a tag, which is resolvable directly).
All 3 jobs failed identically with
"Invalid ref per-job-disk-size for spec/fixtures/modules/provision"
before fetching the real provision_service disk-space fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-code-quality

github-code-quality Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Code coverage overview

Languages: Ruby, Puppet

Ruby / code-coverage/rspec

PE-45867/main/releas... 96ca4be cat-2538-disk-size-g... 0aa621c +/-
84% 84% 0%

Puppet / code-coverage/rspec-puppet

PE-45867/main/releas... 96ca4be cat-2538-disk-size-g... 0aa621c +/-
100% 100% 0%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.


Updated October 29, 2026 10:09 UTC

LukasAud and others added 4 commits October 29, 2026 09:42
Scratch validation only, not for merge. The 2nd migration CI run
failed at the identical tar file/byte-offset as before requesting
disk_size_gb, suggesting GCP sized the disk but the guest OS never
grew its root filesystem into the extra space. This workflow
provisions one VM (no PE install) and inspects lsblk/df/growpart
presence/guest-agent logs to confirm, without burning an hour-long
full migration run per check.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
….yaml

Scratch validation only, not for merge. A brand-new workflow file
isn't dispatchable until it exists on the default branch, so folds the
single-VM disk-resize diagnostic into the already-registered
test-migration.yaml as a second job, gated by a diagnose_only
workflow_dispatch input so it can run without also re-triggering the
3-job migration matrix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…gnostic

Scratch validation only. First diagnostic run failed with an SSH auth
error ~20s after provisioning returned -- the provisioner's
startup script that sets up the litmus user likely hadn't finished
yet, same class of race as other freshly-booted-VM SSH issues seen
before. Adds a short wait/retry loop before the real inspection
command instead of connecting immediately.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…cade

Scratch validation only, not for merge. provision_service#448 (the
disk_size_gb backend support) is only merged into its own PR branch
so far, not main/release, so the task's default facade-release
silently ignores disk_size_gb -- confirmed by a single-VM check
showing the boot disk stayed at its unchanged 10G default. Points
SERVICE_URL at the PR's own test-deploy facade
(facade-per-job-disk-size), created automatically per
docs/testing.md, instead of production.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@LukasAud

Copy link
Copy Markdown
Author

Closing — validation complete and confirmed effective.

Against this branch's test-deploy facade (facade-per-job-disk-size), a single-VM check showed the requested boot disk coming up at the correct size with the filesystem grown to match, and the full migration matrix trimmed to the 3 originally-failing jobs (standard/large/extra-large, 2025.12.0 → 2026.0.0) all passed with no No space left on device failures.

Tracked in CAT-2951. Real rollout depends on puppetlabs/provision_service#448 and puppetlabs/provision#303 getting reviewed, merged, and released — see that ticket for the rollout order. None of this branch's changes (trimmed matrix, .fixtures.yml branch pin, SERVICE_URL override) should be merged.

🤖 Generated with Claude Code

@LukasAud LukasAud closed this Oct 29, 2026
CharithaDunuwille added a commit that referenced this pull request Oct 29, 2026
provision_service now supports a per-job disk_size_gb override
(provision#303, released as provision v3.2.0). Bump the .fixtures.yml
pin and request 100G for standard/large/extra-large -> 2026.0.0
combos, which were confirmed failing with "No space left on device"
while extracting the PE 2026.0.0 tarball on the default 10G boot disk.
Scoped the same way as the existing PROVISION_MACHINE_TYPE override.

Validated in #740.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant