[imaqsood]

Summary

provision_service never retried an HTTP 5xx from the provision service. invoke_cloud_request handled any non-200 response with exit 1, and SystemExit isn't a StandardError, so the RETRY_COUNT rescue loop in #provision never fired. The task died on the first 500.

This PR:

• raises ProvisionService::ServiceError on non-200 responses, and its ServerError subclass on 5xx, instead of calling exit 1;
• retries ServerError, network errors and empty 200 responses, up to 4 attempts. The waits double each time (30 s, 60 s, 120 s) plus up to 50% random jitter. A 4xx still fails fast;
• logs each retry to stderr. Bolt parses stdout as the task result, and litmus reads value['target_names'] from it, so any extra text there breaks that JSON (the old retry message went to stdout);
• keeps the provision_service/service_error error JSON in .run, now with an attempts field so CI logs show whether retries happened;
• drops the currnet_retry_count typo and gives the empty-response raise a message.

Additional Context

• Root cause and the steps to reproduce.
  On the 2026-10-01 supported-modules nightly, provision 500s caused 389 of 451 failed jobs across 34 modules. They're intermittent: about 32% of provisions failed at 00:00 UTC, rising to about 61% by 01:00 UTC, and they hit live image families too (rhel-8/9, windows-2022, debian-12). 21 of 22 sampled failing logs show no retry at all. Example runs: stdlib RedHat-9, acl Windows-2022.
• Thought process behind the implementation.
  Every module's .fixtures.yml clones this repo from main, so this one change takes effect fleet-wide on the next nightly with no module PRs. The jitter matters because all nightly crons fire at 0 0 * * *, and provisioning requests hit the service in a burst. Worst case, the backoff adds about 5 minutes to a job that would have failed anyway.

Reviewer question: can the provision service partly create the VMs and still return a 500? If it can, retrying the POST could leave orphaned VMs behind. Our data doesn't answer this either way.

Update 2026-10-08 (from MODULES-11966 log analysis): The nightly 500s mostly come from failed tofu apply runs in backend-release. The causes are the GCE FIREWALLS quota (500) being exhausted, and earlier in the window a us-west1-c stockout. A failed apply doesn't clean up its firewall rule, and the reaper doesn't remove those rules for 3h. So a 500 can leave orphaned resources behind (firewall rules, not VMs), and each client retry creates another rule. The retries here recover the start of the burst, but they may make the quota plateau worse. The real fix is on the service side (MODULES-11966: raise the quota, or use one shared rule instead of one per request). Reviewers: should this land as-is, or with fewer attempts until 11966 is fixed?

Note on CI: the red "Spec tests (Puppet ~> 8.0)" check is the Codecov upload step added in cat-github-actions 70e9e0be. No coverage file is produced because SIMPLECOV=yes isn't set. All 85 specs pass, and it is not a required check.

Tear-down is unchanged: it still makes a single attempt, and a 5xx now surfaces as the same service_error JSON as before.

Related Issues (if any)

• MODULES-11965 (epic MODULES-11964)
• Retry if provisioning target returns a 500 #193 asked for a 5xx retry in 2022. The retry loop added at the time never covered the 500 path.

Checklist

• 🟢 Spec tests. spec/tasks/provision_service_spec.rb passes 22/22, with new cases for a 500 followed by a 200, retries running out (4 attempts, backoff order), a 4xx not being retried, the .run error JSON and retry_delay bounds. Rubocop is clean.
• 🟢 Acceptance tests.
• Manually verified.

🤖 Generated with Claude Code

[Copilot]

🔵 Needs a closer look

Human confirmation is needed that retrying a partially completed provisioning POST cannot create duplicate or orphaned VMs.

0 open findings

What changed in this PR

Makes Provision Service retries handle HTTP 5xx failures, addressing #193 while preserving Bolt’s JSON error format.

Changes:

• Adds typed service errors and four attempts with exponential backoff and jitter.
• Sends retry logs to stderr and includes attempt counts in service errors.
• Adds specs for HTTP retry behavior, error output, and delay bounds.

File	Description
tasks/​provision_service.rb	Implements retry handling, backoff, and structured errors.
spec/​tasks/​provision_service_spec.rb	Covers retries, fail-fast responses, and error reporting.

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[gavindidrichsen]

Looks good to me! Very nice improvement to the retry mechanism.

[LukasAud]

LGTM