fix(sandbox): retry a transient 502/503/504 from the daemon, not just network failures - #6936
Closed
pedrofrxncx wants to merge 1 commit into
Closed
fix(sandbox): retry a transient 502/503/504 from the daemon, not just network failures#6936pedrofrxncx wants to merge 1 commit into
pedrofrxncx wants to merge 1 commit into
Conversation
… network failures daemonRequest retries thrown network/timeout errors, but a non-2xx response (even 502/503/504 while the daemon pod is restarting behind its proxy) was returned as-is with zero retries — every other transient-failure client in this repo (jira/github/sandbox) already retries transient 5xx per the established pattern; this one didn't. Classifies 502/503/504 as retriable by routing them through the same retry path as a thrown error, and unwraps RetryError's cause so an exhausted transient status still surfaces as the normal ConfigRequestError/etc rather than a misleading [SANDBOX_UNREACHABLE]. 500 is deliberately excluded (an application bug, not an infra hiccup) and still fails on the first attempt.
Collaborator
Author
|
Closing as stale: this PR sat past the bot's 48h merge window, main has moved on, and its CI results no longer reflect the current base. This is a housekeeping close, not a rejection of the change — if the underlying problem still exists, the bot will find it again and open a fresh, rebased PR. [studio-bot:stale-close] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug found while auditing
packages/sandbox/server/daemon-client.ts(brittle-retry-logic focus area) — not tied to a specific issue.The gap:
daemonRequest's retry wrapper (retry()from@decocms/shared/std) only retries whenfetchitself throws (network error, timeout). A non-2xx response — including 502/503/504, which a daemon pod restarting behind its NetworkPolicy/proxy can legitimately return — was returned immediately with zero retries. Every other transient-HTTP-failure client already hardened in this repo (jira/github clients, per the repo's own convention) retries transient 5xx before giving up; this daemon client didn't, so a single restart blip duringpostConfig/postSetupStep/postOrgFsConfigsurfaced as a hard failure instead of self-healing on the next attempt like a network blip already does.The fix: classify 502/503/504 as retriable by routing them through the same retry path as a thrown error (
TransientStatusError), while excluding 500 (an application bug, not an infra hiccup — still fails fast). On exhaustion, unwrapsRetryError.causeso callers still get their normal typed error (ConfigRequestError, etc.) instead of a misleading[SANDBOX_UNREACHABLE].Regression test:
packages/sandbox/server/daemon-client.test.ts— added 3 cases: retries a 503 then succeeds, surfaces the last 503 asConfigRequestErroronce retries are exhausted, and confirms a plain 500 is NOT retried (single attempt).Reviewer command:
bun test packages/sandbox/server/daemon-client.test.tsLocally verified:
bun run fmt,bunx tsc --noEmit(packages/sandbox), the targeted test file (29/29 pass),bunx oxlinton both changed files (0 warnings/errors). Full CI validates the rest.Summary by cubic
Fixes
daemonRequestso transient 502/503/504 responses from the sandbox daemon are retried, matching how thrown network errors are already retried. Previously these statuses were returned immediately, so a daemon restarting behind its proxy surfaced as a hard failure instead of self-healing.Bug Fixes
ConfigRequestError) instead of[SANDBOX_UNREACHABLE].Written for commit cc2a381. Summary will update on new commits.