Skip to content

fix(sandbox): retry a transient 502/503/504 from the daemon, not just network failures - #6936

Closed
pedrofrxncx wants to merge 1 commit into
mainfrom
fix/daemon-client-retry-transient-status-w1
Closed

fix(sandbox): retry a transient 502/503/504 from the daemon, not just network failures#6936
pedrofrxncx wants to merge 1 commit into
mainfrom
fix/daemon-client-retry-transient-status-w1

Conversation

@pedrofrxncx

@pedrofrxncx pedrofrxncx commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Bug found while auditing packages/sandbox/server/daemon-client.ts (brittle-retry-logic focus area) — not tied to a specific issue.

The gap: daemonRequest's retry wrapper (retry() from @decocms/shared/std) only retries when fetch itself throws (network error, timeout). A non-2xx response — including 502/503/504, which a daemon pod restarting behind its NetworkPolicy/proxy can legitimately return — was returned immediately with zero retries. Every other transient-HTTP-failure client already hardened in this repo (jira/github clients, per the repo's own convention) retries transient 5xx before giving up; this daemon client didn't, so a single restart blip during postConfig/postSetupStep/postOrgFsConfig surfaced as a hard failure instead of self-healing on the next attempt like a network blip already does.

The fix: classify 502/503/504 as retriable by routing them through the same retry path as a thrown error (TransientStatusError), while excluding 500 (an application bug, not an infra hiccup — still fails fast). On exhaustion, unwraps RetryError.cause so callers still get their normal typed error (ConfigRequestError, etc.) instead of a misleading [SANDBOX_UNREACHABLE].

Regression test: packages/sandbox/server/daemon-client.test.ts — added 3 cases: retries a 503 then succeeds, surfaces the last 503 as ConfigRequestError once retries are exhausted, and confirms a plain 500 is NOT retried (single attempt).

Reviewer command: bun test packages/sandbox/server/daemon-client.test.ts

Locally verified: bun run fmt, bunx tsc --noEmit (packages/sandbox), the targeted test file (29/29 pass), bunx oxlint on both changed files (0 warnings/errors). Full CI validates the rest.


Summary by cubic

Fixes daemonRequest so transient 502/503/504 responses from the sandbox daemon are retried, matching how thrown network errors are already retried. Previously these statuses were returned immediately, so a daemon restarting behind its proxy surfaced as a hard failure instead of self-healing.

Bug Fixes

  • 500 is deliberately not retried because it indicates an application bug, not an infrastructure hiccup.
  • After retries are exhausted, callers still get the normal typed error (ConfigRequestError) instead of [SANDBOX_UNREACHABLE].
  • Adds regression tests covering retry success, exhausted retries, and no retry on 500.

Written for commit cc2a381. Summary will update on new commits.

Review in cubic

… network failures

daemonRequest retries thrown network/timeout errors, but a non-2xx response (even 502/503/504 while the daemon pod is restarting behind its proxy) was returned as-is with zero retries — every other transient-failure client in this repo (jira/github/sandbox) already retries transient 5xx per the established pattern; this one didn't.

Classifies 502/503/504 as retriable by routing them through the same retry path as a thrown error, and unwraps RetryError's cause so an exhausted transient status still surfaces as the normal ConfigRequestError/etc rather than a misleading [SANDBOX_UNREACHABLE]. 500 is deliberately excluded (an application bug, not an infra hiccup) and still fails on the first attempt.
@pedrofrxncx

Copy link
Copy Markdown
Collaborator Author

Closing as stale: this PR sat past the bot's 48h merge window, main has moved on, and its CI results no longer reflect the current base. This is a housekeeping close, not a rejection of the change — if the underlying problem still exists, the bot will find it again and open a fresh, rebased PR.

[studio-bot:stale-close]

@pedrofrxncx pedrofrxncx closed this Sep 8, 2026
@pedrofrxncx
pedrofrxncx deleted the fix/daemon-client-retry-transient-status-w1 branch September 8, 2026 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant