Skip to content

fix(server): a hung Codex app-server no longer blocks every later Codex run - #26555

Open
MahirAbbas wants to merge 1 commit into
pingdotgg:mainfrom
MahirAbbas:fix/codex-initialize-timeout
Open

MahirAbbas wants to merge 1 commit into
pingdotgg:mainfrom
MahirAbbas:fix/codex-initialize-timeout

Conversation

@MahirAbbas

Copy link
Copy Markdown
Contributor

Problem

All Codex threads on one instance share one codex app-server session. If that app-server hangs before it answers initialize, the request waits forever while it holds the session's initialize permit (CodexAdapterV2.ts, ensureInitialized). Every later Codex run reuses the same live session (ProviderSessionManager.open returns the existing entry) and waits behind the permit. The runs stay in "starting" with startedAt: null until T3 restarts. A new Codex binary path does not help, because the live session is reused. Status probes use fresh processes, so Codex looks healthy the whole time.

This is a small, focused fix for an obvious hang: a request with no timeout that holds a shared lock.

How I established it

This happened on T3 Code Nightly 0.1.2-nightly.20261021 on macOS. A hung syspolicyd made every new exec of codex stop at _dyld_start. The server traces show:

  • The first Codex run opened the shared session at 21:48:51 (CodexAdapterV2.open succeeded). Its first CodexAppServerProtocol.request never got an answer.
  • No CodexAdapterV2.open ran again until the restart at 22:16. Three delegated Codex runs and two other runs each stopped at the first CodexAppServerProtocol.request in ProviderTurnStartService.loadFromProvider. They waited 64 s to 414 s, and each one ended only when it was cancelled.
  • The permit serialized the waiters. Run cbea0d64 sent its request 3 ms after the previous waiter was cancelled.
  • After the restart, the first Codex run completed in 1.5 s.

Claude runs kept working the whole time, so the effect worker was not blocked.

Fix

initialize now times out after 30 seconds. It normally answers in under a second. On timeout, the adapter fails the session's event stream, the same signal #26188 uses when the app-server exits. The session manager then retires the session and closes its scope, which stops the hung app-server. The waiting run fails instead of staying in "starting", and the next run spawns a fresh app-server with the current settings.

Open PR #26253 (one app-server per thread) would limit this hang to one thread. It does not add a timeout, so that thread would still hang.

Verification

  • New test retires the shared session when initialize never answers in CodexAdapterV2.test.ts. A replay app-server never answers initialize. The test checks that ensureThread fails after 30 s (TestClock) and that the session event stream fails.
  • Without the fix, the test hangs until the 10 s test timeout. With the fix, it passes.
  • vp test run src/orchestration-v2/Adapters/CodexAdapterV2.test.ts: 109 passed.
  • tsc --noEmit for apps/server, plus vp lint and vp fmt --check on the two changed files: clean.

Not checked: I did not reproduce the hang end to end in a running T3 instance.

Model: Claude Opus 5.5 in Claude Code, running in T3 Code.

🤖 Generated with Claude Code

…ex run

All Codex threads on one instance share one app-server session. If that
app-server hangs before it answers `initialize`, the request waits forever
while it holds the session's initialize permit. Every later Codex run reuses
the same live session and waits behind it, so runs stay in "starting" until
T3 restarts. Changing the Codex binary path does not help, because the live
session is reused.

`initialize` now times out after 30 seconds. On timeout the adapter fails the
session's event stream, which makes the session manager retire the session
and stop its app-server. The next run spawns a fresh app-server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the vouch:unvouched PR author is not yet trusted in the VOUCHED list. label Oct 28, 2026
@coderabbitai

coderabbitai Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 125b619 and 3139860.

Files selected for processing: 2
  • apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.test.ts
  • apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

Walkthrough

Codex app-server initialization now has a 30-second timeout. If initialization does not receive a response, the adapter fails the session event stream and initialization request. A regression test checks both failures.

Changes: Codex initialization timeout

Layer / File(s) Summary
Initialization timeout and failure handling
apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts, apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.test.ts
The adapter applies a 30-second timeout to initialization. If it expires, the adapter fails the event stream and initialization request with a session-scoped error. The regression test checks both failures.

Priority: Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Possibly related issues

  • pingdotgg/t3code#22422: Both changes fail the event stream when a Codex app-server does not respond, addressing session lifecycle handling.

Suggested labels: bugfix, size:S

Suggested reviewers: juliusmarminge, macroscopeapp

Merge Risk: Minimal · up to 31398

Hung sessions can now be retired and replaced. Runs already queued on the old session may still time out sequentially. No merge-blocking issue was established.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main fix: preventing a hung Codex app-server from blocking later Codex runs.
Description check Passed The description covers the problem, reproduction context, focused fix, scope rationale, verification steps, observed results, limitations, and agent disclosure. It satisfies the required template.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant