›_ Codex Issues
Codex Issues / #22390

Retry transient capacity errors with backoff and retained task state

Open 💬 5 comments Opened May 12, 2026 by kylechinnock

What variant of Codex are you using?

Codex Desktop App, Codex VS Code extension. V 26.506.31421

What feature would you like to see?

Transient capacity errors are retryable.

When Codex receives a backend/model-capacity condition such as:

Selected model is at capacity. Please try a different model.

that should be treated as a temporary admission/capacity failure, not as a terminal task failure.

Codex should keep the task alive, retry with bounded exponential backoff and jitter, and resume from the last safe model/tool boundary.

This is normal distributed-systems behavior for transient overload. Treating a retryable backend condition as a terminal task failure turns automation into babysitting.

Core reliability expectation

Retryable tasks should not fail terminally.

If a task hits a retryable condition, such as transient model capacity, network interruption, server overload, or temporary backend unavailability, Codex should preserve the task state and keep the task resumable.

The state machine should be:

running -> retrying with backoff -> paused / waiting for manual retry

not:

running -> failed

Automatic retry can have limits. That is fine. But exhausting the automatic retry budget should move the task into a retained-state paused state, not a failed state.

At that point the UI should offer:

  • Retry now
  • Continue retrying automatically
  • Switch model and resume
  • Cancel task

The important contract is that retryable errors should not destroy task continuity or force the user to reconstruct context manually.

Current behavior

Codex performs substantial work and then stops on a transient backend capacity condition.

Examples from screenshots:

  • Ran 4 commands, changed 3 files, +175 -53, then stopped with:

> Selected model is at capacity. Please try a different model.

  • Ran 8 commands, changed 9 files, +901 -106, then stopped with:

> Selected model is at capacity. Please try a different model.

This leaves the user with partial work and requires manual intervention.

The user has to notice the failure, reopen or review the task, determine whether the work is safe to continue, and manually act as the retry loop.

Expected behavior

I should be able to start a long-running Codex task, leave it running overnight, and trust that transient backend capacity issues will be handled automatically.

Expected flow:

  1. Codex receives a transient capacity / server-overloaded / model-at-capacity error.
  2. The task remains active.
  3. The UI shows something like:

> Selected model is temporarily at capacity. Retrying in 30s...

  1. Codex retries with exponential backoff and jitter.
  2. Codex resumes from the last safe model/tool boundary.
  3. If automatic retries are exhausted, the task moves to paused / waiting for manual retry, not failed.
  4. The user can optionally:
  • retry now
  • continue retrying automatically
  • cancel
  • manually switch models and resume
  1. Manual intervention should happen only after retry exhaustion or a true non-retryable failure.

Failure modes observed

Failure mode 1: hard stop during long-running work

Codex makes progress, runs commands, modifies files, then stops because the selected model is temporarily at capacity.

The work is not treated as paused/retrying. It is treated as failed.

Failure mode 2: UI pushes manual model switching too early

The error tells the user to try a different model before Codex has exhausted automatic retry.

For automation, the first recovery path should be automatic retry. Manual model switching should be an override or fallback, not the default recovery path.

Failure mode 3: retry exhaustion conflates pause with failure

Even if automatic retry has a bounded budget, exhausting that budget should not mean the task failed.

It should mean Codex has paused automatic retry and retained the task state for manual retry, continued retry, model switch, or cancellation.

Why this matters

The expectation of an automation agent is:

  • start the task
  • leave it alone
  • come back later
  • review completed work or a clear non-retryable failure

The current experience is:

  • start the task
  • worry it may silently stop
  • hover over it
  • come back to partial work blocked by a transient backend condition
  • manually act as the retry loop

That turns automation into babysitting.

This creates reliability, trust, and anxiety problems. The product expectation is that automation means automation.

Expectation:

  • leave long-running tasks overnight
  • build trust
  • relax
  • review results later

Reality:

  • unreliability
  • trust issues
  • hovering
  • anxiety
  • manual recovery from transient backend state

Suggested implementation

This should likely be handled in shared core/protocol code, not separately in each UI.

The overload/capacity condition appears to be represented as a shared error condition and exposed through shared protocol error info. That looks like the right joint point for cross-platform behavior.

Suggested behavior:

  • classify server/model capacity overload as retryable
  • reuse existing retry/backoff machinery
  • emit a retrying/reconnecting event rather than a terminal error
  • only surface the current “try a different model” wording after retry exhaustion
  • avoid rerunning completed commands blindly
  • resume from persisted conversation/tool state
  • if automatic retries are exhausted, emit a paused/retryable state instead of a failed state

Conceptually:

retryable error + retry budget available -> retry with backoff
retryable error + retry budget exhausted -> paused / waiting for manual retry
non-retryable error -> terminal error

Definition of done / acceptance criteria

This issue should not be considered resolved by temporary backend mitigation alone.

A resolution should include durable product behavior that can be observed by users and, where applicable, verified by tests.

Required behavior

  • Transient model-capacity / server-overloaded errors are classified as retryable.
  • A task receiving a retryable capacity error does not immediately enter a terminal failed state.
  • While automatic retries remain, the task enters a visible retrying state.
  • The retrying state shows at least:
  • that Codex is retrying
  • the retry attempt count or retry phase
  • the next retry delay or approximate retry time
  • Retry uses bounded exponential backoff with jitter, or an equivalent documented backoff policy.
  • If the automatic retry budget is exhausted, the task enters a retained-state paused / waiting for manual retry state, not failed.
  • In the paused state, the user can resume without recreating the task, re-prompting from scratch, or manually reconstructing context.
  • The paused state offers clear actions:
  • Retry now
  • Continue automatic retry
  • Cancel
  • Switch model and resume, if model switching is supported for that task type
  • Completed file changes remain visible and intact.
  • Completed command/tool results remain visible and intact.
  • The task retains enough execution state to resume safely from the last safe model/tool boundary.
  • Codex does not blindly rerun shell commands or tool calls that already completed successfully.
  • Non-retryable failures still fail clearly and do not loop forever.
  • Retryable failure handling is observable in logs/debug output.

Required test or verification coverage

A complete fix should include test coverage or an equivalent maintainer-verifiable reproduction showing:

  • A capacity/server-overloaded response during a task causes retry/backoff, not terminal failure.
  • A capacity/server-overloaded response after partial work preserves changed files and command/tool history.
  • Exhausting the automatic retry budget produces a paused/resumable state, not a failed state.
  • Manual retry from the paused state resumes the task without losing state.
  • Non-retryable errors still produce a terminal failure.

Not sufficient to close this issue

The following should not be considered sufficient resolution by themselves:

  • Closing because a specific outage or capacity incident was mitigated.
  • Changing only the error message text.
  • Advising the user to switch models manually.
  • Adding only a banner or warning.
  • Retrying only initial request creation while still allowing mid-task capacity failures to terminate the task.
  • Marking the issue resolved without a linked PR, commit, release note, or explicit explanation of where the retry/pause behavior is implemented.
  • Treating retry budget exhaustion as task failure instead of retained-state pause.

Additional information

Related issues

Primary related issue:

  • Related to openai/codex#22277

openai/codex#22277 was closed as mitigated, but that does not appear to establish durable product behavior. Backend capacity recovery and client/task resilience are separate issues. This request is specifically about the latter: retryable capacity failures must not terminate or discard long-running task state.

Strong supporting issues:

  • Related to openai/codex#19583
  • Related to openai/codex#19579
  • Related to openai/codex#17014
  • Related to openai/codex#19446

Adjacent UI/state issues:

  • Related to [#11635](<https://github.com/openai/codex/issues/11635>)
  • Related to [#11904](<https://github.com/openai/codex/issues/11904>)

Related PRs / implementation precedent

These do not appear to fully resolve this request, but they are relevant implementation context:

  • Related to [#10118](<https://github.com/openai/codex/issues/10118>)
  • model-capacity guidance / client-side capacity messaging
  • Related to [#1947](<https://github.com/openai/codex/issues/1947>)
  • capacity / high-demand error path
  • Related to [#1956](<https://github.com/openai/codex/issues/1956>)
  • retry/backoff tuning
  • Related to [#506](<https://github.com/openai/codex/issues/506>)
  • precedent for retrying mid-stream failures with existing exponential backoff
  • Related to [#12001](<https://github.com/openai/codex/issues/12001>)
  • model reroute notification, adjacent to capacity/routing behavior

Relevant files

Possible shared implementation points:

  • codex-rs/codex-api/src/api_bridge.rs
  • maps backend overload/capacity responses into shared error state
  • codex-rs/protocol/src/error.rs
  • defines shared error classification and retryability
  • codex-rs/core/src/session/turn.rs
  • main turn loop where retryable errors should become retrying/paused, not terminal failure
  • codex-rs/core/src/compact.rs
  • existing retry/backoff precedent using reconnect-style behavior
  • codex-rs/protocol/src/error_tests.rs
  • likely place for retryability regression tests
  • codex-rs/codex-api/src/api_bridge_tests.rs
  • existing mapping tests for server-overloaded responses
  • codex-rs/app-server-protocol/src/protocol/v2/shared.rs
  • shared protocol surface for app/extension clients

Triage metadata

Suggested labels:

  • enhancement
  • bug
  • connectivity
  • app
  • extension
  • CLI, if maintainers want to track the same cross-client behavior

I would avoid treating this primarily as rate-limits. This is about retryable model/server capacity and retained task state, not user quota exhaustion.

Additional information

This is not a request for unlimited capacity, priority access, or automatic quality downgrade.

It is a request for Codex to handle retryable infrastructure conditions like an automation system:

  • retry automatically when safe
  • preserve state
  • pause instead of fail when automatic retry is exhausted
  • let the user resume without reconstructing context manually

A retryable transient backend condition should not become a terminal user-facing task failure.

View original on GitHub ↗

5 Comments

kylechinnock · 3 months ago

Correlation ID: 019e062d-4a0f-7303-b884-caeec48cf88e

Keesan12 · 2 months ago

Retaining task state matters more than the backoff itself.

Without state retention, a retry after capacity failure often just replays the most expensive part of the run and makes the outage costlier than it needed to be.

The two things I'd want here are:

  • retry with preserved execution context/checkpoint, not a soft restart
  • one clear ceiling where the product stops being hopeful and says "this run is blocked by capacity, here's the last good state"

Otherwise users get the worst combo: delay plus duplicate spend.

We've seen the same pattern while building MartinLoop. The lesson keeps being the same: retries are only healthy when they carry forward evidence, not just optimism.

AdrianIp0204 · 1 month ago

I’d like to propose an opt-in follow-up to #31058 rather than duplicate its capacity-retry implementation.

The additional behavior would be a configurable fallback model used only for the remainder of the current agent turn when the selected model returns a structured capacity error.

Proposed recovery modes:

  • preserve the existing/default terminal behavior where required;
  • retry the selected model using the capacity policy established by #31058;
  • after a configured number of primary retries—or immediately when configured—switch once to an explicitly selected fallback model for the rest of that turn.

Suggested fallback semantics:

  • react only to structured server_is_overloaded / slow_down capacity errors;
  • initially support only a fallback under the same provider/account;
  • keep the user’s selected primary model unchanged;
  • use the fallback for all remaining sampling steps in the current turn, including steps after tool calls;
  • return to the selected primary on the next user turn;
  • emit an explicit model-rerouted notice;
  • allow only one fallback transition and no fallback chains;
  • if the fallback is also overloaded, terminate normally;
  • switch only at a safe sampling boundary, without replaying completed output, commands, tool calls, or other side effects;
  • validate context-window, reasoning, modality, and tool compatibility before switching.

I would intentionally exclude switching back to the primary during the same turn. Codex has no passive model-availability signal, so this would require speculative inference probes and model-switch hysteresis.

Would maintainers prefer this follow-up to remain under #22390 or be tracked as a separate enhancement? If the approach aligns with the roadmap, I would be happy to implement a focused PR after #31058 lands, but I will wait for an explicit invitation before writing code.

tylorsaling-source · 1 month ago

Measured confirmation from Windows Desktop that the current capacity path violates the retained-state contract described here.

Codex Desktop 26.721.4979.0, model gpt-5.6-sol (high), ran a turn for 1,501,434 ms (~25 minutes) and then recorded:

type: task_complete
turn_id: 019fa62c-0b6f-7bf0-ba34-e09aad4b8277
last_agent_message: null
error.message: Selected model is at capacity. Please try a different model.
error.codex_error_info: server_overloaded

There was no retry, no retained-state pause, and no user-facing explanation. The Desktop UI displayed only a red failure icon. The user had to inspect the internal rollout to identify the cause.

This adds two concrete acceptance requirements:

  1. server_overloaded must not terminalize a turn as task_complete with a null agent message.
  2. After retry exhaustion, the UI must visibly pause and offer Retry now, Continue retrying, Switch model and resume, or Cancel.

A configuration setting should control whether Codex asks before switching, automatically selects the closest available model, keeps retrying the selected model, or stops with a visible notification. Completed tool calls must not be replayed when the turn resumes.

nyl199310 · 20 days ago

I investigated this against current main and built a local, integration-tested prototype that refreshes and extends the approach in #31058. Thanks @steipete-oai for establishing the capacity-specific retry direction there.

The smallest complete ownership boundary appears to be:

  • classify only structured server_is_overloaded / slow_down responses at the provider boundary (no matching of the user-visible message);
  • defer those failures out of the ordinary HTTP retry loop, then use a separate cancellation-aware patient budget in the turn loop (3 retries at 30s, 120s, and 300s, with positive-only jitter);
  • reuse the same turn and client session so accepted input is recorded once and completed tool outputs are carried forward rather than replayed;
  • intercept capacity before WebSocket-to-HTTP fallback, and never change the selected model implicitly;
  • surface retry and terminal events as structured ServerOverloaded;
  • after exhaustion, keep dependent Tab-queued input paused instead of dispatching it as though the failed turn completed (#33853).

I also traced the same error through local, remote-v1, and remote-v2 compaction (#35911). Pre-turn compaction is the important edge case: it runs before normal input recording, so terminal capacity exhaustion must preserve the already accepted input for the next user-initiated resume, exactly once. Previous-model compaction must also retain its model choice on capacity rather than silently falling back to the current model.

Because the complete sampling + compaction coverage is larger than the repository's 800-line review guideline, I propose two owner-aligned changes:

  1. sampling/provider-boundary patient retry plus terminal dependent-queue safety;
  2. compaction retry/cancellation plus pre-turn input resumption, based on the merged first change.

The prototype has focused integration coverage for independent HTTP/stream/capacity budgets, same-turn input and completed-tool preservation, cancellation during backoff, no capacity-triggered transport fallback, all three compaction paths, previous-model behavior, terminal observability, resumability, and queue safety.

The contribution policy requires an explicit maintainer invitation before opening an external PR. Would a maintainer please confirm whether I may submit stage 1 with this scope, and whether the two-stage split is the preferred review shape? I will not open a duplicate issue or unsolicited PR.