[Codex App][Remote SSH] Thread hydration blocks the app-server queue and causes an endless reconnect loop

Open 💬 4 comments Opened Jul 30, 2026 by yufeinever

💡 Likely answer: A maintainer (github-actions[bot], contributor) responded on this thread — see the highlighted reply below.

What version of the Codex App are you using (From "About Codex" dialog)?

26.721.11231.0; remote Codex CLI/app-server 0.146.0.

What subscription do you have?

ChatGPT subscription (exact tier not confirmed).

What platform is your computer?

Microsoft Windows NT 10.0.26200.0 x64; remote host is Linux 6.8.0-136-generic x86_64.

What issue are you seeing?

Codex Desktop Remote SSH can enter an endless reconnect loop when it automatically hydrates several existing threads. One slow thread/resume request occupies an app-server request slot for 50-65 seconds. Additional hydration and background requests fill all six in-flight slots, queued requests expire after 30 seconds, and the whole WebSocket eventually closes with code 1006.

This affects the entire remote host connection, not only the problematic thread. New threads cannot be created or sent while the queue is blocked. Normal SSH sessions remain stable, the managed remote app-server PID remains alive, and server CPU/memory are not saturated.

The affected session files were not exceptionally large by storage standards (approximately 2-20 MB), but some JSONL records were monolithic: individual records were approximately 1.2-1.64 MB. A 2.28 MB session with only 111 records contained one 1.26 MB record. This suggests that record shape and concurrent hydration, rather than only total session size, trigger the failure.

Representative redacted Desktop log sequence:

app_server_client_request_queue_rejected inFlightRequestCount=6
  method=plugin/list queueWaitMs=30003 reason=queue-expired

Request failed method=thread/resume durationMs=65397
  error="Codex app-server is not available"

app_server_connection.closed code=1006 transport=websocket
proxy_command_failed signal=SIGTERM operation=app_server_proxy

SIGTERM is logged after the WebSocket closes and appears to be cleanup, not the initial cause.

What steps can reproduce the bug?

Configure Codex Desktop on Windows with a Remote SSH Linux project.
Create several threads in that project, including threads containing large tool outputs or other large individual JSONL records.
Leave Remote SSH auto-connect enabled and keep those threads active/unarchived.
Restart Codex Desktop or reconnect the Remote SSH host so Desktop hydrates multiple threads.
Observe several concurrent thread/resume calls. One or more resume calls take tens of seconds.
Observe inFlightRequestCount=6, request queue expiration after 30 seconds, WebSocket close code 1006, and automatic reconnect.
Try to create or send a new thread while this is happening. The new request is blocked behind hydration requests or fails when the app-server connection closes.

The same SSH route becomes responsive immediately after the recovery steps below, without changing the network.

What is the expected behavior?

A slow or malformed thread should only fail or delay that thread's hydration.
Interactive operations such as thread/start and turn/start should not be blocked behind bulk hydration.
Thread history should be streamed, paginated, or processed with bounded record sizes.
Remote reconnect should be bounded and should not repeatedly replay the same failing hydration workload.
The UI should identify the offending thread and offer a reversible recovery action.

Additional information

The effective recovery was:

Identify the thread whose thread/resume is blocking the queue.
Archive the offending large/abnormal thread or threads (reversible; do not delete them).
Cancel their pending resume operations and clear queued hydration requests.
Stop automatic reconnect instead of immediately replaying the same workload.
Restart the managed remote daemon once to clear already in-flight server requests, then reconnect only after the offending threads are archived.

Recommended product behavior: after a hydration timeout or repeated queue saturation, show an Archive task and stop reconnecting recovery action. It should archive the identified thread, cancel pending hydration, clear the affected queue, and pause automatic reconnect until the user explicitly retries. Interactive requests should also receive priority over background hydration.

After applying the recovery above, request latency on the unchanged connection returned to approximately 1-350 ms, turn/start completed in approximately 122 ms, and the connection remained stable. Before recovery, even smaller threads took 25-31 seconds because they were queued behind the blocking resume.

Related but not duplicate:

#20128 tracks Remote SSH fragility behind a network gateway/NLB.
#35217 tracks leaked app-server processes during reconnects with a shared NFS home.

This report is specifically about thread hydration/backpressure causing host-wide request starvation and a reconnect loop.

View original on GitHub ↗

4 Comments

github-actions[bot] contributor · 28 days ago

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

#35096

Powered by Codex Action

yufeinever · 28 days ago

I reviewed #35096. There is clear overlap: both are triggered by Windows Codex Desktop automatically restoring Remote SSH threads.

However, #36189 captures a narrower backend failure mode that #35096 does not document: thread/resume requests saturate all six app-server request slots, queued requests expire after 30 seconds, the WebSocket closes with code 1006, and automatic reconnect repeatedly replays the same workload. The UI does not necessarily become fully unresponsive, but new threads cannot be sent.

#36189 also includes request timings, evidence of monolithic 1.2-1.64 MB JSONL records, and a reproducible recovery: archive the offending threads, cancel hydration, stop reconnecting, clear the queue, and restart the managed daemon once.

These issues may share the same underlying thread-restoration regression, but I am leaving #36189 open as the app-server/backpressure and reconnect-loop case unless maintainers confirm they should be consolidated.

tpn · 25 days ago

Confirming this on a Linux server & Windows ChatGPT App setup, Windows
connecting to the Linux box via "Connections" & ssh, with server-side evidence
from three related incidents. Remote host: Ubuntu (kernel 6.17), managed
standalone app-server 0.145.0 and later 0.146.0; client: Codex/ChatGPT Desktop
on a separate Windows workstation over SSH; current version 26.727.51351 but
this occurred recently on earlier versions over the past week.

Symptom match. Desktop entered the endless reconnect loop described here:
SSH itself fine, but Codex never finished connecting. At the time of the most
recent incident the remote session store was 16 GB across 6,621 rollout files —
largest 492 MB, with 61 files ≥ 10 MB. These are far past the 2–20 MB files
reported above, which may explain why our failure mode progressed further
(below).

Server-side observations across the three incidents (Jul 25, Jul 28, Aug 2):

Each Desktop reconnect attempt spawns codex app-server proxy on the remote

host via SSH. When the app-server stops responding, the proxies never time out
and never exit. We accumulated 4,076 orphaned proxy processes in one
incident and 3,506 in another (spawn cadence ~every 10–50 s), with the
app-server's unix-socket accept queue saturated at 2,731 and 2,906 pending
connections respectively. Same leak shape as #35217, no NFS involved.

The app-server itself failed two different ways: once spinning on futexes at

~7 cores (~46 k futex syscalls/s, SIGTERM ignored, required SIGKILL — matches
#34142), once fully deadlocked at 0% CPU while holding the listener. RSS
reached 7.6 GB with another 3.8 GB in swap before the hangs.

Between incidents the managed app-server also died silently and nothing

restarted it (matches #35295); the next incoming proxy then auto-spawned an
unmanaged replacement (ppid 1) that the daemon refuses to manage, so recovery
required manually killing it.

On the six-slot limit. I searched codex-rs (0.146.0) for the limiter: there
is no 6-permit semaphore anywhere in the app-server — request pools are 256/1024
and the per-connection ConnectionRpcGate is unbounded — and the string
inFlightRequestCount doesn't appear in this repo. So the six in-flight slots
and the 30 s queue expiry appear to live in the closed-source Desktop client,
meaning affected users have no configuration or self-build escape hatch on
either end.

Mitigation data point. Archiving old/large rollouts (active set reduced from
16 GB / 6,621 files to ~4.7 GB, largest active file now ~10 MB) stopped the
reconnect loop immediately. That's consistent with the hydration-cost theory:
six slots are fine when thread/resume is fast; they're fatal when a handful of
resumes take 50+ s.

Requests:

Make the Desktop client's in-flight request limit and queue expiry

configurable — or better, deprioritize bulk thread hydration behind
interactive requests so a slow resume can't starve new-thread creation.

Paginate or lazily hydrate thread/resume (0.146.0 already added paginated

history for forks; resume seems like the natural next candidate).

Have codex app-server proxy fail fast (bounded connect/handshake timeout,

then exit non-zero) when the app-server is unresponsive, so a wedged server
degrades into fast failures instead of thousands of orphaned processes and a
saturated accept queue.

Happy to provide the futex traces, accept-queue captures, or process trees from
any of the three incidents if useful.

raka08 · 4 days ago

Confirming the same user-visible failure on a macOS controller → Windows Remote SSH host setup.

On the same host and SSH configuration:

Short existing chats open successfully.
A long existing chat remains on Loading task....
The app then shows Failed to resume chat — Codex app-server is not available.
The host banner changes to Connection to [host] was lost. Reconnecting..., and retrying does not open that long chat.

This long-vs-short control makes a general SSH authentication/connectivity problem unlikely and points to the thread history hydration/resume path described in this issue. The expected behavior is for a large thread to load incrementally, or fail only that thread with a specific recoverable error, without dropping and replaying the entire remote connection.

I have screenshots of both the loading state and the exact errors. I am intentionally not posting thread IDs, host names, local paths, raw logs, or chat data publicly; sanitized diagnostics can be supplied privately to OpenAI maintainers if requested.