`codex exec` can hang indefinitely before the SSE stream starts (response-header wait has no timeout/retry) — Windows/Linux

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #30526

Powered by Codex Action

Seeing the same "no timeout ever fires" behaviour on a very different setup, which may help
narrow the code path. Also traced where the gap appears to be in the source.

Another environment, same shape

• codex-cli 0.145.0, Linux (GitHub Actions ubuntu-latest), non-interactive codex exec
• Auth: Azure OpenAI via [model_providers.azure], wire_api = "responses"
• No timeout/retry overrides (defaults), so stream_idle_timeout_ms = 300000

Differences from the original report: Linux instead of Windows, Azure instead of ChatGPT auth, and
it stalls at the very start of the turn rather than 23 minutes in.

Log of one occurrence (the job was killed by a 15-minute CI cap, so the stall was still going):

22:52:09  run starts
22:52:27  [{"url":".../issues/comments/…"}]   ← last output: a `gh api` call the prompt asked for
          (fifteen minutes, not one line of output)
23:07:24  ##[error]The operation was canceled.
23:07:24  Terminate orphan process: pid (2418) (codex)

Four occurrences in two days. Every time, a plain rerun of the same commit finished in ~1 minute.

Where the gap appears to be

stream_idle_timeout is only applied after a response is obtained —
codex-rs/codex-api/src/endpoint/responses.rs:

let stream_response = self.session.stream_encoded_json_with(…).await?;  // not covered
Ok(spawn_response_stream(
    stream_response,
    self.session.provider().stream_idle_timeout,   // applied to the stream, once it exists
    …
))

And the HTTP client has no overall deadline to cover that .await —
codex-rs/http-client/src/client_builder.rs exposes connect_timeout only (default None);
there is no .timeout() and no read_timeout.

That matches both reports: the 300s idle default cannot fire because the SSE stream has not started
yet, and nothing else bounds the wait, so the process waits forever and has to be killed.

(For the original report's CLOSE_WAIT observation: a connection closed by the peer while awaiting
response headers would land in exactly this unguarded window.)

Workaround, for anyone hitting this in CI

Wrapping the call so a hang is cut short and retried, rather than raising the job's cap:

timeout --signal=TERM --kill-after=30s 300 codex exec … || status=$?

One note if you do this: with --kill-after, a process that ignores TERM exits 137, not 124 —
and a process wedged on a model call is exactly that kind, so treating only 124 as "timed out"
misses the common case.

Thanks @isamu — this is very useful corroboration. I have updated the title to reflect the cross-platform scope. The Linux/Azure reproduction rules out Windows and ChatGPT authentication as necessary conditions and points to an unbounded phase in the shared Responses HTTP path.

I checked rust-v0.145.0 as well. ResponsesClient::stream_encoded() awaits stream_encoded_json_with(...).await before spawn_response_stream(...) is created, so stream_idle_timeout cannot cover the wait for the HTTP response headers. The lower layer does have an optional per-request timeout, but Provider::build_request() initializes it to None, and this Responses request does not override it. ReqwestTransport::stream() can therefore remain blocked in builder.send().await; because no error is produced, the retry policy has nothing to retry. The same sequencing is still present on current main.

That is consistent with the original CLOSE_WAIT observation, although it does not by itself prove that every occurrence has the same socket-level trigger.

I would not close this as a duplicate of #30526 yet. That issue has a related symptom, but it concerns app-server, includes the additional thread-lifecycle failure (thread not found), and contains no transport or pre-header evidence. The two issues may share this lower-level gap, so cross-linking is useful, but equivalence has not been established.

The appropriate fix appears to be a retryable, per-attempt deadline for establishing the HTTP stream through receipt of the response headers, followed by the existing activity-based SSE idle timeout. A general reqwest request timeout would also cover the response-body lifetime and could incorrectly cap a healthy long-lived SSE stream.

Thanks also for documenting the 137 versus 124 behavior of the CI workaround.

I'm surprised this is not a bigger issue affecting many users

Adding an independent reproduction from the interactive Codex TUI. This looks like the same general unbounded pre-SSE/stream-wait failure, although I was not able to capture the affected socket state in time, so I cannot confirm CLOSE_WAIT for this specific turn.

Environment

• Codex CLI: 0.146.0
• Platform: Windows
• Surface: interactive TUI, not codex exec
• Authentication: ChatGPT
• Model: gpt-5.6-sol
• Reasoning effort: xhigh
• Session ID: 019fb23e-0d89-77a1-ba83-36f5f5fde3f6

The affected thread was relatively large:

• Last request input: approximately 170,064 tokens
• Model context window: 258,400
• Reported primary rate-limit usage: approximately 12%

There was no 429 or quota-exhaustion error.

Timeline

Times below are local time, UTC+8.

• 18:00:00 — User prompt was accepted and the turn started.
• 18:00:16 — First reasoning output item was recorded.
• 18:00:28 — Second reasoning output item was recorded.
• After that, no user-visible assistant response or terminal turn event appeared.
• 18:02:11 — The user interrupted the stalled turn. The interrupt was accepted and the previous task was marked aborted.
• 18:02:12 — A follow-up message was accepted and a new turn started.
• The new turn completed model-cache lookup, skill selection, world-state construction, and hooks.
• Logging then reached:

``text
run_turn
-> run_sampling_request
-> try_run_sampling_request
``

• No SSE/Responses stream event was recorded for the new turn after that point: no reasoning item, output item, structured error, retry, or completion event.

No tool call was active, and the hooks completed normally. This does not appear to have been caused by a shell command, approval request, or project hook.

A separate analytics request failed near the beginning of the first affected turn:

codex_analytics::client:
failed to send events request:
error sending request for url
(https://chatgpt.com/backend-api/codex/analytics-events/events)

That is not proof that the model request failed for the same reason, but it is a contemporaneous connectivity signal.

By the time the session was inspected a few minutes later, the process associated with this thread was no longer present. No corresponding Windows Application Error or Application Hang event was found, so I cannot determine whether it exited normally, was closed by the user, or terminated through an unlogged path.

Why this may be related

The second turn appears to have stalled after request preparation but before the first SSE event, which is consistent with the timeout gap described in this issue.

This may also overlap with #32366, which reports a long-running gpt-5.6-sol thread where the Responses request produced no stream events. The difference here is that this reproduction occurred in Codex CLI 0.146.0 interactive TUI.

The session remains recoverable with:

codex resume 019fb23e-0d89-77a1-ba83-36f5f5fde3f6

I can submit in-product feedback from the affected session if an uploaded trace would help.

Thanks @rullerzhou-afk — this is useful, especially because it occurred in 0.146.0 through the interactive TUI. Yes, please submit in-product feedback from the affected session and reference #31376 plus session ID 019fb23e-0d89-77a1-ba83-36f5f5fde3f6. The uploaded trace is likely the best way to identify the exact wait phase.

I would keep the mechanism classified as related but not yet confirmed identical. In the first turn, the reasoning items at 18:00:16 and 18:00:28 show that its response stream had already started. Only about 103 seconds elapsed between the last recorded item and the interrupt, which is below the default 300-second stream-idle timeout. That first turn therefore does not demonstrate the pre-response-header, unbounded wait reported here.

The second turn is the relevant candidate, but run_sampling_request -> try_run_sampling_request ends above the transport layer. In 0.146.0, the built-in OpenAI provider supports WebSockets, so the absence of response events does not reveal which transport was active. The most decisive trace markers would be:

• transport="responses_http" versus transport="responses_websocket";
• for HTTP, whether Request completed ... status=200 OK or response headers were recorded;
• for WebSocket, whether connection or reuse and request send completed;
• the exact duration of the second wait, including whether it crossed 300 seconds without an idle timeout, retry, or terminal event.

If HTTP was still waiting for response headers, this directly reinforces #31376. If HTTP had already received 200 OK but no events followed, it is closer to #32366. If WebSocket was active, it points instead to the related connect/reuse/send or first-event path. The missing process may also make lifecycle markers in the trace useful.

The analytics request failure is valuable contemporaneous connectivity evidence, but, as you noted, it is a separate request path. In-product feedback is preferable to posting raw logs publicly because response headers can contain sensitive values.

Thanks for capturing this and for offering the trace.

Confirming a closely related failure signature on Windows with additional system-level evidence.

Environment

• Windows 11 Home 25H2, build 26200
• ChatGPT/Codex desktop package version 26.803.10989.0
• Sustained, concurrent agentic work in the Codex desktop app

Observed behavior
The entire PC becomes severely bogged down even though CPU, physical RAM, commit, disk latency/queueing, GPU utilization, and VRAM all retain substantial headroom. Windows has repeatedly logged global ephemeral-port exhaustion:

• TCP/IP Event ID 4231: 4 events in the prior 14 days
• TCP/IP Event ID 4266 (UDP): 10 events in the prior 14 days
• Most recent TCP exhaustion event: 2026-08-10 21:01:11 local time

An elevated endpoint snapshot taken about 83 minutes after that event—after most transient pressure had already cleared—found:

• codex.exe PID 8832 owned 34 of the machine's 35 CLOSE_WAIT sockets
• 32 of those 34 had been stuck for more than one hour
• Median CLOSE_WAIT age was approximately 78 minutes; oldest approximately 117 minutes
• 32 targeted 172.64.155.209:443; 2 targeted 104.18.32.47:443
• Chrome, the ChatGPT UI process, and Claude each had zero CLOSE_WAIT sockets in the same snapshot
• The Windows dynamic TCP range was normal (49152-65535, 16,384 ports), so this was not caused by a narrowed range

This appears consistent with the failure described here: the remote peer closes a pooled connection, but Codex does not close its side or recover cleanly. The concentration and age of the stale sockets also suggest that the issue may accumulate across concurrent/long-running Codex tasks and potentially contribute to ephemeral-port exhaustion rather than only hanging one individual run.

I cannot restart the app immediately because several long-running tasks are in flight. I have full system-performance and endpoint-owner captures and can provide them privately to OpenAI; I am not attaching them publicly because they contain process metadata and endpoint addresses.

Live recurrence after a full Windows reboot, with the desktop app left running under sustained concurrent agentic work:

• Roughly nine hours after reboot, the PC again became severely bogged down.
• Current Get-NetTCPConnection state totals were: Bound=148, Established=67, TimeWait=29, Listen=26, CloseWait=26.
• The current codex.exe process (PID 2272) owned:
• 91 BOUND records
• 24 CLOSE_WAIT records
• 13 ESTABLISHED records
• Therefore Codex owned 24 of all 26 CLOSE_WAIT sockets (92.3%) in the live slow state.
• The only other CLOSE_WAIT owners had one socket each (nvcontainer.exe and backgroundTaskHost.exe).
• The Windows IPv4 TCP dynamic range remains the normal 49152-65535 (16,384 ports).

Important nuance: this particular live check does not show the global TCP pool currently exhausted; the endpoint table contains only hundreds of records, not approximately 16K unique dynamic ports. It does show that the highly concentrated Codex CLOSE_WAIT signature returns quickly after reboot and again coincides with severe degradation. The earlier Windows Event ID 4231 records may reflect transient spikes that have already drained by the time a manual snapshot is taken.

The app cannot be restarted immediately because multiple local tasks are still in flight. This makes a task-level timeout/cleanup path especially important: stale pre-header/half-closed requests should be failed and retried or canceled individually rather than requiring termination of the shared desktop codex.exe process.

Traced this to the root cause the report already hypothesized: the responses/SSE request path never sets a per-request timeout, and the shared reqwest::ClientBuilder never enables HTTP/2 keep-alive.

Confirmed on current main:

• Provider::build_request always sets timeout: None (codex-rs/codex-api/src/provider.rs), so ReqwestTransport::stream's builder.send().await (codex-rs/http-client/src/transport.rs) has no timeout applied for the header-wait phase.
• The shared HttpClientBuilder::base_reqwest_builder() (codex-rs/http-client/src/client_builder.rs), which every outbound Codex HTTP client goes through (responses API, MCP HTTP transports, etc. — traced via create_client_for_route/create_client), never calls http2_keep_alive_interval/http2_keep_alive_timeout/http2_keep_alive_while_idle.

So a pooled HTTP/2 connection the peer silently closes (CLOSE_WAIT, matching the report's Get-NetTCPConnection evidence) can be reused for a new request and hang forever: nothing below the application layer ever notices the dead socket.

Fix: enable HTTP/2 keep-alive pings (bounded ack timeout, including while idle in the pool) on HttpClientBuilder::base_reqwest_builder(), matching the report's suggested fix #2 ("Enable HTTP/2 keepalive PINGs (with timeout) and/or a pool max-idle age"). This bounds detection of a dead connection to roughly interval + timeout instead of the Windows TCP-keepalive fallback (~2h) the report observed, and applies uniformly to every Codex outbound HTTP client since they all funnel through this one builder.

diff --git a/codex-rs/http-client/src/client_builder.rs b/codex-rs/http-client/src/client_builder.rs
index 7cd0b4db2d..0cfd3c873a 100644
--- a/codex-rs/http-client/src/client_builder.rs
+++ b/codex-rs/http-client/src/client_builder.rs
@@ -259,7 +259,11 @@ impl HttpClientBuilder {
     }
 
     fn base_reqwest_builder(self) -> reqwest::ClientBuilder {
-        let mut builder = reqwest::Client::builder();
+        let mut builder = Self::apply_http2_keep_alive(
+            reqwest::Client::builder(),
+            HTTP2_KEEP_ALIVE_INTERVAL,
+            HTTP2_KEEP_ALIVE_TIMEOUT,
+        );
         if let Some(default_headers) = self.default_headers {
             builder = builder.default_headers(default_headers);
         }
@@ -277,8 +281,30 @@ impl HttpClientBuilder {
         }
         builder
     }
+
+    /// Configures HTTP/2 keep-alive pings so a peer that silently drops a connection (for
+    /// example, a socket the peer closed that lingers in `CLOSE_WAIT` on our side) gets detected
+    /// and evicted instead of hanging a request forever while waiting for response headers that
+    /// will never arrive. Pings are also sent while a connection is idle in the pool so a stale
+    /// connection is more likely to be caught before it is reused for a new request.
+    fn apply_http2_keep_alive(
+        builder: reqwest::ClientBuilder,
+        interval: Duration,
+        timeout: Duration,
+    ) -> reqwest::ClientBuilder {
+        builder
+            .http2_keep_alive_interval(interval)
+            .http2_keep_alive_timeout(timeout)
+            .http2_keep_alive_while_idle(/*enabled*/ true)
+    }
 }
 
+/// How often to probe HTTP/2 connections with a keep-alive `PING` frame.
+const HTTP2_KEEP_ALIVE_INTERVAL: Duration = Duration::from_secs(20);
+
+/// How long to wait for a keep-alive `PING` to be acknowledged before closing the connection.
+const HTTP2_KEEP_ALIVE_TIMEOUT: Duration = Duration::from_secs(10);
+
 impl Default for HttpClientBuilder {
     fn default() -> Self {
         Self {

Also added a regression test (codex-rs/http-client/src/client_builder_tests.rs) that accepts a TCP connection and then never reads or writes anything (simulating a peer that silently disappears), and asserts a request over that connection fails within ~150ms once keep-alive detects it, instead of hanging. It doesn't need a real HTTP/2 server: an http2_prior_knowledge client drives its own keep-alive timer as soon as it opens the socket, independent of whether the peer ever responds.

Kept this first change dependency-free (no new crates), so I intentionally didn't add a lower-level test that spins up a real h2 server to reproduce the exact "healthy connection reused, then goes CLOSE_WAIT" sequence end-to-end — happy to add that as a follow-up (via the h2 crate as a dev-dependency) if maintainers want tighter coverage.

Verified:

• cargo check -p codex-http-client and downstream (cargo check -p codex-core -p codex-api -p codex-login)
• just fmt
• just test -p codex-http-client (76/76 passed, including the new test)
• just fix -p codex-http-client

Branch: https://github.com/Excelius-Wang/codex/compare/main...Excelius-Wang:codex:fix/http2-keep-alive-dead-connections

I don't have collaborator access on this repo (PR creation here is restricted to collaborators), so I can't open a PR directly — posting the diff here in case a maintainer wants to pick it up.

Thanks @Excelius-Wang — this is a useful candidate fix, and the shared client builder is a sensible layer to evaluate. I reviewed the linked diff at 69c972e.

One important distinction from the Windows captures: CLOSE_WAIT means the peer's FIN has already reached the local TCP stack and the local process has not completed its side of the close. The added test accepts a TCP connection and then neither reads nor writes for five seconds; it exercises keep-alive detection of an unresponsive/black-holed peer, but it does not reproduce a previously healthy HTTP/2 connection that later receives a FIN and remains in CLOSE_WAIT.

That makes HTTP/2 keep-alive a valuable mitigation candidate, but I would not call the observed field root cause confirmed yet. Exact regression coverage would ideally:

1. establish a real HTTP/2/SSE connection;
2. complete or begin a response and then have the peer close or half-close the connection;
3. exercise the pooled-reuse or cancellation race;
4. assert both bounded request failure and prompt connection/driver cleanup.

A separate bounded pre-header timeout/cancellation path may still be needed. A blanket reqwest request timeout also covers the long-lived response body and could terminate legitimate SSE streams.

I can validate a maintainer/nightly Windows build against the recurring CLOSE_WAIT counts once one is available.