App-server disconnects remote TUI mid-turn when outbound queue fills (128 messages)

Resolved 💬 3 comments Opened Apr 16, 2026 by cipher982 Closed Apr 24, 2026

💡 Likely answer: A maintainer (etraut-openai, contributor) responded on this thread — see the highlighted reply below.

What version of Codex CLI is running?

v0.121.0

What subscription do you have?

None, API Keys

Which model were you using?

gpt-5.4

What platform is your computer?

Darwin 25.4.0 arm64 arm

What terminal emulator and version are you using (if applicable)?

Warp

What issue are you seeing?

The app-server's WebSocket transport disconnects remote TUI clients instantly when its 128-message outbound queue fills, instead of applying backpressure. This makes --remote mode unusable for any turn that produces moderate output volume.

The TUI exits with:

ERROR: remote app server at `ws://127.0.0.1:<port>/` transport failed:
  WebSocket protocol error: Connection reset without closing handshake

The app-server logs:

disconnecting slow connection after outbound queue filled: ConnectionId(0)

This reproduces on stock upstream Codex with a single remote TUI client and no third-party tooling.

What steps can reproduce the bug?

Terminal 1 — start the app-server:

codex app-server --listen ws://127.0.0.1:0
# note the printed ws:// URL

Terminal 2 — attach the remote TUI:

codex --remote ws://127.0.0.1:<port>

Then send a prompt that produces moderate streaming output:

Run `rg -n '120|timeout' ~/.codex` and answer briefly.

The TUI disconnects within seconds. Smaller prompts sometimes survive, but any turn with realistic tool output triggers it.

What is the expected behavior?

The remote TUI should remain connected during normal turns. A client that falls 128 messages behind during streaming is momentarily slower than the producer, not stuck. it should not be terminated instantly.

Stdio clients already handle this correctly: they use blocking .send().await (stdio.rs:35 sets disconnect_sender: None), so they never hit the disconnect path. WebSocket clients should get equivalent resilience, either through a larger queue, a send timeout, or per-connection send tasks.

Additional information

root cause is in send_message_to_connection() at codex-rs/app-server/src/transport/mod.rs line 308-316. WebSocket clients are marked disconnectable (websocket.rs:176 sets disconnect_sender: Some(...)), so when try_send() returns TrySendError::Full on the 128-slot bounded channel, the connection is terminated immediately.

The outbound router is a single select! loop (lib.rs:643), so blocking on a slow WebSocket is not safe. But the remote_control transport already solves this differently, remote_control/websocket.rs uses per-stream BoundedOutboundBuffer with backpressure instead of instant disconnect.

Design options that preserve the non-blocking router:

Per-connection send tasks. Structurally similar to remote_control's existing approach.
Larger queue + send_timeout. Simplest fix, small blast radius.
Priority-based notification dropping under queue pressure. opt_out_notification_methods already has the right shape.

The existing test broadcast_does_not_block_on_slow_connection (line 878) uses channel capacity 1 and validates instant disconnect as expected behavior. That test encodes the wrong contract — it should validate a grace period, not instant termination.

Related: #13949 (static analysis of same path), #15355 (local ingress feature request).

Reproduced on codex-cli 0.120.0 and 0.121.0, both stock upstream and with a third-party wrapper using --remote mode.

View original on GitHub ↗

3 Comments

etraut-openai contributor · 4 months ago

Thanks for the bug report. Excellent analysis and repro steps!

I've introduced some additional buffering and confirmed that the repro steps no longer drop the connection. This will be included in the next release.

joewolly · 20 days ago

Still reproducing on codex-cli 0.147.0 (standalone package) — the original fix did not cover the code path that triggers this.

Root cause: CHANNEL_CAPACITY is still 128, unchanged by #19246.

PR #19246 raised the WebSocket writer buffer, but the per-connection outbound message channel that triggers the disconnect is a separate constant, and it is still:

// codex-rs/app-server-transport/src/transport/mod.rs (line 25)
pub const CHANNEL_CAPACITY: usize = 128;

Verified identical on main, rust-v0.147.0, and rust-v0.148.0-alpha.5 as of 2026-08-07.

The disconnect path is unchanged too (codex-rs/app-server/src/transport.rs):

if connection_state.can_disconnect() {
    match writer.try_send(queued_message) {
        Ok(()) => false,
        Err(mpsc::error::TrySendError::Full(_)) => {
            warn!("disconnecting slow connection after outbound queue filled: {connection_id:?}");
            disconnect_connection(connections, connection_id)
        }
        ...

So remote-control clients (phone/desktop apps via the relay) are still dropped mid-turn whenever the 128-slot queue fills — which happens routinely with two concurrent threads streaming reasoning deltas (each can emit hundreds of events/sec at high effort) through one connection.

Live reproduction (2026-08-07, remote-control connection):

20:23:28 WARN  transport | disconnecting slow connection after outbound queue filled: ConnectionId(25)
20:23:28 WARN  transport | dropping message for disconnected connection: ConnectionId(25)
20:23:28 INFO  client_tracker | forwarding remote control connection closed transport event connection_id=ConnectionId(25)

Trigger: two threads (opencode-go/deepseek-v4-flash, reasoning_effort=max, routed through an OpenAI-compatible proxy) streaming simultaneously to one client connection. The app shows "connection lost" on both phone and laptop; the affected thread survives server-side but the client session is dropped.

Suggested fix: raise the outbound capacity for remote-control connections (e.g., 64K like the writer buffer), or replace try_send + immediate disconnect with async backpressure (await send) and only disconnect after a timeout, so a slow-but-alive client is not killed by a burst.

etraut-openai contributor · 20 days ago

@joewolly, if you're experiencing a problem, please open a new bug report and use /feedback to upload your logs.