›_ Codex Issues
Codex Issues / #37526

App-server still drops remote-control clients when outbound queue fills (128 messages) — #18203 fix missed the code path

Open 💬 2 comments Opened Aug 8, 2026 by joewolly

Summary

codex app-server still disconnects remote-control clients mid-turn with disconnecting slow connection after outbound queue filled on codex-cli 0.147.0 (and the same code exists on current main and rust-v0.148.0-alpha.5). This is the same symptom as #18203, which was closed as fixed by #19246 — but that PR only raised the WebSocket writer buffer. The per-connection outbound message channel that actually triggers the disconnect is still 128 messages, so the bug is unfixed.

Environment

  • codex-cli 0.147.0 (standalone package, Linux x86_64)
  • codex app-server with remote control enabled; phone + desktop apps connected through the chatgpt.com relay (wss://chatgpt.com/backend-api/wham/remote/control/server)
  • model routed through a local OpenAI-compatible proxy (opencode-go/deepseek-v4-flash, reasoning_effort=max)

Root cause

CHANNEL_CAPACITY is still 128 (verified identical on main, rust-v0.147.0, and rust-v0.148.0-alpha.5 as of 2026-08-07):

// codex-rs/app-server-transport/src/transport/mod.rs (line 25)
pub const CHANNEL_CAPACITY: usize = 128;

Disconnect path unchanged (codex-rs/app-server/src/transport.rs):

if connection_state.can_disconnect() {
    match writer.try_send(queued_message) {
        Ok(()) => false,
        Err(mpsc::error::TrySendError::Full(_)) => {
            warn!("disconnecting slow connection after outbound queue filled: {connection_id:?}");
            disconnect_connection(connections, connection_id)
        }
        ...

Reproduction

Run two threads concurrently so both stream reasoning deltas to one remote-control connection (each max-effort thread can emit hundreds of events/sec; relay latency lets the 128-slot queue fill). The app-server then kills the client:

20:23:28 WARN  transport | disconnecting slow connection after outbound queue filled: ConnectionId(25)
20:23:28 WARN  transport | dropping message for disconnected connection: ConnectionId(25)
20:23:28 INFO  client_tracker | forwarding remote control connection closed transport event connection_id=ConnectionId(25)

Both the phone and desktop apps show "connection lost"; the thread keeps running server-side but the client session is dropped.

Expected behavior

A slow-but-alive client should not be disconnected because of a burst. The outbound queue should be sized for streaming workloads, or the writer should apply backpressure (await send) and only disconnect after a timeout.

Suggested fix

  • Raise CHANNEL_CAPACITY for remote-control connections (the WebSocket writer buffer was raised to 64K in #19246), or
  • Replace try_send + immediate disconnect with async backpressure plus a disconnect timeout.

References

View original on GitHub ↗

2 Comments

niklassheth · 19 days ago

I'm experiencing this issue, it's incredibly frustrating because every time the connection drops my side chats expire.

jdcodes1 · 9 days ago

Verified on main (1f41cc5d92): both halves of the analysis hold. CHANNEL_CAPACITY is still 128 (app-server-transport/src/transport/mod.rs#L25, re-exported into the app-server), and the disconnect path is byte-identical — try_send → TrySendError::Full → immediate disconnect_connection (https://github.com/openai/codex/blob/1f41cc5d92/codex-rs/app-server/src/transport.rs#L156-L163). So #19246 enlarged the layer below the one that kills the connection; the mpsc hop above it still hard-drops at 128, and two max-effort reasoning streams through a relay can outrun that trivially.

Beyond raising the constant, the structural issue is that disconnect is the only overflow policy. For a remote-control observer, the queue is full of deltas whose value decays instantly — the correct degradation is coalescing/shedding, not connection teardown: drop oldest delta messages (never lifecycle/completion events) when full, or collapse consecutive deltas for the same item into the latest one. That bounds memory identically while making slow relays lossy-but-alive — a phone missing some reasoning frames is fine; a phone being disconnected mid-turn (then re-syncing everything) is both worse UX and more total bandwidth. A capacity bump alone just moves the cliff; the reproduction with two concurrent threads will find any fixed constant.