app-server: active turn hangs indefinitely when the client disconnects during an in-flight exec (remote unix-socket transport)

Open 💬 2 comments Opened Aug 24, 2026 by yazzaoui

app-server: active turn hangs indefinitely when the client disconnects during an in-flight exec (remote unix-socket transport)

Environment

  • codex-cli / app-server 0.149.0 (standalone install), Debian 13 x86_64
  • daemon started via codex remote-control start; client is codex --remote unix://… on macOS, reaching the daemon's control socket through an SSH-forwarded unix socket
  • long-lived thread (resumed legacy session), agent mid-turn running a long shell command via the exec tool

What happened

The client machine (laptop) went to sleep, severing the SSH tunnel and therefore the websocket-over-unix-socket connection, while the agent's turn had an exec in flight (a long go test run it was monitoring).

Journal timeline (thread rollout):

  • 01:09:2101:11:03 — turn active and healthy: agent emits progress messages every ~60–75 s while polling its long-running command.
  • 01:11:03 — last journal record of the turn. Client connection died at approximately this time.
  • 01:1101:2918 minutes of total silence: no further items, no task_complete, no error, no abort. The turn produced nothing and never finished.
  • 01:29:43 — client reconnects, resumes the thread, submits a new user message (task_started + user_message).
  • 01:29:57turn_aborted recorded — i.e. the stalled turn is only terminated as a side effect of the client-side interrupt at reattach, ~19 minutes after it stopped making progress.

Expected

Per the disconnect design (thread connection_closed only unsubscribes; turns are not tied to connections), the turn should have either:

  1. continued to completion server-side (if its exec is turn-owned), or
  2. — if the in-flight exec is connection-scoped and gets reaped on disconnect — had its pending tool call resolve with an error so the model could react (retry, report, finish the turn).

Actual

The in-flight exec appears to be reaped by the connection-close cleanup, and the turn's pending await on it never resolves. The turn neither completes nor fails: it hangs silently and invisibly (an unattended daemon would hold it forever; ours held it until manual reconnect). No error is journaled at the stall point.

Source reading (0.149.0)

  • app-server/src/lib.rs TransportEvent::ConnectionClosedprocessor.connection_closed(...)
  • app-server/src/message_processor.rs connection_closed drains RPCs, then calls outgoing/fs/command_exec/process_exec/thread processors' cleanup.
  • app-server/src/request_processors/thread_processor.rs connection_closed only removes thread subscriptions (no turn abort) — consistent with the intended "turns survive disconnects" semantics.
  • command_exec.rs / process-exec cleanup terminate sessions owned by the closed connection. When the terminated execution is the one an active turn is awaiting, nothing appears to deliver a completion/error back to the turn — matching the observed permanent stall.

Suggested fix direction

When connection-close cleanup terminates an execution that an active turn is awaiting, resolve the pending tool call with an explicit error (e.g. "execution terminated: owning connection closed") so the turn can proceed and journal the outcome — or, alternatively, don't scope turn-initiated execs to the client connection at all, since the turn itself survives the disconnect by design.

Notes

  • Recovery works as expected: reconnect + interrupt + new turn continues from journaled state; no data loss beyond the stalled turn's unfinished tail.
  • Possibly adjacent to #36185 / #29262 (remote-control disconnect symptoms), but those concern notification delivery; this is about turn liveness.

View original on GitHub ↗

2 Comments

argszero · 3 days ago

Verified against current main (@0d9bb6c34c) — the mechanism is unchanged from your 0.149.0 trace, and I can pin down precisely why the turn's await never resolves.

Confirmed on main:

  • CommandExecManager::connection_closed (codex-rs/app-server/src/command_exec.rs:388-415) removes every session owned by the connection and sends CommandControlRequest { control: CommandControl::Terminate, response_tx: None }.
  • The response_tx: None matters: for a client-initiated command/exec/terminate the request handler still resolves the caller, but on the connection-close path there is intentionally nobody to answer — the session is also dropped from the map (line 398), so any later send_control gets command_no_longer_running_error.
  • run_command then terminates the process and, once the exit code is known, calls outgoing.send_response(request_id, CommandExecResponse { exit_code, stdout, stderr }) (command_exec.rs:557-566). But request_id is a ConnectionRequestId whose connection_id is the just-closed connection — the envelope is addressed to a transport that no longer exists. The completion is emitted, it just has no live destination.

So the stall is not "nothing delivers a result" — the result is delivered into a dead connection, while the turn's pending tool-call item is keyed to the thread and never sees it.

The underlying asymmetry: exec session ownership is keyed by ConnectionProcessId { connection_id, process_id } (command_exec.rs:62-66), i.e. execs are connection-scoped by construction, while turns explicitly survive disconnects (thread_processor::connection_closed only removes thread subscriptions). Those two lifecycle contracts disagree, and the disconnect path resolves the disagreement by reaping the exec but never telling the turn.

Fix directions, refined:

  1. Turn-owned execs should not be connection-scoped. Mark exec sessions initiated by a turn tool call (agent source, tracked as ThreadItem::CommandExecution) as turn-owned, and on connection_closed hand them off instead of terminating them — the response can then be routed by thread/turn rather than by connection, matching the "turns survive disconnects" design. This is your option (b) and I think it is the semantically correct one.
  2. If termination is required (sandbox/resource constraints), the terminate path must resolve the pending ThreadItem::CommandExecution with an explicit error (e.g. "execution terminated: owning connection closed") so the model can react and the turn can finish — your option (a), which is the minimal behavioral fix but keeps the ownership mismatch.
  3. Note that simply keeping request_contexts/telemetry alive (as I initially suspected) is not the fix — those are tracing-only (outgoing_message.rs RequestContext); the actual resolution path is the connection-scoped request_id envelope.

Happy to elaborate with a concrete patch outline for option (1) if that direction is preferred.

yazzaoui · 1 day ago

Follow-up: the forced kill described in #40969 (auto-update sends SIGKILL after a 60s drain budget) is one reliable way to reach this state — the app-server dies while turns are awaiting an exec, and those turns then hang rather than surfacing an error. Observed again on 2026-08-26 when the daemon auto-updated 0.149.0 → 0.150.0 with several agent turns mid-execution.