app-server: active turn hangs indefinitely when the client disconnects during an in-flight exec (remote unix-socket transport)
app-server: active turn hangs indefinitely when the client disconnects during an in-flight exec (remote unix-socket transport)
Environment
- codex-cli / app-server 0.149.0 (standalone install), Debian 13 x86_64
- daemon started via
codex remote-control start; client iscodex --remote unix://…on macOS, reaching the daemon's control socket through an SSH-forwarded unix socket - long-lived thread (resumed legacy session), agent mid-turn running a long shell command via the exec tool
What happened
The client machine (laptop) went to sleep, severing the SSH tunnel and therefore the websocket-over-unix-socket connection, while the agent's turn had an exec in flight (a long go test run it was monitoring).
Journal timeline (thread rollout):
01:09:21…01:11:03— turn active and healthy: agent emits progress messages every ~60–75 s while polling its long-running command.01:11:03— last journal record of the turn. Client connection died at approximately this time.01:11→01:29— 18 minutes of total silence: no further items, notask_complete, no error, no abort. The turn produced nothing and never finished.01:29:43— client reconnects, resumes the thread, submits a new user message (task_started+user_message).01:29:57—turn_abortedrecorded — i.e. the stalled turn is only terminated as a side effect of the client-side interrupt at reattach, ~19 minutes after it stopped making progress.
Expected
Per the disconnect design (thread connection_closed only unsubscribes; turns are not tied to connections), the turn should have either:
- continued to completion server-side (if its exec is turn-owned), or
- — if the in-flight exec is connection-scoped and gets reaped on disconnect — had its pending tool call resolve with an error so the model could react (retry, report, finish the turn).
Actual
The in-flight exec appears to be reaped by the connection-close cleanup, and the turn's pending await on it never resolves. The turn neither completes nor fails: it hangs silently and invisibly (an unattended daemon would hold it forever; ours held it until manual reconnect). No error is journaled at the stall point.
Source reading (0.149.0)
app-server/src/lib.rsTransportEvent::ConnectionClosed→processor.connection_closed(...)app-server/src/message_processor.rsconnection_closeddrains RPCs, then callsoutgoing/fs/command_exec/process_exec/threadprocessors' cleanup.app-server/src/request_processors/thread_processor.rsconnection_closedonly removes thread subscriptions (no turn abort) — consistent with the intended "turns survive disconnects" semantics.command_exec.rs/ process-exec cleanup terminate sessions owned by the closed connection. When the terminated execution is the one an active turn is awaiting, nothing appears to deliver a completion/error back to the turn — matching the observed permanent stall.
Suggested fix direction
When connection-close cleanup terminates an execution that an active turn is awaiting, resolve the pending tool call with an explicit error (e.g. "execution terminated: owning connection closed") so the turn can proceed and journal the outcome — or, alternatively, don't scope turn-initiated execs to the client connection at all, since the turn itself survives the disconnect by design.
Notes
- Recovery works as expected: reconnect + interrupt + new turn continues from journaled state; no data loss beyond the stalled turn's unfinished tail.
- Possibly adjacent to #36185 / #29262 (remote-control disconnect symptoms), but those concern notification delivery; this is about turn liveness.
2 Comments
Verified against current
main(@0d9bb6c34c) — the mechanism is unchanged from your 0.149.0 trace, and I can pin down precisely why the turn's await never resolves.Confirmed on main:
CommandExecManager::connection_closed(codex-rs/app-server/src/command_exec.rs:388-415) removes every session owned by the connection and sendsCommandControlRequest { control: CommandControl::Terminate, response_tx: None }.response_tx: Nonematters: for a client-initiatedcommand/exec/terminatethe request handler still resolves the caller, but on the connection-close path there is intentionally nobody to answer — the session is also dropped from the map (line 398), so any latersend_controlgetscommand_no_longer_running_error.run_commandthen terminates the process and, once the exit code is known, callsoutgoing.send_response(request_id, CommandExecResponse { exit_code, stdout, stderr })(command_exec.rs:557-566). Butrequest_idis aConnectionRequestIdwhoseconnection_idis the just-closed connection — the envelope is addressed to a transport that no longer exists. The completion is emitted, it just has no live destination.So the stall is not "nothing delivers a result" — the result is delivered into a dead connection, while the turn's pending tool-call item is keyed to the thread and never sees it.
The underlying asymmetry: exec session ownership is keyed by
ConnectionProcessId { connection_id, process_id }(command_exec.rs:62-66), i.e. execs are connection-scoped by construction, while turns explicitly survive disconnects (thread_processor::connection_closedonly removes thread subscriptions). Those two lifecycle contracts disagree, and the disconnect path resolves the disagreement by reaping the exec but never telling the turn.Fix directions, refined:
ThreadItem::CommandExecution) as turn-owned, and onconnection_closedhand them off instead of terminating them — the response can then be routed by thread/turn rather than by connection, matching the "turns survive disconnects" design. This is your option (b) and I think it is the semantically correct one.ThreadItem::CommandExecutionwith an explicit error (e.g."execution terminated: owning connection closed") so the model can react and the turn can finish — your option (a), which is the minimal behavioral fix but keeps the ownership mismatch.request_contexts/telemetry alive (as I initially suspected) is not the fix — those are tracing-only (outgoing_message.rsRequestContext); the actual resolution path is the connection-scopedrequest_idenvelope.Happy to elaborate with a concrete patch outline for option (1) if that direction is preferred.
Follow-up: the forced kill described in #40969 (auto-update sends SIGKILL after a 60s drain budget) is one reliable way to reach this state — the app-server dies while turns are awaiting an exec, and those turns then hang rather than surfacing an error. Observed again on 2026-08-26 when the daemon auto-updated 0.149.0 → 0.150.0 with several agent turns mid-execution.