[TUI] In-process app-server can deadlock during automatic skills refresh
Open 💬 2 comments Opened Aug 3, 2026 by Camsyn
What version of Codex CLI is running?
main
What subscription do you have?
pro
What issue are you seeing?
During a Codex run, the TUI sometimes freezes completely. It no longer responds to keyboard input and produces no further output.
Based on GDB analysis and codex assistance, this may be a bounded-channel deadlock:
- The TUI is blocked while awaiting
skills_list. - The in-process app-server is blocked while sending
TurnCompleted.
There is no explicit error message; the failure appears as a silent hang.
What steps can reproduce the bug?
This is an intermittent concurrency bug and cannot currently be reproduced reliably.
The suspected triggering conditions are:
- The TUI starts awaiting an automatic
skills_listrequest and stops consuming events.. - The app-server rapidly produces a large number of
AgentMessageDeltaevents. - The bounded event queues become full.
- The app-server then sends
TurnCompletedto the event queue, before the pendingskills/listresponse. - The app-server to block on
event_tx.send(...), waiting for TUI to consume the events in the queue. - The TUI remains blocked waiting for the response and still stops consuming events.
The suspected timing pattern is:
TUI: await app_server.skills_list(...)
app-server: produce many AgentMessageDelta events
queues: become full
app-server: await event_tx.send(TurnCompleted), cannot give the subsequent skills/list response
TUI: remains blocked waiting for the skills/list responseAdditional information
The following flowchart describes the suspected deadlock cycle (with the help of gdb and codex).
flowchart TD
S["1. Skills auto-refresh<br/>SkillsWatcher::spawn_event_loop"] --> B["2. TUI awaits skills/list"]
C["3. Many AgentMessageDelta events"] --> O["4. Facade event queue full<br/>128 / 128"]
B -- "TUI stops consuming events" --> O
O --> W["5. Facade forwarding worker blocks"]
W --> I["6. Inner event queue full<br/>128 / 128"]
I --> Q{"7. Is TurnCompleted handled before the response?"}
Q -- "No" --> N["8. Response completes normally"]
Q -- "Yes" --> R["9. App-server blocks on<br/>event_tx.send(TurnCompleted)"]
R --> P["10. skills/list response cannot complete"]
P --> B
B --> G["11. TUI stops processing input and output"]
classDef dead fill:#DD9397,stroke:#321E2D
class S,B,R,P dead
2 Comments
I am running into the same issue: codex-cli froze on two different occasions when I Ctrl-C'd out of a
/btwsession back to a long-running codex conversation that probably made things slow at an unexpected point and caused a hang. I could observe the agent was still working (launching processes etc.) but the UI was frozen. I actually had another Sol instance usegdbto attach to the frozen session and successfully revive it (!!), so I am assuming it knows what it is talking about. Here is its (lightly edited) report, which it tells me might be useful here:> I can now confirm this failure twice on two separate live Linux systems running Codex 0.147.0.
>
> Both incidents occurred in the /btw/Ctrl-C side-conversation context after a long-running session. The original Codex processes and their agents remained alive; only the TUI stopped consuming input/output.
>
> In both incidents, symbolized GDB tracing produced the same fingerprint:
>
> - App::handle_event was synchronously awaiting InProcessAppServerClient::request<SkillsListResponse>.
> - The outer Tokio oneshot had strong=2, weak=1, and state 0x1 (RX_TASK_SET, with neither value nor closed state).
> - Its receiver waker exactly matched the parked top-level TUI waker.
> - Invoking the validated Tokio wake_by_ref caused one top-level poll, but the request returned Pending again. Thus this was not merely a lost wake.
> - Closing only that validated response receiver restored the TUI through Codex’s normal request-error path. The bounded top-level poll completed, outgoing app-server messages resumed, and the PID, process start time, and running agent remained intact.
>
> The second recurrence adds direct backpressure evidence. There were 28:
>
> > dropping in-process server notification (queue full)
>
> warnings during 20:27:37–20:27:38 UTC. The 0.147.0 source establishes the relevant default in-process capacity as 128.
>
> Timing around completion was:
>
> 20:27:38.052822 UTC turn/completed emitted
> 20:27:38.052978 UTC final queue-full warning
>
> TurnCompleted is classified as requiring delivery and therefore uses event_tx.send(...).await. That send occurs in the same runtime routing loop that must route responses to pending request oneshots. This strongly supports the cycle described in this issue:
>
> foreground skills/list await
> -> TUI stops consuming app-server events
> -> bounded event queues fill
> -> required notification delivery blocks the router
> -> skills/list response cannot reach its oneshot
> -> TUI cannot resume event consumption
>
> Proof boundary: the queue saturation is directly proved, but neither debugger stopped inside the inner blocked send, so I cannot prove that this particular TurnCompleted was the exact blocking message.
>
> RSS differed substantially between the two observed occurrences—approximately 2.65 GiB and 6.14 GiB—but the future and channel fingerprints were identical. GDB 16.3 and 15.1 both worked with the exact release symbols.
>
> A useful regression test would use a deliberately small/saturated in-process event capacity, stop foreground event consumption while AppCommand::ListSkills is pending, enqueue a required TurnCompleted before the skills response, and verify that response routing or TUI progress cannot deadlock. Decoupling response routing from blocking notification delivery appears more fundamental than adding only a request timeout; making automatic skills refresh background/cancellable would provide an additional containment boundary.
Environment
Reproduction steps
Expected behavior
Assistant output continues streaming, tool-call lifecycle events are visible, and the TUI remains responsive to keyboard input.
Actual behavior
Output stops after the prompt and one assistant line; no further streamed text or tool call appears; no explicit error; session appears silently stalled.
Root-cause hypothesis
0.147.0 embedded app-server client uses a bounded caller-facing event queue. Lossless notifications block in event_tx.send(event).await when the queue is full, while the TUI main loop is occupied awaiting a foreground app-server request, so the worker cannot drain runtime stream events. Evidence: codex-rs/app-server-client/src/lib.rs:141, :222, :468; codex-rs/tui/src/app.rs:1198; codex-rs/tui/src/app/event_dispatch.rs:483; codex-rs/tui/src/chatwidget/input_submission.rs:359. main already changes this queue to unbounded at codex-rs/app-server-client/src/lib.rs:347 and adds regression test unread_lossless_notifications_do_not_block_in_process_requests at :1189.
Excluded possibilities
Markdown renderer line loss is not primary: controller finalizes from full source and 38 scripted ConPTY runs rendered multiple lines and tool completion. Missed redraw cannot stop tool lifecycle events. App-server disconnect has a fatal path, not this silent freeze. Provider/model truncation is not fully ruled out by negative local reproduction but is less consistent with the source-level fix.
Official issue search evidence
gh search issues found no exact-phrase duplicate, but openai/codex#36750 already covers the 0.147.0 bounded-queue deadlock with GDB and queue-full evidence. #32551 and #31340 cover the visible TUI truncation symptom. No open PR matched #36750 and no linked PR appeared in its timeline. Recommended action: do not create a new issue; add this analysis to #36750.