›_ Codex Issues
Codex Issues / #34037

In-process app-server drops AgentMessageDelta under backpressure, truncating TUI output

Resolved 💬 6 comments Opened Jul 18, 2026 by sky-bro Closed Jul 22, 2026
💡 Likely answer: A maintainer (etraut-openai, contributor) responded on this thread — see the highlighted reply below.

What version of Codex CLI is running?

codex-cli 0.144.5

Which model were you using?

gpt-5.6-sol through a custom Responses HTTP provider (supports_websockets = false). The provider does not appear relevant: the complete response was persisted before the TUI transport loss.

What platform is your computer?

Linux xyz 6.18.12-1-MANJARO x86_64 GNU/Linux

Terminal multiplexer: tmux 3.6a

What issue are you seeing?

The interactive CLI TUI can render a truncated assistant answer when its in-process app-server notification queue is full.

In one occurrence, the API response and rollout contained the complete ending:

````markdown
设置后，后续请求会带：

"reasoning": {
  "effort": "low",
  "summary": "detailed",
  "context": "all_turns"
}

````

but the live terminal ended at:

设置后，后续请求 "

The persisted rollout contained the complete assistant response_item and task_complete.last_agent_message, so this is not model/API truncation. The complete HTTP Responses stream was also independently captured. The TUI consolidation log recorded a shorter source (source_len=1902, versus 2008 bytes in the complete persisted answer).

At the same timestamp, logs_2.sqlite contained repeated warnings:

dropping in-process server notification (queue full)
dropping in-process app-server event because consumer queue is full
app-server event consumer lagged

This initially looked like fenced-JSON Markdown corruption, but the JSON block merely occurred after the missing transcript delta.

Deterministic source-level reproduction

Current main at 56395bddaf26eb2829387ca6a417bf9128e5b239 still has inconsistent must-deliver classification between the two in-process queue layers.

Add this test to codex-rs/app-server/src/in_process.rs:

use codex_app_server_protocol::AgentMessageDeltaNotification;

#[test]
fn guaranteed_delivery_helpers_cover_agent_message_deltas() {
    assert!(server_notification_requires_delivery(
        &ServerNotification::AgentMessageDelta(AgentMessageDeltaNotification {
            thread_id: "thread-1".to_string(),
            turn_id: "turn-1".to_string(),
            item_id: "item-1".to_string(),
            delta: "```json\n{\"reasoning\": true}\n```".to_string(),
        })
    ));
}

Then run:

cd codex-rs
cargo test -p codex-app-server guaranteed_delivery_helpers_cover_agent_message_deltas --lib -- --exact

The assertion fails deterministically because codex-rs/app-server/src/in_process.rs::server_notification_requires_delivery currently matches only:

TurnCompleted
ThreadSettingsUpdated
ExternalAgentConfigImportCompleted

As a result, AgentMessageDelta reaches the event_tx.try_send(...) branch and is dropped when that queue is full.

By contrast, codex-rs/app-server-client/src/lib.rs::server_notification_requires_delivery correctly treats the following as lossless and forwards them with send(...).await:

ItemCompleted
AgentMessageDelta
PlanDelta
ReasoningSummaryTextDelta
ReasoningTextDelta

The second layer therefore cannot preserve a transcript delta that the first layer has already dropped.

Expected behavior

Assistant, plan, and reasoning transcript deltas and authoritative item/turn completion notifications should never be dropped under in-process backpressure. The TUI should render the same assistant message that is persisted in the rollout.

Both in-process queue layers should share the same must-deliver classification, or the app-server layer should include all transcript and completion notification variants and use send(...).await for them.

Additional information

Related reports:

  • #20777 reported the same dropping in-process server notification (queue full) warning and was closed as a duplicate of #20601.
  • When closing #20601, maintainers asked users to file a new report if similar symptoms recurred.
  • PR #21870 addressed TUI active-thread routing, but its changed files were limited to codex-rs/tui; it did not align the lower-level helper in codex-rs/app-server/src/in_process.rs.
  • A comment on #20601 explicitly proposed aligning the lower-level helper for ItemCompleted and assistant/plan/reasoning deltas, which matches the remaining gap in current main.

No raw rollout or SQLite database is attached because they contain private prompts and local paths. A sanitized thread/session identifier can be provided privately or through /feedback if useful.

View original on GitHub ↗

6 Comments

TheJhyeFactor · 1 month ago

Hey, I had a look at this against current main (2deed3fb9c) on macOS.

The lower in-process queue only treated turn completion, thread settings, and external-import completion as lossless. That meant assistant, plan, and reasoning deltas could be dropped before the higher-level client had a chance to preserve them.

I've prepared a focused fix that aligns the lower queue with the client's transcript classification. ItemCompleted, AgentMessageDelta, PlanDelta, ReasoningSummaryTextDelta, and ReasoningTextDelta now use send(...).await under pressure. Command-output deltas stay best-effort, so high-volume progress events can still be dropped instead of blocking the whole transport.

The patch is 98 changed lines in one file: 84 additions and 14 deletions. Most of that is expanded regression coverage; the production behavior change is the small must-deliver classification above.

I added regression coverage for the must-deliver variants and for the command-output exception. The focused tests pass, and just fix -p codex-app-server plus just fmt are clean.

I also ran just test -p codex-app-server: 954 tests passed. Eight unrelated integration tests failed locally—four because the scoped build didn't provide the codex / test_stdio_server helper binaries, and four macOS zsh-fork tests timed out.

Prepared commit: https://github.com/TheJhyeFactor/codex/commit/3aeb0aae8ebf5c9ca28a024ee983b8afadb5d5d7

I haven't opened a PR because the contribution guide asks external contributors to wait for an invitation. If this direction looks right, the branch is ready for an invited submission.

rayman22201 · 1 month ago

Independent Windows reproduction: safety modal becomes unresponsive while the backend continues

This incident appears to connect the in-process notification backpressure described here with the cybersecurity-interruption/modal reports in #32225 and #32139.

Environment

  • Codex CLI: 0.144.6
  • Model: gpt-5.6-sol
  • Platform: Windows x64
  • Terminal: SpecterTerm
  • Shell hosting Codex: cmd.exe
  • Workload: authorized defensive design work in the user's own local repository; Windows Filtering Platform / egress-firewall hardening specification, named-pipe peer authentication, recovery behavior, and verification planning
  • No live or third-party target

Affected identifiers:

  • Session/thread: 019f8071-5965-7a81-b17b-0940e48ed03c
  • Turn: 019f81a7-1408-7be1-a8de-449ba0fc1908
  • Frozen Codex PID: 2380

What happened

During a long-running specification amendment, the CLI displayed:

Additional safety checks
This request requires additional safety checks, which can take extra time.

1. Retry with a faster model
2. Keep waiting
3. Learn more

The entire TUI then stopped accepting input. Local Esc, terminal resize, and maximize/restore had no effect, so the user could not choose Keep waiting or dismiss the modal.

The terminal itself and the Codex process remained alive. Independent monitoring showed that the Codex session continued appending distinct events and performing meaningful work behind the stale modal. During incident inspection, logs_2.sqlite showed more than 300 occurrences of:

dropping in-process server notification (queue full)

clustered within approximately six seconds for the affected process.

This was not an immediate context-window failure. The thread reached approximately 213k of a 258.4k context window, successfully auto-compacted to approximately 60k, and continued making substantive edits afterward.

Eventually the backend also stopped making progress. The final recorded event was at 2026-07-21T00:29:41.973Z, at a pending Git commit tool call. No corresponding tool result or task_complete reached the turn. The visible TUI remained stuck at the old safety modal.

Recovery

  1. Verified that the session transcript had stopped advancing and identified the exact Codex PID.
  2. Terminated only PID 2380.
  3. Verified the intended two-file staged diff and git diff --cached --check.
  4. Completed the pending exact-path commit manually.
  5. Ran codex resume 019f8071-5965-7a81-b17b-0940e48ed03c.
  6. The session loaded successfully in a new Codex process and returned to a responsive prompt.

No repository work was lost, but the modal offered no usable recovery path and the final commit had to be completed outside the interrupted turn.

Why this seems related to this issue

  • The producer/backend continued emitting and persisting events.
  • The live TUI stopped consuming/rendering them.
  • The local log recorded rapid in-process notification queue saturation.
  • The failure was more severe than truncated output: the modal and composer became completely non-interactive, and the turn later stranded a pending tool call.

The safety check was the transition immediately preceding the wedge, but the evidence does not establish whether it is the root cause or simply a reliable high-event-pressure trigger. The defensive request may also have been a classifier false positive; the UI deadlock is independently a client reliability bug even if the safety review itself was intentional.

Expected behavior

  • The Additional safety checks modal must remain keyboard-responsive under notification backpressure.
  • Choosing Keep waiting must always be possible.
  • The client should not continue hidden backend execution while presenting a stale blocking modal without accurate state.
  • Modal state, transcript deltas, authoritative item/tool completion, and turn completion should use lossless delivery or be reconstructable from persisted state.
  • If the consumer queue falls behind, the TUI should coalesce best-effort progress events, preserve control/completion events, and expose a recoverable error rather than wedging.
  • A safety event should provide a stable incident identifier and a direct Report false positive action.

Reproduction characteristics

The occurrence was intermittent, but the observed workload had these characteristics:

  1. A long interactive CLI session with many tool and reviewer events.
  2. A large, security-heavy defensive specification involving firewall enforcement and authenticated local IPC.
  3. Context usage above 80%, followed by successful automatic compaction.
  4. The Additional safety checks modal appeared during continued agent work.
  5. Notification queue-full warnings spiked while the TUI became unresponsive.

I am not attaching the raw rollout or SQLite database publicly because they contain private repository content and local paths. The identifiers above are provided so OpenAI can correlate the incident internally. A scoped private upload can be supplied if requested.

etraut-openai contributor · 1 month ago

Thanks for the bug report and analysis. This will be addressed in the next release.

TheJhyeFactor · 1 month ago

Thanks, Eric. I’d already prepared and tested a focused patch for this above. It updates the lower in-process queue to preserve ItemCompleted and assistant, plan, and reasoning transcript events under backpressure, while keeping high-volume command output best-effort. The app-server suite completed with 954 passing tests, alongside the unrelated local failures I documented.

Did the team get a chance to review that work, and does the next-release fix follow the same approach? If my branch is still useful, I’m happy to submit it through the invitation process.

etraut-openai contributor · 1 month ago

@TheJhyeFactor, I didn't look at your branch. Codex wrote the fix that I merged. We generally don't accept outside code contributions because it's way more efficient for Codex to write fixes than for us to review external changes. We do appreciate repro steps and root cause analyses though. Those are helpful!

TheJhyeFactor · 1 month ago

Thanks for the clarification, @etraut-openai .
I understand why the team prefers using Codex to implement fixes internally.

I’d still really like to become more involved with the project beyond occasionally reporting bugs. I’m happy to focus on the areas that are most useful to the team reliable reproductions, root-cause analysis, regression tests, validating fixes across platforms, or identifying gaps between related components.

I use Codex heavily and enjoy digging into these issues personally,
so I’d appreciate any guidance on the best way to contribute consistently and potentially work toward being invited to submit changes in the future.