›_ Codex Issues
Codex Issues / #29066

Windows CLI can hang after shell output with quotes/non-ASCII corrupts function_call_output JSONL

Open 💬 6 comments Opened Jun 19, 2026 by ooiuuii

Summary

On Windows, I hit a Codex CLI turn that never produced a final response even though the visible shell command (rg) had already completed successfully in ~0.4s.

The local session transcript shows that the function_call_output line for that shell command was written as malformed JSONL: the command output contained source lines with quotes and non-ASCII/mojibake text, and at least one quote inside payload.output was persisted unescaped. After that tool output, Codex started the next responses streaming request and then remained stuck with no final message and no task_complete event.

This looks like a runtime/session serialization + stream watchdog issue rather than the user command hanging.

Environment

  • OS: Windows build 22631, 64-bit
  • Observed session metadata: cli_version: 0.138.0-alpha.7
  • Locally installed package at time of report: @openai/codex 0.142.0-alpha.1 (not yet confirmed whether this newer version still reproduces)
  • Model in stuck turn: gpt-5.5, reasoning effort xhigh
  • Transport in logs: responses_http against chatgpt.com/responses

What happened

In one real session, the final turn had four parallel shell calls:

  1. Get-Content ...SKILL.md
  2. git status --short
  3. Get-Content package.json
  4. rg -n "type DemoViewId|demoTabs|ProofPackPanel|PaymentGateView|runReplay|openTab|setCaption\(page" src\App.tsx scripts\record-full-demo.mjs README.md

The first three calls returned normally. The rg call also returned normally:

Exit code: 0
Wall time: 0.4 seconds
Output:
scripts\record-full-demo.mjs:76:async function setCaption(page, text) {
scripts\record-full-demo.mjs:93:async function openTab(page, label) {
...
src\App.tsx:1081:              <button className="primary-action" onClick={runReplay} disabled={isRunning}>
src\App.tsx:1203:            <ProofPackPanel compact />

However, the persisted JSONL record for that function_call_output was invalid. A sanitized excerpt of the problematic shape:

{"timestamp":"...","type":"response_item","payload":{"type":"function_call_output","call_id":"...","output":"Exit code: 0\nWall time: 0.4 seconds\nOutput:\n... await setCaption(page, \"<non-ASCII text>");\n..."}}

The opening quote in the source line appears escaped as \", but the later source quote before ); is bare inside the JSON string, so parsing the JSONL line fails. PowerShell ConvertFrom-Json fails around that position with an invalid object error.

Immediately after that, the Codex logs show a new model streaming request:

run_sampling_request -> model_client.stream_responses_api
model=gpt-5.5 wire_api=responses transport="responses_http"
http.method="POST" api.path="responses"

The HTTP client connected successfully, but no later response_item/message, task_complete, or final output appeared. The thread state remained updated at the tool-output timestamp. The CLI process kept an established HTTPS connection to chatgpt.com:443 from the time of that request.

Expected behavior

  • Shell command output should always be serialized as valid JSONL, regardless of quotes, backslashes, CRLF, Unicode, mojibake, ANSI sequences, or replacement characters.
  • If a model stream stalls after tool output, Codex should eventually surface a clear transport/stream timeout or retryable error instead of leaving the CLI turn silently hanging.
  • A corrupt transcript line should not make the user-visible turn look like an indefinitely running command.

Actual behavior

  • The shell command finished quickly.
  • Its function_call_output was persisted as malformed JSONL.
  • Codex then opened the next responses stream and never emitted a final response or completion event.
  • From the user's perspective, the CLI appeared stuck on the previous rg command, even though rg.exe was no longer running.

Suggested fixes

  1. Make transcript/event persistence a strict round-trip invariant.
  • Add a regression test that writes and reads back function_call_output.output containing:
  • double quotes in source code ("...")
  • backslashes
  • CRLF and LF
  • non-ASCII text
  • mojibake / replacement characters
  • long multi-line rg output
  • Assert each JSONL line parses with a standard JSON parser after append.
  1. Avoid any manual JSON/string interpolation in event persistence.
  • Ensure function_call_output is serialized only through a real JSON serializer (for Rust, serde_json::to_writer / to_string, not format!).
  • If there is a UI/transcript truncation path, make sure truncation happens before serialization on the raw string, not by slicing already-serialized JSON.
  1. Harden Windows shell-output decoding.
  • Normalize subprocess stdout/stderr into valid UTF-8 before constructing the event.
  • If bytes cannot be decoded cleanly, use replacement decoding or a structured fallback, but still let JSON serialization own all escaping.
  1. Add a stream watchdog after tool outputs.
  • Once the follow-up responses request is sent, detect lack of first SSE event / no response progress after a timeout.
  • Emit a terminal event like turn_failed or stream_timeout so the CLI can show a final error instead of hanging forever.
  1. Make transcript replay tolerant of bad historical lines.
  • If a JSONL line is corrupt, record/report the corrupt offset and skip or quarantine the line, rather than leaving the turn state ambiguous.

Notes

The user command itself was not the problem. Process inspection showed no rg.exe remaining; only the Codex CLI process chain was still alive. This is why the issue is filed as a Codex CLI/runtime robustness bug rather than a project-level command hang.

View original on GitHub ↗

6 Comments

ooiuuii · 2 months ago

Additional data point from the same Windows CLI environment, which suggests the hang is not limited to malformed JSONL output.

Second occurrence

Same long-running CLI session/thread. The user saw the CLI stuck at:

Get-Content src\styles.css | Select-Object -Skip 1140 -First 110
Get-Content src\styles.css | Select-Object -Skip 2040 -First 75
Working (17m ...)

Local inspection showed both visible shell commands had already completed successfully:

2026-06-19T08:27:02.098Z function_call  call_oQSRxNVPUtvptjsaWSPJrEOj
  command: Get-Content src\styles.css | Select-Object -Skip 1140 -First 110

2026-06-19T08:27:02.643Z function_call_output call_oQSRxNVPUtvptjsaWSPJrEOj
  Exit code: 0
  Wall time: 0.5 seconds

2026-06-19T08:27:02.142Z function_call  call_JdhPx6f2jOVTjWaM2hHkTGve
  command: Get-Content src\styles.css | Select-Object -Skip 2040 -First 75

2026-06-19T08:27:02.694Z function_call_output call_JdhPx6f2jOVTjWaM2hHkTGve
  Exit code: 0
  Wall time: 0.5 seconds

There was no remaining Get-Content/PowerShell child command running for those reads. The thread state stopped at the tool-output timestamp:

updated_utc: 2026-06-19 08:27:02
tokens_used: 30276894
model: gpt-5.5
reasoning_effort: xhigh

Immediately after those tool outputs, Codex started the follow-up model stream:

2026-06-19 08:27:03 run_sampling_request
model_client.stream_responses_api
model=gpt-5.5 wire_api=responses transport="responses_http"
http.method="POST" api.path="responses"
connecting to 104.18.32.47:443
connected to 104.18.32.47:443
http1 handshake complete
connection is ready
checkout dropped for ("https", chatgpt.com)

After that, there were no further logs for the thread, no response_item/message, no task_complete, and no final output. The CLI kept showing Working.

Why this matters

The first occurrence had a malformed function_call_output JSONL line, so transcript serialization was a clear bug. This second occurrence looks slightly different: the final visible Get-Content tool outputs were present and parseable, but the post-tool responses_http request still became an unbounded wait.

That makes the stream watchdog / terminal failure event a separate necessary fix, not just a nice-to-have around the JSONL corruption case.

Refined fix recommendation

I would split the fix into two independent invariants:

  1. Transcript/event invariant
  • Every appended JSONL event must be generated by a real JSON serializer and immediately round-trip parseable in tests.
  • Add regression fixtures with Windows PowerShell output containing quoted CSS/TSX/JS strings, backslashes, CRLF, non-ASCII text, mojibake/replacement characters, and long multi-line rg/Get-Content output.
  • If transcript append fails or produces invalid JSON, emit a terminal turn error instead of continuing with ambiguous state.
  1. Post-tool stream progress invariant
  • After the client sends tool outputs to the Responses API, the CLI should require one of:
  • first response/SSE event within a bounded timeout,
  • normal stream completion,
  • explicit transport/model error,
  • or an emitted turn_failed/stream_timeout terminal event.
  • A connected HTTP socket with no model events should not keep the CLI in Working indefinitely.
  • The watchdog should report the phase, e.g. waiting_for_first_model_event_after_tool_outputs, plus model id, turn id, and elapsed time.

This would make the user-visible behavior much clearer: either the turn completes, or the CLI says the post-tool model stream stalled and exits the turn cleanly.

ooiuuii · 2 months ago

Additional observed pattern from the same reproduction: a bad/non-terminal turn leaves the session in a degraded state, and later turns in the same session repeatedly hang at the same post-tool continuation phase, even when the later tools themselves succeed.

Observed chained session degradation / replay robustness issue

This should not be treated as a single isolated command hang. In this Windows CLI session, the first observed stuck turn had a malformed function_call_output JSONL line. After that, the same long-running thread repeatedly got stuck at the same phase: after a successful tool output, while waiting for the follow-up Responses stream to produce the next model event/final answer.

The later hangs were not caused by the later commands still running:

  • Second occurrence: two Get-Content ... | Select-Object ... reads completed in ~0.5s, with parseable function_call_output records, but the follow-up responses_http stream hung indefinitely.
  • Third occurrence: a Playwright check using system Chrome completed successfully, closed the browser, and printed proof-lines=6:
2026-06-19T08:50:53.547Z function_call call_NReOVKAFcYZbndRFpomb3Rhl
  command: node script launching @playwright/test chromium with executablePath C:\Program Files\Google\Chrome\Application\chrome.exe

2026-06-19T08:50:58.129Z function_call_output call_NReOVKAFcYZbndRFpomb3Rhl
  Exit code: 0
  Wall time: 4.6 seconds
  Output: proof-lines=6

Immediately after that successful tool output, the session recorded that the model still needed a follow-up:

post sampling token usage ... model_needs_follow_up=true has_pending_input=false needs_follow_up=true
built_tools: node_repl tool_count=3, codex_apps tool_count=204
stream_request: model_client.stream_responses_api
transport="responses_http"
http.method="POST" api.path="responses"

Then no final message, task_complete, or terminal error appeared. The CLI stayed in Working until the user interrupted it several minutes later.

There is now another recurrence in the same environment/session family. The old large session transcript shows repeated JSONL corruption, not just the original rg output case:

session: rollout-2026-06-19T12-34-43-019ede28-e71a-74f1-a34f-a6bed66be08b.jsonl
last_write: 2026-06-19 17:01:29 +08:00
length: 72,587,926 bytes
tail 300 lines: 221 parseable JSONL events, 79 parse errors

The last valid events in that same old session again end immediately after tool output / token count, with no terminal turn event after it:

2026-06-19T09:01:29.084Z response_item function_call        call_EDgQgeCKWR7hX57ml67M1mOI view_image
2026-06-19T09:01:29.162Z response_item function_call_output call_EDgQgeCKWR7hX57ml67M1mOI
2026-06-19T09:01:29.166Z event_msg     token_count

At the same local time, the standalone CLI process still had an established HTTPS connection to the Responses endpoint host:

2026-06-19 17:01:29 +08:00
process: codex.exe PID 98348
remote: 104.18.32.47:443
state: Established

So the observed chain is:

  1. An earlier turn corrupts or fails to cleanly terminate session state, for example by appending malformed JSONL and/or not emitting a terminal turn event.
  2. Later turns in the same thread continue replaying/loading that large, partially bad historical state.
  3. Later shell/tool commands still complete successfully, but the CLI repeatedly hangs after their outputs while waiting for the post-tool model continuation.
  4. Starting a fresh short thread and importing only a clean handoff note avoids replaying the old huge/corrupt session transcript and is the practical workaround.

The exact internal causality still needs maintainer confirmation, but the repeated user-visible degradation is observed: the same thread keeps getting stuck after unrelated successful tool calls once the transcript has corrupt/non-terminal history.

Repair suggestions

In addition to the transcript JSONL round-trip invariant and post-tool stream watchdog already mentioned above, I would suggest adding explicit session/replay robustness checks:

  1. Treat corrupt historical transcript state as quarantinable, not silently reusable.
  • On session load/replay, validate every JSONL event before using it to construct the next model request.
  • If one line is corrupt, report the session path + byte/line offset and either quarantine that event or force a clean terminal error.
  • Do not let a malformed historical function_call_output, agent_message, or user_message produce an ambiguous live turn that looks like a running shell command.
  1. Add a per-turn terminal-state invariant.
  • Every turn should end in exactly one terminal state such as turn_completed, turn_failed, turn_aborted, or stream_timeout.
  • If a process is interrupted or the stream drops after tool output, persist that terminal state explicitly.
  • On the next turn, detect prior non-terminal turns and either repair them or warn the user that the thread is degraded.
  1. Add a degraded-session guardrail.
  • If a thread has a corrupt JSONL event, a previous non-terminal turn, or repeated post-tool stream timeouts, show a user-facing warning like: This session transcript is degraded; start a new thread or continue from a clean summary.
  • Optionally provide a built-in codex session export-summary / codex session repair path that writes a clean continuation summary without replaying corrupt tool-output payloads.
  1. Make the watchdog phase-aware.
  • The timeout must specifically cover waiting_for_first_model_event_after_tool_outputs, including after tool-list rebuild/plugin discovery and after non-shell tool outputs such as screenshots/images.
  • If this phase times out, the CLI should name the completed tool call id(s), elapsed time since tool output, model id, turn id, transcript validation status, and whether an older replay validation error exists.
  1. Regression test the chained failure mode.
  • Fixture A: append a malformed historical function_call_output, agent_message, or user_message line, then start a new turn in that same session and assert the CLI does not silently hang.
  • Fixture B: simulate a successful tool output followed by no first Responses/SSE event and assert a bounded stream_timeout terminal event.
  • Fixture C: combine A+B in one long session and assert the CLI surfaces a degraded-session or replay error instead of leaving the UI on Working.

This would turn the current failure mode from ?old thread keeps getting stuck after unrelated successful commands? into either a clean recovery path or a clear session-corruption diagnostic.

amit777 · 2 months ago

Adding a macOS reproduction of what looks like the same post-tool-result stream stall, from a different client surface (codex app-server, not the TUI). It corroborates the conclusion in the comments above that this isn't actually tied to the malformed-JSONL framing in the title.

Environment: codex-cli 0.142.3 in app-server mode (driven via an app-server broker over a unix socket), model gpt-5.5, model_reasoning_effort = xhigh, macOS arm64.

Same signature as the comments here:

  • The hang occurs immediately after a successful tool/function-call result is returned to the model, while waiting for the follow-up Responses stream.
  • The stream then produces nothing further — no message, no terminal event. In app-server terms, no turn/completed or turn/failed is ever emitted, so the driving client reports running forever.
  • A later turn/interrupt fails with thread not found — the thread is gone server-side but never emitted a terminal event.
  • The token_count heartbeat events that fired on every prior tool round go silent at the stall instant.

Two differences that strengthen the "not JSONL corruption" conclusion:

  1. The function_call_output JSONL here is well-formed (no bare quotes / no corruption) and it still hung. So malformed serialization looks like a separate, real bug — but not the cause of this stall.
  2. It's cross-platform (macOS, not Windows) and on a different client surface (app-server, not TUI), which points at the model/stream layer rather than anything OS- or TUI-specific.

Two reproductions in one session:

  • Thread 019f1192-99f2-77a1-b8ea-dd0b5a2d8ec1: stalled mid-session after a function_call_output; the rollout's last record is a lone reasoning item, then the rollout stops appending. ~443K input tokens on that call.
  • Thread 019f119f-f8ed-7dd3-af83-182a45ffc98c: stalled on the first turn, right after an MCP tool's initial_instructions returned — tiny context. This rules out large context as the trigger.

Same fix suggested as here: a stream-idle / turn-level watchdog that surfaces a terminal turn/failed (and/or retries) instead of hanging silently, plus guaranteeing a thread always emits a terminal event before it is reaped.

(I filed this separately as #30526 before finding this issue; closing that as a duplicate of this one.)

ooiuuii · 1 month ago

Additional data point from Windows codex-cli 0.142.4.

I hit the same user-visible failure mode again on 2026-07-01 in a normal Windows PowerShell session: a tool-heavy turn appeared to stall with no final assistant response after local shell work had completed or returned. The visible last operation was not a long-running user process; it was a broad inspection batch around an OpenClaw checkout, including Get-Content over several instruction/source files and an rg search whose output was large enough to be truncated by the client.

Observed shape:

  • OS/client: Windows PowerShell, codex-cli 0.142.4.
  • The turn used multiple shell reads/searches, including broad Get-Content and rg output.
  • The local commands themselves returned or were interrupted by output-size handling rather than continuing as live child processes.
  • The Codex turn did not produce a terminal user-visible answer, so from the user's perspective the agent was stuck and had to be resumed/restarted.

This looks consistent with the issue already discussed here: after successful or oversized tool output, the follow-up model/Responses phase can fail to produce a terminal event. It also reinforces that the workaround is to keep shell reads narrowly bounded, but the CLI/runtime should still surface a bounded turn_failed/stream_timeout or output-size diagnostic instead of leaving the session ambiguous.

Expected invariant remains:

  • oversized/truncated shell output should end in a clear tool result or tool error;
  • the follow-up model stream should either emit a message/completion or a terminal failure event;
  • the UI should not leave the user thinking the shell command is still running after the command has completed or been client-truncated.
niuzhirui · 1 month ago

I reproduced the same post-tool stall on a newer Windows Codex Desktop build, and the evidence points specifically to an unbounded HTTP response-header wait before the SSE stream exists.

Environment

  • Codex Desktop: 26.707.9981.0
  • Bundled codex-cli 0.144.2
  • Windows 11 x64
  • Model: gpt-5.6-sol, reasoning effort high
  • Transport on the affected attempts: responses_http
  • ChatGPT authentication

Four stalls observed on 2026-07-14

All times below are Asia/Shanghai. In every case, the tool result had already been persisted successfully, then the follow-up model request produced no model event until the user interrupted the turn.

| Last tool result | Turn interrupted | Silent interval |
|---|---:|---:|
| 09:57:43 | 10:23:58 | 26m 15s |
| 10:33:16 | 10:42:55 | 9m 39s |
| 10:51:19 | 10:57:53 | 6m 35s |
| 11:01:56 | 11:19:17 | 17m 21s |

A fifth follow-up request was silent for about 215 seconds and then recovered by itself.

The rollout JSONL files were parsed line-by-line with an explicit UTF-8 JSON parser and had zero parse errors, so these occurrences are not caused by malformed transcript serialization.

For each stalled round, the logs show:

  1. a completed function_call_output / custom_tool_call_output;
  2. the next sampling round starting;
  3. HTTP client setup for the follow-up request;
  4. no established stream/response event and no terminal turn event before interruption.

Source boundary

In rust-v0.144.2:

  • Provider::build_request() initializes Request.timeout to None:

https://github.com/openai/codex/blob/rust-v0.144.2/codex-rs/codex-api/src/provider.rs#L77-L85

  • EndpointSession::stream_encoded_json_with() waits on transport.stream(req).await inside the existing request retry wrapper:

https://github.com/openai/codex/blob/rust-v0.144.2/codex-rs/codex-api/src/endpoint/session.rs#L122-L152

  • ReqwestTransport::stream() does not return until builder.send().await has produced the HTTP response/status/headers:

https://github.com/openai/codex/blob/rust-v0.144.2/codex-rs/http-client/src/transport.rs#L129-L160

  • stream_idle_timeout is only applied after that future returns and spawn_response_stream() is created:

https://github.com/openai/codex/blob/rust-v0.144.2/codex-rs/codex-api/src/endpoint/responses.rs#L140-L162

This explains why stream_idle_timeout_ms never fires in these cases: the SSE byte stream has not been created yet. It also explains why the existing request retry policy does not help: no TransportError::Timeout is produced.

Suggested narrow fix

Add a dedicated response-header/stream-establishment timeout, for example:

response_header_timeout_ms = 60000

The timeout should wrap only the establishment future (transport.stream(req)) and map elapsed time to TransportError::Timeout, so the existing request retry policy can retry it.

It should not use reqwest's whole-request timeout for the streaming request, because that would also terminate a healthy long-lived SSE body after 60 seconds.

Suggested regression tests:

  1. A test server accepts the POST but never sends response headers; assert timeout, retry, and a bounded terminal failure.
  2. A test server sends headers immediately and then streams for longer than the header timeout; assert the healthy SSE stream is not cancelled.

This appears to be the same user-visible failure described here, but the phase-specific evidence suggests a response-header/stream-establishment watchdog is needed in addition to any post-header stream-idle watchdog.

shleder · 11 days ago

This is a strong codex-rescue field case because you have both malformed persisted function_call_output JSONL and later turns that stop after successful tool outputs without a terminal event. I’m testing exactly this distinction: corrupted record vs. interrupted-but-parseable turn.

If you still have one affected local session, could you run the read-only path first?

pipx install codex-rescue
codex-rescue sessions
codex-rescue doctor --latest

If doctor identifies an unfinished state and you want to test recovery:

codex-rescue salvage --latest --fork
codex-rescue verify <rescue-id>

Rescue should not replay unknown tool side effects or rewrite the source rollout. Sanitized diagnostics, versions and exit codes are enough; please don’t post raw JSONL/SQLite, tool output containing repo data, prompts, credentials, or private paths. Repo: https://github.com/shleder/codex-rescue