Auto-review can deadlock behind unbounded post-TurnComplete rollout flush on Windows

Open 💬 7 comments Opened Jul 12, 2026 by svincent-nv

Summary

On Codex Desktop for Windows, the automatic approval reviewer can become blocked behind an unbounded rollout-file flush. The affected tool operation never starts, and the approval eventually reports:

The automatic permission approval review did not finish before its deadline.

This is not simply a slow model response or a full submission/event channel. Local logs and rollout files from thirty-two incidents causally localize the stall to the serialized rollout writer.

Related symptom report: #24204. This report adds a source-level causal chain and Windows incident evidence.

Environment

Codex Desktop packages observed: 26.707.3351.0, 26.707.9981.0, 26.715.4045.0, 26.715.9868.0, 26.715.10079.0, 26.721.4979.0
Embedded core versions observed: 0.144.0-alpha.4, 0.144.2, 0.145.0-alpha.18, 0.145.0-alpha.30
Windows desktop host
approvals_reviewer = auto_review
Workspace-write sandbox
Source comparison: rust-v0.142.5, rust-v0.144.0-alpha.4, latest stable rust-v0.144.1, and current main

No reliable intentional reproduction is available yet; the failure is intermittent after repeated automatic reviews.

User impact

Unrelated tool calls can appear stuck for roughly two minutes or several hours.
The underlying tool is not launched.
Interrupt/review-deadline cleanup may itself fail to complete.
Restarting Codex is the dependable recovery observed so far.

Confirmed causal chain

A guardian review turn completes.
Session::on_task_finished persists and delivers TurnComplete.
After delivery, current code performs another unconditional flush_rollout().await in codex-rs/core/src/tasks/mod.rs.
flush_rollout ultimately calls RolloutRecorder::flush in codex-rs/rollout/src/recorder.rs.
That method queues RolloutCmd::Flush to the thread's single writer and awaits a oneshot acknowledgement without a timeout.
The writer serially awaits its file flush. If that flush remains pending, later writer commands queue behind it.
The guardian caller has already received TurnComplete, so the hidden review session is reused.
The next UserInput is received and enters submission dispatch. Before starting its RegularTask, it emits the mandatory thread_settings_applied event through send_event_raw_without_materializing_rollout.
That event must be persisted before delivery. Its append/flush queues behind the stuck terminal flush, so the review task never starts.

Relevant paths:

codex-rs/core/src/tasks/mod.rs
codex-rs/core/src/session/handlers.rs
codex-rs/core/src/session/mod.rs
codex-rs/thread-store/src/local/live_writer.rs
codex-rs/rollout/src/recorder.rs

Incident evidence

The table is maintained one row per confirmed stall so recurrence frequency, affected versions, and lifecycle changes remain easy to compare.

| # | Start (UTC / PDT) | Visible stall duration | Request type | Lifecycle evidence | Requested action | Recovery / retry | Codex package / core |
|---:|---|---:|---|---|---|---|---|
| 1 | 2026-07-11 11:29:05.119 UTC / 04:29:05.119 PDT | 6h 28m 46.5s (23,326.5s) | shell_command | Valid model allow payload arrived; terminal review lifecycle then remained blocked until writer release | Did not run; no side effects observed | Manual termination surfaced the reviewer-deadline error; no orphan process | 26.707.3351.0 / 0.144.0-alpha.4 |
| 2 | 2026-07-11 21:10:17.868 UTC / 14:10:17.868 PDT | 6h 29m 32.5s (23,372.5s) | apply_patch | Submission was received; thread_settings_applied was not physically written/delivered until teardown released the writer | Did not run; no side effects observed | Manual termination; same Codex app process survived; no related process leak | 26.707.3351.0 / 0.144.0-alpha.4 |
| 3 | 2026-07-12 06:39:21.306 UTC / 2026-07-11 23:39:21.306 PDT | 1m 47.1s (107.1s) | apply_patch | Submission was received; delayed thread_settings_applied and task start appeared only after writer release | Did not run; no side effects observed | Reviewer-deadline error returned; subsequent work continued without side effects | 26.707.3351.0 / 0.144.0-alpha.4 |
| 4 | 2026-07-12 18:11:42.544 UTC / 11:11:42.544 PDT | 3h 21m 59.3s (12,119.3s) | apply_patch | Reviewer deadline was recorded internally after about 95.5s, but the parent tool result was not delivered for another ~3h20m | Did not run; no side effects observed | Manual termination surfaced the deadline error; retry completed in ~13.5s | 26.707.3351.0 / 0.144.0-alpha.4 |
| 5 | 2026-07-13 00:57:27.145 UTC / 2026-07-12 17:57:27.145 PDT | 41m 6.8s (2,466.8s) | apply_patch | Reviewer decline was recorded internally after 95.56s, but the parent tool result was not delivered for another 39m31.3s | Did not run; no side effects observed | Delayed result surfaced; stale cell was already gone; retry completed in ~15s | 26.707.3351.0 / 0.144.0-alpha.4 |
| 6 | 2026-07-13 12:15:24.796 UTC / 05:15:24.796 PDT | 2h 30m 3.2s (9,003.2s) | apply_patch | Automatic review did not finish before its deadline; the parent tool call remained pending for roughly 2.5 hours before the timeout surfaced | Did not run; fresh inspection found no side effects | Delayed timeout returned; retry after one minute completed in 3.8s | 26.707.3351.0 / 0.144.0-alpha.4 |
| 7 | 2026-07-13 18:24:40 UTC / 11:24:40 PDT (derived) | 1h 30m 24.2s (5,424.2s) | apply_patch | Automatic review did not finish before its deadline; the parent tool call remained pending for about 90 minutes before cancellation surfaced the timeout | Did not run; target remained unchanged | Stale request cancelled; one retry completed in about 12.2s | 26.707.3351.0 / 0.144.0-alpha.4 |
| 8 | 2026-07-14 05:03:27.944 UTC / 2026-07-13 22:03:27.944 PDT | 4m 22.1s (262.1s) | apply_patch | thread_settings_applied was recorded, but task_started was delayed 261.952s until cancellation; automatic review then surfaced its deadline timeout | Did not run; target remained unchanged | Stalled call cancelled; timeout surfaced; first retry completed successfully | 26.707.3351.0 / 0.144.0-alpha.4 |
| 9 | 2026-07-14 15:18:56.585 UTC / 08:18:56.585 PDT | 23m 20.0s (1,400.0s) | apply_patch | Reviewer decline was recorded internally after 95.601s, but the parent tool result was not delivered for another 21m 44.3s | Did not run; no side effects observed | Delayed timeout surfaced; first retry completed in 14.644s | 26.707.3351.0 / 0.144.0-alpha.4 |
| 10 | 2026-07-15 04:55:17.085 UTC / 2026-07-14 21:55:17.085 PDT | 1h 18m 3.7s (4,683.7s) | apply_patch | Reviewer decline was recorded internally after 95.407s, but the parent tool result remained pending for another ~1h16m28s until cancellation | Did not run; no side effects observed | Stalled call cancelled; timeout surfaced; first retry completed successfully in 14.139s | 26.707.9981.0 / 0.144.2 |
| 11 | 2026-07-15 07:21:39.349 UTC / 00:21:39.349 PDT | 1h 22m 18.7s (4,938.7s) | apply_patch | Reviewer decline was recorded internally after 95.390s, but the completed parent tool result was not delivered for another 1h 20m 43.3s | Did not run; target remained unchanged | Delayed timeout surfaced after new user activity; first retry completed successfully in 15.249s | 26.707.9981.0 / 0.144.2 |
| 12 | 2026-07-15 18:47:50.077 UTC / 11:47:50.077 PDT | 1h 9m 35.2s (4,175.2s) | apply_patch | Reviewer decline was recorded internally after 95.426s, but the parent tool result was not delivered for another 1h 7m 59.7s | Did not run; target remained unchanged | New user activity surfaced the pending cell; manual termination returned the timeout; first retry completed in 4.1s | 26.707.9981.0 / 0.144.2 |
| 13 | 2026-07-16 01:53:39.695 UTC / 2026-07-15 18:53:39.695 PDT | 1h 35m 49.4s (5,749.4s) | apply_patch | Reviewer decline was recorded internally after 95.394s, but the parent tool result was not delivered for another 1h 34m 14.0s | Did not run; target remained unchanged | Manual termination returned the timeout; first retry completed successfully in 2.740s | 26.707.9981.0 / 0.144.2 |
| 14 | 2026-07-16 07:00:23.191 UTC / 00:00:23.191 PDT | 4m 20.6s (260.587s) | apply_patch | Automatic review did not finish before its deadline; the parent tool call remained pending for 260.587s before the timeout surfaced | Did not run; target remained unchanged | Timeout surfaced; after a one-minute delay, the first retry completed in 10.063s | 26.707.9981.0 / 0.144.2 |
| 15 | 2026-07-16 18:37:57.745 UTC / 11:37:57.745 PDT | 54m 6.3s (3,246.313s) | apply_patch | thread_settings_applied was recorded immediately, but task_started was delayed 3,246.186s until cancellation; the turn aborted 0.127s later | Did not run; target remained unchanged | Manual termination after new user activity returned the reviewer-deadline timeout; first retry completed successfully | 26.707.9981.0 / 0.144.2 |
| 16 | 2026-07-17 03:44:32.000 UTC / 2026-07-16 20:44:32.000 PDT | 24m 39.0s (1,479.034s) | apply_patch | Review task started and produced reasoning, but no terminal allow/decline or task_complete; cancellation produced turn_aborted 91ms later | Did not run; target remained unchanged | Manual termination surfaced the reviewer-deadline timeout; first retry completed successfully | 26.707.9981.0 / 0.144.2 |
| 17 | 2026-07-17 04:40:33.434 UTC / 2026-07-16 21:40:33.434 PDT | 1h 5m 1.5s (3,901.535s) | apply_patch | Review task started and produced reasoning, but no terminal allow/decline or task_complete; cancellation produced turn_aborted 115ms later | Did not run; target remained unchanged | Reviewer-deadline timeout surfaced after interruption; first retry completed successfully on a fresh reviewer session | 26.707.9981.0 / 0.144.2 |
| 18 | 2026-07-17 10:26:54.517 UTC / 03:26:54.517 PDT | 4h 33m 55.8s (16,435.8s) | apply_patch | Reviewer decline was recorded internally after 95.430s, but the parent tool result was not delivered for another 4h 32m 20.4s | Did not run; no side effects observed | Delayed timeout surfaced; first retry was terminated during a user pause before completion | 26.707.9981.0 / 0.144.2 |
| 19 | 2026-07-18 01:44:23.766 UTC / 2026-07-17 18:44:23.766 PDT | 3m 6.4s (186.376s) | apply_patch | Automatic review did not finish before its deadline; the parent tool call remained pending until a waiter surfaced the timeout | Did not run; target remained unchanged | Timeout surfaced; first identical retry completed successfully in 10.046s | 26.707.9981.0 / 0.144.2 |
| 20 | 2026-07-18 23:06:58.952 UTC / 16:06:58.952 PDT | 2h 29m 10.7s (8,950.7s) | apply_patch | Reviewer decline was recorded internally after 95.432s, but the parent tool result was not delivered for another 2h 27m 35.3s | Did not run; target remained unchanged | Delayed timeout surfaced after cancellation; first retry completed successfully in 13.945s | 26.707.9981.0 / 0.144.2 |
| 21 | 2026-07-19 01:49:59.780 UTC / 2026-07-18 18:49:59.780 PDT | 2h 43m 31.0s (9,811.0s) | apply_patch | Reviewer decline was recorded internally after 95.490s, but the parent tool result was not delivered for another 2h 41m 55.5s | Did not run; target remained unchanged | Delayed timeout surfaced after cancellation; first retry completed successfully in 15.042s | 26.707.9981.0 / 0.144.2 |
| 22 | 2026-07-19 07:33:05.531 UTC / 00:33:05.531 PDT | 1h 56m 49.7s (7,009.7s) | apply_patch | Reviewer decline was recorded internally after 95.438s, but the parent tool result was not delivered for another 1h 55m 14.3s | Did not run; target remained unchanged | Delayed timeout surfaced after cancellation; first retry completed successfully in 14.905s | 26.707.9981.0 / 0.144.2 |
| 23 | 2026-07-19 11:17:36.068 UTC / 04:17:36.068 PDT | 6h 8m 56.6s (22,136.6s) | shell_command | Automatic review did not finish before its deadline; the parent tool call remained pending for 22,136.6s before manual cancellation surfaced the timeout | Did not run; no side effects observed | Manual termination surfaced the timeout; corrected direct read-only retry completed in 1.1s | 26.707.9981.0 / 0.144.2 |
| 24 | 2026-07-19 19:37:16.897 UTC / 12:37:16.897 PDT | 40m 6.6s (2,406.6s) | shell_command | Automatic review did not finish before its deadline; the parent tool call remained pending for 2,406.6s before manual termination surfaced the timeout | Did not run; no side effects observed | Manual termination surfaced the timeout; corrected bounded read-only retry completed in 0.8s | 26.715.4045.0 / 0.145.0-alpha.18 |
| 25 | 2026-07-20 00:44:26.141 UTC / 2026-07-19 17:44:26.141 PDT | 2h 9m 42.9s (7,782.884s) | apply_patch | Reviewer decline was recorded internally after 95.445s, but the parent tool result was not delivered for another 2h 8m 7.4s | Did not run; target remained unchanged | Manual termination surfaced the timeout; first identical retry completed successfully within one minute | 26.715.4045.0 / 0.145.0-alpha.18 |
| 26 | 2026-07-20 05:31:33.494 UTC / 2026-07-19 22:31:33.494 PDT | 3h 27m 37.4s (12,457.4s) | apply_patch | Reviewer decline was recorded internally after 95.435s, but the parent tool result was not delivered for another 3h 26m 2.0s | Did not run; target remained unchanged | Delayed timeout surfaced; first identical retry completed successfully in 15.648s | 26.715.4045.0 / 0.145.0-alpha.18 |
| 27 | 2026-07-20 18:26:58.360 UTC / 11:26:58.360 PDT | 10m 30.4s (630.384s) | apply_patch | Guardian returned allow after 6.230s, but the approval lifecycle still recorded decline after 95.428s; the parent result was delayed for 623.368s | Did not run; target remained unchanged | Deadline timeout surfaced; first identical retry completed successfully in 13.546s | 26.715.4045.0 / 0.145.0-alpha.18 |
| 28 | 2026-07-21 01:38:13.266 UTC / 2026-07-20 18:38:13.266 PDT | 2h 39m 29.5s (9,569.5s) | apply_patch | Reviewer decline was recorded internally after 95.476s, but the parent tool call did not yield for another 9,474.066s | Did not run; target remained unchanged | Waiter surfaced the deadline timeout; first retry completed successfully in 14.139s | 26.715.4045.0 / 0.145.0-alpha.18 |
| 29 | 2026-07-21 07:28:15.714 UTC / 2026-07-21 00:28:15.714 PDT | 24m 7.7s (1,447.725s) | apply_patch | Reviewer decline was recorded internally after 95.486s, but the parent tool call did not yield for another 1,352.239s | Did not run; target remained unchanged | Cancellation surfaced the deadline timeout; first retry completed successfully in 15.184s | 26.715.4045.0 / 0.145.0-alpha.18 |
| 30 | 2026-07-23 03:43:14.936 UTC / 2026-07-22 20:43:14.936 PDT | 1h 34m 15.1s (5,655.1s) | apply_patch | Reviewer decline was recorded internally after 95.638s, but the parent tool call did not yield for another 5,559.499s | Did not run; target remained unchanged | Waiter surfaced the deadline timeout; first retry completed successfully in 16.201s | 26.715.9868.0 / 0.145.0-alpha.30 |
| 31 | 2026-07-24 15:57:30.042 UTC / 08:57:30.042 PDT | 1h 43m 55.0s (6,235.0s) | shell_command | Review submission was accepted, but no reviewer decision arrived; hidden-session closed-channel teardown surfaced only after 6,235.0s | Did not run; no side effects observed | Deadline timeout surfaced; first identical retry completed successfully in 247.971s | 26.715.10079.0 / 0.145.0-alpha.30 |
| 32 | 2026-08-03 ~13:25:04 UTC / ~06:25:04 PDT (derived) | ~1m (approx.; terminal at 13:26:04.192 UTC) | apply_patch | Automatic review did not finish before its deadline; the parent result surfaced near the review bound | Did not run; target remained unchanged | First identical retry completed successfully; terminal recorded at 13:27:13.762 UTC | 26.721.4979.0 / core not exposed |

Incidents 2 and 3 stopped after the Submission debug row but before the review task's turn span. In both:

The thread_settings_applied JSONL record retained the original submission timestamp.
The record was not physically written/delivered until teardown released the writer.
task_started followed immediately after writer release.
The task was then interrupted/aborted because the approval deadline had already expired.

Incident 1 reached a valid model allow payload but did not complete its terminal lifecycle until teardown. Incidents 4 and 5 add a distinct result-delivery symptom: the reviewer result was already recorded internally, yet the parent tool call remained pending for hours or tens of minutes. Incident 6 independently adds another roughly 2.5-hour parent-call delay before the automatic-review timeout surfaced; its table row intentionally makes no stronger internal log-timing claim. Incident 7 adds a 90.4-minute parent-call delay with the same automatic-review deadline error; its start time is derived from the observed end time and tool-call duration, and its row likewise makes no stronger internal log-timing claim. Incidents 9 through 32 extend recurrence across six observed builds: incident 9 on package 26.707.3351.0 / core 0.144.0-alpha.4, incidents 10 through 23 fourteen times on package 26.707.9981.0 / core 0.144.2, and incidents 24 through 29 six times on package 26.715.4045.0 / core 0.145.0-alpha.18, incident 30 once on package 26.715.9868.0 / core 0.145.0-alpha.30, and incident 31 once on package 26.715.10079.0 / core 0.145.0-alpha.30; incident 12 recorded the reviewer decline after 95.426 seconds and then withheld the completed parent result for another 4,079.726 seconds, incident 13 recorded the decline after 95.394 seconds and withheld the parent result for another 5,653.970 seconds, incident 14 adds a shorter 260.587-second current-version recurrence where the timeout surfaced directly, and incident 15 again shows thread_settings_applied followed by a 3,246.186-second delay before task_started appeared only when cancellation released the blocked lifecycle. Incident 16 adds a 1,479.034-second reviewer turn that started and produced reasoning but never emitted a terminal decision or task_complete; cancellation yielded turn_aborted 91 milliseconds later, and the underlying apply_patch remained unapplied. Incident 17 adds a 3,901.535-second reviewer turn with the same started-and-reasoning-only lifecycle; cancellation yielded turn_aborted 115 milliseconds later, the underlying apply_patch remained unapplied, and the first retry completed on a fresh reviewer session. Incident 18 adds a 16,435.8-second parent-call delay after the reviewer had already recorded its decline at 95.430 seconds; the underlying apply_patch did not run, and the first retry was terminated during a user pause before completion. Incident 19 adds a 186.376-second current-version recurrence where the automatic-review deadline surfaced through a waiter; the underlying apply_patch did not run, and the first identical retry completed in 10.046 seconds. Incident 20 adds an 8,950.7-second parent-call delay after the reviewer recorded its decline at 95.432 seconds; the underlying apply_patch did not run, and the first retry completed successfully in 13.945 seconds. Incident 21 adds a 9,811.0-second parent-call delay after the reviewer recorded its decline at 95.490 seconds; the underlying apply_patch did not run, and the first retry completed successfully in 15.042 seconds. Incident 22 adds a 7,009.7-second parent-call delay after the reviewer recorded its decline at 95.438 seconds; the underlying apply_patch did not run, and the first retry completed successfully in 14.905 seconds. Incident 23 adds a 22,136.6-second parent-call delay before manual cancellation surfaced the automatic-review timeout; the underlying shell_command did not run, and a corrected direct read-only retry completed in 1.1 seconds. Incident 24 adds a 2,406.6-second parent-call delay on package 26.715.4045.0 / core 0.145.0-alpha.18 before manual termination surfaced the automatic-review timeout; the underlying shell_command did not run, and a corrected bounded read-only retry completed in 0.8 seconds. Incident 25 adds a 7,782.884-second parent-call delay after the reviewer recorded its decline at 95.445 seconds; the underlying apply_patch did not run, and the first identical retry completed successfully within one minute. Incident 26 adds a 12,457.4-second parent-call delay after the reviewer recorded its decline at 95.435 seconds; the underlying apply_patch did not run, and the first identical retry completed successfully in 15.648 seconds. Incident 27 adds a 630.384-second approval-lifecycle delay even though the guardian returned allow after 6.230 seconds; the lifecycle recorded decline after 95.428 seconds, the underlying apply_patch did not run, and the first identical retry completed successfully in 13.546 seconds. Incident 28 adds a 9,569.5-second parent-call delay after the reviewer recorded its decline at 95.476 seconds; a waiter surfaced the automatic-review timeout, the underlying apply_patch did not run, and the first retry completed successfully in 14.139 seconds. Incident 29 adds a 1,447.725-second recurrence after the reviewer recorded its decline at 95.486 seconds; cancellation surfaced the automatic-review timeout, the underlying apply_patch did not run, and the first retry completed successfully in 15.184 seconds. Incident 30 adds a 5,655.1-second recurrence on package 26.715.9868.0 / core 0.145.0-alpha.30 after the reviewer recorded its decline at 95.638 seconds; a waiter surfaced the automatic-review timeout, the underlying apply_patch did not run, and the first retry completed successfully in 16.201 seconds. Incident 31 adds a 6,235.0-second recurrence on package 26.715.10079.0 / core 0.145.0-alpha.30: the review submission was accepted, no reviewer decision arrived, hidden-session closed-channel teardown surfaced only at the end of the stall, the underlying shell_command did not run, and the first identical retry completed successfully in 247.971 seconds. Incident 32 adds an approximately one-minute recurrence on package 26.721.4979.0 with the bundled core version not exposed by the inspected local surface: the automatic-review deadline surfaced near its bound, the underlying apply_patch did not run, and the first identical retry completed successfully.

This rules out, for these incidents:

submission-channel capacity
event-channel capacity (the event channel is unbounded)
model inference delay for the pre-task incidents
an unprocessed interrupt submission
command length, filename length, or path length
a coded 47th/48th-review retirement limit

Regression signal

rust-v0.142.5 did not perform the second unconditional flush after terminal event delivery. rust-v0.144.0-alpha.4 added it. Current main and stable rust-v0.144.1 still retain the path.

The terminal event's normal persistence already flushes through the local thread store before delivery, making the additional post-delivery unbounded flush both redundant and uniquely dangerous: the caller can reuse the session while it is pending.

Expected behavior

Rollout persistence must not block approval processing indefinitely.
A reviewer deadline must complete near its configured bound.
A persistence failure/timeout must retire the affected guardian session.
A later approval must start on a healthy session.
Interrupt cleanup must not depend on the same stalled session-loop/persistence path.

Suggested fixes

Any complete fix should address the failure class, not only this call site:

Remove the redundant post-terminal flush, or bound it with a short timeout.
Mark a guardian session unusable if its persistence barrier times out.
Ensure pre-task events cannot wait indefinitely on rollout persistence.
Make deadline cancellation independent of the blocked submission/persistence path.
Preserve an explicit diagnostic identifying the blocked persistence phase.

Suggested regression tests

Use an injected writer that deliberately withholds a flush acknowledgement:

Complete review N and deliver TurnComplete.
Block only its post-terminal flush.
Submit review N+1.
Assert N+1 either proceeds on a fresh session or fails within a strict bounded timeout.
Assert the next review after timeout succeeds.
Assert no hidden reviewer task/session leaks.
Run on Windows in addition to unit-level injected-writer coverage.

Privacy

This report intentionally excludes transcripts, company/workspace names, local paths, credentials, tokens, proprietary source, and unrelated application evidence.

View original on GitHub ↗

7 Comments

WG-Mojo · 1 month ago

Cross-project evidence from OpenClaw's Codex app-server integration appears to match this failure family.

Environment: native Windows, OpenClaw 2026.7.1, whose matched @openclaw/codex plugin pins @openai/codex 0.144.3; Guardian auto-review was active.
The affected child review recorded task_started at 2026-07-16T21:33:11.020Z and emitted a schema-valid final allow decision at 21:33:15.400Z.
Exactly 90 seconds after task start, at 21:34:41.019Z, the structured Codex trace recorded the timeout's Op::Interrupt submission, interrupt received, and aborting running task. These map to session/handlers.rs:722-727, session/mod.rs:3960-3963, and tasks/mod.rs:829-850 in the exact 0.144.3 source.
At 21:34:41.127Z, the trace warned that the task had not completed gracefully after the configured 100 ms interruption grace period (constant, wait/abort path).
No assistant response_item, task_complete, child TurnComplete/TurnAborted, or parent item/autoApprovalReview/completed followed before OpenClaw's outer watchdog intervened. Only then, at 21:39:30.542Z, did the child rollout record TurnAborted with duration_ms: 379521.

This narrows the OpenClaw observation: the Guardian deadline did fire on schedule, but the child interrupt/abort-drain/terminal propagation path remained blocked. It is a strong symptom-level match for this issue—especially the incident where a valid allow arrived—but does not yet prove that the same rollout-writer flush is the underlying I/O block.

Full redacted event sequence, exact dependency contract, and OpenClaw-side behavior: https://github.com/openclaw/openclaw/issues/87744#issuecomment-4997128953

_Edited with Codex assistance after direct source and local trace verification._

hassan-k101 · 1 month ago

Corroborating Windows Desktop incident — 2026-07-22/23

I encountered this same failure class during a long-running Codex Desktop repository-coordination task on Windows.

Environment visible to the task:

Codex Desktop on Windows
sandbox_mode = workspace-write
approvals_reviewer = auto_review
Local Git repository with several Codex worktrees
App/core version was not exposed to the agent runtime

Observed behavior:

After the user explicitly authorized cleanup, Codex submitted narrowly scoped escalated git worktree remove commands for already-merged, clean Codex-owned worktrees.
The underlying commands did not start. Parent tool calls remained pending for minutes; reported wall times included approximately 261.9 seconds and 1704.9 seconds.
Cancelling/waiting surfaced the exact error:

``text The automatic permission approval review did not finish before its deadline. Do not assume the action is unsafe based on the timeout alone. You may retry once, or ask the user for guidance or explicit approval.``

Retrying after explicit user authorization hit the same approval deadline. Inspection showed no deletion or branch mutation from the rejected calls.
Separate read-only filesystem/Git calls (Get-Content, rg, git status, git merge-base) also intermittently yielded with no output for tens of seconds or longer; some completed only when rerun outside the restricted sandbox.
During the same incident, three read-only subagents stopped/disappeared from the collaboration status after their shell calls stalled. Replacement agents instructed to use escalated bounded reads completed successfully. This may be a downstream turn/session lifecycle symptom rather than a separate root cause.
Some yielded execution cells became unavailable after steering/cancellation (exec cell ... not found), making recovery and observability harder.

User impact:

Explicitly authorized, narrowly scoped cleanup could not be completed.
The user repeatedly saw all background agents stop and reasonably concluded that Codex had died.
The coordinator could not reliably distinguish a running command, a blocked approval, and a dead subagent without repeated status probes.
No source work was lost, but the experience was not reliable enough for serious multi-agent repository work.

This report is intentionally sanitized: repository names, user names, local workspace paths, proprietary source, credentials, and transcript content are omitted. The originating user explicitly requested that this reliability failure be reported to the Codex team.

rmorse · 28 days ago

Additional independent app-server incident — Windows, Codex CLI 0.144.6

I observed another strong symptom-level match from an independent Codex app-server client on Windows.

Environment:

codex-cli 0.144.6
Native Windows
workspace-write
approvals_reviewer = auto_review
Persistent app-server JSON-RPC session

Sanitized event sequence on 2026-07-30 UTC:

13:11:40.125: app-server emitted item/started for a commandExecution wrapping an apply_patch operation.
13:11:40.160: item/autoApprovalReview/started arrived for that item.
13:13:15.157: stderr emitted the exact automatic-review deadline error, approximately 95 seconds after review start.
No matching item/autoApprovalReview/completed, item/completed, or turn/completed followed for another 80 minutes. The review remained pending for a total of approximately 4,895.8 seconds.
14:33:15.678: the client explicitly sent turn/interrupt.
14:33:15.954: item/autoApprovalReview/completed and the parent item/completed arrived, 276 ms after the interrupt.
14:33:16.022: turn/completed(status=interrupted) arrived, 344 ms after the interrupt.
14:33:16.136: the same app-server process emitted the next turn/started; subsequent tools and automatic reviews completed normally. No process restart was required.

This independently confirms that the Guardian deadline itself fires on schedule, while completion of the review, parent item, and turn can remain blocked indefinitely. An explicit turn interruption immediately released the terminal lifecycle and left the process reusable.

Because this client only observes the external app-server protocol, this does not independently prove that the rollout-writer flush is the blocked internal operation. It is, however, a close behavioral match for the incidents documented here and provides an exact externally observable regression shape.

One possibly related stderr signal also repeated during the stalled window:

failed to refresh available models: timeout waiting for child process to exit

It first appeared about 61 seconds after review start and then recurred roughly every three minutes. I cannot establish whether that signal shares the same cause, so I’m including it only as a secondary observation.

Local paths, repository details, tool payloads, IDs, and transcript content have been omitted.

schickling-assistant · 17 days ago

Linux CLI evidence: ordinary sessions can wedge behind the same rollout writer

I have a sanitized Linux CLI incident that appears to confirm the same underlying failure class while ruling out several Windows/Guardian-specific explanations.

Environment and impact

codex-cli 0.145.0, interactive TUI, Linux x86_64
full-access permissions; no automatic approval reviewer involved
session rollouts on a ZFS dataset; global history/state on a separate local filesystem
affected rollout was only about 1.6 MB

The previous turn durably ended in task_complete. Two later prompts were accepted by the TUI and appended to global message history, and local TUI commands remained responsive. However, the session rollout never advanced: there was no later user_message, task_started, or model request.

Kernel-level evidence

During the stall, one Codex Tokio worker was repeatedly in Linux D state. Its live syscall was:

write(fd, ..., 1442)

The file descriptor resolved to the affected session's rollout JSONL. The kernel stack was:

vfs_write
  -> zpl_iter_write
  -> zfs_write
  -> dmu_tx_assign
  -> dmu_tx_wait

The storage pool was healthy but severely saturated: I/O PSI was approximately some=96%, full=78-79%. In one fleet sample, four other Codex processes had six blocked I/O threads. Several unrelated processes were simultaneously blocked in 4 KiB SQLite reads against a separate roughly 19 GB database on the same two-disk ZFS pool. The pool was serving roughly 400 small reads/s at only about 3 MiB/s, with approximately 300-500 ms total read wait dominated by queueing. A control of 30 ordinary 17-byte appends usually completed in 1-2 ms, so the incident was an extreme tail/starvation event rather than every small write being uniformly slow.

The session eventually recovered without a restart. The delayed thread_settings_applied event retained the prompt's original timestamp, while task_started was recorded 49m22.787s later. Once the persistence wait released, the queued prompt started and the turn completed normally. This makes the failure bounded only by eventual filesystem recovery; Codex itself supplied no deadline, diagnostic, or alternate recovery path.

This does not show which not-yet-written JSONL record was in the 1,442-byte buffer, so I am not claiming it was specifically the clean post-terminal Flush command. It does prove that a rollout writer operation immediately after the completed turn blocked queued session progress, and that the writer/session had no bounded recovery path.

Why global history contained the prompts

The source explains the otherwise confusing split. After submission, the TUI separately requests the cross-session history append (input_submission.rs), and that append is spawned independently (thread_routing.rs). A history entry therefore does not mean core created or persisted the turn.

On the core side, normal completion clears the active turn, awaits terminal rollout persistence, and only afterward starts queued work (tasks/mod.rs). The local store holds its per-thread writer lock across the durable operation and awaits the recorder flush (live_writer.rs). The recorder uses one FIFO writer; flush() waits on an acknowledgement with no timeout (recorder.rs), while every record awaits both write_all and file.flush (writer loop, write_line). One blocked filesystem operation therefore serializes every later persistence command for that thread.

Version check

I compared this path in rust-v0.145.0, rust-v0.146.0, latest stable rust-v0.147.0, and current main as of 2026-08-10. The same unbounded chain remains in all four: post-terminal persistence precedes pending-work start, the live-writer lock spans the awaited durable operation, and writer write_all/flush plus acknowledgement have no timeout. So an upgrade is desirable for other fixes, but the latest stable version does not appear to fix this class.

Two related changes seem worth revisiting:

#30144 added the post-terminal durability barriers. The durability goal is valid, but the barrier is currently unbounded on a liveness-critical path.
#30668 would have removed redundant clean-writer filesystem work and added useful writer-stage spans, but was closed unmerged. It would reduce exposure and improve diagnosis, though it would not bound a genuinely blocked write_all/per-record flush.

Suggested deterministic regression coverage

In addition to the withheld-flush-ack test proposed in this issue, please inject a writer whose next write_all blocks:

Complete ordinary interactive turn A and deliver its terminal event.
Block the next rollout write before completion cleanup can start queued work.
Submit ordinary turn B, with no Guardian/approval path.
Confirm the submission is accepted, but current code never reaches B's user_message/task_started.
Require fixed code to either complete a bounded persistence barrier and start B, or transition the thread to an explicit recoverable persistence error within a strict timeout.
Release the writer and verify ordered recovery or clean writer-generation retirement without duplication.

A black-box Linux repro can use a disposable delayed/FUSE mount for only $CODEX_HOME/sessions, leaving history/state local. This reproduces the accepted-history/frozen-rollout split without using real transcripts or stressing production storage.

Questions for maintainers:

Should rollout append/flush waits have a deadline and explicit persistence blocked thread state?
Can pending-work scheduling be decoupled from the unbounded post-terminal barrier while preserving the durable ordering contract?
Can the per-thread live-writer lock stop spanning filesystem I/O?
Is there a replacement planned for #30668's clean-flush elimination and writer-stage tracing?

This evidence intentionally omits transcript content, thread IDs, local paths, user/workspace names, hostnames, account details, and infrastructure identifiers.

Given the independent CLI/Linux/full-access reproduction, this issue may also warrant CLI/Linux labeling and a title that reflects the general rollout-writer failure class rather than only Windows auto-review.

ShadowTheFrog · 16 days ago

Additional current-build recurrence

I observed a new Windows Codex App recurrence with the same externally visible failure shape.

Codex App package: 26.803.10989.0
Platform: Microsoft Windows NT 10.0.26200.0 x64
Permission mode: approvals_reviewer = auto_review
Sandbox: workspace-write
Requested operation: native apply_patch with an Update File hunk against a non-sensitive ignored shadow Markdown candidate
Dispatch timestamp: 2026-08-12T07:46:09.552Z
Manual interruption/result timestamp: 2026-08-12T10:51:48.323Z
Observed pending duration: 11,138.8 s (3h 05m 38.8s)
Parent result after interruption: aborted by user after 11138.8s
Mutation reconciliation: the target remained unchanged, no patch_apply_end event was recorded for the stalled call, and the tracked worktree remained clean
Recovery: restarting/interruption was required; after the recovery handshake, rebuilding the shadow candidate with a fresh atomic delete/add patch completed in about 2 seconds

While the call was pending, the task remained shown as in progress and could not autonomously report status, change strategy, or recover because the tool call had not returned.

This is consistent with the issue's “underlying tool did not run / no side effects / parent call remained pending for hours” symptom class. I do not have sufficient hidden-review lifecycle evidence to assert the same internal causal stage, so I am reporting the measured parent-call behavior only.

Repository names, local paths, prompt contents, user identifiers, and full rollout logs are intentionally omitted. A narrowly redacted event pair can be provided if useful.

mennya · 12 days ago

Deterministic unit-level regression prototype

I prototyped a two-file regression test against main at commit 9ded177ce7c1. It adds test infrastructure only; there is no change to normal runtime behavior and no production fix.

The harness adds a one-shot gate to InMemoryThreadStore. In this fixture, it holds the second explicit ThreadStore::flush_thread call—the call reached after TurnComplete—and then starts persistence/delivery of a subsequent thread_settings_applied event.

On the base commit, the new test reaches its bounded assertion:

next thread_settings_applied event blocked behind post-terminal flush

The gate is then released unconditionally, and the queued persistence task completes before the test reports the failure. One nextest invocation produced the initial failure and an automatic retry; both failed at the expected assertion in 0.310s and 0.318s.

The two adjacent terminal completion/abort durability tests passed in separate focused runs. just fmt completed, and git diff --check reported no whitespace errors. I have not run the full workspace test suite.

This is a unit-level reproduction of the persistence-ordering dependency. It does not reproduce the intermittent Windows/filesystem stall, does not exercise Guardian deadline or session-retirement behavior, and does not establish that every incident reported here has the same cause.

Would this test shape be useful to the team? I have not opened a PR because external contributions are invitation-only. If this aligns with the intended approach, I can prepare a PR if a maintainer explicitly invites one.

shleder · 11 days ago

This is exactly the kind of interrupted persisted-turn case I’m field-testing with codex-rescue. It won’t unblock Codex’s rollout writer or Guardian reviewer; the useful question is whether the durable parent/reviewer session can be diagnosed after a stall without assuming the pending tool actually executed.

If you still have an affected local session, could you start with the read-only path?

pipx install codex-rescue
codex-rescue sessions
codex-rescue doctor --latest

If doctor identifies an unfinished state and you actually need recovery, the optional path is:

codex-rescue salvage --latest --fork
codex-rescue verify <rescue-id>

The source rollout remains read-only, and unknown tool side effects are not auto-replayed. Please share only sanitized findings, versions and exit codes—no raw rollouts/SQLite, prompts, tool payloads, credentials, or private paths. Repo: https://github.com/shleder/codex-rescue