Auto-review can deadlock behind unbounded post-TurnComplete rollout flush on Windows

Summary

On Codex Desktop for Windows, the automatic approval reviewer can become blocked behind an unbounded rollout-file flush. The affected tool operation never starts, and the approval eventually reports:

The automatic permission approval review did not finish before its deadline.

This is not simply a slow model response or a full submission/event channel. Local logs and rollout files from twenty-seven incidents causally localize the stall to the serialized rollout writer.

Related symptom report: #24204. This report adds a source-level causal chain and Windows incident evidence.

Environment

• Codex Desktop packages observed: 26.707.3351.0, 26.707.9981.0, 26.715.4045.0
• Embedded core versions observed: 0.144.0-alpha.4, 0.144.2, 0.145.0-alpha.18
• Windows desktop host
• approvals_reviewer = auto_review
• Workspace-write sandbox
• Source comparison: rust-v0.142.5, rust-v0.144.0-alpha.4, latest stable rust-v0.144.1, and current main

No reliable intentional reproduction is available yet; the failure is intermittent after repeated automatic reviews.

User impact

• Unrelated tool calls can appear stuck for roughly two minutes or several hours.
• The underlying tool is not launched.
• Interrupt/review-deadline cleanup may itself fail to complete.
• Restarting Codex is the dependable recovery observed so far.

Confirmed causal chain

1. A guardian review turn completes.
2. Session::on_task_finished persists and delivers TurnComplete.
3. After delivery, current code performs another unconditional flush_rollout().await in codex-rs/core/src/tasks/mod.rs.
4. flush_rollout ultimately calls RolloutRecorder::flush in codex-rs/rollout/src/recorder.rs.
5. That method queues RolloutCmd::Flush to the thread's single writer and awaits a oneshot acknowledgement without a timeout.
6. The writer serially awaits its file flush. If that flush remains pending, later writer commands queue behind it.
7. The guardian caller has already received TurnComplete, so the hidden review session is reused.
8. The next UserInput is received and enters submission dispatch. Before starting its RegularTask, it emits the mandatory thread_settings_applied event through send_event_raw_without_materializing_rollout.
9. That event must be persisted before delivery. Its append/flush queues behind the stuck terminal flush, so the review task never starts.

Relevant paths:

• codex-rs/core/src/tasks/mod.rs
• codex-rs/core/src/session/handlers.rs
• codex-rs/core/src/session/mod.rs
• codex-rs/thread-store/src/local/live_writer.rs
• codex-rs/rollout/src/recorder.rs

Incident evidence

The table is maintained one row per confirmed stall so recurrence frequency, affected versions, and lifecycle changes remain easy to compare.

| # | Start (UTC / PDT) | Visible stall duration | Request type | Lifecycle evidence | Requested action | Recovery / retry | Codex package / core |
|---:|---|---:|---|---|---|---|---|
| 1 | 2026-07-11 11:29:05.119 UTC / 04:29:05.119 PDT | 6h 28m 46.5s (23,326.5s) | shell_command | Valid model allow payload arrived; terminal review lifecycle then remained blocked until writer release | Did not run; no side effects observed | Manual termination surfaced the reviewer-deadline error; no orphan process | 26.707.3351.0 / 0.144.0-alpha.4 |
| 2 | 2026-07-11 21:10:17.868 UTC / 14:10:17.868 PDT | 6h 29m 32.5s (23,372.5s) | apply_patch | Submission was received; thread_settings_applied was not physically written/delivered until teardown released the writer | Did not run; no side effects observed | Manual termination; same Codex app process survived; no related process leak | 26.707.3351.0 / 0.144.0-alpha.4 |
| 3 | 2026-07-12 06:39:21.306 UTC / 2026-07-11 23:39:21.306 PDT | 1m 47.1s (107.1s) | apply_patch | Submission was received; delayed thread_settings_applied and task start appeared only after writer release | Did not run; no side effects observed | Reviewer-deadline error returned; subsequent work continued without side effects | 26.707.3351.0 / 0.144.0-alpha.4 |
| 4 | 2026-07-12 18:11:42.544 UTC / 11:11:42.544 PDT | 3h 21m 59.3s (12,119.3s) | apply_patch | Reviewer deadline was recorded internally after about 95.5s, but the parent tool result was not delivered for another ~3h20m | Did not run; no side effects observed | Manual termination surfaced the deadline error; retry completed in ~13.5s | 26.707.3351.0 / 0.144.0-alpha.4 |
| 5 | 2026-07-13 00:57:27.145 UTC / 2026-07-12 17:57:27.145 PDT | 41m 6.8s (2,466.8s) | apply_patch | Reviewer decline was recorded internally after 95.56s, but the parent tool result was not delivered for another 39m31.3s | Did not run; no side effects observed | Delayed result surfaced; stale cell was already gone; retry completed in ~15s | 26.707.3351.0 / 0.144.0-alpha.4 |
| 6 | 2026-07-13 12:15:24.796 UTC / 05:15:24.796 PDT | 2h 30m 3.2s (9,003.2s) | apply_patch | Automatic review did not finish before its deadline; the parent tool call remained pending for roughly 2.5 hours before the timeout surfaced | Did not run; fresh inspection found no side effects | Delayed timeout returned; retry after one minute completed in 3.8s | 26.707.3351.0 / 0.144.0-alpha.4 |
| 7 | 2026-07-13 18:24:40 UTC / 11:24:40 PDT (derived) | 1h 30m 24.2s (5,424.2s) | apply_patch | Automatic review did not finish before its deadline; the parent tool call remained pending for about 90 minutes before cancellation surfaced the timeout | Did not run; target remained unchanged | Stale request cancelled; one retry completed in about 12.2s | 26.707.3351.0 / 0.144.0-alpha.4 |
| 8 | 2026-07-14 05:03:27.944 UTC / 2026-07-13 22:03:27.944 PDT | 4m 22.1s (262.1s) | apply_patch | thread_settings_applied was recorded, but task_started was delayed 261.952s until cancellation; automatic review then surfaced its deadline timeout | Did not run; target remained unchanged | Stalled call cancelled; timeout surfaced; first retry completed successfully | 26.707.3351.0 / 0.144.0-alpha.4 |
| 9 | 2026-07-14 15:18:56.585 UTC / 08:18:56.585 PDT | 23m 20.0s (1,400.0s) | apply_patch | Reviewer decline was recorded internally after 95.601s, but the parent tool result was not delivered for another 21m 44.3s | Did not run; no side effects observed | Delayed timeout surfaced; first retry completed in 14.644s | 26.707.3351.0 / 0.144.0-alpha.4 |
| 10 | 2026-07-15 04:55:17.085 UTC / 2026-07-14 21:55:17.085 PDT | 1h 18m 3.7s (4,683.7s) | apply_patch | Reviewer decline was recorded internally after 95.407s, but the parent tool result remained pending for another ~1h16m28s until cancellation | Did not run; no side effects observed | Stalled call cancelled; timeout surfaced; first retry completed successfully in 14.139s | 26.707.9981.0 / 0.144.2 |
| 11 | 2026-07-15 07:21:39.349 UTC / 00:21:39.349 PDT | 1h 22m 18.7s (4,938.7s) | apply_patch | Reviewer decline was recorded internally after 95.390s, but the completed parent tool result was not delivered for another 1h 20m 43.3s | Did not run; target remained unchanged | Delayed timeout surfaced after new user activity; first retry completed successfully in 15.249s | 26.707.9981.0 / 0.144.2 |
| 12 | 2026-07-15 18:47:50.077 UTC / 11:47:50.077 PDT | 1h 9m 35.2s (4,175.2s) | apply_patch | Reviewer decline was recorded internally after 95.426s, but the parent tool result was not delivered for another 1h 7m 59.7s | Did not run; target remained unchanged | New user activity surfaced the pending cell; manual termination returned the timeout; first retry completed in 4.1s | 26.707.9981.0 / 0.144.2 |
| 13 | 2026-07-16 01:53:39.695 UTC / 2026-07-15 18:53:39.695 PDT | 1h 35m 49.4s (5,749.4s) | apply_patch | Reviewer decline was recorded internally after 95.394s, but the parent tool result was not delivered for another 1h 34m 14.0s | Did not run; target remained unchanged | Manual termination returned the timeout; first retry completed successfully in 2.740s | 26.707.9981.0 / 0.144.2 |
| 14 | 2026-07-16 07:00:23.191 UTC / 00:00:23.191 PDT | 4m 20.6s (260.587s) | apply_patch | Automatic review did not finish before its deadline; the parent tool call remained pending for 260.587s before the timeout surfaced | Did not run; target remained unchanged | Timeout surfaced; after a one-minute delay, the first retry completed in 10.063s | 26.707.9981.0 / 0.144.2 |
| 15 | 2026-07-16 18:37:57.745 UTC / 11:37:57.745 PDT | 54m 6.3s (3,246.313s) | apply_patch | thread_settings_applied was recorded immediately, but task_started was delayed 3,246.186s until cancellation; the turn aborted 0.127s later | Did not run; target remained unchanged | Manual termination after new user activity returned the reviewer-deadline timeout; first retry completed successfully | 26.707.9981.0 / 0.144.2 |
| 16 | 2026-07-17 03:44:32.000 UTC / 2026-07-16 20:44:32.000 PDT | 24m 39.0s (1,479.034s) | apply_patch | Review task started and produced reasoning, but no terminal allow/decline or task_complete; cancellation produced turn_aborted 91ms later | Did not run; target remained unchanged | Manual termination surfaced the reviewer-deadline timeout; first retry completed successfully | 26.707.9981.0 / 0.144.2 |
| 17 | 2026-07-17 04:40:33.434 UTC / 2026-07-16 21:40:33.434 PDT | 1h 5m 1.5s (3,901.535s) | apply_patch | Review task started and produced reasoning, but no terminal allow/decline or task_complete; cancellation produced turn_aborted 115ms later | Did not run; target remained unchanged | Reviewer-deadline timeout surfaced after interruption; first retry completed successfully on a fresh reviewer session | 26.707.9981.0 / 0.144.2 |
| 18 | 2026-07-17 10:26:54.517 UTC / 03:26:54.517 PDT | 4h 33m 55.8s (16,435.8s) | apply_patch | Reviewer decline was recorded internally after 95.430s, but the parent tool result was not delivered for another 4h 32m 20.4s | Did not run; no side effects observed | Delayed timeout surfaced; first retry was terminated during a user pause before completion | 26.707.9981.0 / 0.144.2 |
| 19 | 2026-07-18 01:44:23.766 UTC / 2026-07-17 18:44:23.766 PDT | 3m 6.4s (186.376s) | apply_patch | Automatic review did not finish before its deadline; the parent tool call remained pending until a waiter surfaced the timeout | Did not run; target remained unchanged | Timeout surfaced; first identical retry completed successfully in 10.046s | 26.707.9981.0 / 0.144.2 |
| 20 | 2026-07-18 23:06:58.952 UTC / 16:06:58.952 PDT | 2h 29m 10.7s (8,950.7s) | apply_patch | Reviewer decline was recorded internally after 95.432s, but the parent tool result was not delivered for another 2h 27m 35.3s | Did not run; target remained unchanged | Delayed timeout surfaced after cancellation; first retry completed successfully in 13.945s | 26.707.9981.0 / 0.144.2 |
| 21 | 2026-07-19 01:49:59.780 UTC / 2026-07-18 18:49:59.780 PDT | 2h 43m 31.0s (9,811.0s) | apply_patch | Reviewer decline was recorded internally after 95.490s, but the parent tool result was not delivered for another 2h 41m 55.5s | Did not run; target remained unchanged | Delayed timeout surfaced after cancellation; first retry completed successfully in 15.042s | 26.707.9981.0 / 0.144.2 |
| 22 | 2026-07-19 07:33:05.531 UTC / 00:33:05.531 PDT | 1h 56m 49.7s (7,009.7s) | apply_patch | Reviewer decline was recorded internally after 95.438s, but the parent tool result was not delivered for another 1h 55m 14.3s | Did not run; target remained unchanged | Delayed timeout surfaced after cancellation; first retry completed successfully in 14.905s | 26.707.9981.0 / 0.144.2 |
| 23 | 2026-07-19 11:17:36.068 UTC / 04:17:36.068 PDT | 6h 8m 56.6s (22,136.6s) | shell_command | Automatic review did not finish before its deadline; the parent tool call remained pending for 22,136.6s before manual cancellation surfaced the timeout | Did not run; no side effects observed | Manual termination surfaced the timeout; corrected direct read-only retry completed in 1.1s | 26.707.9981.0 / 0.144.2 |
| 24 | 2026-07-19 19:37:16.897 UTC / 12:37:16.897 PDT | 40m 6.6s (2,406.6s) | shell_command | Automatic review did not finish before its deadline; the parent tool call remained pending for 2,406.6s before manual termination surfaced the timeout | Did not run; no side effects observed | Manual termination surfaced the timeout; corrected bounded read-only retry completed in 0.8s | 26.715.4045.0 / 0.145.0-alpha.18 |
| 25 | 2026-07-20 00:44:26.141 UTC / 2026-07-19 17:44:26.141 PDT | 2h 9m 42.9s (7,782.884s) | apply_patch | Reviewer decline was recorded internally after 95.445s, but the parent tool result was not delivered for another 2h 8m 7.4s | Did not run; target remained unchanged | Manual termination surfaced the timeout; first identical retry completed successfully within one minute | 26.715.4045.0 / 0.145.0-alpha.18 |
| 26 | 2026-07-20 05:31:33.494 UTC / 2026-07-19 22:31:33.494 PDT | 3h 27m 37.4s (12,457.4s) | apply_patch | Reviewer decline was recorded internally after 95.435s, but the parent tool result was not delivered for another 3h 26m 2.0s | Did not run; target remained unchanged | Delayed timeout surfaced; first identical retry completed successfully in 15.648s | 26.715.4045.0 / 0.145.0-alpha.18 |
| 27 | 2026-07-20 18:26:58.360 UTC / 11:26:58.360 PDT | 10m 30.4s (630.384s) | apply_patch | Guardian returned allow after 6.230s, but the approval lifecycle still recorded decline after 95.428s; the parent result was delayed for 623.368s | Did not run; target remained unchanged | Deadline timeout surfaced; first identical retry completed successfully in 13.546s | 26.715.4045.0 / 0.145.0-alpha.18 |

Incidents 2 and 3 stopped after the Submission debug row but before the review task's turn span. In both:

• The thread_settings_applied JSONL record retained the original submission timestamp.
• The record was not physically written/delivered until teardown released the writer.
• task_started followed immediately after writer release.
• The task was then interrupted/aborted because the approval deadline had already expired.

Incident 1 reached a valid model allow payload but did not complete its terminal lifecycle until teardown. Incidents 4 and 5 add a distinct result-delivery symptom: the reviewer result was already recorded internally, yet the parent tool call remained pending for hours or tens of minutes. Incident 6 independently adds another roughly 2.5-hour parent-call delay before the automatic-review timeout surfaced; its table row intentionally makes no stronger internal log-timing claim. Incident 7 adds a 90.4-minute parent-call delay with the same automatic-review deadline error; its start time is derived from the observed end time and tool-call duration, and its row likewise makes no stronger internal log-timing claim. Incidents 9 through 27 extend recurrence across three observed builds: incident 9 on package 26.707.3351.0 / core 0.144.0-alpha.4, incidents 10 through 23 fourteen times on package 26.707.9981.0 / core 0.144.2, and incidents 24 through 27 four times on package 26.715.4045.0 / core 0.145.0-alpha.18; incident 12 recorded the reviewer decline after 95.426 seconds and then withheld the completed parent result for another 4,079.726 seconds, incident 13 recorded the decline after 95.394 seconds and withheld the parent result for another 5,653.970 seconds, incident 14 adds a shorter 260.587-second current-version recurrence where the timeout surfaced directly, and incident 15 again shows thread_settings_applied followed by a 3,246.186-second delay before task_started appeared only when cancellation released the blocked lifecycle. Incident 16 adds a 1,479.034-second reviewer turn that started and produced reasoning but never emitted a terminal decision or task_complete; cancellation yielded turn_aborted 91 milliseconds later, and the underlying apply_patch remained unapplied. Incident 17 adds a 3,901.535-second reviewer turn with the same started-and-reasoning-only lifecycle; cancellation yielded turn_aborted 115 milliseconds later, the underlying apply_patch remained unapplied, and the first retry completed on a fresh reviewer session. Incident 18 adds a 16,435.8-second parent-call delay after the reviewer had already recorded its decline at 95.430 seconds; the underlying apply_patch did not run, and the first retry was terminated during a user pause before completion. Incident 19 adds a 186.376-second current-version recurrence where the automatic-review deadline surfaced through a waiter; the underlying apply_patch did not run, and the first identical retry completed in 10.046 seconds. Incident 20 adds an 8,950.7-second parent-call delay after the reviewer recorded its decline at 95.432 seconds; the underlying apply_patch did not run, and the first retry completed successfully in 13.945 seconds. Incident 21 adds a 9,811.0-second parent-call delay after the reviewer recorded its decline at 95.490 seconds; the underlying apply_patch did not run, and the first retry completed successfully in 15.042 seconds. Incident 22 adds a 7,009.7-second parent-call delay after the reviewer recorded its decline at 95.438 seconds; the underlying apply_patch did not run, and the first retry completed successfully in 14.905 seconds. Incident 23 adds a 22,136.6-second parent-call delay before manual cancellation surfaced the automatic-review timeout; the underlying shell_command did not run, and a corrected direct read-only retry completed in 1.1 seconds. Incident 24 adds a 2,406.6-second parent-call delay on package 26.715.4045.0 / core 0.145.0-alpha.18 before manual termination surfaced the automatic-review timeout; the underlying shell_command did not run, and a corrected bounded read-only retry completed in 0.8 seconds. Incident 25 adds a 7,782.884-second parent-call delay after the reviewer recorded its decline at 95.445 seconds; the underlying apply_patch did not run, and the first identical retry completed successfully within one minute. Incident 26 adds a 12,457.4-second parent-call delay after the reviewer recorded its decline at 95.435 seconds; the underlying apply_patch did not run, and the first identical retry completed successfully in 15.648 seconds. Incident 27 adds a 630.384-second approval-lifecycle delay even though the guardian returned allow after 6.230 seconds; the lifecycle recorded decline after 95.428 seconds, the underlying apply_patch did not run, and the first identical retry completed successfully in 13.546 seconds.

This rules out, for these incidents:

• submission-channel capacity
• event-channel capacity (the event channel is unbounded)
• model inference delay for the pre-task incidents
• an unprocessed interrupt submission
• command length, filename length, or path length
• a coded 47th/48th-review retirement limit

Regression signal

rust-v0.142.5 did not perform the second unconditional flush after terminal event delivery. rust-v0.144.0-alpha.4 added it. Current main and stable rust-v0.144.1 still retain the path.

The terminal event's normal persistence already flushes through the local thread store before delivery, making the additional post-delivery unbounded flush both redundant and uniquely dangerous: the caller can reuse the session while it is pending.

Expected behavior

• Rollout persistence must not block approval processing indefinitely.
• A reviewer deadline must complete near its configured bound.
• A persistence failure/timeout must retire the affected guardian session.
• A later approval must start on a healthy session.
• Interrupt cleanup must not depend on the same stalled session-loop/persistence path.

Suggested fixes

Any complete fix should address the failure class, not only this call site:

1. Remove the redundant post-terminal flush, or bound it with a short timeout.
2. Mark a guardian session unusable if its persistence barrier times out.
3. Ensure pre-task events cannot wait indefinitely on rollout persistence.
4. Make deadline cancellation independent of the blocked submission/persistence path.
5. Preserve an explicit diagnostic identifying the blocked persistence phase.

Suggested regression tests

Use an injected writer that deliberately withholds a flush acknowledgement:

• Complete review N and deliver TurnComplete.
• Block only its post-terminal flush.
• Submit review N+1.
• Assert N+1 either proceeds on a fresh session or fails within a strict bounded timeout.
• Assert the next review after timeout succeeds.
• Assert no hidden reviewer task/session leaks.
• Run on Windows in addition to unit-level injected-writer coverage.

Privacy

This report intentionally excludes transcripts, company/workspace names, local paths, credentials, tokens, proprietary source, and unrelated application evidence.