›_ Codex Issues
Codex Issues / #16300

RolloutRecorder shutdown race can drop tail rollout items and emit `channel closed`

Open 💬 5 comments Opened Mar 31, 2026 by hawkingrei
💡 Likely answer: A maintainer (github-actions[bot], contributor) responded on this thread — see the highlighted reply below.

What issue are you seeing?

During normal session shutdown, Codex can emit:

failed to record rollout items: failed to queue rollout items: channel closed

This appears to be more than a noisy shutdown log.

From reading the current codex-core implementation, the race is:

  • Session::persist_rollout_items(...) clones the rollout recorder handle under the shared services.rollout mutex and then performs the enqueue outside that mutex.
  • Session::shutdown(...) later takes the shared recorder out of services.rollout and calls RolloutRecorder::shutdown().
  • RolloutRecorder clones share the writer mpsc::Sender, but do not share an explicit shutdown lifecycle flag.

That means a stale recorder clone can still attempt to enqueue after another clone has already started shutdown and closed the writer channel, which then surfaces as channel closed.

The effect is not just logging noise: the tail rollout write for a session can be dropped during normal teardown, which can affect resume/history fidelity.

Relevant code paths in the pinned source tree:

  • codex-rs/core/src/codex.rs
  • persist_rollout_items(...)
  • shutdown(...)
  • codex-rs/core/src/rollout/recorder.rs
  • record_items(...)
  • persist(...)
  • flush(...)
  • shutdown(...)

What steps can reproduce the bug?

A deterministic reproducer exists at the RolloutRecorder level and does not require UI interaction.

Minimal reproduction logic:

  1. Create a RolloutRecorder.
  2. Clone it so there is a stale clone.
  3. Call shutdown() on the original recorder.
  4. After shutdown starts, call record_items() / persist() / flush() on the stale clone.

Expected old behavior before a fix:

  • the stale clone can race into the closed writer channel and return the same class of error as above.

In code, the reproducer is essentially:

let recorder = RolloutRecorder::new(path).await?;
let stale = recorder.clone();
recorder.shutdown().await?;
stale.record_items(vec![item]).await?;
stale.persist().await?;
stale.flush().await?;

In a real session, the same race shows up when teardown begins while late rollout items are still being appended.

What is the expected behavior?

Normal session shutdown should not fail tail rollout persistence because of stale recorder clones racing a closed writer channel.

More specifically:

  • once shutdown begins, late recorder clones should observe shutdown state and return cleanly
  • rollout shutdown should be deterministic
  • normal teardown should not surface failed to queue rollout items: channel closed
  • resume/history/replay should not lose the last rollout items because of this race

Additional information

I have a local patch that fixes this by giving RolloutRecorder clones a shared shutdown state:

  • add a shared shutdown_started flag across recorder clones
  • set it before closing the writer task in shutdown()
  • make late record_items() / persist() / flush() calls treat shutdown as a clean no-op instead of racing the closed channel

This is currently carried in an AgentHub-specific Codex patch branch while waiting for an upstream fix.

View original on GitHub ↗

5 Comments

github-actions[bot] contributor · 5 months ago

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

  • #16271

Powered by Codex Action

hawkingrei · 5 months ago

I dug into the current codex-core implementation and built a local fix for this race.

What appears to be happening:

  • Session::persist_rollout_items(...) clones the rollout recorder handle under services.rollout and then performs the enqueue outside that mutex.
  • Session::shutdown(...) later takes the shared recorder out of services.rollout and calls RolloutRecorder::shutdown().
  • RolloutRecorder clones share the writer mpsc::Sender, but they do not share an explicit shutdown lifecycle flag.

That lets a stale recorder clone enqueue after another clone has already started shutdown and closed the writer channel, which then surfaces as:

failed to record rollout items: failed to queue rollout items: channel closed

I do not think this is only logging noise. The real risk is that tail rollout items can be dropped during normal session teardown, which can affect resume/history fidelity.

I have a local patch that fixes this by making recorder clones share shutdown state:

  • add a shared shutdown_started flag across RolloutRecorder clones
  • flip it before closing the writer task in shutdown()
  • make late record_items() / persist() / flush() calls observe shutdown and return cleanly instead of racing a closed channel

I also added a focused reproducer at the recorder layer:

  1. create a RolloutRecorder
  2. clone it
  3. call shutdown() on the original
  4. call record_items() / persist() / flush() on the stale clone

That is enough to deterministically model the race without needing a full UI/session repro.

For reference, I currently carry the patch here while validating it downstream:

  • branch: hawkingrei/codex:agenthub/patches
  • patch commit: 18eaa6b8cdefd89a7a8ad8a0e1b0791fc33267bf

If this direction matches what you want upstream, I can reshape the patch to whatever structure/style you prefer.

JaysonGeng · 4 months ago

Actual patch details:

compare: https://github.com/openai/codex/compare/main...JaysonGeng:codex:fix/rollout-recorder-shutdown-race
commit: 397bf851fe97ff4d714051ceebd73b06fc02b108

Fix summary:

  • share shutdown_started across RolloutRecorder clones
  • - flip it before shutdown() enqueues Shutdown
  • - - make late record_items() / persist() / flush() return cleanly after shutdown starts
  • - - - flush and exit the writer on Shutdown
  • - - - - add a recorder-level regression test for stale-clone shutdown behavior

Validation:

  • cargo test -p codex-rollout
  • - just fmt
  • - - just fix -p codex-rollout
  • - - - git diff --check

just argument-comment-lint still requires local bazel, so I could not run that target here.

PR creation is still invitation-only from my account, so I’m posting the compare link here.

Necmttn · 2 months ago

The recorder needs an explicit lifecycle state and a final flush invariant.

Before shutdown completes, every queued rollout item should either be acked as persisted with last item id/offset, or reported as dropped with a terminal error. Resume can then validate the tail instead of silently accepting a transcript that may have lost final tool/result items.

---

_Generated with ax._

shleder · 14 days ago

The shutdown race you identified is directly relevant to recovery because a dropped tail write can leave the durable rollout behind the actual tool/turn state even when teardown looks otherwise normal.

I’m testing Codex Rescue against real interrupted-tail cases like this. It cannot restore items that were never persisted, but it can inspect the surviving boundary, preserve unknown execution state, and avoid replaying any side effects. The source rollout remains untouched and verification stays fail-closed.

If you have an actual rollout from a session that logged failed to queue rollout items: channel closed, could you run:

pipx install codex-rescue==0.1.0a3
codex-rescue sessions
codex-rescue doctor --latest

Sanitized output only is sufficient—no raw rollout, database, prompts, tool contents, paths, or secrets.

https://github.com/shleder/codex-rescue