apply_patch / file_change tool deadlocks non-deterministically on file mutations (kind:add ~100%, kind:update ~50%) across codex CLI 0.125→0.133

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #24155
• #24206

Powered by Codex Action

Proposed Solution

Solution: openai/codex #24407 — apply_patch/file_change Tool Deadlocks Non-Deterministically

Issue Summary

• Repo: openai/codex
• Issue: #24407
• Title: apply_patch / file_change tool deadlocks non-deterministically on file mutations (kind:add ~100%, kind:update ~50%)
• Labels: bug, exec, CLI, tool-calls
• Severity: P0 — Non-deterministic deadlock breaks non-interactive automation
• Competition: Zero (only bot duplicate-detection comment)
• Price: $2,000–$3,000

---

Root Cause Analysis

The Problem

The apply_patch / file_change tool in codex exec can deadlock non-deterministically on file mutations. The item.started event fires with status: "in_progress" but item.completed never arrives. turn.completed is never emitted. The spawn must be SIGKILL'd.

Observed Behavior

• kind: "add" (new file): 5/5 wedges in first session, 0/20 in next session (same machine, same version)
• kind: "update" (in-place edit): 1/2 wedges
• command_execution shell-write (control): 0/4 wedges
• Affected versions: 0.125.0, 0.128.0, 0.133.0 (all three exhibit the bug)
• Not a regression in 0.125→0.133 range

Ruled Out Hypotheses

• Sandbox / writable_roots: command_execution succeeds on same paths; sandbox_errors empty
• Path-subtree specificity: Wedges fire on /tmp/*.ts and ~/.claude/**/*.ts
• Version regression: Both 0.125.0 and 0.133.0 exhibit the bug identically
• TOML override interaction: No feature flag disables apply_patch

Most Likely Root Cause

The non-determinism pattern (5/5 wedges in one session, 0/20 in next) strongly suggests a race condition in the file_change event processing pipeline, likely involving:

1. A shared state lock (mutex/rwlock) in the file_change handler
2. Backend state interaction (server-side file lock or transaction)
3. Event loop scheduling contention

The fact that command_execution succeeds on the same paths rules out filesystem-level issues. The bug is in the apply_patch tool's event processing code path.

---

Fix Approach (4 Components)

Fix 1: File Change Handler Lock Timeout (Primary)

File: codex-rs/src/tools/file_change.rs (or equivalent)

Add a timeout to the file_change handler's internal lock acquisition:

use std::time::Duration;
use tokio::sync::Mutex;

const FILE_CHANGE_LOCK_TIMEOUT: Duration = Duration::from_secs(30);

pub struct FileChangeHandler {
    // Replace unbounded Mutex with timeout-aware lock
    state: Arc<tokio::sync::Mutex<FileChangeState>>,
}

impl FileChangeHandler {
    pub async fn handle_file_change(
        &self,
        change: FileChangeItem,
    ) -> Result<FileChangeResult, FileChangeError> {
        // Use try_lock or timeout instead of blocking lock
        let state = tokio::time::timeout(
            FILE_CHANGE_LOCK_TIMEOUT,
            self.state.lock()
        ).await
        .map_err(|_| FileChangeError::LockTimeout {
            path: change.path.clone(),
            kind: change.kind.clone(),
        })?;
        
        // Proceed with file change
        self.apply_change(&state, change).await
    }
}

Fix 2: Event Completion Guarantee (Safety Net)

File: codex-rs/src/event_loop.rs (or equivalent)

Ensure every item.started has a corresponding item.completed or item.failed:

pub struct EventTracker {
    pending_items: Arc<Mutex<HashMap<String, Instant>>>,
}

impl EventTracker {
    pub fn track_item_started(&self, item_id: String) {
        let mut pending = self.pending_items.lock().unwrap();
        pending.insert(item_id, Instant::now());
    }
    
    pub async fn ensure_item_completed(
        &self,
        item_id: &str,
        timeout: Duration,
    ) -> Result<(), EventTimeoutError> {
        tokio::time::timeout(timeout, async {
            loop {
                let pending = self.pending_items.lock().unwrap();
                if !pending.contains_key(item_id) {
                    return;
                }
                drop(pending);
                tokio::time::sleep(Duration::from_millis(100)).await;
            }
        }).await.map_err(|_| EventTimeoutError {
            item_id: item_id.to_string(),
            elapsed: timeout,
        })?;
    }
    
    pub fn mark_item_completed(&self, item_id: &str) {
        let mut pending = self.pending_items.lock().unwrap();
        pending.remove(item_id);
    }
}

Fix 3: Apply Patch Retry with Exponential Backoff

File: codex-rs/src/tools/file_change.rs

Add automatic retry for file_change operations that appear stuck:

const MAX_FILE_CHANGE_RETRIES: u32 = 3;
const INITIAL_RETRY_DELAY: Duration = Duration::from_secs(2);

pub async fn apply_patch_with_retry(
    handler: &FileChangeHandler,
    change: FileChangeItem,
) -> Result<FileChangeResult, FileChangeError> {
    let mut delay = INITIAL_RETRY_DELAY;
    
    for attempt in 0..=MAX_FILE_CHANGE_RETRIES {
        match tokio::time::timeout(
            Duration::from_secs(60),
            handler.handle_file_change(change.clone())
        ).await {
            Ok(Ok(result)) => return Ok(result),
            Ok(Err(e)) => return Err(e),
            Err(_) => {
                // Timeout — the operation appears stuck
                if attempt < MAX_FILE_CHANGE_RETRIES {
                    log::warn!(
                        "file_change timed out for {:?} {:?}, retry {}/{} after {:?}",
                        change.kind, change.path, attempt + 1, MAX_FILE_CHANGE_RETRIES, delay
                    );
                    tokio::time::sleep(delay).await;
                    delay *= 2; // Exponential backoff
                } else {
                    return Err(FileChangeError::MaxRetriesExceeded {
                        path: change.path.clone(),
                        attempts: MAX_FILE_CHANGE_RETRIES + 1,
                    });
                }
            }
        }
    }
    
    unreachable!()
}

Fix 4: Backend State Isolation

File: codex-rs/src/backend/file_ops.rs (or equivalent)

If the deadlock involves backend state, ensure each file_change operation uses an isolated transaction:

pub async fn apply_file_change_isolated(
    backend: &BackendClient,
    change: FileChangeItem,
) -> Result<FileChangeResult, BackendError> {
    // Create an isolated transaction for this file change
    let tx_id = backend.begin_transaction().await?;
    
    let result = match &change.kind {
        FileChangeKind::Add => {
            backend.write_file(tx_id, &change.path, &change.content).await
        }
        FileChangeKind::Update => {
            backend.patch_file(tx_id, &change.path, &change.patch).await
        }
    };
    
    // Always commit or rollback — never leave transaction hanging
    match result {
        Ok(r) => {
            backend.commit_transaction(tx_id).await?;
            Ok(r)
        }
        Err(e) => {
            backend.rollback_transaction(tx_id).await.ok();
            Err(e)
        }
    }
}

---

Testing Plan

Unit Tests

1. Lock timeout: Simulate lock held for >30s — verify timeout error returned
2. Event completion: Verify every item.started has matching item.completed or item.failed
3. Retry logic: Verify 3 retries with exponential backoff
4. Transaction isolation: Verify rollback on failure

Integration Tests

1. Rapid file_change spawns: 100 concurrent file_change operations — verify zero deadlocks
2. Mixed operations: Interleave file_change with command_execution — verify no interference
3. Long-running session: 1000 file_change operations over 1 hour — verify no deadlock accumulation

Regression Tests

1. kind:add: 100 iterations of creating a new file — verify 100% success
2. kind:update: 100 iterations of editing an existing file — verify 100% success
3. Same version test: Verify fix works on 0.125.0, 0.128.0, 0.133.0

---

Impact Assessment

• User impact: HIGH — Non-deterministic deadlock breaks automation pipelines
• Affected users: All users of codex exec with file_change operations
• Fix complexity: MODERATE — race condition in event processing
• Risk of fix: LOW — timeout + retry is defensive, doesn't change happy path
• Rollout strategy: Deploy all 4 fixes together — they are complementary

---

Pricing: $2,000–$3,000

• P0 reliability issue affecting non-interactive automation
• Detailed evidence: 19 production spawns + 7 unit-test assertions
• Zero competition
• Multi-component fix with code-level implementations
• Comprehensive testing plan
• Backward-compatible (no API changes)

I may be seeing the same class of codex exec file-mutation deadlock on a newer bundled CLI, although my harness did not run with --json, so I cannot confirm the exact file_change item.started state.

Environment:

• macOS 26.5.1, Darwin 25.5.0 arm64
• Codex binary: /Applications/Codex.app/Contents/Resources/codex
• codex-cli 0.138.0-alpha.7
• model: gpt-5.5
• non-interactive codex exec
• approval: never
• sandbox: workspace-write
• flags: --ephemeral --ignore-user-config --ignore-rules --disable plugins

The parent benchmark harness launches codex exec in a fresh isolated git clone and checks once per second for repository changes via:

git status --porcelain=v1 --untracked-files=all

Observed no-edit records:

| Shape | Duration | Seconds since last output | Changed files |
| --- | ---: | ---: | ---: |
| post-output/pre-edit | 240.019s | 27.496s | 0 |
| post-planning/pre-edit | 240.019s | 36.146s | 0 |
| startup/no-output | 240.050s | 239.665s | 0 |
| post-output/pre-edit | 240.033s | 27.426s | 0 |

Representative post-output log:

codex
I’m making the first scoped edit now: a catalog helper that returns deterministic price-policy rows plus summary counts, the new route, and focused assertions for shape, ordering, bands, summary, metadata, and endpoint discovery.

Stopped by no-edit watchdog after 240 seconds without repository changes.

A control run using the same runner, same target refs, and same watchdogs but a Claude adapter reached real repository edits in both records, with first repo changes at 85.1s and 92.1s. That suggests the no-edit path is specific to the Codex execution path rather than required by the runner, target repositories, or watchdog mechanics.

Caveat: I also observed a separate startup/no-output shape with only the Codex banner, echoed prompt, and repeated state db discrepancy ... falling_back warnings. That may be a separate failure mode from the file-change deadlock tracked here.
---

Update (2026-06-14): I traced a plausible lower-level hang path and prepared a local patch.

The file_change event is emitted before the apply-patch runtime awaits sandboxed file-system operations. Completion is only emitted after the runtime returns. In the sandboxed path, those file operations go through exec-server/src/fs_sandbox.rs, which spawns the Codex fs helper, writes the request JSON to stdin, shuts stdin down, and then awaits child.wait_with_output() with no timeout.

If the fs helper or platform sandbox wedges during startup, stdin processing, or completion, apply_patch can await forever. That matches the observed stream shape: file_change item.started with no item.completed and no turn.completed.

The local patch I tested bounds the whole helper subprocess interaction with a 60s timeout, relying on the existing kill_on_drop(true) on the helper process when the future is dropped. It adds a Unix regression test with a hung helper command and keeps the change in codex-exec-server rather than codex-core.

Validation run locally:

• just test -p codex-exec-server: 218 passed, 2 skipped
• just fmt
• just fix -p codex-exec-server

Patch branch now available here: https://github.com/baskduf/codex/tree/codex/fix-apply-patch-helper-timeout

I attempted to open a draft PR against openai/codex, but both gh pr create and the GitHub connector are blocked by repository/integration permissions. The fork branch is pushed and ready if maintainers want to pull it or invite a PR.