›_ Codex Issues
Codex Issues / #34013

Windows sandbox helpers race while updating deny_read_acl_state.json

Open 💬 4 comments Opened Jul 18, 2026 by starriet9

What version of Codex CLI is running?

codex-cli 0.144.5

Update (2026-08-13): The transaction-lock root cause remains present in current main, but later Windows validation found that the linked historical candidate's NamedTempFile::persist step is not the final replacement primitive needed for an existing/open destination. The mutex design remains valid; the persistence portion is superseded by the ReplaceFileW/MoveFileExW design and validation described in this follow-up.

The failure was captured from a local build based on the exact rust-v0.144.5 source. Its unrelated context-window changes did not modify codex-rs/windows-sandbox-rs. The same unlocked state implementation remains in main at 56395bdda.

What subscription do you have?

Pro

Which model were you using?

gpt-5.6-sol

What platform is your computer?

Microsoft Windows NT 10.0.19045.0 x64

What terminal emulator and version are you using (if applicable)?

Windows Terminal 1.24.11321.0 with PowerShell 7.6.3 (Core, x64). No terminal multiplexer.

Codex doctor report

Relevant results from codex doctor --json are summarized below; unrelated entries are omitted.

{
  "schemaVersion": 1,
  "overallStatus": "warning",
  "codexVersion": "0.144.5",
  "checks": {
    "system.environment": {
      "status": "ok",
      "os": "Windows 10 Pro 10.0.19045, 64-bit"
    },
    "sandbox.helpers": {
      "status": "ok",
      "approvalPolicy": "OnRequest",
      "filesystemSandbox": "restricted",
      "networkSandbox": "restricted"
    },
    "state.paths": {
      "status": "ok",
      "databaseIntegrity": "state, log, goals, and memories checks passed"
    },
    "network.provider_reachability": {
      "status": "ok"
    },
    "network.websocket_reachability": {
      "status": "ok"
    }
  }
}

The overall warning came from 19 old rollout files that contained no parseable rollout items. It is unrelated to the sandbox state-file race.

What issue are you seeing?

When several sandboxed tool calls start in parallel on Windows, their codex-windows-sandbox-setup.exe processes concurrently update the shared file:

%USERPROFILE%\.codex\.sandbox\deny_read_acl_state.json

In the captured run, six setup helpers started within a few milliseconds. Two failed with:

setup error: apply deny-read ACLs

Caused by:
  0: parse deny-read ACL state <CODEX_HOME>\.sandbox\deny_read_acl_state.json
  1: EOF while parsing a value at line 1 column 0

The affected commands failed with helper_unknown_error: apply deny-read ACLs. Retrying the same command by itself succeeded.

The root cause is in codex-rs/windows-sandbox-rs/src/deny_read_state.rs:

  1. sync_persistent_deny_read_acls() loads the shared state.
  2. It applies new ACLs and revokes stale ACLs.
  3. store_state() calls std::fs::write() with no inter-process lock.
  4. std::fs::write() truncates the destination before writing the new JSON.

Another helper can read during that truncate/write interval and receive empty bytes, producing the EOF parse error. Even when every read parses successfully, two helpers updating different principals can both load the same old state and overwrite one another, losing one update.

What steps can reproduce the bug?

Runtime reproduction:

  1. Use the Windows workspace-write sandbox.
  2. Start multiple filesystem tool calls in parallel so several setup helpers run at the same time.
  3. Inspect the failed tool result and the daily file under <CODEX_HOME>\.sandbox\sandbox.YYYY-MM-DD.log.

A deterministic state-layer regression test also reproduces the lost-update form without changing real ACLs:

  1. Synchronize eight state updates so they all load the same initial JSON.
  2. Have every update add a distinct principal and then store the result.
  3. On unmodified main, the final state contains 1 principal instead of 8.

The candidate also launches six child test processes against one state file to verify the fix across real process boundaries.

No thread ID is required; this occurs in the local Windows helper before the sandboxed command starts.

What is the expected behavior?

Concurrent setup helpers should serialize the full persistent state transaction. Every helper should see complete JSON, updates for different principals should not be lost, and parallel shell/file operations should not fail during sandbox setup.

Additional information

Historical candidate branch:

https://github.com/starriet9/codex/tree/fix/windows-sandbox-deny-read-state-race

Candidate commit:

https://github.com/starriet9/codex/commit/0280e7b8ed1b58f95f2d4da732c601bbf02b7310

The historical candidate:

  • holds a Windows named mutex across the complete load -> ACL apply/revoke -> store transaction
  • treats an abandoned mutex as acquired, allowing recovery after a process exits
  • writes the JSON to a temporary file in the same directory, flushes it, and atomically replaces the previous state
  • adds thread-level and child-process concurrency regression tests

Validation on Windows:

Original synchronized test: final principal count 1, expected 8
Patched thread test: 8 of 8 principals preserved
Patched process test: 6 of 6 principals preserved
cargo fmt -p codex-windows-sandbox -- --check: passed
cargo clippy --offline -p codex-windows-sandbox --all-targets -- -D warnings: passed
cargo build --offline --release -p codex-windows-sandbox --bins: passed

The process regression test uses actual child processes and a Windows kernel mutex without changing real filesystem ACLs. I did not run a full parallel setup-helper stress against the machine's live sandbox accounts, ACLs, firewall, and Windows Filtering Platform (WFP) state because that would mutate system security configuration.

Related issue #30251 reports a state file that is already malformed or filled with NUL bytes and has a candidate that treats malformed JSON as empty state and uses atomic replacement. Atomic replacement prevents readers from observing a partially written file, but it does not serialize the complete read-modify-write transaction: two helpers can still load the same old state and atomically replace it with different updates, losing one update. The candidate in this report adds that missing transaction lock; it intentionally does not recover a state file that was already corrupt before startup.

Related issue #30540 likewise covers recovery after the JSON has already been corrupted, such as after a crash or power interruption. This report is distinct: it identifies a live concurrency path that can corrupt or lose the state during normal parallel tool execution.

I have not opened a pull request because docs/contributing.md says external pull requests are by invitation only. Do not treat the linked historical branch as a final current-main candidate; the follow-up above records the corrected Windows replacement design that should be used when refreshing it.

View original on GitHub ↗

4 Comments

starriet9 · 15 days ago

Correction and stronger validation from porting this fix to official rust-v0.147.0:

The transaction-wide named mutex remains necessary, and current main still does the unlocked load -> reconcile -> std::fs::write sequence:

https://github.com/openai/codex/blob/1f4ea798538d2c947ce215a65bc166db517a2df6/codex-rs/windows-sandbox-rs/src/deny_read_state.rs#L50-L86

Later Windows testing found that the original public candidate's NamedTempFile::persist step is not a complete replacement implementation for an already-existing destination on this path. In particular, replacement can fail while Windows readers have the destination open. The branch linked in the issue should therefore be treated as a historical proof of the transaction-lock design, not as the final persistence implementation.

The 0.147 maintenance port uses this stronger sequence while retaining the mutex across the complete ACL reconciliation and store transaction:

  • Readers open the state file with delete sharing so an atomic replacement can proceed while they hold a read handle.
  • Serialize into a same-directory temporary file, flush it, and call sync_all before installation.
  • Replace an existing destination with ReplaceFileW.
  • If the destination does not yet exist, use MoveFileExW with MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH.
  • Treat WAIT_ABANDONED as acquired so a crashed writer does not permanently block future reconciliation.

The strengthened tests cover concurrent threads, child processes, abandoned owners, repeated replacement while a reader remains open, complete JSON, and preservation of every principal update. The source-built 0.147 package also completed isolated packaged sandbox smoke and controlled live helper validation.

This correction does not change the issue's root cause or its distinction from recovery of an already-corrupt file in #30251/#30540. It narrows the final Windows persistence primitive needed after the transaction lock has prevented lost updates.

jakobboss1010-source · 10 days ago

Additional Windows host evidence and a tested source candidate:

  • Windows 11 x64 host with the elevated sandbox enabled repeatedly failed before process creation at apply deny-read ACLs.
  • The persisted deny_read_acl_state.json was exactly 22 NUL bytes. Local sandbox logs also showed overlapping setup-helper invocations. This matches the corruption shape reported in #34276.
  • Current source performs the deny-read state load/reconcile/write transaction without cross-process serialization and uses a non-atomic std::fs::write, allowing stale reads, lost updates, and truncated state.

Tested candidate commit: https://github.com/jakobboss1010-source/codex/commit/30b0c6f1da1dd9e10256125dd57260f476d3c591

The candidate:

  1. serializes the full load -> ACL reconciliation -> store transaction with a named Windows mutex, treating WAIT_ABANDONED as acquired;
  2. writes and sync_alls a same-directory temporary file, then installs it with ReplaceFileW, falling back to MoveFileExW(MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH) when the destination is absent;
  3. opens readers with read/write/delete sharing so atomic replacement is possible;
  4. quarantines invalid JSON and regenerates an empty valid state instead of permanently blocking sandbox startup.

Focused Windows regression evidence: 4/4 passed (22-NUL recovery, replacement with an open delete-shared reader, eight concurrent threads, six concurrent processes). Crate-scoped cargo clippy --fix --tests completed successfully. The broader package run reached 147 existing passing tests before one unrelated host-sensitive legacy delete test failed and one elevated integration test was stopped after remaining silent for over 30 seconds.

No live ACL race/fault injection was performed, no unsigned helper was installed, and no PR is being opened in accordance with the repository contribution policy.

kakabanga · 1 day ago

Corroborating evidence from a currently shipping Windows build:

  • Codex CLI: 0.150.0-alpha.8
  • Codex Desktop: 26.820.7780.0
  • OS: Windows 11 Pro 10.0.26200, x64
  • Failure: every sandboxed process launch failed before command execution with helper_unknown_error: apply deny-read ACLs.
  • codex doctor --json reported sandbox.helpers: fail / elevated Windows sandbox provisioning recorded a structured failure.

The direct cause was %USERPROFILE%\.codex\.sandbox\deny_read_acl_state.json: it was exactly 22 bytes and every byte was 0x00. Sandbox logs consistently reported:

parse deny-read ACL state ...\deny_read_acl_state.json
expected value at line 1 column 1

Historical log evidence supports the race described here. At the earliest captured failure, four codex-windows-sandbox-setup.exe processes were spawned against the same state within about 1.3 seconds. One observed EOF while parsing a value at line 1 column 0, while the other helpers reported successful setup completion. Additional bursts of four to six concurrent helpers occurred later. The state file was created on May 21, last written on May 26, and remained persistently corrupt across Desktop restarts and later app/CLI updates.

Checks that did not explain the failure:

  • Worktree ACL verification completed without malformed ACLs.
  • Both PowerShell and cmd.exe failed before process creation.
  • codex sandbox reproduced independently of the Desktop tool call.
  • codex-windows-sandbox-setup.exe and codex-command-runner.exe existed, had valid OpenAI signatures, and had normal executable ACLs.
  • Defender Operational logs contained no matching detection, ASR, or Controlled Folder Access block events.

Verified recovery: quarantine/rename only the corrupt state file, then let the next managed sandbox launch regenerate it. Codex regenerated valid JSON with the expected top-level principals key. Afterward:

  • managed sandbox command/read succeeded;
  • apply_patch add and delete both succeeded;
  • codex doctor --json changed sandbox.helpers to ok (sandbox configuration is readable).

This confirms that restart/reinstall behavior which preserves CODEX_HOME cannot self-heal this corruption. In addition to serializing and atomically replacing the transaction, the helper should quarantine malformed/NUL-filled state and regenerate a valid empty state rather than permanently blocking all sandbox launches.

Luke-Evan · 1 day ago

Corroboration from a current desktop build, plus a variant not yet reported here:
the corruption can also be transient and self-healing, not only persistent.

Environment

  • Codex Desktop 26.820.7780.0, command-runner 0.150.0-alpha.8, elevated sandbox
  • Windows 11 25H2, build 10.0.26200, x64

Observations (2026-08-27)

  1. node_repl kernel launches failed; CODEX_HOME/.sandbox/sandbox.<date>.log:

```text
setup error: apply deny-read ACLs
Caused by:
0: parse deny-read ACL state ...\deny_read_acl_state.json
1: EOF while parsing a value at line 1 column 0
setup refresh: exited with status ExitStatus(1)
Note the serde error: EOF while parsing ... column 0 — the state file was
0 bytes at parse time (naturally truncated by the race), unlike the
22-NUL-byte shape in #39453 / #34841 (expected value at line 1 column 1).

  1. Without any manual intervention, the file later contained the valid 22-byte

skeleton {"principals": {}} again and node_repl worked.

  1. The same cycle repeated three times in one morning (failures at ~08:22,

~09:27, ~09:46; healed by successful rewrites at 08:31:13 and 09:46:50 per
file mtime).
Interpretation: besides the persistent 22-NUL corruption after interrupted
writes, the unlocked truncate-then-write also produces a transient 0-byte
window — kernel launches that parse during the window fail; the next completed
write heals it. User-visible effect: "JS tools unavailable" on every task for a
few minutes, spontaneous recovery, then recurrence.
Triage hint: during the corrupted window, ordinary shell commands (read-acl-only
mode) still succeed; only the deny-read path that parses this state file fails.
In the desktop app this surfaces solely as node_repl/JS tools being unavailable,
which is easy to misattribute.
Minor side observation: every helper launch also logs
hide users: failed to hide current user profile dir (C:\Users\Default): SetFileAttributesW failed ... error 5 — the sandbox user's profile dir appears
to resolve to C:\Users\Default. Non-fatal here; mentioning in case it helps.