Change Codex system instruction about unexpected worktree changes for better support of concurrent agent edits

The current behavior is intended, so I don't think we'd consider this a bug. I'm going to change this to a feature request and see if others upvote it.

We run a multi-agent orchestrator that spawns 5-15 concurrent CLI agents (Claude Code, Codex, Gemini CLI, etc.) against the same repo. We hit exactly this problem early on and learned the hard way that "detect and stop" is fundamentally the wrong primitive for concurrent agents. The real fix is isolation, not tolerance.

Git worktrees as the isolation primitive. Each agent gets git worktree add .sdd/worktrees/{session_id} -b agent/{session_id} — a fully independent working directory on its own branch, branched from HEAD at spawn time. The agents never see each other's changes during execution. No shared index, no shared working tree, no "unexpected changes" to reason about. This eliminates the entire class of problem described in this issue without requiring agents to have opinions about concurrent modifications.

Merge queue instead of shared-branch commits. When an agent finishes, its branch enters a FIFO merge queue with a lock so only one merge runs at a time. We run git merge-tree as a pre-flight check (zero working-tree side effects) to detect conflicts before attempting the real merge. Clean merges go through automatically. Conflicts get routed to a resolver agent that has the conflict context. This is strictly better than agents trying to commit to a shared branch because merge conflicts become structured data rather than runtime surprises.

Scope constraints as a second layer. Each task carries owned_files — an explicit list of paths the agent is expected to touch. The orchestrator assigns non-overlapping file scopes to concurrent agents. Even without worktree isolation, this reduces conflict probability to near zero for well-decomposed tasks. The combination of worktree isolation + file scope constraints means we almost never see real conflicts in practice.

Why "continue unless conflicting" is still fragile. The suggested fix — only stop on semantic conflicts — requires the agent to reliably distinguish "another agent touched an unrelated file" from "someone force-pushed and I'm now working on stale state" from "a build tool regenerated lockfiles" from "the user manually edited something." That's a lot of heuristic surface area. In our experience, agents are bad at making these distinctions under pressure, and the failure mode (silently working on inconsistent state) is worse than the current failure mode (stopping too eagerly).

Concrete gotchas we hit with worktrees:

• Stale worktree cleanup on crash. If the orchestrator dies, you get orphan worktrees that block subsequent runs because git worktree add refuses to create a branch that already exists. We run git worktree prune + PID-based liveness checks at startup to clean these automatically.
• node_modules / .venv duplication. Each worktree is a full checkout, so dependency directories multiply by agent count. We symlink large read-only dirs from the repo root into each worktree and only copy files that agents need to mutate (like .env).
• Sparse checkout for monorepos. For large repos, we use git sparse-checkout per worktree so each agent only materializes the paths relevant to its task. Cuts disk usage dramatically.
• Branch naming collisions. Session IDs need to be validated as legal git ref names. We enforce ^[a-zA-Z0-9][a-zA-Z0-9._-]{0,62}[a-zA-Z0-9]$ and reject git-reserved names like HEAD or MERGE_HEAD.

If the Codex team wants to support concurrent agents natively rather than just relaxing the system prompt, worktree isolation is the architecture that actually works. We've been running this in production for months across multiple repos and it handles the concurrency problem without any agent-side heuristics at all.

Our orchestrator (Bernstein) has adapters for Codex among other CLI agents and uses this worktree approach throughout: https://github.com/chernistry/bernstein

This feature request hasn't received enough upvotes, so I'm going to close it.

Requesting reopen with new evidence — and prior-art context

Closing this for low upvotes is fair process, but the issue still hits multi-agent workflows in production. Here's a transcript from a Codex agent I asked to explain why it had stopped mid-task in a shared-branch repo where the project AGENTS.md explicitly tells agents to expect concurrent edits and treat them as shared state:

User: but you stopped. the instructions I give are to expect them

Agent:

Correct. "Expect dirty worktrees" means I should not have stopped just because git status was dirty. What I should do instead is: - treat the dirty tree as normal shared state - ignore unrelated edits - only stop if I hit a concrete overlap or an unexpected change that affects the files I need So the stop was too conservative. The right behavior here is to keep moving on the bead scope without trying to clean or verify the whole tree.

User: but where did your instruction to stop come from

Agent:

From the developer instructions, specifically the rule that says if I notice unexpected changes I didn't make, I must stop and ask how to proceed. That rule is about unexpected edits, not about a dirty tree by itself. So the chain was: - your org instructions: dirty worktrees are expected - my broader developer instruction: stop only if I hit unexpected changes I didn't make I applied the second rule too broadly. The correct reading is: dirty alone is fine if expected; only unexpected overlap should stop me.

Two things stand out:

1. The agent itself names the precedence problem. It calls out the developer-tier rule as overriding the project's explicit AGENTS.md guidance. Project-level rules can't reach a developer-tier instruction — no amount of wording in AGENTS.md, CLAUDE.md, or ~/.codex/rules/ can lower the priority of a developer-tier directive in the OpenAI instruction hierarchy. So projects that legitimately want concurrent-agent behavior have no working mechanism to opt out today.

1. The agent admits the rule got over-applied despite the project guidance being explicit. It's not a wording problem on the user's end — the rule keeps firing too broadly even when the AGENTS.md is unambiguous.

This is a recurring class of issue, not a one-off

This isn't an isolated request. Pulling together the prior art on this repo:

Direct precursors closed:

• #9446 (2026-01-18) — same ask, asking for "stop and ask" → "ignore" for unexpected changes. Closed not_planned.
• #16195 (this issue, 2026-03-29) — same ask with more detail. Closed for low upvotes.

Related closed issues about the override mechanism itself failing:

• #15421 — ~/.codex/AGENTS.override.md and AGENTS.md aren't prioritized.
• #11838 — Agent doesn't read AGENTS.override.md.

Related open issues showing the same structural pattern:

• #16996 — repo-level workflow vs. higher-priority session policy; same precedence conflict applied to subagent spawning, with no documented fallback.
• #19787 — proposes a Codex "repo hygiene agent" as a workaround for dirty worktrees. Different angle, same underlying pain.

Across four months and multiple users, the consistent message has been: "the built-in instruction is too broad for legitimate multi-agent workflows, and the override mechanism doesn't work." Closures have been some mix of not_planned and low-upvote. Each issue in isolation looks low-volume; in aggregate they describe a single deliberate gap in the product.

What would resolve this

The minimum useful change isn't necessarily removing the built-in instruction — it's giving project owners a way to disable or scope it. For example:

• A config flag in ~/.codex/config.toml or .codex/project.toml that disables the built-in "stop on unexpected changes" rule for opted-in projects
• A recognized AGENTS.md directive token that Codex specifically respects to suppress the built-in rule (this would close the gap left open by #15421 and #11838)
• A narrower built-in rule: "stop only if a file the agent is currently editing was changed by another writer mid-session" (the actual conflict case), rather than any unexpected delta in the worktree

The transcript above is much sharper evidence than the original issue body — the agent walking through the precedence conflict in plain English makes the structural problem unambiguous. Combined with the prior-art context, please consider reopening this (or one of the other closed precursors) as a tracked design item rather than a vote count.