Shared models_cache.json oscillates between originator-specific catalogs across app-server processes

Open 💬 3 comments Opened Jul 16, 2026 by nerdchanii
💡 Likely answer: A maintainer (github-actions[bot], contributor) responded on this thread — see the highlighted reply below.

What version of Codex CLI is running?

codex-cli 0.144.5

What subscription do you have?

ChatGPT authentication. The exact subscription tier is not relevant to the
reproduction.

Which model were you using?

gpt-5.6-sol

What platform is your computer?

macOS 26.5.2, Darwin 25.5.0, Apple Silicon.

What issue are you seeing?

Two long-lived app-server processes built from the exact same Codex binary and using the same CODEX_HOME, provider, auth mode, and ChatGPT account receive different model catalogs when only their app-server originator differs.

Both processes persist their originator-specific response to the same global $CODEX_HOME/models_cache.json. The file therefore alternates between two
valid, but semantically incompatible, catalogs:

  • Codex Desktop / codex_cli_rs: gpt-5.6-sol has

multi_agent_version = v1 and tool_mode = direct.

  • codex-chrome-extension-sidepanel: the same model has

multi_agent_version = v2 and tool_mode = code_mode_only.

The app-server background refresh worker repeats the overwrite approximately
every three minutes. Depending on which process wrote last, newly started
sessions and runtime metadata consumers can see a different multi-agent/tool
configuration.

This is not a corrupt server response and is not explained by a CLI version,
provider, or account mismatch. It is a cache-identity and cross-process storage
problem.

Confirmed A/B measurements

The two app-server processes had:

  • the same executable SHA-256;
  • Codex CLI 0.144.5;
  • the same CODEX_HOME, auth file, provider, token, account, request URL, and

User-Agent;

  • different app-server client identities/originators.

Direct /models?client_version=0.144.5 requests changing only originator
returned:

| Originator | Models ETag | Sol multi-agent | Sol tool mode |
| --- | --- | --- | --- |
| Codex Desktop | W/"6cbd9267ee12889ea6614486cc5879ff" | v1 | direct |
| codex_cli_rs | W/"6cbd9267ee12889ea6614486cc5879ff" | v1 | direct |
| codex-chrome-extension-sidepanel | W/"68e28197d956a749f12435108b07ff25" | v2 | code_mode_only |

Observed cache writes included:

| Time (UTC) | Writer | Request ID | Result |
| --- | --- | --- | --- |
| 11:33:36 | Codex Desktop | 25cc13a9-0991-4bc8-a5e6-ed9a125bd759 | V1/direct catalog written |
| 11:34:03 | Chrome sidepanel | bb12724f-9a1d-4ef3-a256-1fa3c9fe3b1e | V2/code-mode catalog written |
| about 3 minutes later | Chrome sidepanel | cfaa5c42-34e5-4600-8e1e-73df23bb9132 | the alternating pattern repeated |

No token, account identifier, raw session transcript, or auth file is included
in this report.

What steps can reproduce the bug?

Reproduction A: two app-server clients
  1. Use one Codex binary, one CODEX_HOME, and one ChatGPT account.
  2. Start app-server A and initialize it with

clientInfo.name = "Codex Desktop".

  1. Start app-server B and initialize it with

clientInfo.name = "codex-chrome-extension-sidepanel".

  1. Request the model catalog from both processes.
  2. Record the request originator, ETag, and the gpt-5.6-sol fields

multi_agent_version, tool_mode, and use_responses_lite.

  1. Observe that the server returns different ETags/catalog metadata.
  2. Observe both processes writing to the same

$CODEX_HOME/models_cache.json.

  1. Wait for multiple app-server refresh intervals and observe the file

alternating again.

Reproduction B: Desktop plus Chrome extension
  1. Fully quit ChatGPT Desktop and Chrome.
  2. Start ChatGPT Desktop and create a Codex task.
  3. Record app-server ancestry, initialize identity, /models originator, ETag,

and the cache hash/content.

  1. Start Chrome and open the ChatGPT extension side panel.
  2. Confirm that the extension's native host starts another Codex app-server.
  3. Record the same data and monitor models_cache.json.
  4. Close only the side panel, then Chrome, and repeat to determine the precise

native-host lifecycle.

In the observed run, the second app-server was a child of the ChatGPT for
Chrome native extension host, not a browser tab merely visiting chatgpt.com.

Reproduction C: deterministic regression test

Use two model managers sharing one temporary CODEX_HOME. Mock /models so
the same provider/account/version returns catalog A for originator A and
catalog B for originator B. Alternate online refreshes, drop both managers,
recreate them offline, and verify that each surface can recover its own
catalog. Current single-file behavior cannot preserve both.

Root cause

The request identity and cache identity do not match.

/models uses the app-server originator, and the service can return a different
catalog and ETag for each client surface. The persisted cache is nevertheless a
single global file. Cache eligibility on current main does not include
originator/client surface, and the file is last-writer-wins.

There are three interacting layers:

  1. Semantic isolation: originator-specific catalogs share one cache slot.
  2. Cross-process coordination: independent app-servers have separate

in-memory ETag/catalog state but overwrite the same disk file.

  1. Write safety/amplification: the cache is rewritten as a complete file,

including matching-ETag TTL renewal, without a per-identity slot. Related
reports also show truncated/EOF reads and excessive write churn.

The app-server refresh worker makes the latent design issue persistent: each
long-lived process refreshes immediately and then approximately every three
minutes.

How the code reached this state

  • #7722 introduced one disk cache, a five-minute TTL, and ETag storage.
  • #8491 added refresh on a model ETag mismatch.
  • #8873 and #8988 made app-server clientInfo.name the process originator,

allowing /models requests to represent the actual client surface.

  • #9174 renews cache freshness on matching ETags by rewriting the cache.
  • #10414 rejects caches from another client version, but does not preserve

multiple producer identities.

  • #18950 moved model discovery under providers. A Codex P1 review explicitly

warned that cache eligibility needed provider identity or separate cache
files. The PR was subsequently approved and merged without adding that
isolation, leaving the single-slot design debt in place:
https://github.com/openai/codex/pull/18950#pullrequestreview-4158982727

  • #28699 added immediate and three-minute app-server background refreshes to

avoid cold model-list latency. That change did not create the originator
split, but it turns occasional last-writer-wins contamination into a
recurring oscillation.

  • #30984 is a current draft that adds provider_id, auth_mode, and

account_id to cache eligibility. It does not include originator and still
uses a single file, so both producers in this reproduction have the same key
and one surface's catalog still evicts the other's.

Related reports:

  • #32482 and #33146: a different runtime/version overwrites the shared cache.
  • #32496: matching-ETag TTL renewal causes high write volume.
  • #30864: truncated/EOF cache reads.
  • #32850: a running UI and a fresh catalog disagree.
  • #33575 and #33547: model-visible MCP tools disappear after runtime metadata

changes.

  • #32086: a separate downstream bug where Responses Lite makes deferred V1

tools unreachable because tool_search is not exposed correctly.

The last distinction matters: cache isolation explains why a surface/session
can unexpectedly receive V1 versus V2 metadata. If the server intentionally
returns V1/Responses Lite to a surface, #32086 still needs its own fix.

What is the expected behavior?

Different catalog identities must not overwrite or reuse one another's model
metadata. A client surface should consistently receive and recover the catalog
that was fetched for its own request identity, including after process restart
or while temporarily offline.

Suggested fix

Treat the catalog scope as a first-class identity derived from the same
snapshot used to make the request:

provider_id
endpoint identity/fingerprint
effective auth mode
account/principal
exact originator/client surface
client-version compatibility

The smallest robust storage change is independent per-identity cache slots,
preferably separate files keyed by a hash of that tuple. Each slot should own
its catalog, ETag, and freshness. Storing the account ID directly in the
filename is unnecessary.

Also recommended:

  • snapshot auth and originator once per fetch, then use that snapshot for both

the HTTP request and cache key;

  • start the refresh worker only after the app-server initialize identity is

fixed;

  • use atomic temporary-file replacement;
  • use a same-key cross-process lock or another conflict-safe strategy;
  • avoid rewriting the full catalog for every equal-ETag freshness renewal,

for example by using a small freshness sidecar or coalescing writes;

  • keep in-memory catalog/ETag state partitioned by the same identity.

Adding only an originator validation field to the existing single file would prevent applying a mismatched catalog, but it would still cause eviction, repeated network fetches, and loss of the previous surface's offline state.

Validation plan

Please include regression coverage for:

  1. Same binary/provider/account/version, two originators, different ETags/catalogs.
  2. Alternating online writes in one CODEX_HOME.
  3. Dropping and recreating both managers, followed by offline lookup for each originator.
  4. Multiple background-worker cycles.
  5. Concurrent writers always leaving parseable, complete JSON.
  6. Failure injection during atomic replacement.
  7. Matching-ETag renewal affecting only the correct identity and obeying a write-rate bound.
  8. Initialization races where a worker starts before client identity is known.
  9. A separate #32086 regression proving that V1/Responses Lite retains a usable top-level tool_search route.

Why this is a separate root-cause report

Existing issues cover symptoms or adjacent dimensions: V1 tool serialization, MCP tools disappearing, version skew, provider/account contamination, write amplification, and truncated files. This reproduction isolates a missing cache dimension that is not covered by those reports: the exact same binary, provider, auth mode, account, and client version receive different catalogs
solely because their originators differ, then continuously overwrite one shared cache file.

View original on GitHub ↗

3 Comments

github-actions[bot] contributor · 1 month ago

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

  • #32482

Powered by Codex Action

nerdchanii · 1 month ago

Follow-up: this is not an app-server version mismatch

I rechecked the two live processes and their executable files directly.

| Surface | Executable | Reported version | SHA-256 |
| --- | --- | --- | --- |
| Codex Desktop | /Applications/ChatGPT.app/Contents/Resources/codex | codex-cli 0.144.5 | bdcb530615d44fcc7b35d12fe00f30c3025c25fc22a21193591dcdb064304385 |
| Chrome extension native host | ~/.codex/plugins/.plugin-appserver/codex | codex-cli 0.144.5 | bdcb530615d44fcc7b35d12fe00f30c3025c25fc22a21193591dcdb064304385 |

cmp also reports that the two executable files are byte-for-byte identical. Both processes use the same ~/.codex/models_cache.json.

The catalog is returned by the remote ChatGPT Codex backend endpoint:

GET https://chatgpt.com/backend-api/codex/models?client_version=0.144.5

This is not a locally generated catalog and it is not a request to api.openai.com. The observed HTTP response is served through the chatgpt.com backend (the edge response identifies Cloudflare). The exact internal OpenAI service name behind that endpoint is not visible from client
logs.

The live sequence makes the identity mismatch explicit:

  1. At 2026-07-16 13:13:16 UTC, Chrome app-server PID 60805 requested

/models?client_version=0.144.5 and received:

``text
ETag: W/"68e28197d956a749f12435108b07ff25"
``

This is the Chrome-sidepanel V2 / code_mode_only catalog.

  1. At 13:13:29 UTC, Desktop app-server PID 57870 loaded that freshly written cache. It accepted it because the cached and expected client_version were both 0.144.5, then applied ETag 68e28197....
  1. At 13:13:30 UTC, Desktop requested the same URL with the same client_version and received:

``text
ETag: W/"6cbd9267ee12889ea6614486cc5879ff"
``

This is the Desktop V1 / direct catalog, which was then persisted back to
the same cache file.

The material request difference is the client originator:

Codex Desktop
codex-chrome-extension-sidepanel

Therefore multi_agent_version = v1 versus v2 is not the version of the app-server executable. It is server-provided model runtime metadata inside two originator-specific catalogs.

The failure condition is consequently:

same app-server binary and client_version
+ different originator
-> different server catalog and ETag
+ one cache slot validated only by client_version
-> cross-surface catalog reuse and last-writer-wins oscillation

If the two processes received identical catalog content and ETags, sharing the cache would not create this semantic contamination. Here, however, executable version equality does not protect the cache because originator affects the server response but is absent from the cache identity.

nerdchanii · 1 month ago

I put together a reference implementation of the originator-scoped cache described above:

https://github.com/nerdchanii/codex/tree/models-cache-originator-scope (single commit: nerdchanii/codex@78060a73ddc220139bd8f5bbc712db66304c55cf)

What it does:

  • Stores the cache per originator at CODEX_HOME/models_cache/<sha256(originator)>.json
  • Records the originator inside each cache entry and rejects legacy or mismatched entries on load and TTL renewal
  • Writes via a temp file and atomic rename
  • Adds a regression test where two managers with different originators keep independent catalogs across restarts (targeted model-manager and core TTL tests pass)

Known limitations, intentionally out of scope for this PoC:

  • No migration or cleanup of the legacy models_cache.json
  • No cross-process locking for same-originator read-modify-write
  • The full-file rewrite on TTL renewal is unchanged
  • Originator alone is not the complete identity — provider, auth mode and account still belong in the key. Long term this probably folds into the ModelsCacheKey proposed in #30984 rather than living as a separate path scheme.

Happy to turn this into a PR if the direction looks reasonable to the maintainers.