Stale app-server survives CLI update and causes 0.147.0 TUI to use pre-0.147 `spawn_agent` model filtering
What version of Codex CLI is running?
codex-cli 0.147.0
Platform
Ubuntu 24.04.4 LTS, Linux x86_64.
Codex installation:
running standalone on linux-x86_64
current executable:
/home/keyhr/.codex/packages/standalone/releases/0.147.0-x86_64-unknown-linux-musl/bin/codex
version:
0.147.0
The SHA256 of the installed binary exactly matches the rust-v0.147.0 GitHub release binary:
cb0a15567e9a60a5820d54b0f6ae86d504dc3805c1eab21a47f70e3eb7b73a40 installed codex
cb0a15567e9a60a5820d54b0f6ae86d504dc3805c1eab21a47f70e3eb7b73a40 rust-v0.147.0 release binary
What issue are you seeing?
After updating Codex CLI from 0.146.0 to 0.147.0, a normal TUI session still behaves as if it is using the pre-0.147 multi-agent model filtering.
Specifically, from a gpt-5.6-sol parent, spawning gpt-5.6-luna fails:
Unknown model gpt-5.6-luna for spawn_agent.
Available models: gpt-5.6-sol, gpt-5.6-terra
However, starting the same 0.147.0 binary with:
codex --disable tui_app_server -m gpt-5.6-sol
makes the exact same Luna spawn succeed:
Started `/root/luna_spawn_check`
...
LUNA_SPAWN_OK
This appears to be caused by a stale 0.146.0 app-server surviving the CLI update and being reused by the normal TUI path.
Process inspection shows:
PID=12089
codex -c features.code_mode_host=true app-server --listen unix://
/home/keyhr/.codex/packages/standalone/releases/0.146.0-x86_64-unknown-linux-musl/bin/codex
while the currently installed CLI is 0.147.0.
There is also a 0.147.0 app-server proxy process:
codex app-server proxy
/home/keyhr/.codex/packages/standalone/releases/0.147.0-x86_64-unknown-linux-musl/bin/codex
So the system can end up with a 0.147.0 frontend/proxy while a 0.146.0 app-server remains alive.
Why this is particularly visible in 0.147.0
rust-v0.147.0 includes #36892, "Support leaf models in multi-agent v2".
Before that change, a V2 parent such as gpt-5.6-sol rejected the V1/leaf gpt-5.6-luna model. After #36892, Luna should be accepted as a leaf worker.
The effective model catalog on the affected machine is correct:
gpt-5.6-sol list v2 true
gpt-5.6-terra list v2 true
gpt-5.6-luna list v1 true
The bundled model catalog reports the same values.
So this is not a stale model catalog.
The stale app-server makes a 0.147.0 CLI exhibit exactly the old behavior:
Available models: gpt-5.6-sol, gpt-5.6-terra
Reproduction
- Have Codex
0.146.0installed and an app-server running.
- Upgrade Codex to
0.147.0without manually terminating the existing app-server.
- Verify the shell CLI is updated:
codex --version
# codex-cli 0.147.0
- Start a normal Sol session:
codex -m gpt-5.6-sol
- Call:
spawn_agent(
model="gpt-5.6-luna",
fork_turns="none",
reasoning_effort="low",
message="Return exactly LUNA_SPAWN_OK"
)
- Observe:
Unknown model gpt-5.6-luna for spawn_agent.
Available models: gpt-5.6-sol, gpt-5.6-terra
- Inspect running processes:
for pid in $(pgrep -f 'codex.*app-server'); do
echo "PID=$pid"
tr '\0' ' ' < /proc/$pid/cmdline
echo
readlink -f /proc/$pid/exe
done
- Observe that an app-server from the old 0.146.0 installation is still running.
- Start the same installed 0.147.0 binary while bypassing the app-server-backed TUI:
codex --disable tui_app_server -m gpt-5.6-sol
- Repeat exactly the same
spawn_agentcall.
- It succeeds:
LUNA_SPAWN_OK
Additional A/B verification
On macOS a Homebrew-installed Codex 0.147.0 successfully spawns Luna from a Sol parent.
The parent session log contains:
{
"name": "spawn_agent",
"arguments": {
"model": "gpt-5.6-luna",
"reasoning_effort": "low"
}
}
and the resulting child session confirms:
{
"model": "gpt-5.6-luna",
"effort": "low"
}
So this is not a silent fallback to Sol.
On the affected Linux machine, a local source build of rust-v0.147.0 also fails through the normal TUI path, but the official 0.147.0 binary succeeds when tui_app_server is disabled. This further points to app-server reuse/version skew rather than the 0.147.0 executable itself.
Feature state
Both working and failing environments report:
multi_agent stable true
multi_agent_v2 stable false
so the difference is not explained by that feature flag.
Expected behavior
When the active CLI is upgraded, the normal TUI should not silently execute turns through an incompatible older app-server.
At minimum, Codex should detect:
CLI version != app-server version
and either:
- restart/replace the stale app-server automatically,
- refuse to reuse it and start a matching app-server, or
- show a clear warning that the active session is running against a different app-server version.
A user seeing:
OpenAI Codex (v0.147.0)
should not unknowingly get 0.146.0 backend behavior.
Workaround
This immediately avoids the problem:
codex --disable tui_app_server
Manually terminating the stale 0.146.0 app-server and allowing a current one to start should also resolve the version skew.
Related issues / changes
- #34721 — old app-server/remote-control daemon survives a CLI update
- #24034 — active CLI and app-server can resolve to different versions
- #36892 — 0.147.0 adds leaf-model support for Multi-Agent V2
This may share the same root cause as #34721 / #24034, but this report demonstrates a concrete functional regression in a normal TUI session: a stale app-server causes a feature added in the currently displayed CLI version to remain unavailable.
2 Comments
Cross-reference from Windows Desktop: I can reproduce the same Sol/Terra-only Luna rejection after explicitly ruling out a stale 0.146 app-server.
On one Windows 11 host/account:
OpenAI.Codex 26.803.5235.0client_version = 0.147.0list/v2, Terralist/v2, Lunalist/v1model_catalog_jsonoverrideChatGPT.exeandcodex.exeprocesses were force-terminated and verified absentDesktop still reports:
By contrast, standalone
codex-cli 0.147.0on the same machine successfully spawns Luna from Sol using the same official catalog.I posted the full A/B reproduction in #34964. This suggests the stale-server mechanism documented here is a real cause of the symptom, but not the only one: Windows Desktop 0.147's app-server/spawn path can still exhibit the pre-#36892 allowlist behavior even after a complete process teardown/restart with a freshly generated 0.147 catalog.
A particularly strong clue is that a local catalog copy with only Luna's
multi_agent_versionchangedv1 -> v2makes Desktop spawning succeed again. That points toward an app-server/Desktop-specific model-filtering path still requiring exact V2 metadata rather than accepting visible V1 models as leaf workers.This is the persistent-daemon version-skew family: the TUI you launch is 0.147.0, but it connects to the long-lived
codex app-serverdaemon over the control socket — and that daemon keeps executing whatever binary it was started from until something restarts it. An update replacespackages/standalone/currenton disk but doesn't restart a running daemon, sospawn_agentmodel filtering (server-side logic in the daemon process) stays pre-0.147 while your shell'scodex --versionsays otherwise. Same underlying mechanism as #37754 (old daemon servinglist_turnserrors after update) — and #37893 documents the worst case, where the daemon becomes unmanageable so evencodex app-server daemon restartcan't fix it ("running but not managed").Immediate workaround:
codex app-server daemon restart(or kill thecodex app-serverprocess) after upgrading — your spawn should then acceptgpt-5.6-luna.Repo-side fix worth centralizing across these issues: on connect, the client already probes the daemon's version over the socket; when it doesn't match the client binary, either auto-restart the daemon (safe when no live turns) or at minimum surface "daemon is running 0.146.0, restart to apply the update" instead of letting stale behavior masquerade as current-version bugs. Verifying
probe_app_server_version(app-server-daemon/src/lib.rs#L80) against the client version at TUI bootstrap is the one-line detection.