stdio MCP servers accumulate under a live app-server on 26.810.52044, after the #18881 / #19753 shutdown fix

Open 💬 8 comments Opened Aug 17, 2026 by twkjtwkj

💡 Likely answer: A maintainer (github-actions[bot], contributor) responded on this thread — see the highlighted reply below.

Filing separately from the closed #18881 because the shutdown path that PR #19753
fixed now works on this build, while accumulation during a live session does
not. Cross-referencing #12491 (open, GUI), #20349, #25015, #26984.

Environment

| | |
|---|---|
| Codex (ChatGPT.app) | 26.810.52044, bundle 6662 |
| codex-cli | 0.148.0-alpha.9 |
| macOS | 26.6.1 (25G76), arm64 |
| Surface | Codex desktop app, codex ... app-server |

Two stdio MCP servers configured in ~/.codex/config.toml, both launched via a
wrapper script that execs (so the wrapper leaves no process of its own):

[mcp_servers.fin]
command = "/Users/<user>/.ai_ops/bin/mcp-launch.sh"
args = ["fin"]          # execs a python stdio server

[mcp_servers.email]
command = "/Users/<user>/.ai_ops/bin/mcp-launch.sh"
args = ["email"]        # execs `npx -y @codefuturist/email-mcp stdio`

What happens

A long-lived app-server spawns a fresh stdio MCP server roughly every three
minutes and never closes the previous one. The old servers stay parented to the
app-server with their stdio pipes still held open (lsof shows fds 0/1/2 as
PIPE), at 0.0% CPU and ~1s of accumulated CPU time. They are idle but retained.

This is the same shape as #18881, but that issue was closed by PR #19753
(merged 2026-04-28) and this build postdates it.

What PR #19753 did fix, verified here

Shutdown draining works. When the app-server exits, its MCP children go with it.
Observed directly: app-server PID 43065 was restarted, and all ~33 of its
accumulated MCP children terminated along with it, requiring no manual cleanup.

So this is not a regression of the shutdown path. It is the in-session path,
where servers are replaced but the superseded ones are never shut down.

Measurements

Two independent windows, one app-server each, machine otherwise idle:

| | |
|---|---|
| Rate | ~1 new stdio server per 3 minutes, sustained |
| Single app-server (PID 14353), 71 min uptime | 24 fin_mcp children |
| Across both configured servers, ~1 hour after a manual clear from 11 processes | 137 processes |
| RSS at that point | ~5.4 GB |
| Free system memory at that point | 141 MB |
| After killing the superseded children | 4480 MB free |

Reproduced twice, hours apart, across an app-server restart in between.

Reproduction

Configure one or more stdio MCP servers in ~/.codex/config.toml.
Start the Codex desktop app and leave a project session open.
Watch the children of the app-server process:

APPSRV=$(pgrep -f "Resources/codex .*app-server" | head -1)
watch -n 30 "pgrep -P $APPSRV -f mcp | wc -l"

The count climbs monotonically and never decreases while the app-server lives.

Expected

When a stdio MCP server is replaced, the superseded one is shut down and its
process reaped, so the count tracks the number of configured servers rather than
session age.

Notes

Each server is cheap alone (~28 to 90 MB here) and the fault is only visible over
hours. With two servers configured it took roughly one hour to consume several GB
and drive the machine to 141 MB free. #12491 reports the same end state at much
larger scale (1319 processes, 37 GB), which suggests the ceiling is however long
the app stays open.

The three fix directions proposed in #12491 (process groups, startup reaping of
stale trees, heartbeat self-termination) would each cover this case too. The
narrowest fix specific to what is seen here is shutting down the superseded
server at the point of replacement, rather than only at session shutdown, which
is what #19753 addressed.

View original on GitHub ↗

8 Comments

github-actions[bot] contributor · 11 days ago

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

#38754
#38825

Powered by Codex Action

jdcodes1 · 11 days ago

Traced the in-session path on main @ 1f41cc5d92, and the code structure matches your observation precisely: the shutdown that #19753 added exists only on the session-end path; the in-session replacement path has no shutdown at all, and its Drop-based fallback is a no-op for exactly the processes you're seeing accumulate.

1. Replacement never shuts superseded servers down. A refresh goes through McpRuntime::replace → publish, which builds a new McpConnectionSet (reusing previous connections only when their identity matches) and then just swaps the published pointer:

https://github.com/openai/codex/blob/1f41cc5d92722748e45cae9cecc6d883a4e7cbb1/codex-rs/codex-mcp/src/runtime.rs#L183-L231

The proper cleanup — McpConnectionSet::shutdown, which terminates each stdio child (connection_manager.rs#L813-L828) — is invoked from exactly one place: McpRuntime::shutdown (runtime.rs#L441), i.e. the path #19753 fixed. Superseded connections on the replace path are simply dropped.

2. The Drop fallback cannot kill a started server. Drop for McpServerConnection only cancels the startup token:

https://github.com/openai/codex/blob/1f41cc5d92722748e45cae9cecc6d883a4e7cbb1/codex-rs/codex-mcp/src/connection_manager.rs#L148-L152

and that token only guards the startup future (.or_cancel(...) in rmcp_client.rs#L370). Once startup_complete is set, cancelling it does nothing. Actual child termination happens in StdioServerProcessHandleInner::drop / terminate() (stdio_server_launcher.rs#L497+), which requires the last Arc<RmcpClient> clone to go away — and clones live inside the Shared startup future, cached bindings, and tool-catalog plumbing. Your lsof evidence (superseded children with all three stdio pipes still open in the live app-server) demonstrates that some clone does survive in practice, so the children are never terminated until process exit. This is why the leak is monotonic while the app-server lives yet fully drains on restart.

3. Why a fresh server spawns at all (the ~3-minute cadence). Reuse requires the new McpServerConnectionIdentity to equal the old one, and the identity includes volatile inputs — notably the raw auth token (runtime_auth_token: Option<String>) and the resolved values of referenced environment variables:

https://github.com/openai/codex/blob/1f41cc5d92722748e45cae9cecc6d883a4e7cbb1/codex-rs/codex-mcp/src/server.rs#L95-L110

Meanwhile the session's MCP prewarm worker marks the runtime dirty and re-publishes on every auth-change tick of the auth manager's watch channel (start_mcp_prewarm_worker in core/src/session/mcp_prewarm.rs). So: periodic token refresh → watch tick → refresh → identity mismatch (token string changed) → brand-new child, old one orphaned by (1)+(2). A testable prediction for your setup: stat -f %m ~/.codex/auth.json should advance at the same ~3-minute cadence as new fin_mcp children appear. If it does, that's the trigger confirmed end-to-end.

Fix outline.

In publish, diff the previous set against the new one and explicitly shutdown() every previous connection that wasn't carried over — the in-session analog of #19753, ideally spawned as a detached task like set-level shutdown already does so an interrupt can't cancel cleanup.
Harden Drop for McpServerConnection to schedule a full shutdown() for startup-complete clients instead of only cancelling the startup token, so no future refactor can silently reintroduce the leak.
Reduce the churn itself: for stdio servers the spawned child's command/env doesn't change when the ambient auth token rotates, so the token belongs in the identity only for transports that actually consume it (HTTP bearer, Codex Apps). Comparing spawn-relevant inputs only would make token rotation reuse the existing child.
Log respawns with the identity field that differed — this issue took real forensics to see; a single info! line would have surfaced it immediately.

twkjtwkj · 10 days ago

Windows 11 repro, Codex desktop 26.810.7004.0 (ChatGPT.exe hosting codex.exe), 8 stdio servers, ChatGPT auth.

Fresh boot, one Codex thread, no authentication rotation or config changes. Codex started at 11:12:04. By 11:25:46, the same codex.exe had 70 MCP descendant processes using 1,924 MB. It retained three chains each for Azure DevOps, ServiceNow, and Power BI, plus six Snowflake chains for two configured Snowflake servers. This indicates startup created and retained three complete MCP server sets for a single thread.

Earlier the same day, over a 35-minute watched run of MCP tool calls: auth.json mtime never changed (Aug 10 17:42 throughout), a fourth complete set spawned at the next turn boundary after a config.toml edit was reverted, and one superseded set stayed alive beside the replacement for the rest of the run. So on this platform the leak reproduces without any auth change; each publish leaves the previous set alive, matching point (1). Point (3) alone would not prevent it.

ryofukutani · 9 days ago

macOS最新版でも同じlive app-server内のMCPランタイム蓄積を再現しました。

環境:

ChatGPT.app 26.814.41407
bundled codex-cli 0.148.0-alpha.15
macOS 26.5.2 arm64
32GB Apple Silicon Mac

観測結果（2026-08-19 JST）:

ChatGPT.app本体は約360MBだが、子のcodex ... app-server --analytics-default-enabledはRSS最大約5.1GB、physical footprint peak 4.7GB。
app-server配下の直接子プロセスは最大53個で、node_repl、gitnexus、Slack、Context7、Pencil、mem0などの同じstdio MCP群が複数世代存在した。
各MCPプロセスは個別にはほぼidle（CPU 0%前後）だが、同じapp-serverの下に時間差で4グループ以上残っていた。
app-serverは/Volumes/RAID/cache/codex_sessionsのrollout JSONLを複数開いており、2026-07-23開始のファイルが約2.03GiBまで増加し、調査中も追記されていた。
app-serverのCPUは100〜300%に達し、システムswapは92〜98%使用、残量は約0.5〜1.3GiBまで低下した。
app-serverを再起動すると、蓄積したMCPプロセス群はまとめて消えるため、MCP個別実装の常時CPUリークというより、live app-serverのthread/MCP runtime保持と履歴再水和の問題と考えている。

補足:

ChatGPT.app内蔵のapp-serverバイナリはMach-O arm64で、rustc/tokioの痕跡があり、Rust化は根本対策にならない。
期待動作は、threadがinactive/archive/unsubscribeになった時点でMCP runtimeだけをdetach/shutdownし、履歴は保持してもstdioプロセスと巨大履歴の再水和を無制限に保持しないこと。
既存の#38925にある「superseded connectionの明示shutdown」「spawn-relevant identityだけで再利用」「idle runtimeの上限/TTL」という修正案と一致する。

再現時のcredential・会話本文は共有していません。

(codex-lead レーンによる報告です)

ryofukutani · 9 days ago

追加のアプリ層比較です（同一macOSホスト上の読み取り専用プロセス集計）。

ChatGPT.app本体の子孫: 112プロセス、合計RSS約9.1GB
そのapp-server単体の子孫: 102プロセス、合計RSS約7.7GB
Claude.app本体の子孫: 12プロセス、合計RSS約310MB
Claude側のCLI/MCPは別プロセスツリーとして分離され、ChatGPTのapp-serverのように全スレッドのMCPを一つの長寿命親へ集約していない

この比較から、個別MCPの実装だけではChatGPT.appだけで肥大する差を説明できません。ChatGPT.appのアプリ層が、複数threadのruntimeとMCP stdioプロセスを単一の長寿命app-serverへ保持する設計・ライフサイクルが増幅要因です。

期待する修正境界は、GUIのrenderer最適化ではなく、app-serverのthread unload / unsubscribe / MCP runtime ownershipです。論理threadのresume用状態を保持しても、MCP runtimeとstdio子プロセスは切り離して、必要時にlazy再生成できるべきです。

(codex-lead レーンによる追加監査です)

dingaiminGIT · 8 days ago

Cross-reference from #37453: I reproduced the restart/resume amplification on macOS and posted a tested lifecycle design and before/after PID evidence here: https://github.com/openai/codex/issues/37453#issuecomment-5353125274

The overlap with this issue is the retention of superseded or idle local stdio connection generations. In the prototype, turn completion publishes a dormant replacement first, allows in-flight bindings to retain the old connection set, and shuts down only host-local stdio transports after those references drain. HTTP MCPs, remote transports, Codex Apps, and active calls are preserved.

I am keeping the detailed discussion in #37453 to avoid duplicating the same analysis across both issues.

lazforprez · 7 days ago

Ran a full forensic pass on this on macOS 26.5.2, desktop 26.814.41407 / CLI 0.148.0, after a month of recurring Too many open files (os error 24) outages (11 distinct days). Findings that may help triage:

The failing process is the shared codex app-server. Its fd numbers capped at exactly 255 (lsof) while holding 230–234 descriptors — i.e. it filled a 256 soft-limit table. 126 of 234 were pipes; 13 node_repl helpers were alive for 3 in-progress threads, ages spread over 74 minutes — per-turn spawn without reap, exactly as described here.
All 650 "too many open files" lines in 48h of unified log came from that one PID. Kernel file table was at 8,737 of 122,880 during a live failure — not system-wide.
Confirming the earlier observation in this thread: killing the app-server took its whole accumulated helper tree down with it, and a fresh instance re-hit the 256 ceiling within ~1.5 h of ordinary use.
Nothing between rust-v0.148.0 and rust-v0.149.0-alpha.7 touches fd limits or MCP reaping (commit sweep), and there is no RLIMIT_NOFILE handling anywhere in the codebase.

Workaround (installed today; immediate verification passed — raised limit confirmed end to end, zero errors since restart — but no long-term soak yet): a launcher wrapper at ~/.local/bin/codex doing ulimit -S -n 32768 before exec'ing the real binary (the SSH-workspace bootstrap resolves codex via PATH, so the app-server inherits it), plus a small watcher for when an update rewrites the symlink. Write-up, verification method, and scripts: https://github.com/lazforprez/codex-fd-exhaustion-fix

Two asks for maintainers: reap superseded MCP helpers, and raise the soft limit at app-server startup (kern.maxfilesperproc is 61440 by default — 256 is the launchd floor, not a real constraint).

lordisp · 1 day ago

Another surface for the same in-session accumulation: the standalone codex app-server launched by the Claude Code companion plugin (openai/codex-plugin-cc), i.e. no ChatGPT.app involved.

Environment: codex-cli 0.147.0 (npm), macOS 26.6.2 (25G83) arm64, 36 GB RAM, 8 stdio MCP servers in ~/.codex/config.toml.

The plugin keeps a broker (app-server-broker.mjs, orphaned under launchd after the originating session ended) that holds one long-lived codex app-server. Observed after ~26 h of uptime:

~130 generations of each configured stdio server (roughly one new full set every 12 minutes, sustained), all still parented to the app-server, idle.
1,015 live direct children under the single app-server PID.
At the point macOS raised its "system has run out of application memory" dialog, the JetsamEvent report (2026-08-27 08:48 CEST) showed 1,640 node processes at 72.6 GB resident (lifetimeMax sum 114 GB); swap was at 42.4 of 44 GB.

Confirming the shutdown-path observation from the OP on this surface too: killing the app-server took the whole accumulated tree down with it; swap dropped from 42.4 GB to 3.7 GB immediately, free memory went from 38 % to 65 %.

One triage note: the fd-limit wrapper from lazforprez/codex-fd-exhaustion-fix does not help this manifestation — memory, not descriptors, was the binding constraint here, so raising the fd ceiling only extends how long the accumulation can run before the OOM dialog appears.