🩹 Codex app-server leaks MCP process stacks for subagents, causing linear memory growth on Linux

The failure mode here looks less like one memory bug and more like missing admission control for helper lifetimes.

A practical split that has worked better for us is:

• tool inventory can be inherited or cached cheaply
• helper process startup should require a live lease
• the lease should name owning_thread, owning_subagent, last_tool_call_at, and idle_timeout_at
• teardown should key off lease expiry, not off whether the broader app-server is still alive

That matters because operators usually need two different answers:

1. why did this helper start at all?
2. why is it still admitted to stay alive?

Without those receipts, every quiet helper stack looks like a leak, even when some are just over-scoped lifecycle decisions. We ran into the same boundary in MartinLoop, and making helper admission explicit reduced both real leaks and false-positive debugging.

Confirming the same leak on macOS — the mechanism described here is not Linux-specific. Evidence below from the ChatGPT desktop app's bundled app-server.

Environment

• macOS (Darwin 25.5.0), Apple M5 Max, 128 GB RAM
• ChatGPT.app Codex build 26.707.62119, running codex -c features.code_mode_host=true app-server
• 3 stdio MCP servers configured: chrome-devtools-mcp, @upstash/context7-mcp, xcodebuildmcp (via build-ios-apps plugin), all spawned as npx <pkg>@latest

Observation (app-server pid 55230, uptime ~47 min at capture)

Six complete generations of MCP server sets were alive simultaneously as children of the app-server — apparently one per thread/subagent, never reaped after completion. Three sets were spawned within 7 seconds of each other (parallel subagents):

55230  ChatGPT.app → codex app-server            (uptime 47 min)
  ├─ set 1: npm exec xcodebuildmcp@latest + @upstash/context7-mcp + chrome-devtools-mcp@latest   (age 38:38)
  ├─ set 2: same triplet                                                                          (age 38:34)
  ├─ set 3: same triplet                                                                          (age 38:31)
  ├─ set 4: same triplet                                                                          (age 22:08)
  ├─ set 5: same triplet                                                                          (age 17:10)
  └─ set 6: same triplet                                                                          (age 16:07)

Each set = 3 npm exec wrapper processes + 3 actual server processes ≈ 600–900 MB RSS. At capture the machine had 52 node processes totalling ~5 GB RSS, the majority of them these leaked sets; swap was at 3.2/4 GB.

Related datapoint: a standalone @openai/codex CLI process (npm install, v24.13.1 node) had been orphaned headless (PPID 1, no TTY, idle) for 9.7 days, holding its full MCP server set alive the entire time — consistent with #28794 and the accumulation reported for macOS in #25744.

Impact beyond memory: because each set is spawned via npx <pkg>@latest, every leaked-then-respawned generation also performs a fresh npm registry resolution, adding CPU burst and network churn on every thread start.

Workarounds we verified: quitting the ChatGPT app reaps all leaked sets; pinning MCP commands to globally-installed binaries (instead of npx @latest) reduces per-set cost and spawn time but does not stop the leak itself.

Deterministic root-cause closure on current macOS Desktop build

I reproduced and source-traced this on the current macOS Desktop package. The visible symptom contains two separate lifecycle failures:

1. Primary / deterministic: completed or unsubscribed thread sessions remain resident, and every resident session owns an eagerly started full MCP process set. This creates unbounded aggregate process/RSS growth across many root tasks, even if process termination itself never fails.
2. Secondary / observed: when a real shutdown is finally attempted, macOS killpg(SIGTERM) can return EPERM; the current local-process terminator then suppresses escalation and permanently marks the handle terminated, so that MCP group may survive.

This is a resource/lifecycle leak (unbounded retention), not evidence of a single allocator leak inside one Node process.

Environment

• macOS 26.5.2 (25F84), Apple silicon / arm64
• ChatGPT/Codex Desktop 26.715.21425, build 5488
• Embedded codex-cli 0.145.0-alpha.18
• Exact release source: tag rust-v0.145.0-alpha.18, commit f84f9a6406cc55b210395f71b4c6aed236fc7ebb

Source-level causal chain

A. Every thread/session eagerly starts its own MCP stack

Session owns a session-local MCP manager. The source even contains a TODO to extract session state for sharing between subagents: session.rs L1035-L1100.

During session construction, Codex calls McpConnectionManager::new(...) before any model turn or MCP tool call: session.rs L1211-L1263. The manager iterates all enabled servers and starts them eagerly: connection_manager.rs L128-L225.

Therefore the cost is:

loaded root sessions + resident subagent sessions × all enabled stdio MCP servers

This happens even when an agent never invokes an MCP tool.

B. V1 completion does not end the child session

The V1 completion watcher waits for a final status and notifies the parent, but it does not call shutdown_live_agent, remove_thread, or release the registry entry: control.rs L431-L518.

Actual V1 close_agent is a different path and explicitly sends Op::Shutdown, waits, removes the thread, forgets residency, and releases the registry entry: legacy.rs L3-L25.

So “agent completed” and “agent session/MCP runtime shut down” are not the same event.

C. V2 intentionally keeps completed agents resident

V2 stores resident subagents in an AgentControl-scoped VecDeque. It shuts down an unloadable completed/errored/interrupted agent only when a later reservation finds that this root session's residency capacity is full and LRU eviction runs: residency.rs L79-L149, residency.rs L216-L230.

The cap is derived from max_concurrent_threads_per_session, not a global Desktop/app-server process or memory budget: config/mod.rs L1462-L1475.

This behavior was intentionally introduced by PR #26632. PR #26994 then renamed the V2 model-facing close_agent operation to interrupt_agent specifically so the agent remains registered and reusable.

The design is coherent for durable logical identity, but it accidentally couples logical durability to a heavyweight live MCP runtime. With many root tasks, each root can remain under its per-session cap while the app-server's global process/RSS total grows without a bound.

D. Desktop unsubscribe also retains the whole session for 30 minutes

The app-server contract explicitly says that thread/unsubscribe keeps the thread loaded for 30 minutes after the last subscriber/activity: app-server README L162.

Keeping lightweight thread state resident may be reasonable; keeping every session's eager stdio MCP processes resident for the same window is the expensive coupling.

Isolated black-box reproduction

I used a separate temporary CODEX_HOME and a separate app-server. I did not touch the user's live app-server or tasks.

Configuration:

• plugins, apps, and remote plugins disabled
• exactly one bundled local stdio MCP server enabled
• no model turn and no MCP tool invocation
• five new ephemeral threads

Observed direct children of the isolated app-server:

| Step | Loaded threads | MCP child processes | MCP RSS |
|---|---:|---:|---:|
| Baseline | 0 | 0 | 0 |
| Start one thread | 1 | 1 | ~35 MiB |
| Start five threads | 5 | 5 | ~170-174 MiB |
| Unsubscribe all five; wait and re-list | 5 | 5 | ~170 MiB |

All five thread/unsubscribe calls returned status: "unsubscribed". A subsequent thread/loaded/list still returned all five IDs, and all five MCP PIDs remained alive.

Control test: I started one additional persisted thread, producing a sixth MCP child. thread/delete changed that thread to notLoaded and immediately reaped exactly its corresponding MCP PID; the five unsubscribed ephemeral sessions and their five MCP children remained. Finally exiting only the isolated app-server reaped all five remaining diagnostic children.

This separates the questions cleanly:

• process termination works when the true session-shutdown boundary is reached;
• completion/unsubscribe does not reach that boundary, so the heavy runtime remains resident by design.

The explicit shutdown implementation does call the MCP manager's shutdown path: handlers.rs L597-L615, connection_manager.rs L461-L472.

Field corroboration (sanitized)

In one live app-server snapshot I counted 81 repeated, session-shaped helper bundles. Seventy-six shared one workload cwd. Each complete bundle had the same shape: one Node REPL plus four plugin MCP servers. The local spawn graph contained 7,847 closed and 1,113 open edges; one high-fanout workload had 6,414 closed and 120 open edges across five root parents.

I am not treating an open graph edge as proof that a process is live. The live-process conclusion comes from the OS process tree; the graph counts only corroborate the exposure pattern and show why a per-root cap cannot bound a long-lived desktop globally.

Secondary cleanup defect: EPERM becomes a silent terminal state

Local stdio servers are launched in their own process group: stdio_server_launcher.rs L263-L280.

On shutdown, if terminate_process_group() errors, the implementation logs a warning and sets should_escalate = false, so no delayed SIGKILL is attempted: stdio_server_launcher.rs L327-L344. The outer handle has already set its terminated atomic and returns Ok(()) for the local path: stdio_server_launcher.rs L385-L404.

On this exact build I found four warnings with the exact shape:

Failed to terminate MCP process group ...: Operation not permitted (os error 1)

This is not needed to explain the primary accumulation above, but it can explain survivors after a genuine close_agent/shutdown attempt. The origin of the intermittent macOS EPERM is not yet isolated; the confirmed bug is that the error is swallowed, prevents fallback, and makes the handle non-retryable.

Why PR #19753 did not fully solve this

PR #19753 correctly added explicit MCP shutdown and process handles. The current explicit deletion test proves that path can work. The remaining primary gap is when shutdown is invoked: ordinary agent completion, V2 durable residency, and app-server unsubscribe intentionally retain the session, so the new shutdown method is never reached.

Version attribution

This is not introduced by Desktop build 5488.

I downloaded the official build 5484 package and compared it with the installed 5488 package:

• both embed codex-cli 0.145.0-alpha.18 arm64;
• the 264,898,656-byte codex binaries are byte-for-byte identical through the complete executable payload up to the LC_CODE_SIGNATURE offset (262,827,104); only the signature region differs;
• the desktop main-process app-server protocol chunk is byte-for-byte identical;
• the renderer chunk containing thread/loaded/list, thread/unsubscribe, thread/read, and thread/list differs in only 138 bytes, all of which are generated import/source-map filename hashes; the lifecycle logic is identical;
• main-process JavaScript differs only in an unrelated two-hash allow-list rotation plus generated source-map filenames and build/version metadata.

The same defect family is documented on much older versions in #17832 and this issue. The recent amplification point is the June 8 V2 durable-residency/interrupt design in PRs #26632 and #26994, combined with larger plugin MCP sets and many simultaneously loaded/restored root tasks. It is a long-standing lifecycle defect made more expensive and easier to trigger, not a one-day 5488 regression.

As of current main commit 315195492c80fdade38e917c18f9584efd599304, the relevant completion, residency, manager-shutdown, and local-process-terminator files retain the same behavior.

Suggested fix boundary

1. Decouple logical agent residency from MCP-runtime residency. On final status or last unsubscribe, shut down/detach the MCP runtime while keeping durable agent/thread identity. Recreate it lazily on the next turn/tool use.
2. Add a global app-server budget/TTL, not only a per-root agent cap: process count, owned RSS, last tool call, owning root/thread, and idle deadline.
3. Start stdio MCP servers lazily when their catalog/tool is actually needed, rather than in every Session::new().
4. Do not swallow local termination failure. Preserve a retryable state, validate the recorded PID/PGID identity, and use a safe direct-child or SIGKILL fallback when group termination fails.
5. Add regression tests that start a blocking PID-reporting mock MCP server and assert:

• N completed/unsubscribed sessions converge to a bounded MCP baseline;
• many separate V2 root sessions are bounded globally, not only per root;
• explicit close/delete reaps every owned process;
• a forced killpg failure is surfaced and retried/fallback-cleaned rather than reported as success.

Evidence boundary

The primary mechanism is deterministically confirmed by exact release source, a minimal isolated reproduction, version-to-version binary/chunk comparison, and the live process-tree shape. The intermittent cause of macOS EPERM is the only part I am leaving as an explicitly separate unresolved question; its error-handling defect is confirmed in source.

this is the fan-out-inherits-everything pattern. each subagent forks the parent's full stdio MCP set at spawn, so process count tracks how wide you fan out, not whether the task ever calls a tool. the leak is a second bug stacked on that: close_agent reaps the agent but the child stdio trees hang off a different lifecycle, so they outlive it. the shared per-project pool in your expected-behavior list kills both, spawn the servers once, hand subagents handles, nothing to reap on close. linear RSS growth is the tell that spawn and teardown aren't symmetric.

Confirming the same failure mode on Windows 11 Codex Desktop. This appears to be cross-platform session/MCP-runtime retention rather than a Linux-only problem.

Environment

• Windows 11 Pro x64, 10.0.26100
• 31.92 GiB physical RAM
• Codex Desktop AppX package OpenAI.Codex 26.721.11231.0
• Long-running desktop session with multi-agent use
• Enabled stdio/local MCP families include Chrome DevTools, Playwright, Context7, Sentry, and Node REPL

Read-only field snapshot

Snapshot taken 2026-07-30 10:33 AWST while active tasks were left untouched:

• System physical memory: 30.80 / 31.92 GiB used
• Windows committed memory: 73.05 / 73.97 GiB
• Main desktop codex app-server tree (child of ChatGPT.exe): approximately 345 processes, 17.22 GiB private bytes, and 4.70 GiB working set
• A nearby process census contained 347 descendants:
• 175 × node.exe
• 128 × cmd.exe
• 16 × node_repl.exe
• 21 × PowerShell / pwsh
• The command-line/process shape showed at least 16 repeated session-shaped MCP sets:
• 16 Context7 instances
• 16 Playwright instances
• 16 Sentry MCP instances
• 16 Node REPL instances
• 48 Node command lines matching chrome-devtools-mcp (the npm wrapper/launcher/server layers produce multiple matching processes per set)

No processes were killed and the app was not restarted because long-running tasks were active.

Field reproduction pattern

1. Keep Codex Desktop running for an extended period.
2. Use multiple root tasks/subagents with several stdio MCP servers enabled.
3. Completed or inactive agent/session generations leave their full MCP process sets resident under the long-lived app-server.
4. Repeated use creates additional nearly identical sets.
5. Node/npm/cmd process count and Windows commit grow until the machine approaches commit exhaustion.

This matches the lifecycle explanation in the macOS evidence above: logical thread/subagent residency remains coupled to an eagerly started heavyweight MCP runtime. On Windows, npm-backed MCPs also add substantial cmd.exe wrapper fan-out.

Expected behavior would be for inactive/completed sessions to release or detach their MCP runtimes, for MCPs to start lazily, or for the app-server to enforce a global process/private-byte/idle-time budget. A Windows-safe diagnostic/cleanup mechanism would also help until the lifecycle is fixed.

I can gather a controlled before/after Windows reproduction later, but I did not disturb the live tasks for this field capture.

Additional high-scale macOS field confirmation on a newer Desktop build (credential-redacted).

Environment

• ChatGPT/Codex Desktop 26.727.51351 (build 6119)
• Bundled codex-cli 0.146.0-alpha.9.2
• macOS 26.5.2 (25F84), arm64
• Long-lived app-server with multi_agent and multi_agent_v2 enabled
• Multiple local stdio MCP servers

Observed impact and cleanup

• Initial operator snapshot: 518 MCP-related OS processes and 27.69 GB aggregate RSS.
• A saved credential-redacted inventory at 2026-07-31 20:41 MDT contained 451 external MCP processes / 19.63 GiB aggregate RSS; 406 were Codex-owned in that snapshot.
• After revalidation, 402 stale Codex-owned MCP processes were selected for cleanup. Of those, 399 exited after SIGTERM and 3 exited during the same termination cascade; none survived. They accounted for 17.91 GiB aggregate RSS immediately before termination.
• After follow-up cleanup, a sample taken three seconds later showed zero Codex-owned external MCP processes. This was a point-in-time result; retained contexts could initialize new stacks again.

Lifecycle correlation

• 84 retained node_repl execution contexts were alive in the same Codex app-server process tree.
• In the sampled process table, 80/80 distinct start-time clusters contained a retained context plus matching configured MCP roots.
• All 84 context starts were within three seconds of a logged context-creating event: 46 spawn_agent, 31 exec, and 7 wait/message/follow-up events. 71 were also within three seconds of rollout-file creation.
• Session logs contained 6,941 tool calls across 35 sessions, while there were 84 retained contexts. Matched rollouts contained 3,035 tool calls, with up to 211 calls in one rollout.
• Together with the process clustering, this is inconsistent with one MCP stack per tool call and instead points to eager MCP initialization per retained rollout/execution context.
• Matching MCP stacks remained after many associated subagent tasks had completed or been interrupted, indicating that teardown/reaping was not reliably reached or completed.

Config reload behavior

Disabling one stdio server changed most later context initializations but did not terminate already-running copies. Config disablement therefore is not a cleanup mechanism.

This field dataset matches the session/MCP-runtime retention mechanism already source-traced above and confirms that the defect remains observable in the newer bundled 0.146.0-alpha.9.2 build. I have not independently source-traced that exact build's callback path, so the callback-level conclusion here is intentionally limited to the process and timestamp evidence.

I can provide credential-redacted PID, PPID, RSS, start-time inventories, and timestamp-correlation tables if useful. No endpoints or credentials are included.

Current-build reproduction (2026-08-14)

macOS, Codex Desktop app 26.803.61601 (build 6396), CLI 0.147.0-alpha.6.5. The main app-server had been alive ~22.5 h at peak. Parallel workflows / sub-agent threads each spin up their own stdio MCP runtime (opencode-codebase-index-mcp --host codex, launched as an npm exec wrapper plus its node server child). At peak: 218 matching processes (~109 npm+node pairs), all descendants of the live app-server, zero with PPID 1. Live snapshot at time of writing (machine in active use, after the cleanup below): 19 npm exec wrappers + 19 node servers, all still direct children of the same long-lived app-server.

New causal evidence

Read-only aggregates from the local Codex state database (thread_spawn_edges joined with threads): 4,555 spawn edges (3,680 open, 875 closed); 5,016 threads with thread_source = 'subagent' (1,785 not archived); 158 child threads created in the last 6 hours; 222 parent threads with 5 or more children. Bursts of child threads are born within the same second/minute during parallel workflows, and each thread carries a full MCP runtime that outlives the sub-agent.

Controlled cleanup

Selection based on start-time + cwd correlation with thread metadata (created ±10 min, same cwd, thread not archived, activity <60 min to preserve active ones): 142 stale runtimes stopped, 74 preserved, only 2 respawned afterwards for genuinely active threads. Age alone is a dangerous criterion: one runtime ~14 h 48 old was still serving an active conversation.

Interpretation

This is retention of MCP runtimes per thread/sub-agent under a live app-server, not an EOF-propagation defect of the MCP server and not primarily a watcher storm (see #30408 for the general per-thread angle). Individual processes are mostly idle, but their count and their ~29 OS threads each grow without bound.

Expected invariant

After a sub-agent terminates/closes/is unloaded, the number of MCP runtimes should return toward the number of genuinely active threads; runtimes retained for resume must be detached or lazily recreated.

Evidence availability

Happy to provide a sanitized PID/PPID/start/cwd table (paths hashed) and the aggregation SQL; I won't share the state database or rollouts.

Current stable Linux Desktop confirmation (2026-08-19)

The same session-shaped MCP multiplication is still observable on a newer Linux Desktop release.

Environment

• Ubuntu 24.04.4 LTS, x86_64
• Kernel 6.14.0-37-generic
• systemd 255.4-1ubuntu8.17
• AMD Ryzen 5 PRO 8540U, 6 cores / 12 threads
• 30.6 GiB RAM, 8 GiB swap
• ChatGPT/Codex Desktop package 26.814.41957
• Active app-server: codex-cli 0.147.0 launched as codex -c features.code_mode_host=true app-server --analytics-default-enabled
• The package also bundles codex-cli 0.148.0-alpha.15, but this app-server was configured to use the local 0.147.0 binary.
• Exactly two enabled stdio MCP entries from codex mcp list --json: node_repl and claude_collab

Live process census

At a fixed capture after approximately 54 minutes of app-server uptime:

• app-server itself: 1,173,956 KiB RSS, 64 OS threads
• node_repl: 47 direct children, 375.2 MiB aggregate RSS, 705 OS threads
• claude_collab stdio server: 37 direct children, 1,845.0 MiB aggregate RSS, 259 OS threads
• combined configured MCP roots: 84 direct children, ~2.17 GiB aggregate RSS, 964 OS threads
• app-server plus those MCP roots: ~3.32 GiB RSS and 1,028 OS threads

The tree also contained nine deliberately long-lived project helper servers (~372 MiB RSS); I excluded them from the MCP totals above. There were no direct children in uninterruptible D state at capture.

This field snapshot does not prove that every one of the 84 instances was stale. It does show that, on the current stable Linux Desktop build, a long-lived app-server can retain dozens of complete per-session MCP runtimes and more than 1,000 associated OS threads in under an hour, despite having only two configured MCP families. The direct PPID relationship and repeated start-time pairs match the per-loaded-session/session-retention mechanism already source-traced above.

Operational impact observed on this machine

During a separate high-pressure incident, user@1000.service reached 29.0 GiB memory peak and 7.9 GiB swap peak. Ubuntu's default systemd-oomd recorded user-slice pressure at 80.66% > 50% for over 20 seconds, then killed the user's init.scope, collapsing the GNOME session. The candidate report included a 21.6 GiB high-pressure application scope (PSI avg10 79.93) and live Codex/node_repl processes.

There were also multiple concurrent Rust compile/link jobs in that incident, so I am not attributing the entire 21.6 GiB scope or logout solely to MCP retention. The narrower conclusion is that retained per-session MCP runtimes materially consume memory, swap, PID, and thread headroom, amplifying otherwise legitimate parallel workloads.

Diagnostics that would make future reports easier to action

It would help if app-server diagnostics exposed:

1. loaded/subscribed/executing/idle session counts;
2. the owner session/thread ID and last-use time for every MCP process root;
3. global app-server gauges for MCP process count, aggregate RSS, and OS threads;
4. an invariant/test that completed, unsubscribed, archived, or evicted sessions converge to a globally bounded MCP-runtime baseline.

The latest OpenAI documentation describes MCP configuration consistently across Codex surfaces, but currently there is no user-visible mapping from a spawned local MCP PID back to the task/session retaining it. That ownership mapping would also make a safe temporary cleanup tool possible without age-based guessing.

Additional field report from Linux. This is provided as a diagnostic snapshot only; I am not able to perform additional reproduction work or respond to follow-up QA requests.

Environment:

• Codex CLI: codex-cli 0.149.0
• Running app-server reported by codex doctor: 0.147.0
• Linux 5.15.0-187-generic x86_64
• 8 GiB RAM, 4 GiB swap
• Long-lived app-server / IDE integration

Observed behavior:

• One app-server remained alive for about 8.5 days under one systemd session scope.
• Its direct children contained at least seven separately-started generations of the same MCP suite, launched at distinct times across multiple days.
• Repeated launcher roots included Serena MCP, Chrome DevTools MCP, Playwright MCP, a search MCP, and a Node MCP server; Serena generations also retained separate gopls and telemetry descendants.
• All generations remained children of the live app-server and in the same systemd session scope. There was no per-thread/per-generation cgroup boundary visible.
• Several older generations showed no CPU-time increase during a five-second observation window but remained alive.

Resource impact at the snapshot:

• System swap: approximately 4.0 / 4.0 GiB used.
• Development-tool process groups collectively retained approximately 2.8 GiB of swap.
• Approximate swap by group: Chrome DevTools MCP 915 MiB, Codex 660 MiB, gopls 534 MiB, Playwright MCP 430 MiB, Serena MCP 313 MiB.
• The long-lived app-server process itself retained approximately 413 MiB of swap, almost entirely anonymous memory.
• Example older individual generations retained about 68 MiB per Serena process, 83 MiB per Chrome DevTools process, and 113-118 MiB per gopls process.
• Current memory PSI was zero and swap I/O was nearly idle, indicating retained cold pages rather than active thrashing; however, swap capacity was exhausted.

The process topology is consistent with this issue: new MCP generations are added beneath a live app-server, while previous generations are not reaped. Because all generations share the same long-lived parent and session scope, the OS cannot distinguish completed thread/session ownership and will not reclaim them automatically.

Expected behavior: when a thread/subagent/session is completed, cancelled, disconnected, or reinitialized, Codex should terminate and wait for the owned MCP process tree, with a grace period and descendant-kill fallback. Per-generation ownership metadata or process-group/cgroup isolation would also make cleanup deterministic.

Again, this comment is an evidence contribution only. No additional reproduction or follow-up QA support is available from this reporter.