MCP stdio servers leak pipe fds + orphan child processes → cumulative EMFILE ("Too many open files", os error 24)

Open 💬 13 comments Opened Jun 8, 2026 by jacobcxdev

💡 Likely answer: A maintainer (github-actions[bot], contributor) responded on this thread — see the highlighted reply below.

MCP stdio servers leak pipe fds + orphan child processes → cumulative EMFILE ("Too many open files", os error 24)

What version of Codex CLI is running?

codex-cli 0.137.0 (also reproduced symptom on long-running 0.12x sessions)

What subscription do you have?

Pro

Which model were you using?

n/a (process/transport bug, model-independent)

What platform is your computer?

Darwin 25.5.0 (macOS 26.5.1) arm64

What issue are you seeing?

Long-running sessions that use stdio MCP servers slowly accumulate open file descriptors in the codex parent process until they hit the per-process soft limit (256 on a default macOS login shell), at which point every subsequent open()/spawn() fails with Too many open files (os error 24).

This is almost certainly the underlying fd leak behind the now-closed #17806 ("Repeat Os { code: 24 } crashes"). That issue was closed via #12216, which only hardened the panic in AbsolutePathBuf::parent() so EMFILE no longer crashes the TUI — it did not address why fds are exhausted. The leak is still present.

Evidence (live system)

A single long-lived TUI session (codex resume, soft limit 256) sitting 3 fds under the ceiling:

$ lsof -p <codex_pid> | awk 'NR>1{print $5}' | sort | uniq -c | sort -rn
 186 PIPE        <-- the leak
  38 REG         (state_*.sqlite WAL/shm, rollout jsonl, logs_*.sqlite)
   8 KQUEUE
   7 unix
   5 IPv4
   ...
 253 total       (soft NOFILE = 256 -> next open() = EMFILE)

186 anonymous PIPE fds for a session with ~4–5 configured MCP servers (expected ≈3 pipes/server = ~15). The excess maps almost 1:1 to orphaned MCP child processes that the codex parent still holds read pipes to.

System-wide, orphaned stdio MCP servers far outnumber the live codex sessions that should own them:

$ pgrep -f slack-mcp-server  | wc -l   # 30
$ pgrep -f xcodebuildmcp     | wc -l   # 18
$ pgrep -f "codex app-server"| wc -l   #  9

These are spawned as npm exec <server> / npx, i.e. the actual server (node) is a grandchild under an npm wrapper.

Root-cause analysis (code, codex-rs @ 8f1aad5)

Cleanup of stdio MCP servers has three gaps that compound under the MCP refresh/rebuild churn.

1. Teardown is fire-and-forget — never awaits process exit

rmcp-client/src/stdio_server_launcher.rs LocalProcessTerminator::terminate (unix):

let should_escalate = terminate_process_group(pgid) /* killpg SIGTERM */ ...;
if should_escalate {
    spawn(move || {                 // detached std::thread
        sleep(PROCESS_GROUP_TERM_GRACE_PERIOD /* 2s */);
        kill_process_group(pgid);   // killpg SIGKILL
    });
}

RmcpClient::shutdown (rmcp-client/src/rmcp_client.rs:678) calls this and returns immediately — the SIGKILL lands ≥2 s later on a detached thread. Nothing awaits actual child exit or pipe EOF. If the runtime/parent is torn down (or the next refresh races) within that window, the SIGKILL thread can be lost, leaving the group alive.

2. Cancelled-startup path skips explicit terminate entirely

On MCP refresh, core/src/session/mcp.rs::refresh_mcp_servers_inner:

// :341  cancel old startup token
guard.cancel();
// :344  build a brand-new manager — spawns the ENTIRE MCP server set again
let (refreshed_manager, cancel_token) = McpConnectionManager::new(...).await;
// :377  swap in
let mut old_manager = std::mem::replace(&mut *manager, refreshed_manager);
// :379  shut down the old one
old_manager.shutdown().await;

For any old client still mid-handshake, AsyncManagedClient::shutdown (codex-mcp/src/rmcp_client.rs:240) resolves the shared startup future to Err(Cancelled):

match self.client().await {
    Ok(client) => client.client.shutdown().await,
    Err(StartupOutcomeError::Cancelled) => {}   // <-- terminate() never called
    Err(error) => warn!(...),
}

Cleanup then depends entirely on Drop for StdioServerProcessHandleInner, which only fires once the last Arc clone is dropped — and which itself uses the same fire-and-forget terminate from gap #1.

The same full-set spawn-then-shutdown churn also runs in the ephemeral connector-discovery manager (core/src/connectors.rs:273 … :359).

3. Group signalling misses `npm`/`npx` grandchildren

Spawn uses bare command.process_group(0) (stdio_server_launcher.rs:263) and signals pgid == direct-child PID. The direct child is the npm/npx wrapper; the real server is node. When the wrapper exits (or the server reparents) the original group no longer covers node, so killpg(SIGTERM/SIGKILL) misses it → orphaned node (the 30/18 above). Note utils/pty/src/process_group.rs already has detach_from_tty/setsid-style helpers, but the stdio MCP launcher does not use a wait-and-verify reap.

Net effect

Each refresh/rebuild cycle spawns a fresh server set and tears down the old one without awaiting death and without reliably reaping the npm grandchild. Every orphan keeps its stdout/stderr write ends open, so the codex-side read pipe + the detached stderr-reader task (stdio_server_launcher.rs:274) never reach EOF and never close. Pipe fds in the codex parent grow monotonically until EMFILE.

Suggested fixes

Await actual exit on teardown. Make shutdown()/Drop waitpid/try_wait the group after SIGTERM→SIGKILL instead of fire-and-forget, so pipes are closed deterministically.
Terminate on the cancelled-startup arm. In AsyncManagedClient::shutdown, call the process terminator on Err(Cancelled) rather than relying on Drop.
Reap the whole group reliably. Either spawn the server binary directly (resolve npm exec/npx to the real entrypoint) or setsid + track the session and kill+wait the session, not just the direct child's PID-as-PGID.
(Hardening) Bound the stderr-reader task to transport lifetime / close the read half on terminate so a lingering grandchild can't pin the pipe.

Workaround for users

Raise the per-process limit before launching (ulimit -n 65536, and/or sudo launchctl limit maxfiles 65536 524288 for GUI-launched servers) and periodically kill orphaned *-mcp-server / npx processes. This only delays the leak.

#17806 (closed) — same symptom; closed via panic-hardening only, leak not addressed
#12216 (closed) — EMFILE panic hardening in AbsolutePathBuf::parent()
#25243 (open) — distinct root cause (Codex.app re-exec storm vs syspolicyd), not MCP

View original on GitHub ↗

13 Comments

github-actions[bot] contributor · 1 month ago

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

#25744
#25742
#25697
#26773

Powered by Codex Action

mochafreddo · 1 month ago

I can reproduce the same EMFILE failure on Codex Desktop on macOS.

Environment:

Codex Desktop is still running when the failure occurs.
ulimit -n: 256
launchctl limit maxfiles: previously observed as 256 unlimited
Failure observed from Codex tool execution:

Failed to create unified exec process: Too many open files (os error 24)

Current evidence:

Even trivial process creation can fail intermittently, including commands like git log and launchctl limit maxfiles.
Codex GUI main process had ~259 lsof entries while the soft limit is 256.
Related child process counts observed:
chrome-devtools-mcp: 33
node_repl: 12
SkyComputerUseClient: 13
codex app-server --listen stdio: 5
codex app-server --listen unix: 2

This matches the hypothesis in this issue: long-lived Codex Desktop sessions with MCP/tooling accumulate child processes and file descriptors until the parent process reaches the macOS soft NOFILE limit. At that point, new open()/spawn() calls start failing with EMFILE.

I did not paste full ps output because it contains local workspace paths and prompt content, but I can provide more sanitized diagnostics if useful.

Impact:
This blocks normal development because once the parent process reaches the FD limit, Codex cannot reliably run even simple shell commands. Restarting Codex temporarily recovers the session, but the counts grow again in long-running sessions.

twentyOne2x · 1 month ago

Adding a current Desktop datapoint and linking the broader report: #27601.

Environment:

Codex Desktop 26.608.12217 / build 3722
Bundled CLI 0.138.0-alpha.7
macOS 26.3 / Darwin 25.3.0 arm64
launchctl maxfiles soft limit: 256

Current helper counts after some recovery work:

codex app-server --listen stdio:// helpers: 40
node_repl helpers: 40

Earlier in the same investigation, helper counts were higher and turn/start requests timed out with high pending counts. This lines up with a cumulative helper/stdio lifecycle pressure class: Desktop eventually gets into a state where thread loads take minutes and turn/start times out, then hard quitting/reopening Desktop improves it temporarily.

mochafreddo · 1 month ago

I prepared and locally verified a candidate fix for this on a fork branch:

Summary of the fix:

local MCP stdio shutdown now closes stdin first, waits briefly for EOF-driven clean exit, then terminates only if needed
local stdio child/process-group cleanup now waits/reaps the child and escalates to kill after the grace period
executor process API now has process/closeStdin, so MCP stdio launched through an executor can also receive stdin EOF before termination
remote coverage exercises the websocket exec-server path: RemoteProcess -> ExecServerClient -> process/closeStdin -> server close_stdin_process, which is the path relevant when local Codex controls a remote executor over SSH/exec-server
added regression coverage for local stdio, executor stdio, remote executor stdio, wrapper/grandchild cleanup, and initialized shutdown with an in-flight operation

Verification run locally:

just test -p codex-core unified_exec
just test -p codex-rmcp-client --test process_group_cleanup shutdown_closes_remote_executor_stdio_before_terminating_server
just test -p codex-exec-server
just test -p codex-rmcp-client
just fmt
just fix -p codex-exec-server
just fix -p codex-rmcp-client

I cannot open an upstream PR from this account because the repository currently limits PR creation to collaborators. If this approach aligns with the intended direction, please invite/enable a PR from this branch or feel free to cherry-pick the branch.

Nicolas0315 · 24 days ago

Additional live data point: in a long-running macOS Codex Desktop session, the failure can progress to the point where Codex cannot spawn even trivial diagnostics from the session. exec_command is now failing with Failed to create unified exec process: Too many open files (os error 24), including for commands such as pwd or ulimit -n.

That matches the EMFILE behavior described here, but the practical impact is that once the fd ceiling is reached, the affected session cannot collect normal follow-up evidence or run local mitigation commands through Codex itself. I’m avoiding raw process/log output here because it may include local paths or private MCP details.

Nicolas0315 · 24 days ago

Additional live datapoint from the same macOS Desktop session:

ulimit -n: 256
launchctl maxfiles: 256 unlimited
Codex GUI process: 1
codex helper processes: 6
node processes: 37
node_repl processes: 6
chrome-devtools-mcp processes: 13
Codex GUI lsof entries: 233

Immediately before this capture, the session hit Failed to create unified exec process: Too many open files (os error 24) while trying to run read-only diagnostics such as git diff and nl. Closing a background subagent reduced pressure enough that pwd and the sanitized count commands started working again.

This seems consistent with a soft-limit pressure pattern rather than a single permanently wedged command path: the parent can be below the 256 soft limit after recovery, but transient helper/subagent/MCP activity can still push it over the edge and make process creation fail. I’m still avoiding raw ps/lsof details because they can include local paths or private MCP arguments.

Nicolas0315 · 23 days ago

Adding a current macOS Desktop/CLI datapoint from a long-running multi-surface setup.

Snapshot time: 2026-06-28 11:26 JST

Environment:

Codex Desktop: 26.623.42026, build 4514
Codex CLI / app-server: 0.142.3
Platform: macOS arm64
codex doctor --json: overallStatus=ok
MCP config: 13 configured servers, 4 disabled, 7 stdio, 6 streamable HTTP
launchctl limit maxfiles: 256 unlimited
shell ulimit -n: 1048575

Current process / FD pressure:

standalone codex app-server --listen unix:// lsof entries: 258
chrome_devtools_mcp processes: 59
playwright_mcp processes: 30
computer_use_mcp processes: 41
computer_use_event_stream processes: 22
node_repl processes: 22
xcodebuildmcp processes: 40
codex_app_server matched processes: 20

The important operational detail is that codex doctor is green while the long-lived app-server/helper surface is still over the launchd soft FD limit. In practice, once this state is reached, the session can feel slow or fail at ordinary process/tool startup even though the basic health checks pass.

I am keeping this sanitized and omitting raw process args, full local paths, and private MCP/project details. Happy to collect a narrower maintainer-requested probe if there is a preferred diagnostic shape.

CountofGlamorgan · 7 days ago

Confirming this leak is still present on the current stable 0.144.3, and adding a second reproduction surface: the long-lived Desktop SSH app-server, not just a TUI / codex resume session.

Reproduction surface: the shared Desktop SSH app-server

Codex Desktop (SSH remote) multiplexes every project window onto a single long-lived parent:

codex -c features.code_mode_host=true app-server --listen unix://

That one process holds the read pipes to the stdio MCP child of every session it has ever served. Live snapshot from a ~27h-old app-server (0.144.1):

# orphaned stdio MCP proxy processes still parented to the app-server
$ ps -axo pid,ppid,command | grep '[m]cp-proxy' | awk '$2==<app_server_pid>' | wc -l
31

# fds held by the app-server parent
$ lsof -p <app_server_pid> | awk 'NR>1{print $5}' | sort | uniq -c | sort -rn
 255 PIPE     <-- the leak
  39 REG
   5 unix
   3 KQUEUE
   ...

255 anonymous PIPE fds for what should be a handful of live MCP servers. On a default macOS login shell (RLIMIT_NOFILE soft = 256) this parent would already be throwing os error 24; it only survives because we pre-raised the soft limit to 65536. Peak before a manual sweep was 96 orphaned proxy processes machine-wide (35 under the app-server), ~5 GB RSS.

Because all Desktop projects share this one app-server, when it finally hits EMFILE every project's Codex dies simultaneously — spawn, openpty, and DNS resolution all start failing at once.

Still leaking on 0.144.3

A freshly-started 0.144.3 app-server accumulated 3 orphaned proxies within ~2h of uptime. This is consistent with the changelog: rust-v0.144.1..rust-v0.144.3 contains only a Guardian auto-review revert (#32672) plus a "version-only" 0.144.3 with no merged changes — nothing touches MCP transport or process teardown.

Extra data point for the root-cause (stdin is never closed)

The stdio MCP here is a plain node proxy.mjs (not an npx/npm exec grandchild), and the script explicitly handles clean shutdown:

process.stdin.on('end', () => process.exit(0));

…but that 'end' never fires, because the app-server never closes the child's stdin write end (nor awaits its exit) on session teardown. The child blocks forever on a stdin that will never EOF. So beyond the "fire-and-forget terminate, never awaits exit" gap already noted here: even a well-behaved child that self-exits on stdin EOF cannot clean itself up, because the parent keeps the pipe's write end open. That narrows it away from any npm/npx-wrapper theory — a single-process node child leaks the same way.

Env: macOS 26 (Darwin 25.x) arm64 · codex-cli 0.144.1 → 0.144.3 · Desktop SSH remote host.

MQ-EL · 6 days ago

Adding a precise multi-agent lifecycle datapoint from stable codex-cli 0.144.3 on macOS arm64. This confirms the same failure and narrows why subagents amplify it.

Failure

After a long multi-agent session, even:

pwd

failed before spawn with:

Failed to create unified exec process: Too many open files (os error 24)

The kernel-wide file table was healthy (8154 / 122880), so this was process-local.

Parent FD and child-process snapshot

numeric FDs: 254
highest FD: 254
PIPE: 174
REG: 43
soft maxfiles limit: 256

The Codex parent had 58 direct local MCP helper children:

node_repl: 15
codebase-memory-mcp: 15
chrome-devtools-mcp: 14
mcp/server.mjs: 14

The correlation is exact:

58 live stdio MCP children × 3 parent-side stdio pipe FDs = 174 PIPE FDs

Why subagents multiply MCP processes

These subagents are logical Codex threads inside one Codex OS process, not one OS process per subagent. The control plane is shared across the agent tree, but each spawned thread creates its own session and MCP connection manager. Consequently, every enabled local stdio MCP entry normally starts a separate helper process for every resident logical thread. (Remote HTTP/SSE MCPs do not create local helper processes.)

A current source checkout also shows an important lifecycle detail: V2 residency treats completed, errored, and interrupted threads as unloadable, but LRU unloading is attempted only when the configured logical-agent capacity is reached. An interrupt only cancels the current turn; it is not a shutdown, so I am not counting lack of cleanup on interrupt itself as the bug.

In this run, the agent capacity exposed to the session was 200, while four local stdio MCPs cost roughly 12 parent pipe FDs per resident thread. The process therefore hit the macOS soft FD limit long before the logical-agent residency limit could trigger eviction.

Trigger evidence

The logical tree had 15 non-root agents, including agents already completed and therefore eligible for unload.
Their MCP helpers remained live and sleeping so the resource count accumulated across the session.
The first additional read-only agent started a complete four-helper MCP bundle.
The second additional agent started only two of four helpers before the parent reached 254 FDs.
The root and both new agents then failed to spawn even a trivial command.
A second independent long-lived Codex process showed the same signature: 252 numeric FDs, 189 PIPE FDs, and 63 direct MCP children.

Interpretation and suggested regression boundary

Dedicated per-thread MCP runtimes may be intentional for cwd, permissions, cancellation, event routing, and stateful tools. The failure is the resource-budget mismatch: residency is bounded by logical thread count, but does not account for per-thread stdio process/FD cost or the process RLIMIT_NOFILE. Partial MCP startup also needs atomic rollback.

A robust fix could keep persisted agent metadata while unloading eligible thread runtimes before the FD low-water mark, and should close/await all helper processes plus parent pipe handles. A regression test should spawn and complete N agents with K local stdio MCPs, then assert that helper-process and parent-FD counts remain bounded and return to baseline after eviction/shutdown; it should also force failure partway through an MCP bundle and verify rollback.

Turtle-Hwan · 4 days ago

Confirming this leak is still present on 0.144.5 (newer than the 0.144.3 confirmations above), and adding two things I haven't seen upthread:

an accumulation rate datapoint — the fd fingerprint others reach in ~27h, this host reached in 12 minutes; and
two timer/turn-driven spawn paths that leak independently of session or MCP-refresh churn, one of which has its own open issue (#23119) where the fd angle has never been raised.

Environment

codex-cli 0.144.5
macOS arm64 (Darwin 25.5.0)
launchctl limit maxfiles: 256 unlimited
shell ulimit -n: 1048576
surface: codex -c features.code_mode_host=true app-server --listen unix://

Failure (app-server stderr, on 0.144.4)

ERROR codex_core::tools::router: error=exec_command failed for `/bin/zsh -lc '<gradle test cmd>'`:
  CreateProcess { message: "Rejected(\"Failed to create unified exec process: Too many open files (os error 24)\")" }
ERROR codex_core::tools::router: error=exec_command failed for `/bin/zsh -lc '<node -e cmd>'`:
  CreateProcess { message: "Rejected(\"Failed to create unified exec process: Too many open files (os error 24)\")" }

Kernel-wide file table was healthy throughout, so this is process-local, as established upthread:

kern.num_files: 7625 / kern.maxfiles: 122880   (6%)
kern.maxfilesperproc: 61440

Live snapshot — a 12-minute-old app-server

Process uptime at capture: 12m17s. One stdio MCP server of interest (node_repl, the OpenAI-bundled one shipped inside the desktop app).

$ lsof -p <app_server_pid> | awk 'NR>1{print $5}' | sort | uniq -c | sort -rn
 142 PIPE        <-- the leak
  49 REG
  11 IPv4
   6 unix
   3 KQUEUE
   ...
 216 total       (soft NOFILE = 256)

Every node_repl was still parented to that one app-server, spawning steadily and never exiting:

$ ps -eo ppid,command | grep [n]ode_repl | awk '{print $1}' | sort | uniq -c
   9 <app_server_pid>

spawn times (all children of that single app-server):
  14:34:36  14:34:48  14:34:59  14:35:02  14:38:07
  14:40:21  14:42:51  14:43:28  14:43:56

216 fds / 142 PIPE / 9 orphaned MCP children in ~12 minutes → 84% of the 256 ceiling, i.e. ~18 fd/min. At that rate this app-server would have hit EMFILE in under half an hour. That is far faster than the long-session reports upthread, which points at a driver that is not proportional to session count or user activity.

---

Two spawn paths that leak on a timer / per turn, not per session

The OP pins teardown on LocalProcessTerminator::terminate being fire-and-forget — nothing awaits child exit or pipe EOF. I want to point at two callers that hit that path repeatedly and unattended, which I think explains the rate above and the "it leaks even while idle" reports.

(a) `codex_models_manager` model refresh — see #23119

This host logged 63 occurrences of the following over ~43h, on a base period of ~183s (measured intervals cluster hard at 185s and 366s ≈ 2×183, i.e. a ~3-minute timer with skips):

ERROR codex_models_manager::manager: failed to refresh available models: timeout waiting for child process to exit

That string is CodexErr::Timeout from codex-rs/protocol/src/error.rs — it means the parent gave up waiting for the child to exit. That is definitionally a child the parent stopped reaping, on a timer, forever, for the entire life of the app-server.

Both EMFILE failures above were immediately preceded by a run of this exact error.

This is filed separately as #23119 ("...every 15-45s", open since May, zero comments), where it is treated purely as stderr noise — the word "fd" does not appear in it. I think the two issues are the same bug seen from opposite ends: #23119 is the symptom log line, this issue is the resource consequence. Worth cross-linking, and #23119's reported 15-45s cadence is ~4-12× faster than this host's 183s, which would make it a much more aggressive leak driver than anything measured in this thread so far.

(b) The `notify` hook — one unreaped child per turn

Default desktop config wires the Computer Use client in as the turn-end notifier:

notify = ["<...>/SkyComputerUseClient.app/Contents/MacOS/SkyComputerUseClient", "turn-ended"]

This host had 26 orphaned SkyComputerUseClient processes, all reparented to init (ppid=1), spawned across two days 19 days before capture and still resident — they survived every desktop app restart in that window and had to be reaped by hand. Their spawn timestamps are scattered a few minutes apart across two long evening sessions, which is exactly the shape of per-turn notify invocations rather than anything periodic.

So notify looks like a third path onto the same unreaped-child problem, and a nastier one: these orphans outlive the app entirely, so a user who "fixes" EMFILE by quitting and relaunching Codex still carries them.

Why the mitigations discussed upthread don't hold

Raising launchctl limit maxfiles only moves the ceiling — with a ~3-minute timer and a per-turn notify both leaking unconditionally, growth stays monotonic and unbounded in session length. Same for quitting/relaunching Desktop: it resets the app-server's fd table but leaves the (b) orphans behind.

Turtle-Hwan · 4 days ago

@jif-oai — following up on my datapoint above with a code-level root cause, since this sits directly on top of #29608 and #30101.

Short version: the guarantee #29608 added no longer holds on main. #30101 replaced the explicit shutdown of a superseded manager with an Arc-lifetime rule, but nothing in the current teardown path actually reaps a stdio child when that Arc is released — so superseded managers keep their MCP processes (and the parent's pipe fds) forever.

What #29608 established

// core/src/session/mcp.rs, PR #29608
-        self.services.mcp_connection_manager.store(Arc::new(refreshed_manager));
+        let superseded_manager = self.services.mcp_connection_manager.swap(Arc::new(refreshed_manager));
+        superseded_manager.shutdown().await;

Explicit, awaited, refcount-independent.

What `main` does now

#30101 moved publication into ServiceState::publish_mcp_runtime (core/src/state/service.rs:127), which is synchronous and therefore cannot .await a shutdown:

pub(crate) fn publish_mcp_runtime(&self, ..., manager: McpConnectionManager) -> Arc<McpRuntimeSnapshot> {
    let manager = Arc::new(manager);
    self.mcp_connection_manager.store(Arc::clone(&manager));   // store, not swap
    ...
    self.mcp_runtime.store(Some(Arc::clone(&runtime)));
    runtime
}

swap(...) + superseded_manager.shutdown().await is gone, and grepping core/src there is now no shutdown of a superseded manager anywhere — the only surviving manager.shutdown() calls are connectors.rs:334 (the short-lived codex_apps-only discovery manager) and an unrelated conversation.shutdown().

That is deliberate: #30101 states the intended rule as "Lets an old runtime live only while an in-flight step or request still holds its Arc." The intent is sound. The problem is the second half of that sentence never happens.

Why releasing the last `Arc` does not reap the child

Drop for McpConnectionManager (codex-mcp/src/connection_manager.rs:997) is synchronous, so it can only drop Arcs:

fn drop(&mut self) {
    self.startup_cancellation_token.cancel();
    self.clients.clear();
}

The only drop path that actually signals a local child is Drop for StdioServerProcessHandleInner (rmcp-client/src/stdio_server_launcher.rs:407), which fires when the last Arc<StdioServerProcessHandleInner> dies. But there are two live clones of that handle, created at launch:

rmcp_client.rs:372 — RmcpClient.stdio_process takes transport.process_handle(), an Arc clone;
rmcp_client.rs:930 — the transport itself (holding the original handle and the TokioChildProcess pipes) is moved into service::serve_client(...) and lives on inside ClientState::Ready { service: Arc<RunningService<..>> }.

So clients.clear() drops one clone and the Arc<RunningService> handle; it does not stop rmcp's service, and the handle's refcount does not reach zero, so nothing SIGTERMs the child and the transport's pipe fds stay open in the parent.

The asymmetry is telling: RmcpClient::shutdown() (rmcp_client.rs:737) does not rely on Drop — it calls process.terminate() explicitly and only then drops the previous state. That async path is the only teardown that reliably works, and after #30101 nothing calls it for a superseded manager.

This matches the runtime behavior exactly

The orphans I snapshotted are live processes, not zombies, all still parented to the app-server, with the parent holding their pipes — precisely what "handle refcount never hits zero, so no SIGTERM, and the transport is never dropped" predicts:

codex-cli 0.144.5, app-server uptime 12m17s, soft NOFILE = 256

  216 total fds   (142 PIPE)
    9 node_repl children, all ppid = <app_server_pid>, none exiting

~18 fd/min, unbounded in session length. Under the Arc rule as written, a refresh-heavy session leaks one stdio child per superseded manager, permanently.

The regression test encodes only half the invariant

core/tests/suite/mcp_refresh_cleanup.rs was inverted by #30101, from "superseded server is shut down" to:

async fn refresh_keeps_superseded_mcp_server_alive_for_in_flight_calls() {
    ...
    assert!(process_is_alive(&superseded_pid)?);
    assert!(process_is_alive(&replacement_pid)?);
}

It asserts the keep-alive half and never asserts the eventually-reaped half, so the leak is invisible to CI. A complementary assertion — after the in-flight call completes and the step ends, superseded_pid is reaped within some bound — would fail on main today, and would pin the invariant #30101 intended.

Possible directions

Make the Arc rule enforceable rather than implicit (closest to #30101's intent). Have publish_mcp_runtime swap() instead of store() and hand the superseded manager to an owner that can .await — e.g. spawn a cleanup task that waits for in-flight holders to drain (an explicit in-flight guard on McpRuntimeSnapshot is more honest than polling Arc::strong_count) and then calls shutdown().await. Preserves #30101's semantics; restores #29608's guarantee.
Drain superseded managers at a step/turn boundary — keep a session-level registry of superseded runtimes and shut them down once no step holds them. Simpler to reason about than lifetime-driven cleanup, at the cost of an explicit list.
Independently, make teardown itself sound. Even when shutdown() is called, LocalProcessTerminator::terminate (stdio_server_launcher.rs:328) is fire-and-forget: SIGTERM the process group, then a detached std::thread sleeps 2 s and SIGKILLs, while shutdown() returns immediately and nothing awaits child exit or pipe EOF. Closing stdin first and awaiting exit before escalating (as @mochafreddo prototyped upthread) would make the reap deterministic. This is orthogonal to (1)/(2) — it fixes teardown quality, not the fact that teardown is never invoked.

I'd treat (1) or (2) as the actual fix here and (3) as hardening.

Caveat on rigor: the fd/process numbers above are measured on this host, but the ownership analysis is read from main — I have not built a patched binary to confirm, and I've since disabled the offending server locally, so I no longer have a live repro to bisect against.

dchjmichael · 3 days ago

Any progress? I've encountered four container crashes caused by the "too many open files" error this week, which has seriously affected my work.

ZhangJinH · 3 days ago

I also encountered the same problem. Could you please tell me how the repair is progressing.

MCP stdio servers leak pipe fds + orphan child processes → cumulative EMFILE ("Too many open files", os error 24)