[Windows Codex app] Local stdio MCP servers are repeatedly spawned and not reaped within a single task
What version of the Codex App are you using (From “About Codex” dialog)?
Codex app: 26.810.6296.0
What subscription do you have?
ChatGPT Plus
What platform is your computer?
Microsoft Windows NT 10.0.26200.0 x64
What issue are you seeing?
Within one Codex task, each new turn/tool environment starts another full
set of local tool processes without terminating previous instances.
Observed after restart:
- qq_mail_mcp: 6 logical MCP instances / 12 Python processes
- node_repl.exe: 6 instances
- all instances share the same codex.exe app-server parent
- creation timestamps increase sequentially within the same task
With cua-driver enabled, each leaked mcp --direct instance also creates a
full-screen Cua.AgentCursorOverlay. At 12-14 instances, the Windows cursor
stutters severely. Restarting Codex clears the processes and restores smooth
cursor movement. Disabling cua-driver prevents those overlay processes but
other stdio MCP instances still accumulate.
What steps can reproduce the bug?
- On Windows, configure at least one local stdio MCP server in Codex. In my case, I had:
- qq_mail_mcp
- cua-driver using
mcp --direct
- Fully exit and restart the Codex desktop app so that no old MCP child processes remain.
- Open a single Codex task. Do not create additional tasks.
- Send several consecutive messages in the same task that require tool-enabled turns, for example asking Codex to run a read-only PowerShell command.
- After each turn, inspect the child processes of
codex.exein Task Manager or PowerShell.
- Observe that every new turn/tool environment starts another instance of each configured stdio MCP server, while the instances created by previous turns remain running.
- After six turns in one task, I observed:
- 6 logical
qq_mail_mcpinstances (12 Python processes because each instance has a launcher and runtime child) - 6
node_repl.exeinstances - All instances had the same
codex.exe app-serverparent.
- When
cua-driver mcp --directwas enabled, every leaked Cua instance also created a visible full-screenCua.AgentCursorOverlaycovering the dual-monitor virtual desktop.
- After approximately 12-14 accumulated Cua instances, the Windows mouse cursor started stuttering severely in normal applications.
- Fully exiting and restarting Codex removed all accumulated child processes and immediately restored smooth cursor movement. Disabling the Cua MCP prevented its overlay processes from returning, but other stdio MCP instances continued to accumulate.
What is the expected behavior?
Codex should reuse the existing stdio MCP connection within the same task, or terminate the previous MCP child process when its tool/session context is disposed.
Additional information
_No response_
18 Comments
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
macOS reproduction, root cause, and tested patch available
I reproduced the same per-turn stdio MCP accumulation on macOS in the ChatGPT desktop app (bundled Codex app-server, Codex CLI
0.148.0-alpha.9).Reproduction evidence
Using
ps -axo pid,ppid,etime,stat,command, I confirmed that the accumulating processes were direct descendants of the long-lived ChatGPT-ownedcodex app-server, rather than Hermes, a LaunchAgent, or another watcher/service.A real session accumulated approximately:
node_replprocessesAfter terminating only those verified app-server-owned children, a few additional tool-enabled turns created new stdio MCP /
node_replpairs that remained alive. A separately parented instance of the same MCP server continued running, confirming that process-name-wide cleanup would be unsafe.This appears to be the same cross-platform lifecycle problem reported here: a single desktop task repeatedly creates stdio MCP generations instead of converging to a bounded baseline.
Root cause found
I traced two cooperating lifecycle gaps:
Superseded connection generations additionally need lease-aware retirement: an in-flight tool call must retain its exact connection, while the process should be cleaned up once the last lease is released.
Reference implementation
A tested reference implementation is available for maintainers to inspect or adapt:
najy97:codex/mcp-stdio-lifecycle-cleanupmain: openai/codex compare view0a8fb85691daa48072f4The focused patch:
SIGTERM -> grace period -> SIGKILL -> exit confirmationon only the owned Unix process group<details>
<summary>Implementation map and lifecycle invariants</summary>
stdio_server_process.rsStdioServerProcessHandleclones share one retryable cleanup state and target only the process group, Windows Job Object/process handle, or executor handle captured at spawn.Idle | Failed -> Running -> Complete | Failed; every concurrent caller joins the same running attempt.connection_manager.rsruntime.rsprocess_group.rsEPERMmember fallback stay scoped to the captured process group.connection_manager_tests.rsprocess_group_cleanup.rsmcp_refresh_cleanup.rsstdio_message_limits.rsSuggested adoption order: spawn-owned process cleanup, pending reuse, lease-aware retirement/final barrier, then platform-specific regression coverage.
</details>
2026-08-20 hard-exit and cleanup-barrier update
The reference branch now includes two additional review-sized commits:
20e4d940d2: process-owned stdio cleanup across runtime and hard parent exits0a8fb85691: explicit connection leases, retryable retirement, and the final session cleanup barrierAdditional lifecycle hardening:
Arc::strong_countpolling for in-flight calls, resources, and event streamsSIGKILLstill triggers owned process-groupSIGTERM, bounded wait, andSIGKILLCurrent validation on macOS:
SIGKILLcodex-mcp: 183/183codex-utils-pty: 28/282026-08-21 latest-main rebase and native Windows validation
The branch is now rebased onto upstream
daa48072f4withrmcp3.1.3 and the new hosted MCP event-streaming path preserved alongside explicit connection leases.Native Windows validation on exact commit
0a8fb85691:cargo build -p codex-cli --bin codex: passcodex-utils-pty: 23/23 passcodex-rmcp-client: 251/251 pass, 7 skippedcodex-mcp: 190/190 passtest_stdio_server.execount0 -> 0after every runLegacyandV20260728protocol modes covered across success, failure, timeout, and cancellationCODEX_CLI_PATHProcess/User/Machine state, and permanent User/Machine PATH hashes remained unchangedProcess enumeration was used only as an after-the-fact baseline assertion. Cleanup itself remained scoped to spawn-captured Job/process ownership.
Validation
The patch has been validated with:
node_repl, then both reached zero on app-server shutdown while the separately owned service process survivedcodex-rmcp-client231/231,codex-mcp182/182,codex-utils-pty22/22, plus the focused core integration test1e3286d51f:codex-utils-pty21/21,codex-rmcp-client226/226, andcodex-mcp183/1833b45c29062(mainat validation time, includingrmcp3.1.2); the lifecycle-scoped run passed 449/450 tests and the focused core integration test passed, with the sole unrelated OAuth redirect failure reproduced identically on the upstream basegit diff --checkThe Windows verification used spawned-process handles/Job ownership for cleanup. Exact-path process enumeration was used only as an after-the-fact assertion that the baseline returned to zero.
Related reports include #17832 and #12491.
Under the repository's current contribution policy, external code contributions and pull requests are not accepted, so I will not open one. I am leaving the branch and validation evidence available as a technical reference, and I would be happy to provide additional diagnostics or test a maintainer-provided candidate build on macOS and Windows.
Corroborating on macOS Desktop, with a controlled reproduction and a regression window.
Environment: macOS (Apple Silicon, 36 GB). ChatGPT/Codex Desktop
26.810.41047→26.810.52044, embeddedcodex-cli 0.148.0-alpha.9; ~18 configured MCP servers (mostly stdio). Also tested: signed26.721.81911(embedded0.146.0-alpha.3.1).Generation leak (this issue): With a single visible task, every recorded
task_started/task_completeboundary spawned a complete new stdio MCP generation (~14 processes) with the prior generation retained — 9 generations in ~31 min, timestamps matching boundaries to the second. Incident peak: ~107 complete suites / ~1,500 children / ~41 GB RSS under one app-server, ending in a kernel watchdog reset. The leak also reproduces on the older0.146.0-alpha.3.1line (5 generations in ~10 min), so it predates the current release line.New controlled finding — reaping strands the task (confirms the #28704/#35486 cross-refs): We ran an external reaper to contain memory (SIGTERM stale generations, newest preserved). A same-task/new-task A/B with timestamps: after a sweep, the existing task's tool calls fail in ~6 ms with
Transport closedpermanently, while a newly created task (fresh generation) passes the same calls. This matches the source: stdio transports are excluded from retry/reconnect (codex-rs/rmcp-client/src/streamable_http_retry.rs—PendingTransport::Stdio => false), so any workaround that kills leaked children permanently kills those tasks' tools. Operators are stuck: reap and strand tasks, or don't and hit the watchdog.Regression window that may help bisecting: embedded
0.147.0-alpha.6.5ran multi-agent sessions against the same 18-server MCP config for days with no melt.0.148.0-alpha.9additionally fails all stdio tool calls even after a fresh cold restart (single generation, direct JSON-RPCinitialize/tools/listagainst the same wrappers passes) — that fresh-start transport failure does not reproduce on0.146.0-alpha.3.1.Aggravator: per #37548, the app re-enables Sparkle auto-update on every launch; the working
0.147host bundle was overwritten in place, and there is no official downgrade path — so recovery from this regression required living with the older, still-leaking line.Asks: (1) consider inviting @najy97's prepared lifecycle-cleanup branch as a PR — it targets exactly the retained-generation mechanism; (2) stdio reconnect/rebind parity with streamable HTTP, or a documented lifecycle contract; (3) a named candidate build — we have a deterministic 10-boundary qualification harness and will report results back.
Дополнение: воспроизводится на stateful Java stdio MCP и ломает состояние workspace.
Окружение:
java -Xmx8g -jar bsl-language-server-1.0.8-exec.jar -c ../.bsl-language-server.json mcpПосле чистого перезапуска Codex был один дочерний Java PID 47376. Один вызов
list_workspace_foldersсоздал второй PID 42824; оба остались живы. В предшествующем прогоне накопилось 11 Java-процессов, созданных с 22:52:51 по 22:57:36; все имели общим родителемcodex.exe.Фрагмент
C:\Users\volos\.codex\logs_2.sqlite:Для трёх последовательных инициализаций в одном Codex process:
BSL получает от Codex:
Последствие:
register_workspace_folderхранит root только в конкретном экземпляре BSL. При следующем создании процесса список workspace пуст, поэтому stateful MCP-инструменты невозможно использовать надёжно. Тот же JAR в Claude Code работает без этой проблемы.I can reproduce this on a newer Windows build of the Codex App, and in my case the accumulation eventually causes severe system-wide stuttering/freezing as system commit approaches the limit.
Environment:
26.810.7004.010.0.26200.0 x64I captured a live process/memory snapshot while Windows was already starting to stutter.
The Codex process tree had accumulated exactly 17 copies of several configured local MCP/runtime processes.
Direct children of the Codex process roots included:
npx.exenode.exenode_repl.exepython.exeThe repeated groups included Playwright MCP, Chrome DevTools MCP, Context7, filesystem MCP, node_repl, and a local Python MCP server.
The repeated count of exactly 17 instances across multiple configured runtimes is particularly suspicious. This does not look like a single leaking Node heap. It looks like complete MCP/runtime generations are being created repeatedly and retained instead of being reused or reaped.
At the time of the snapshot, the whole system had:
node.exesystem-wide accounted for:I am not claiming that all 162
node.exeprocesses belonged to Codex, because other development applications were also running. The direct-child attribution above is from a PID/PPID snapshot and is the portion I can attribute directly to the Codex process roots.Kernel pool usage was also elevated during the incident:
For comparison, a previous clean post-reboot baseline on the same machine was approximately:
So combined kernel pool usage was roughly 2.3x above that previous baseline.
However, this incident looks different from a previous Codex-related failure I captured on the same machine.
In that earlier incident, the dominant symptom was kernel paged-pool growth:
Tokealone reached ~12 GiBSeAt,SeTd,SeDt, andSeTlwere also elevatedgit.exe,powershell.exe,conhost.exe,cmd.exe, andnode.exelaunches associated with CodexAfter reboot in that earlier case:
Tokereturned to approximately 7–8 MiBThe current incident is therefore not dominated by the same extreme kernel
Tokegrowth.This time, the much larger contributor is accumulated user-space process private commit: approximately 46 GiB of aggregate private bytes with system commit at approximately 54.3 / 59.9 GiB.
The machine becomes progressively less responsive as this state accumulates. By the time of the snapshot, system commit was above 90% and Windows was visibly stuttering/freezing.
This occurred during a long-running Codex workload rather than immediately after startup. The MCP/runtime process population appears to accumulate over time instead of returning to a bounded baseline when previous tool/session contexts are no longer needed.
The process snapshot strongly suggests repeated MCP/runtime generations are involved because multiple independent configured runtimes were present in exactly 17 copies at the same time.
I have preserved sanitized diagnostics from the degraded state, including:
I can provide the sanitized snapshots if they are useful for debugging.
---
ALSO!
As a live workaround, I terminated several old MCP generations using
taskkill /PID <pid> /T /Fwhile leaving the main Codex app-server and active Long Horizon task running.This reduced the system from:
node.exeto:
node.exeThe active Codex Long Horizon task continued running.
This suggests that old MCP generations and their descendant process trees account for a substantial portion of the accumulated memory/process pressure, and that they can be removed independently of the active app-server/task.
Request for maintainer triage / canonical fix path
This issue now appears to be one of the clearest central reports for the Desktop stdio MCP lifecycle failure also seen in #38825, #38765 and #38877.
The external patch already linked above is unusually actionable: it targets pending-connection reuse, shared shutdown completion, cancellation-safe cleanup and lease-aware retirement, and it has focused validation showing repeated tool calls stay bounded at one MCP server + one
node_repl.Could a Codex maintainer please confirm whether this is aligned with the intended fix direction and, if so, invite the PR or point to the internal/public PR that supersedes it?
For affected Windows users this is now a production-blocking reliability issue rather than a cosmetic leak: related reports include system commit exhaustion, severe stutter/freezes and an OS bugcheck/reboot. A named candidate build would also be useful; I can run a controlled Windows qualification against it and report process-count/commit behavior.
Additional current-version reproduction: Python STDIO transports continuously multiply
Reporter: @terrydwisely
This is a same-machine, same-app-server reproduction on Codex Desktop
26.810.7004.0. It adds exact process ancestry, timestamps, memory impact, post-restart behavior, successful live integration tests, stale transport routing, Windows launcher behavior, an absolute-path mitigation, and independent Claude Code verification.The Outlook and Google Workspace MCP servers must remain enabled. The user is living with the leak until OpenAI ships an application-level lifecycle fix.
---
Windows Codex MCP process leak and duplicate-spawn report
Title
[Windows][Desktop] STDIO MCP servers multiply within one session, survive app-server restart, and bare commands can launch through cmd.exeEnvironment
OpenAI.Codex_26.810.7004.0_x640.148.0-alpha.9ChatGPT.execodex.exe -c features.code_mode_host=true app-server --analytics-default-enabledinbox-watchmanandrefresh-terry-s-morning-briefingboth pausedConfigured STDIO servers
The commands originally used bare
python. They were changed to the verified Python 3.14.2 executable above. The original config was backed up before editing.Actual behavior
There are separate duplicate-spawn, cleanup, transport-routing, and Windows-launch defects.
1. Duplicate servers inside one app-server lifetime
The current desktop app-server, PID 53644, started at 1:49:59 PM. In its first 25 minutes it produced eight MCP server launch events instead of one Outlook launch and one workspace-mcp launch. Three Outlook launches occurred within four seconds: 1:59:40 PM, 1:59:41 PM, and 1:59:44 PM.
The duplicate behavior reproduced again while this report was being prepared:
This proves the application is creating a fresh server manager/process set on a later turn or steer event instead of reusing or disposing the earlier set.
Post-restart verification at 2:33 PM
The desktop MCP restart did not correct the duplicate-spawn defect:
cmd.exeor WindowsApps Python alias launchers.This is a clean post-restart reproduction: normal application startup, an MCP restart, and ordinary integration use each retained a separate server set instead of converging on one instance per configured server.
Independent verification by Claude Code at 2:39 PM
Claude Code independently inspected the same machine and confirmed:
codex.exePID 30216 started at 2:33:16 PM.cmd.exewrappers and zero WindowsApps Python alias launchers were present.Current mitigation: living with it until OpenAI fixes it
Both MCP servers must remain enabled because Outlook and Google Workspace are required integrations. The absolute-path configuration is being retained because it removes the alias chain and target console-window launcher. There is no available configuration setting that prevents Codex Desktop from continuously spawning and retaining duplicate transport pairs, so the process leak remains active pending an application fix from OpenAI.
Continued growth while filing the issue
A Codex-side process snapshot at 2:43:39 PM found the same app-server PID 30216 still running with seven Outlook server instances and six workspace-mcp entrypoints. Additional Outlook launches appeared at 2:41:40 PM, 2:43:01 PM, and 2:43:36 PM; additional workspace-mcp launches appeared at 2:41:40 PM and 2:43:03 PM. The newest Outlook process had appeared only three seconds before the snapshot and its corresponding workspace-mcp entrypoint had not yet appeared, showing that startup is neither deduplicated nor serialized as one atomic pair. The bad-launcher count remained zero.
2. Children survive an app-server restart
An earlier app-server PID 11916 started an Outlook and workspace-mcp set at 1:23:30-1:23:35 PM. The desktop restarted its app-server at 1:49:59 PM and replaced it with PID 53644, but all 12 processes in the 1:23 set remained alive with the now-nonexistent PID 11916 recorded as the parent of the two
cmd.exeroots.Relevant desktop log excerpt:
The stale 1:23 tree was later terminated explicitly: 12 processes, 146.8 MB working set. The stale 1:59 tree was also terminated after the 2:13 duplicate appeared: 8 processes, 332.3 MB working set. No server files or configuration entries were removed.
3. Tool routing remains bound to the stale copies
After the stale 1:59 pair was terminated, the new 2:13 pair remained live and used the corrected absolute commands. However, Outlook and workspace-mcp test calls in the active task both returned:
An automatic retry did not rebind to the already-running 2:13 servers. The application spawned replacement copies but kept the active tool router attached to the older transports.
4. Bare commands use an unnecessarily deep Windows launch chain
With
command = "python", the observed workspace-mcp process tree was:The Outlook server similarly used
cmd.exe -> WindowsApps Python alias -> real Python.5. Console windows can flash
The 1:23 launches used commands such as:
Each
cmd.exehad aconhost.exechild. With Windows Terminal configured as the console host, this produced a visible terminal flash.Configuration-fix verification
An isolated app-server 0.148.0-alpha.9 probe loaded the edited config, started an ephemeral thread, and queried MCP status.
ready, server version 1.26.0.ready, server version 3.4.7.workspace-mcp.exeentrypoint ran.cmd.exeprocesses were used by either server.This verifies the absolute-path config workaround. It does not fix the desktop application's duplicate-spawn or stale-routing logic.
Expected behavior
thread/start, turn start, steer, status refresh, and config reload cannot race into duplicate launches..execommands directly with redirected stdio andCREATE_NO_WINDOW; do not usecmd.exefor executable commands.User impact
Suggested attachments
Reporting route
/in the Codex composer and choose Feedback. Include this task/session with the report.https://github.com/openai/codex/issues.https://github.com/openai/codex/issues/new/chooseand attach the report plus reviewed evidence.---
Related active reports found before submitting this evidence: #30408, #33946, #35485, #38526, #38614, #38925, and #38981.
Corroborating on Windows Desktop Codex app 26.814.5517.0.
I originally suspected a project build-directory assemble/teardown race, but that race removes build artifacts during parallel tests and produces missing-file/worker failures; it does not spawn or retain these processes.
Independent current-build Windows reproduction: 659 descendants / 30.94 GiB working set
Corroborating this on a current Windows build with a continuous live capture.
Environment
OpenAI.Codex 26.814.5517.0x640.148.0-alpha.1510.0.26200features.code_mode; the tool-enabled environment is initiated by Desktop/runtime.User-visible impact
After several hours of ordinary Codex work, Windows developed severe system-wide lag. Moving normal desktop windows became difficult and input/UI latency was very high. Codex could not be restarted immediately because two project tasks were still running, which allowed the same app-server lifetime to be measured continuously.
No Codex-owned processes were terminated during this capture.
Continuous growth under one app-server
The same app-server PID remained alive throughout, started at
2026-08-19 16:43:56local time.| Local time | Codex descendants | Working set | Private memory | node_repl | node | cmd | python | Handles | Threads |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 19:27 | 352-358 | 16.5-16.7 GiB | 12.2-12.3 GiB | 15 | 165 | 114 | 30 | 67,383 | 3,318 |
| 19:38 | approximately 510 | 24.0 GiB | 17.31 GiB | 22 | 238 | 167 | 44 | not sampled | not sampled |
| 19:46 | 659 | 30.94 GiB | 22.04 GiB | 30 | 303 | 219 | 58 | 128,480 | 5,701 |
Host memory pressure increased over the same interval:
Windows appears to have expanded the commit limit during the capture. The application was approaching the host commit limit rather than merely retaining a harmless number of idle processes.
Repeated stdio generations
At 19:46:
node_repl.exeinstances were direct children of the same app-serverserena.exeentrypoints and 58 associated Python processes remained liveThis is not a process-name attribution from the whole machine. Counts were taken only from the recursively enumerated descendant tree of the verified Codex app-server.
Shutdown evidence
The active local app-server log contains successful terminal tool outcomes followed by logical session shutdown records, including:
dispatch_tool_call_with_terminal_outcomeop.dispatch.shutdownShutting down Codex instanceIn a bounded recent log window there were 80 explicit
Shutting down Codex instancerecords, while the corresponding stdio generations remained in the live app-server process tree.This independently supports the lifecycle diagnosis in this issue: logical session/executor shutdown is occurring, but the owned stdio MCP process generations are not converging to a bounded pool and are not being reaped.
Saturation checks
A separate 15-second sample during the same reproduction showed:
The immediate degradation therefore aligns much more strongly with retained process/memory growth than with sustained CPU, DPC, GPU, or disk saturation.
Expected behavior
Completed tool/session contexts should either reuse an unchanged MCP connection or deterministically reap their owned Windows process tree after the final lease is released. The process count and committed memory must converge to a bounded baseline during a long-running Desktop session.
This also provides a concrete mechanism for the Windows-wide lag previously reported in #29187 and is consistent with the retained MCP batches reported in #32797.
Raw task transcripts, complete databases, environment values, and private paths are intentionally omitted. Sanitized aggregate CSV samples and a bounded log summary can be provided if maintainers request them.
A temporary containment method on Windows is:
Limitation: this interrupts active tasks and only contains the leak. Avoid killing processes globally by executable name, because the same MCP runtime may belong to another application.
macOS reproduction with turn-count, process-tree, swap, and restart A/B evidence
I reproduced the same unbounded per-turn stdio MCP generation growth on macOS, including system-wide swap exhaustion and an OOM incident.
Environment
codex-cli 0.148.0-alpha.15mcp-fetchProcess evidence before a full Desktop restart
The long-lived Desktop app-server owned 526 direct child processes. Exact live instance counts included:
mcp-server.cjsprocessesnpm execparents / 105 server childrennpm execparents / 105 server childrenThese were live sleeping processes retained under the same app-server lifetime, not Unix zombies.
The corresponding Desktop log window contained:
turn/start: 99thread/start: 7thread/resume: 25shutdown,close,exit,SIGTERM): 0The near 1:1 match between 99
turn/startcalls and approximately 105 retained Context7/fetch generations supports the per-turn lifecycle correlation reported here; this was not 105 concurrently active user tasks.System impact
At peak:
The claude-mem Bun worker itself was idle at approximately 100 MiB physical footprint with queue depth 0, so the current pressure was not a Bun work queue or model-request backlog.
Restart A/B
The only cleanup intervention was fully quitting and reopening the Desktop app.
Immediately after restart:
This is a current-build macOS reproduction affecting both user-configured stdio MCP servers and a plugin-provided MCP server. It also shows that the retained generations remain owned by the app-server and are released when that owner exits.
I can provide sanitized process-count commands and log-correlation commands if useful for a maintainer-provided candidate build.
macOS Apple Silicon corroboration on current builds:
Local mitigation only reduced blast radius: disabled a stale duplicate registration, scoped an unrelated server out of the project, moved a credential to a mode-600 file-backed source, and performed exact-PID cleanup. It does not fix lifecycle ownership in Codex.
I can provide a sanitized process-tree capture or test a candidate build if useful.
Same defect reproduced on the CLI (not the desktop app), with a per-subagent trigger and a controlled census. Filing the detail here rather than keeping a separate issue open — full write-up was #39982.
Environment: Codex CLI 0.149.0, standalone install,
windows-x86_64, Windows 11 26200. 4 configured stdio MCP servers (1 disabled). Auth ChatGPT, modelgpt-5.6-sol.What is different from the report above: your accumulation is per turn within one task. Ours is per subagent, and the server involved starts lazily (only on first tool use), so this is not eager startup — the server is spawned fresh for a subagent that actually calls it, as a new direct child of the same live
codex.exe, instead of attaching to the instance already running under that supervisor.Controlled run (two subagents, one lazily-started stdio server):
It is not one misbehaving server. After that server was removed from the config entirely, the same live supervisor still held duplicate fleets of four unrelated stdio servers (a Node REPL runtime,
uvx mcp-obsidian, and twobun-launched plugin-cache servers). An earlier snapshot on the same machine had one live supervisor holding 5 concurrent instances of a single server and a second holding 3 more. Any stdio server reachable from more than one subagent accumulates the same way.Relationship to the existing fix: #19753 ("Terminate stdio MCP servers on shutdown") is merged, and it does address shutdown. It does not cover this case, because the supervisor never shuts down — it stays live for the whole session while duplicates pile up underneath it. #38925 makes the same observation.
Why this cannot be cleaned up from outside the client: the supervisor is live the whole time, so nothing in the process tree distinguishes a redundant duplicate from a fleet in active use. There is no lease, no connection id, and no terminal state to read, so terminating a duplicate is a guess that can tear down a server another subagent is mid-call on. Our own cleanup tooling reclaims only fleets whose supervisor is gone; for duplicates under a live supervisor it is deliberately report-only — it counts them and refuses to act.
Ask (either one closes the class): reuse one fleet per active client/project so a second subagent attaches to the running server, or close stdin/the child when the owning client/subagent ends so the server sees EOF. Independently useful: expose a stable terminal state for a connection or lease, so external tooling can act on facts instead of heuristics.
Closely related, same root cause from different surfaces: #38693 (subagent-scoped trees, Desktop), #38925 (accumulation under a live app-server), #37870 (CLI, completed subagents), #38353 (proposes the pooling/teardown design).
Source-level root cause, condensed here since our thread was folded into this one. Full write-up with the complete chain: https://github.com/openai/codex/issues/39982#issuecomment-5373515471 — read against
main@51ebf5b1842d44a8e2c955e8b5cd2a589d41e71e.Short version: this isn't a missing teardown. Teardown is correct where it runs — it just never runs for a subagent that merely finishes.
Per-thread fleets. Every thread, root or subagent, owns its own
Session.services.mcp_runtime(codex-rs/core/src/session/mcp_runtime.rs) → its ownMcpConnectionManager→ its own stdio children. Subagents do not share the parent's.Only one reap path. Children are terminated only by
mcp_runtime.shutdown()inshutdown_session_runtime(codex-rs/core/src/session/handlers.rs:397, call at:417), reachable only viaOp::Shutdown. For a subagent that arrives from exactly two places:close_agent→shutdown_live_agent(codex-rs/core/src/agent/control/legacy.rs:8,:48)codex-rs/core/src/agent/control/residency.rs:117)A subagent that simply reaches
Completedhits neither. It stays resident on purpose so the parent canwait/resume/send_input, and its whole MCP fleet stays alive with it.Why the count climbs. Eviction is spawn-triggered only (
residency.rs:49) — nothing proactively reaps a completed resident. Capacity ismax_concurrent_threads_per_session - 1, default 4 → 3 (codex-rs/core/src/config/mod.rs:224,effective_agent_max_threadsat:1507). Steady state ≈ (live root threads) × (1 + up to 3) fleets. Our controlled two-subagent pilot took a process census 10 → 12 with nothing reaped on completion, and removing one server left other duplicate fleets standing — the duplication is per-thread, not per-server.Second-order.
impl Drop for McpServerConnection(codex-rs/codex-mcp/src/connection_manager.rs:148) only callscancel_token.cancel(); it never awaitsshutdown(). Any drop not preceded by an explicit shutdown is therefore not a guaranteed reap — relevant on Windows, where an orphaned child isn't cleaned up by process-group semantics.Suggested fix (smallest defensible). Split "resident so history/resume work" from "holds live OS processes": when a subagent reaches a terminal state — the predicate already exists as
is_unloadable(residency.rs:233:Completed/Errored/Interrupted, no active turn, no pending mailbox items) — callmcp_runtime.shutdown()and mark the runtime dormant, re-publishing on the next turn via the existingpublish_mcp_runtime/request_mcp_runtime_refreshpath. Resume, history and transcript reads never touch the MCP runtime, so nothing user-visible regresses, andensure_v2_agent_loadedalready rebuilds fully-evicted threads — a dormant-runtime resume is strictly smaller than what resume handles today.Workaround for anyone hitting this now: set
agents.max_concurrent_threads_per_session = 1to shrink the resident cache to zero (costs subagent concurrency), or have the parent callclose_agenton each subagent when done — that path does reap immediately.Following up on my earlier root-cause comment above with a concrete patch.
docs/contributing.mdsays external PRs are not accepted, so this is a comment rather than a pull request — the branch is linked at the bottom if it is easier to read there.Read against
main@ad32eba832.Recap of the cause
Completed subagent threads are deliberately kept resident, and each resident thread owns a full MCP fleet.
shutdown_session_runtimereaps stdio children correctly, but it is only reached viaOp::Shutdown— sent fromclose_agentand from residency eviction, neither of which runs when a subagent simply finishes. So the steady state per root thread is up tomax_concurrent_threads_per_session - 1completed-but-resident subagents, each holding a duplicate fleet.The two properties conflated are "the thread stays resident so history/resume work" and "the thread holds live OS processes". The patch below separates them.
The change
Three files, ~74 lines.
1.
codex-rs/codex-mcp/src/runtime.rs— a way to retire the published generation without shutting it down. FactoredPublishedMcpRuntime::emptyout ofMcpRuntime::emptyand added:Retiring rather than calling
shutdown()matters: an out-of-bandcall_mcp_toolcan be running when a turn ends, and it holds theArc<McpConnectionSet>it captured. Retiring gives it exactly the lifetimereplace()already gives a superseded generation — which is whatrefresh_keeps_superseded_mcp_server_alive_for_in_flight_callspins down. The stdio children then exit when that last reference drops, through thekill_on_drop(true)and the Windows job object already set up instdio_server_launcher.rs, so descendants go too.2.
codex-rs/core/src/session/mcp.rs— the policy, next tomark_mcp_runtime_dirty:The guard mirrors the
is_unloadablepredicate already inresidency.rs.mark_mcp_runtime_dirty(notrequest_mcp_runtime_refresh) is deliberate: it must not schedule a prewarm, or the fleet respawns immediately. The nextrefresh_mcp_if_dirty— which every tool-call path and turn start already goes through — republishes.3.
codex-rs/core/src/tasks/mod.rs— one call site, at the end ofon_task_finished:Ordering is load-bearing: if queued work started a new turn,
active_turnis set again and the release is skipped.Root threads are untouched,
McpStartupPolicy::LazyWhenCachedstill applies to subagents, and disabled servers are unaffected because nothing about the projection changes — the next publish is the same publish that would have happened anyway.Tests
Four cases added to
codex-rs/core/tests/suite/mcp_refresh_cleanup.rs, in the style of the existing test there (real stdio server,MCP_TEST_PID_FILE, process-liveness assertions):call_mcp_toolstill holds its connection when the turn completes, and the process exits only after that call is released;What I ran
cargo clippy -p codex-core -p codex-mcp --tests -- -D warningsis clean. The suite is#[cfg(unix)], so the lifecycle tests ran on Linux even though the original report was Windows 11 / CLI 0.149.0:The last one is your existing test, included as a control that the retire path does not change superseded-generation lifetime.
Branch, if a diff is easier to read than a comment: https://github.com/yotamleo/codex/tree/fix/mcp-runtime-release-on-terminal-subagent — no PR opened, per the contributing policy. Happy to re-measure the process census against a build if that would help.
One thing I did not change
impl Drop for McpServerConnectionstill only callscancel_token.cancel()and does not awaitshutdown(). That is fine for this path because the transport iskill_on_drop, but it does mean any other drop-without-shutdown path depends on that rather than on an explicit reap.Windows corroboration: process/thread/handle exhaustion reaches Win32 1816
I have a separate Windows 11 long-running Codex reproduction that appears to extend this same lifecycle family from duplicate stdio/MCP children into eventual process-creation failure.
Sanitized live snapshot before cleanup
node,node_repl, Python bridge/MCP helpers, and Codex-owned/helper descendantsNot enough quota is available to process this commandThose aggregate counts are system-wide; I am not claiming every process/handle is Codex-owned. The current cleanup pass is deliberately tracing PID, PPID, executable, creation time, command fingerprint, and owning task/lease before terminating anything.
The user-visible pattern is consistent with this issue: long-running Codex work creates helper generations faster than terminal task/turn cleanup returns them to baseline. Restart/cleanup temporarily restores process creation.
Current-main code audit
I also checked current public
openai/codexsource rather than relying only on the observed process tree.codex-rs/rmcp-client/src/stdio_server_launcher.rsnow has useful Windows Job Object/process-handle containment, butStdioServerProcessHandlestill uses a singleterminated: AtomicBool: the first local caller marks termination complete, signals the Job/process, and returns without that method waiting for a verified tree exit/reap; later callers return immediately once the flag is set. That still leaves a cancellation/concurrent-shutdown correctness gap of the kind described in the reference implementation already posted above.Separately,
codex-rs/codex-mcp/src/runtime.rs::replace_fresh()publishes withprevious: None, and the current session MCP refresh path callsreplace_fresh(). That means hard refreshes intentionally forgo connection reuse, making correct retirement/reaping of old generations especially important.Additional acceptance criterion suggested by the Windows failure
Please consider a Windows regression that repeatedly performs tool-enabled turns/subagents and asserts all of the following return to a bounded baseline after their owners reach terminal state:
node_repl/MCP generationsCreateProcesswithout Win32 1816Cleanup should remain spawn/lease-owned; process-name-wide cleanup would be unsafe because unrelated same-name Node/Python processes can be active at the same time.
I am preserving the detailed local process census privately and can add further sanitized before/after measurements once the ownership-bound cleanup run completes.
Follow-up after current-main history review: this reproduces after the existing Windows process-tree fixes
I checked the relevant landed history so this report does not ask for work Codex already has:
9daa491f7c27a5513fec554473a7122d88fca367/ #37366 (2026-08-07): Harden local MCP server process tree cleanup — Windows non-breakaway Job Objects, process-handle fallback, descendant cleanup tests.8751fd3fcb8031d42b62670a6131872074635c9b/ #29608: explicitly shut down superseded MCP managers on refresh.4e0cf945b7f43f1f9c1d09faadac6def76bdefbf/ #19753: explicit stdio MCP shutdown/draining on session shutdown/refresh.82c981cafc57dfed8383e72c6bbf6082622a3b4a/ #10710: original process-group cleanup.So the Aug 24 Windows exhaustion reproduction should be treated as a remaining lifecycle-generation/retirement/postcondition defect, not simply “Codex lacks process-tree cleanup.”
The remaining current-main concerns are:
StdioServerProcessHandleInnerstill collapses shutdown state into oneterminated: AtomicBool; the first caller signals termination and returns, while later callers can treatterminated=trueas terminal without sharing/waiting for a verified cleanup result.McpRuntime::replace_fresh()withprevious: Noneon the current session refresh path, so any missing retirement edge has multiplicative impact.The useful regression therefore is not merely “grandchild dies when handle drops.” It is a long-lived app-server task/subagent sequence that measures active MCP generations + process/handle/listener counts and requires them to return to a bounded baseline after each owner reaches terminal state, then verifies a fresh CreateProcess still succeeds.
I’ll add exact before/after ownership-bound Windows cleanup measurements once the local census finishes.
Additional finding: Win32 1816 should be treated as an operation failure requiring attribution, not a machine-capacity diagnosis
A recent Windows reproduction initially surfaced Win32 1816 during a Codex-managed process launch. Further investigation showed that the machine remained capable of launching substantial workloads; the failure was specific to the Codex execution path and required tracing the actual failing operation.
Recommended reliability behavior:
A concrete local recovery path required fixing the failing component rather than relying on small targeted cleanup events. Cleanup should be considered successful only when the original capability is restored or ownership state is proven healthy.
Suggested acceptance criterion:
process launch failure -> ownership attribution -> targeted repair -> successful retryrather than:
process launch failure -> Win32 error -> assume environment blocker.