CLI: subagents memory leak

Open 💬 6 comments Opened May 18, 2026 by sigyulee

💡 Likely answer: A maintainer (github-actions[bot], contributor) responded on this thread — see the highlighted reply below.

What version of Codex CLI is running?

0.130.0

What subscription do you have?

Pro Lite

Which model were you using?

_No response_

What platform is your computer?

Apple Silicon

What terminal emulator and version are you using (if applicable)?

_No response_

Codex doctor report

What issue are you seeing?

Spawning many subagents causes the native codex process memory and thread count to grow rapidly. The CLI can become temporarily unusable and may eventually exit/crash.

This appears to happen inside the native codex process, not in the Node wrapper or external child processes.

Observation:

codex RSS grew from ~23 MB to 315 MB+
vmmap physical footprint reached ~363.7 MB
MALLOC_SMALL resident grew from ~29.8 MB to ~229.0 MB
malloc allocation count grew from ~79k to ~691k
thread count grew from 22 to 68

When the session becomes large enough, RAM leak worsens indefinitely, eventually emitting "codex(...) MallocStackLogging: can't turn off malloc stack logging because it was not enabled." and make system unusable. (See #17139 )

What steps can reproduce the bug?

Spawn subagents in a large session

What is the expected behavior?

Subagent fanout should have bounded memory growth.

Completed subagents should not remain fully live in memory indefinitely. Their event queues, session loops, thread resources, and model/tool state should either be drained, cold-unloaded, or released after completion while preserving enough metadata/results for the parent thread.

The CLI should remain responsive and should not exit/crash.

Additional information

Cause 1: v1 child event queue may not be drained

codex::spawn creates a per-session unbounded event channel:

session/mod.rs:482, async_channel::unbounded()
session/mod.rs:747, next_event() consumes from that receiver.

When a child thread is spawned, the thread manager consumes the initial SessionConfigured event, but it does not drain later child events:
thread_manager.rs:1197
thread_manager.rs:1245

The completion watcher only subscribes to child status, waits for a terminal status, and injects a completion message into the parent. It does not consume the child event stream:
agent/control.rs:320
agent/control.rs:943
agent/control.rs:1012
There is also TODO about this:
agent/control.rs:308

Cause 2: completed child sessions are not released
The completion watcher notifies the parent, but the child session remains live. Actual release appears to happen through the explicit close path:
agent/control.rs:715, shutdown_live_agent
agent/control.rs:737, close_agent
agent/control.rs:79, spawn slot reservation
agent/control.rs:99, release path

This means terminal children can keep their session loop, event channel, metadata, and associated runtime allocations alive until the user or client explicitly closes them.

Subagents getting the whole session transcript

View original on GitHub ↗

6 Comments

github-actions[bot] contributor · 3 months ago

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

#23219

Powered by Codex Action

emilfihlman · 1 month ago

Confirming repeated native codex OOM kills on Linux with Codex CLI 0.144.1 and 0.144.3.

Retained kernel journals contain three events in which the kernel recorded constraint=CONSTRAINT_NONE, global_oom, and killed the native codex process with SIGKILL:

| Confirmed event | CLI/session state | Cumulative Codex agent/session tree | Rollout set on disk | Native codex anonymous RSS |
|---|---|---:|---:|---:|
| 1 | 0.144.1 | 34 nodes | 154.1 MiB | 3.42 GiB |
| 2 | 0.144.1 | 72 nodes | 345.6 MiB | 3.35 GiB |
| 3 | 0.144.3 process after resuming root state created on 0.144.1 | 155 nodes | 1,022.6 MiB | 2.84 GiB |

The final two events belong to the same root session across a restart/resume. The tree counts are cumulative root, child, and nested session nodes, not concurrent workers.

At the three events, the host had 11.6 GiB usable RAM, no swap, approximately 3.9–4.1 GiB of shared/tmpfs memory, and approximately 0.10–0.15 GiB free. The kernel OOM task tables showed codex as the largest individual anonymous-memory process. Its oom_score_adj was 0.

The shell printed only Killed codex. Codex logs ended approximately 15 seconds, 4 minutes 35 seconds, and 14 seconds before the corresponding kernel records. I found no Rust panic, backtrace, fatal Rust error, clean-shutdown marker, or Codex coredump for these events.

Environment: Linux x86_64, GNU Screen 5.0.1, multi-agent CLI workload.

Sanitized codex doctor --json summary:

{
  "schemaVersion": 1,
  "codexVersion": "0.144.3",
  "overallStatus": "ok",
  "statusCounts": {
    "ok": 18
  }
}

No raw doctor output, logs, rollout files, terminal snapshot, process listings, or log database are attached because they contain local paths, project content, and session metadata.

fa1k3 · 1 month ago

Another Linux data point, with kernel-side evidence for how bad this gets when MCP servers are configured.

Environment

codex-cli 0.144.3 (npm, x86_64 musl vendor build)
Linux 6.17, 128 GB RAM, 512 MB swap
3 MCP servers in config.toml: 2 stdio (one of them loads a local embedding model), 1 streamable-http

What happened

A research fan-out with 12 parallel codex exec subagents (read-only, web/GitHub research). Within ~10 minutes the box allocated ~30-40 GB and went into a global memory livelock. It was unreachable for 8 h 35 min, then recovered on its own.

This matches the mechanism described in #23749: each subagent receives the full parent transcript and starts its own complete MCP stack. With K subagents that is K × the entire stack, including K copies of an embedding-model-backed server.

Timeline (from sar, 10-minute sampling)

| time | mem avail | page cache | load-1 | blocked | majflt/s | pgpgin/s |
|---|---|---|---|---|---|---|
| 12:00 | 45.3 GB | 31.0 GB | 0.94 | 0 | 0.00 | 0.7 |
| 12:10 | 39.8 GB | 19.7 GB | 1.16 | 0 | 0.23 | 1,684 |
| 12:20 | 2.06 GB | 4.0 GB | 11.6 | 64 | 213 | 59,797 |
| 12:40 | 1.94 GB | 3.9 GB | 674 | 87 | 4,044 | 1,196,746 |
| 21:10 | 29.3 GB | 7.3 GB | 0.67 | 0 | 24.7 | 11,144 |

sadc could not write a single sample between 12:40 and 21:00 despite a 10-minute timer.

Independent confirmation of the freeze duration

An unrelated container logged a line whose application-internal timestamp is 12:25:42, while the Docker-side write timestamp is 21:00:06. The process was blocked for 8.5 hours before it could flush one line to stdout.

Why the OOM killer took 8.5 hours

The kernel kept reclaiming page cache instead of killing anything (cache is always nominally reclaimable), and with only 512 MB of swap there was nowhere to put anonymous memory. Cumulative PSI full memory stall was ~8.95 h. It eventually fired:

oom-kill:constraint=CONSTRAINT_NONE,...,global_oom,task_memcg=/user.slice/...
Out of memory: Killed process <pid> (MainThread)

The moment the global OOM killer ran, the machine recovered. So this is not only "codex uses a lot of memory" - with no cgroup limit on the user session it degrades into a whole-host livelock rather than a failed codex run.

Peak of the cgroup the fan-out ran in: 64.5 GiB.

Workaround

Disabling all MCP servers for fan-out runs:

codex exec \
  -c mcp_servers.<name>.enabled=false \
  ... (one flag per server)

Research subagents do not need local MCP context anyway.

Note on a related footgun: -c mcp_servers="{}" looks like it should clear the table but is a no-op on 0.144.3 - codex mcp list -c mcp_servers="{}" still reports every server as enabled. Only the per-server enabled=false form works. A profile file layered via --profile also did not toggle them in my testing. Happy to open that separately if it is not already tracked.

Suggestion: sharing one MCP stack across subagents of the same run, or at minimum a documented per-run switch, would remove the K× multiplier that makes this dangerous on machines that otherwise have plenty of RAM.

Patchy · 17 days ago

Hit this at 17.3 GB RSS after a 32 hour multi-agent CLI session. Root cause:

Every observed thread gets a lazily created ThreadEventChannel (tui/src/app/thread_routing.rs) that buffers notifications while the thread isn't the active view. Channels for closed background agents are never removed. The buffer is also bounded by entry count (THREAD_EVENT_CHANNEL_CAPACITY = 32768, raised from 1024 in #9957), not bytes, and buffered ItemCompleted notifications carry full payloads (aggregated_output is ~1MiB, MCP results can be bigger). So a finished subagent can leave gigabytes pinned until the session ends.

vmmap on the 17.3 GB process shows 12.6 GB of MALLOC_LARGE across 1115 allocations (~11 MB average), which lines up with retained buffered payloads. Probably also behind #23749, #24668 and #20740.

I've been running this fix locally: free the channel when a background thread emits ThreadClosed (switching to it later just backfills from server history), plus a smaller capacity. https://github.com/Patchy/codex/commit/97dccc8fa5 if useful.

fa1k3 · 17 days ago

This matches what I hit. In my run the retained memory was dominated by buffered ItemCompleted notifications from finished background codex exec subagents — the tool-call results in those payloads are the largest contributor, well past the ~1 MiB aggregated_output, so an entry-count cap rather than a byte cap let a handful of closed agents pin most of the resident growth.

Both parts of your fix look independently useful: freeing the channel on ThreadClosed stops retention from scaling with the number of finished subagents, and the 32768 → 4096 cap bounds the live-thread case where a single busy agent buffers large results.

Any chance you'd open it as a PR against openai/codex? It currently only lives in your fork, and #23749 / #24668 / #20740 look like the same root cause.

Patchy · 16 days ago

Thanks for confirming. I'd happily open a PR but contribution rules say PRs are invite only (docs/contributing.md), uninvited ones get closed. i see @etraut-openai triaged this issue before, so flagging the root cause analysis above. The change is contained to the TUI app layer (thread_routing.rs plus one constant). Also the const change to THREAD_EVENT_CHANNEL_CAPACITY is more of a arbitrary suggestion i'm just running with.