Memory leak when running multiple agents
Open 💬 4 comments Opened May 20, 2026 by aniketyamleintern
💡 Likely answer: A maintainer (github-actions[bot], contributor)
responded on this thread — see the highlighted reply below.
What version of the Codex App are you using (From “About Codex” dialog)?
Codex
What subscription do you have?
Pro
What platform is your computer?
Darwin 25.5.0 arm64 arm
What issue are you seeing?
My codex crashed after running the application to 128GB memory usage on Macbook pro. Can you verify if it's memory leak or somethin when you run parallel agents.
What steps can reproduce the bug?
Feedback ID: 019e4666-b55f-7482-ba26-9db616a3c078
What is the expected behavior?
_No response_
Additional information
Feedback 019e4666-b55f-7482-ba26-9db616a3c078
4 Comments
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
Fix: Memory Leak When Running Multiple Agents — Crashed at 128GB on MacBook
Issue References
---
Executive Summary
When running parallel subagents, the Codex CLI process accumulates unbounded memory because:
async_channel::unbounded()event channel, but after the subagent completes, the channel is never consumed or closed, causing events to accumulate indefinitely in memory.ThreadManagerState.threadsHashMap retainsArc<CodexThread>references to completed subagents, preventing garbage collection of their session state, event channels, and associated runtime allocations.tokio::spawntask inmaybe_start_completion_watcherretainsArcreferences toAgentControlandThreadManagerState, creating a reference cycle that prevents cleanup.AgentRegistry.active_agents.agent_treepersist for the lifetime of the parent session.---
Root Cause Analysis
Cause 1: Unbounded Event Channel Never Drained
File:
codex-rs/core/src/session/mod.rsThe
tx_eventsender is cloned into theSessionand used throughout the session lifecycle to emit events (turn completions, tool calls, status changes, etc.). Therx_eventreceiver is exposed viaCodex::next_event()(line 733), but only the first event (SessionConfigured) is consumed during thread finalization:After the initial
SessionConfiguredevent, no consumer ever drains therx_eventchannel for subagents. The parent thread'sCodexstruct holds therx_eventreceiver, but the parent never callsnext_event()on child threads. This means:Eventsent viatx_event.send()during the subagent's lifetime is buffered in the unbounded channelTurnComplete,ToolCall,ToolCallOutput,AgentMessage,TokenUsage, etc.Cause 2: Completed Child Sessions Not Released
File:
codex-rs/core/src/thread_manager.rsWhen a subagent completes its work, the
ThreadManagerState.threadsHashMap is never automatically cleaned up. The only removal paths are:close_agent()call —agent/control.rs:785-796— requires the model to explicitly invokeclose_agenttoolshutdown_live_agent()—agent/control.rs:763-781— only called during explicit shutdownhandle_thread_request_result()onInternalAgentDied—agent/control.rs:754-758— only on errorThe completion watcher (
maybe_start_completion_watcher) notifies the parent but does not clean up the child's resources:Cause 3: Reference Cycle via Completion Watcher
File:
codex-rs/core/src/agent/control.rsThe completion watcher spawns a detached
tokio::taskthat holds a clone ofAgentControl:While
AgentControlusesWeak<ThreadManagerState>to avoid a direct reference cycle, the detached task still:Arc<AgentRegistry>alive during its lifetimeAgentRegistryholdsactive_agents.agent_treewith metadata for all spawned threadsused_agent_nicknamesHashSet grows without bound as new agents are spawnedCause 4: Forked Rollout Items Carry Full Parent History
File:
codex-rs/core/src/agent/control.rsWhen using
SpawnAgentForkMode::FullHistory, the child receives a copy of the parent's entire rollout history:For large sessions with many turns, each forked subagent carries a full copy of the parent's conversation history in its
InitialHistory::Forked(...)variant.---
Proposed Fix
Fix 1: Auto-Drain Event Channel on Subagent Completion
Add an event drain task that runs after the subagent's submission loop terminates, consuming and discarding any remaining events.
File:
codex-rs/core/src/thread_manager.rsFix 2: Auto-Cleanup Completed Subagents
Modify the completion watcher to automatically clean up completed subagents by calling
shutdown_live_agentafter notification.File:
codex-rs/core/src/agent/control.rsFix 3: Bounded Event Channel for Subagents
Replace the unbounded event channel with a bounded channel for subagent sessions, with a size that balances memory usage against the risk of blocking the submission loop.
File:
codex-rs/core/src/session/mod.rsNote: With a bounded channel, the
send_eventcalls must handle theSendErrorcase where the channel is full. Since the drain task (Fix 1) will consume events, this should only happen under extreme load. Consider adding a warning log when events are dropped.Fix 4: Truncate Forked History for Subagents
Add a configuration option to limit the amount of parent history copied to forked subagents.
File:
codex-rs/core/src/agent/control.rsFix 5: Periodic AgentRegistry Cleanup
Add a periodic cleanup mechanism for the
AgentRegistryto remove stale metadata entries.File:
codex-rs/core/src/agent/registry.rs---
Implementation Plan
Phase 1: Critical Fixes (High Impact, Low Risk)
codex-rs/core/src/agent/control.rsmaybe_start_completion_watcher)codex-rs/core/src/thread_manager.rsfinalize_thread_spawn)Phase 2: Defensive Fixes (Medium Impact, Medium Risk)
codex-rs/core/src/session/mod.rsCodex::spawn_internal)codex-rs/core/src/agent/control.rsspawn_forked_thread)Phase 3: Monitoring & Prevention (Low Impact, Low Risk)
codex-rs/core/src/agent/registry.rsAgentRegistryimpl)---
Testing Plan
Memory Profiling Tests
Automated Memory Test
Integration Test Scenario
---
Edge Cases & Considerations
1. Parent Thread Exits Before Child Completes
If the parent thread shuts down before the child completes, the completion watcher's
Weak<ThreadManagerState>upgrade will fail. The current code already handles this by returning early. With the fix, the cleanup code path must also handle this gracefully.2. Subagent Crashes (Non-Terminal Status)
If a subagent crashes without reaching a terminal status, the completion watcher may hang indefinitely. Consider adding a timeout:
3. Nested Subagents (Depth > 1)
Subagents that spawn their own subagents must also clean up their children. The auto-cleanup in
maybe_start_completion_watcherhandles this recursively because each subagent has its own watcher.4. Explicit
close_agentCalled Before Auto-CleanupIf the user/model explicitly calls
close_agentbefore the completion watcher triggers, the thread is already removed from the HashMap. The auto-cleanup must handleThreadNotFoundgracefully:5. Event Channel Drain Task Panics
If the drain task panics, it should not affect the parent process. The
tokio::spawnhandles this by default (panics in spawned tasks don't propagate).6. Bounded Channel Blocking
If the bounded event channel fills up faster than the drain task can consume,
send_eventcalls will block. This could stall the submission loop. Mitigations:send_eventcalls for subagents---
Prevention Strategies
1. Add Memory Budget Configuration
2. Add Telemetry for Memory Monitoring
3. Add Health Check Endpoint
4. Document Best Practices
Add documentation for users about:
close_agentexplicitly vs. relying on auto-cleanup---
Summary of Changes
| File | Lines | Change |
|------|-------|--------|
|
codex-rs/core/src/thread_manager.rs| 1239-1280 | Add event drain task for subagents ||
codex-rs/core/src/agent/control.rs| 991-1064 | Auto-cleanup completed subagents in completion watcher ||
codex-rs/core/src/agent/control.rs| 410-415 | Truncate forked history for memory safety ||
codex-rs/core/src/session/mod.rs| 480-481 | Use bounded event channel for subagents ||
codex-rs/core/src/agent/registry.rs| 79-167 | Add periodic cleanup for stale entries |---
Expected Impact
Based on the observations from issue #23260:
| Metric | Before Fix | After Fix |
|--------|-----------|-----------|
| RSS growth (23 MB → 315 MB+) | Unbounded | Bounded to ~50 MB per active subagent |
| malloc allocation count (79k → 691k) | Unbounded | Stabilizes after cleanup |
| Thread count (22 → 68) | Accumulates | Returns to baseline (~22) |
| vmmap physical footprint | ~363.7 MB+ | ~50-100 MB per active subagent |
| Event channel buffer | Unbounded | 0 (drained) or bounded (256) |
The fix should prevent the 128GB memory crash scenario by ensuring that completed subagents release all their resources promptly.
Hello from #23260
My understanding is that memory usage spikes because each subagent receives the full transcript from the parent, and MCPs are started for each subagent. Keeping the context and MCP catalog clean may help.
Corroborating evidence: native
app-serverheap, attributed to streamed-response buffers across concurrent subagent sessionsReproduced on the VS Code extension and captured the memory composition while it ballooned.
Environment: VS Code Codex extension
openai.chatgpt 26.609.30741, macOS 26.3.1, Apple Silicon, 17 GB RAM,gpt-5.5,model_reasoning_effort = "xhigh",[features] multi_agent = true.Repro: asked Codex to perform a large code-audit task. Native multi-agent fanned out to 5 new concurrent
session_loopsessions spawned within 18 seconds (from the app-server's own logs):Memory went from 1.7 GB to 14 GB in ~2 minutes; only an explicit
op: Interruptreleased it. Left unattended it freezes the machine (hard restart required).Memory attribution (
vmmapon the live nativecodex app-serverprocess):The footprint is almost entirely heap. The
MALLOC_REALLOCmass (buffers repeatedly doubled) points to unbounded accumulation, and the busiest log targets during the spike werecodex_api::endpoint::responses_websocketandcodex_api::sse::responses, i.e. streamed response bodies retained per concurrent session and not released until every session completes.sampleshowed hot frames incodex_core::tools::parallel::ToolCallRuntimeandcodex_app_server_transport::transport::serialize_outgoing_message.Net: N concurrent subagent sessions each retain their full streamed-response context in the parent
app-serverheap with no cap. On a 17 GB host, roughly 5 to 6 wide is enough to exhaust RAM and freeze the OS. A per-session buffer cap, or releasing completed-stream buffers before all siblings finish, would bound it.