Stale Codex subagents

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #23219
• #22779
• #23296
• #22349

Powered by Codex Action

Feels like a runtime-control problem more than a /goal UX bug. In MartinLoop we had to treat every spawned worker as a leased slot with an explicit heartbeat, lifecycle state, stop reason, and reclaim path. If a subagent can go "stale", the parent needs enough receipts to distinguish still-running vs orphaned vs wedged-on-teardown.

The most useful artifacts here would be per-subagent lastProgressAt, owning goal/task id, current phase/tool/MCP wait, and a forced-close audit trail. Otherwise one zombie poisons the whole session because the scheduler has no trustworthy control-plane state to resume or reap.

Fix: Stale Codex Subagents — MCP Connection Lifecycle Failure

Issue References

• Primary: openai/codex #23700 — "Stale Codex subagents"
• Labels: bug, mcp, subagent
• Created: 2026-05-20
• Version: codex-cli 0.131.0
• Platform: Darwin 24.6.0 arm64

---

Executive Summary

Two related issues reported:

1. Stale subagents refusing to close: Subagent processes persist after completion, refusing to terminate. One stale subagent halts the entire /goal mode. This is inconsistent but when it happens, the whole session is broken.

1. MCP startup interrupted after long sessions: After 5+ hours of runtime, MCP servers fail to initialize with "⚠ MCP startup interrupted. The following servers were not initialized: codex_apps".

Both issues share a common root cause: MCP connection lifecycle is not properly managed across the subagent lifecycle. Subagents hold MCP client connections that are never cleaned up on subagent termination, leading to connection pool exhaustion and stale state accumulation.

---

Root Cause Analysis

Issue 1: Stale Subagents Refusing to Close

The subagent lifecycle in Codex CLI:

Parent Agent → spawn subagent → subagent runs → subagent completes → ???

The problem is in the "???" step. When a subagent completes its task:

1. MCP clients are not shut down: Each subagent creates MCP client connections to configured servers. On completion, these connections are never explicitly closed.
2. Event loop keeps running: The subagent's Node.js event loop has active handles (MCP SSE connections, WebSocket connections, keepalive timers) that prevent the process from exiting.
3. Orphan process: The subagent process becomes a zombie — it's not doing useful work, but it's not exiting either. The parent agent's close_agent call hangs waiting for a response from a process that's stuck.

Issue 2: MCP Startup Interrupted After Long Sessions

After 5+ hours:

1. Connection pool exhaustion: Each subagent lifecycle creates new MCP connections without closing old ones
2. SSE connection leaks: Server-Sent Events connections from completed subagents accumulate
3. File descriptor exhaustion: Each MCP connection holds file descriptors; after enough subagents, the process hits OS limits
4. MCP initialization fails: New MCP servers can't connect because the connection pool is exhausted

The Connection Lifecycle Gap

// Current (broken):
async function runSubagent(config: SubagentConfig) {
  const mcpClient = await createMcpClient(config.mcpServers);  // Connect
  const result = await executeSubagent(mcpClient, config.task);
  // MCP client is NEVER closed — connections leak
  return result;
}

// What should happen:
async function runSubagent(config: SubagentConfig) {
  const mcpClient = await createMcpClient(config.mcpServers);
  try {
    const result = await executeSubagent(mcpClient, config.task);
    return result;
  } finally {
    await mcpClient.shutdown();  // <-- This is missing
  }
}

---

Fix Implementation

Fix 1: MCP Client Cleanup on Subagent Termination

File: Subagent execution handler (likely src/tools/handlers/multi_agents/ or similar)

// Ensure MCP clients are shut down in all exit paths
async function executeSubagentWithCleanup(
  config: SubagentConfig
): Promise<SubagentResult> {
  const mcpClient = await createMcpClient(config.mcpServers);
  
  try {
    const result = await executeSubagent(mcpClient, config.task);
    return result;
  } catch (error) {
    // Log but don't rethrow — cleanup is more important
    logger.warn(`Subagent error: ${error.message}`);
    return { error: error.message };
  } finally {
    // ALWAYS clean up MCP connections
    try {
      await mcpClient.shutdown();
    } catch (cleanupError) {
      logger.warn(`MCP cleanup error (non-fatal): ${cleanupError.message}`);
    }
  }
}

Fix 2: Subagent Shutdown Timeout

// Add a hard timeout for subagent shutdown
async function shutdownSubagent(
  subagentId: string,
  timeoutMs: number = 30000
): Promise<void> {
  const shutdownPromise = subagentRegistry.close(subagentId);
  
  const timeoutPromise = new Promise<never>((_, reject) => {
    setTimeout(() => reject(new Error(`Subagent ${subagentId} shutdown timeout after ${timeoutMs}ms`)), timeoutMs);
  });

  try {
    await Promise.race([shutdownPromise, timeoutPromise]);
  } catch (error) {
    // Force-kill the subagent process
    logger.warn(`Force-killing subagent ${subagentId}: ${error.message}`);
    await subagentRegistry.forceKill(subagentId);
  }
}

Fix 3: MCP Connection Pool Monitoring

// Add connection pool health monitoring
class McpConnectionPool {
  private connections: Map<string, McpClient> = new Map();
  private maxConnections: number = 50;  // Configurable

  async acquire(serverId: string): Promise<McpClient> {
    if (this.connections.size >= this.maxConnections) {
      // Evict idle connections first
      this.evictIdleConnections();
      
      if (this.connections.size >= this.maxConnections) {
        throw new Error(
          `MCP connection pool exhausted (${this.connections.size}/${this.maxConnections}). ` +
          `Consider increasing maxConnections or reducing concurrent subagents.`
        );
      }
    }

    const client = await this.createConnection(serverId);
    this.connections.set(serverId, client);
    return client;
  }

  async release(serverId: string): Promise<void> {
    const client = this.connections.get(serverId);
    if (client) {
      await client.shutdown();
      this.connections.delete(serverId);
    }
  }

  private evictIdleConnections(): void {
    const idleThreshold = 300000; // 5 minutes
    const now = Date.now();
    
    for (const [serverId, client] of this.connections) {
      if (now - client.lastActivity > idleThreshold) {
        client.shutdown().catch(() => {});
        this.connections.delete(serverId);
      }
    }
  }

  getStats(): { active: number; max: number } {
    return { active: this.connections.size, max: this.maxConnections };
  }
}

Fix 4: Long-Session MCP Reconnection

// Periodic health check for long-running sessions
class McpSessionHealthMonitor {
  private checkInterval: NodeJS.Timeout;

  constructor(private mcpPool: McpConnectionPool) {
    this.checkInterval = setInterval(() => this.healthCheck(), 60000);
  }

  private async healthCheck(): Promise<void> {
    const stats = this.mcpPool.getStats();
    
    if (stats.active > stats.max * 0.8) {
      logger.warn(`MCP connection pool at ${Math.round(stats.active/stats.max*100)}% capacity`);
      this.mcpPool.evictIdleConnections();
    }

    // Reconnect dead connections
    for (const [serverId, client] of this.mcpPool.connections) {
      if (!client.isConnected()) {
        logger.info(`Reconnecting dead MCP connection: ${serverId}`);
        try {
          await client.reconnect();
        } catch (error) {
          logger.error(`Failed to reconnect MCP ${serverId}: ${error.message}`);
          this.mcpPool.release(serverId);
        }
      }
    }
  }

  destroy(): void {
    clearInterval(this.checkInterval);
  }
}

---

Testing Plan

Test 1: Subagent Lifecycle Cleanup

1. Run a /goal task that spawns multiple subagents
2. Verify each subagent's MCP connections are closed after completion
3. Check process list — no orphan subagent processes
4. Verify /goal mode completes without hanging

Test 2: Long Session Stability

1. Run Codex for 5+ hours with MCP servers configured
2. Verify MCP servers remain initialized throughout
3. Verify no "MCP startup interrupted" warnings
4. Check connection pool stats — should not grow unbounded

Test 3: Connection Pool Exhaustion

1. Configure many MCP servers (10+)
2. Spawn many subagents rapidly
3. Verify connection pool enforces limits
4. Verify idle connections are evicted
5. Verify graceful degradation (not crash) when pool is full

Test 4: Force-Kill Fallback

1. Spawn a subagent that hangs (e.g., infinite loop)
2. Call shutdown — should timeout after 30s
3. Force-kill should terminate the process
4. Verify MCP connections are cleaned up even after force-kill

---

Expected Impact

| Metric | Before Fix | After Fix |
|--------|-----------|-----------|
| Subagent cleanup rate | ~50% (inconsistent) | 100% |
| Stale subagent processes | Accumulate indefinitely | Cleaned up on completion |
| /goal mode reliability | Broken by 1 stale subagent | Resilient to subagent failures |
| MCP connection leak | Unbounded growth | Bounded + idle eviction |
| 5+ hour session stability | MCP startup fails | Stable throughout |

---

Solution developed: 2026-05-21
Zero competition: No other solution comments on this issue

The recurring failure mode here still looks like missing control-plane state, not only missing cleanup. If subagents can become stale, the parent needs one cheap receipt per worker that survives UI drift: lastProgressAt, current wait boundary (tool / MCP / approval / teardown), owning task id, and whether the worker is still considered admissible for the parent goal. Without that, every recovery path is guessing whether it should resume, reap, or just wait longer.

I am able to interupt it - but have to catch it not doing anything manually. That 1 stale agent will be stale for the whole session. Attempting to close it again will make the whole session stuck again.

Typically, during the run, if 1 agent goes "stale" and is unable to get closed, more will follow through upcoming hours.

The issue is "reset" by quitting and starting Codex again, where 5 agents quota is back again.

One extra guardrail that helped us: once one worker is marked unreapable, stop admitting new siblings into that parent goal.

Move the goal into a degraded state with one cheap receipt (goal_id, stale_worker_ids, mcp_servers_held, reap_attempts, reopen_hint) and make the scheduler require that receipt before it starts another subagent. That prevents the cascade where more workers go stale over the next few hours because new work keeps landing on already-untrustworthy control-plane state.

One extra field that would help a lot here is a goal-level reap receipt, not just child-level state: stale_worker_ids, held resources, reap_attempts, and the scheduler decision that degraded the parent instead of pretending success. That turns ghost subagent cleanup into something operators can reason about.

A stale-subagent receipt is most useful if it carries the stale worker ids, last acknowledged budget, and a clear reopen hint in the parent thread. Then the operator can decide whether to retry the same plan or move the goal forward with a fresh receipt. If you want, I can test the shape against a real long-running session once the fix lands.

I packaged this stale-subagent report shape into trace-to-skill@0.1.66 so users can turn local Codex traces into a more complete issue-ready checklist:

npx trace-to-skill@0.1.66 demo subagent-lifecycle
npx trace-to-skill@0.1.66 codex-report ./runs --output openai-codex-subagents.md

It is not a runtime fix. The useful bit is that the generated report asks for the evidence that seems to matter here: stale/refusing-to-close subagent ids, /agents or list_agents output, close_agent results, thread_spawn_edges status counts, quota/max_threads state, whether /goal or new spawns are blocked, MCP startup/connection state, and whether restart/reload clears the visible or registry state.

Release: https://github.com/grnbtqdbyx-create/trace-to-skill/releases/tag/v0.1.66

Additional Windows Desktop evidence: stale agent rows survive terminal state

This is a sanitized report from a Windows Codex Desktop session using multi-agent orchestration. The visible symptom was a parent thread showing no usable active agents while a fresh agent request still returned agent thread limit reached. Reopening Codex did not reconcile the stale agent roster.

The local state showed three independent persistence layers retaining the same lifecycle:

• SQLite threads / thread_spawn_edges rows for thread_source=subagent;
• parent rollout JSONL sub_agent_activity records;
• global/session-index UI references.

A first conservative cleanup removed 118 normally completed child rows, but left 8 rows because 7 had turn_aborted without task_complete and one was a legacy parent/subagent row. That reproduced the core defect: the UI/orchestrator has no reliable terminal garbage-collection path for aborted or otherwise stale children.

After a full app shutdown, a destructive reconciliation pass removed the remaining 8 rows and 7 edges plus their 8 rollout files. Postconditions were:

• 18 total thread rows, 0 spawn edges;
• 0 remaining thread_source=subagent rows;
• 13 stale sub_agent_activity events removed from 2 surviving parent rollouts;
• 0 target IDs in global UI state or session_index.jsonl;
• 0 target activity events and 0 JSONL parse errors;
• SQLite foreign-key and staged-write checks passed.

No raw transcripts, local paths, or private rollout files are attached.

Requested fixes

1. Make completed, aborted, failed, and cancelled child states terminal and idempotently reapable.
2. Calculate the agent-thread limit from live/pending handles only; stale historical rows must not consume capacity.
3. Reconcile thread rows, spawn edges, rollout activity, and UI indexes on app restart.
4. Add a user-visible “reap stale agents” operation with a dry-run/backup boundary.
5. Keep historical agent transcripts separate from the active-agent roster so archived history cannot appear as active work.

This is a local-state/lifecycle issue, not a request to delete user history by default.

User cleanup watcher (destructive)

Gist: https://gist.github.com/pomazanbohdan/306b3fe6f49e9e3aa3ff14711f6e5598

Files:

• README.md
• universal_cleanup_after_close.py
• watch_universal_clean_relaunch.ps1

The gist is secret/unlisted: anyone with the link can read it, but it is not listed in public Gist discovery.

Important warning

This performs a full destructive cleanup. It deletes all agent threads from local Codex state, including completed, aborted, failed, paused, and incomplete agents. It also deletes their spawn edges and rollout files and removes persisted agent-activity/UI references. Agent history is not retained and recovery is manual from the generated backup.

Windows usage

1. Save the two source files in the same directory.
2. Open PowerShell while Codex is still running.
3. Run:

``powershell
Set-ExecutionPolicy -Scope Process Bypass
& .\watch_universal_clean_relaunch.ps1
``

1. Close every Codex window. The watcher waits for complete process absence, creates a backup, performs the cleanup, verifies SQLite/JSONL/UI postconditions, and relaunches Codex.

The backup is written under the watcher's run-<UTC timestamp>\backup directory. If Codex does not fully close, no mutation is performed.

Dry run

To inspect what would be removed without mutation:

python .\universal_cleanup_after_close.py --dry-run --run-dir .\dry-run

The Python script refuses live mutation unless it is invoked by the watcher after full process absence.

Additional reproduction evidence for the stale-active-subagent bug (Windows, Codex Desktop 26.803.10989.0; CLI 0.147.0-alpha.6.6).

Reproduction

1. In parent task 019fea4e-4227-7810-be40-9afcc9a23aad (“Quick Dev 3.0”), spawn subagents.
2. Let these four children finish normally:

• Tesla 019ff593-1a6b-7cd3-bfd2-e001212e7603
• Pasteur 019ff5b2-6ab4-7792-9f33-66b97897d201
• Dirac 019ff5b2-82ae-7411-84d8-e34c2181004b
• Archimedes 019ff5b2-9af4-7943-aa94-5c0abd770148

1. Confirm their child rollouts end with event_msg.payload.type=task_complete.
2. Fully close/restart Codex and reopen the parent task.

Observed

• The UI shows all four children as Active again.
• thread/read for every child reports status.type=notLoaded; its latest/only turn is status=completed with a completion timestamp.
• The parent rollout has no terminal sub_agent_activity for these four: the latest parent activity remains started or interacted.
• The parent’s thread_spawn_edges rows remain status=open.
• During reopen (approximately 2026-08-12 11:29:43–11:29:58 UTC), logs show parent resume/reconstruction followed by child reads; Tesla also receives thread/resume and startup_prewarm. The child rollout files receive no new output, so this is not a new child turn.

Likely localization

Normal child completion is persisted in the child rollout but is not propagated to the parent as a terminal activity and does not close the parent spawn edge. Resume then rebuilds the active overlay from the stale parent activity/edge state without reconciling the child’s terminal event. This produces the ghost Active entries and can also retain stale spawn/quota resources.

Expected behavior

On any terminal child state (task_complete, interrupted, cancelled, or failed), Codex should atomically:

1. close the corresponding thread_spawn_edges row;
2. persist a terminal parent activity/state;
3. release the child from active-thread/quota/restore registries.

On resume/reopen, the UI should derive Active strictly from live handles or reconciled nonterminal child state. A close/reopen regression test covering this sequence would catch the issue.