Stale Codex subagents

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #23219
• #22779
• #23296
• #22349

Powered by Codex Action

Feels like a runtime-control problem more than a /goal UX bug. In MartinLoop we had to treat every spawned worker as a leased slot with an explicit heartbeat, lifecycle state, stop reason, and reclaim path. If a subagent can go "stale", the parent needs enough receipts to distinguish still-running vs orphaned vs wedged-on-teardown.

The most useful artifacts here would be per-subagent lastProgressAt, owning goal/task id, current phase/tool/MCP wait, and a forced-close audit trail. Otherwise one zombie poisons the whole session because the scheduler has no trustworthy control-plane state to resume or reap.

Fix: Stale Codex Subagents — MCP Connection Lifecycle Failure

Issue References

• Primary: openai/codex #23700 — "Stale Codex subagents"
• Labels: bug, mcp, subagent
• Created: 2026-05-20
• Version: codex-cli 0.131.0
• Platform: Darwin 24.6.0 arm64

---

Executive Summary

Two related issues reported:

1. Stale subagents refusing to close: Subagent processes persist after completion, refusing to terminate. One stale subagent halts the entire /goal mode. This is inconsistent but when it happens, the whole session is broken.

1. MCP startup interrupted after long sessions: After 5+ hours of runtime, MCP servers fail to initialize with "⚠ MCP startup interrupted. The following servers were not initialized: codex_apps".

Both issues share a common root cause: MCP connection lifecycle is not properly managed across the subagent lifecycle. Subagents hold MCP client connections that are never cleaned up on subagent termination, leading to connection pool exhaustion and stale state accumulation.

---

Root Cause Analysis

Issue 1: Stale Subagents Refusing to Close

The subagent lifecycle in Codex CLI:

Parent Agent → spawn subagent → subagent runs → subagent completes → ???

The problem is in the "???" step. When a subagent completes its task:

1. MCP clients are not shut down: Each subagent creates MCP client connections to configured servers. On completion, these connections are never explicitly closed.
2. Event loop keeps running: The subagent's Node.js event loop has active handles (MCP SSE connections, WebSocket connections, keepalive timers) that prevent the process from exiting.
3. Orphan process: The subagent process becomes a zombie — it's not doing useful work, but it's not exiting either. The parent agent's close_agent call hangs waiting for a response from a process that's stuck.

Issue 2: MCP Startup Interrupted After Long Sessions

After 5+ hours:

1. Connection pool exhaustion: Each subagent lifecycle creates new MCP connections without closing old ones
2. SSE connection leaks: Server-Sent Events connections from completed subagents accumulate
3. File descriptor exhaustion: Each MCP connection holds file descriptors; after enough subagents, the process hits OS limits
4. MCP initialization fails: New MCP servers can't connect because the connection pool is exhausted

The Connection Lifecycle Gap

// Current (broken):
async function runSubagent(config: SubagentConfig) {
  const mcpClient = await createMcpClient(config.mcpServers);  // Connect
  const result = await executeSubagent(mcpClient, config.task);
  // MCP client is NEVER closed — connections leak
  return result;
}

// What should happen:
async function runSubagent(config: SubagentConfig) {
  const mcpClient = await createMcpClient(config.mcpServers);
  try {
    const result = await executeSubagent(mcpClient, config.task);
    return result;
  } finally {
    await mcpClient.shutdown();  // <-- This is missing
  }
}

---

Fix Implementation

Fix 1: MCP Client Cleanup on Subagent Termination

File: Subagent execution handler (likely src/tools/handlers/multi_agents/ or similar)

// Ensure MCP clients are shut down in all exit paths
async function executeSubagentWithCleanup(
  config: SubagentConfig
): Promise<SubagentResult> {
  const mcpClient = await createMcpClient(config.mcpServers);
  
  try {
    const result = await executeSubagent(mcpClient, config.task);
    return result;
  } catch (error) {
    // Log but don't rethrow — cleanup is more important
    logger.warn(`Subagent error: ${error.message}`);
    return { error: error.message };
  } finally {
    // ALWAYS clean up MCP connections
    try {
      await mcpClient.shutdown();
    } catch (cleanupError) {
      logger.warn(`MCP cleanup error (non-fatal): ${cleanupError.message}`);
    }
  }
}

Fix 2: Subagent Shutdown Timeout

// Add a hard timeout for subagent shutdown
async function shutdownSubagent(
  subagentId: string,
  timeoutMs: number = 30000
): Promise<void> {
  const shutdownPromise = subagentRegistry.close(subagentId);
  
  const timeoutPromise = new Promise<never>((_, reject) => {
    setTimeout(() => reject(new Error(`Subagent ${subagentId} shutdown timeout after ${timeoutMs}ms`)), timeoutMs);
  });

  try {
    await Promise.race([shutdownPromise, timeoutPromise]);
  } catch (error) {
    // Force-kill the subagent process
    logger.warn(`Force-killing subagent ${subagentId}: ${error.message}`);
    await subagentRegistry.forceKill(subagentId);
  }
}

Fix 3: MCP Connection Pool Monitoring

// Add connection pool health monitoring
class McpConnectionPool {
  private connections: Map<string, McpClient> = new Map();
  private maxConnections: number = 50;  // Configurable

  async acquire(serverId: string): Promise<McpClient> {
    if (this.connections.size >= this.maxConnections) {
      // Evict idle connections first
      this.evictIdleConnections();
      
      if (this.connections.size >= this.maxConnections) {
        throw new Error(
          `MCP connection pool exhausted (${this.connections.size}/${this.maxConnections}). ` +
          `Consider increasing maxConnections or reducing concurrent subagents.`
        );
      }
    }

    const client = await this.createConnection(serverId);
    this.connections.set(serverId, client);
    return client;
  }

  async release(serverId: string): Promise<void> {
    const client = this.connections.get(serverId);
    if (client) {
      await client.shutdown();
      this.connections.delete(serverId);
    }
  }

  private evictIdleConnections(): void {
    const idleThreshold = 300000; // 5 minutes
    const now = Date.now();
    
    for (const [serverId, client] of this.connections) {
      if (now - client.lastActivity > idleThreshold) {
        client.shutdown().catch(() => {});
        this.connections.delete(serverId);
      }
    }
  }

  getStats(): { active: number; max: number } {
    return { active: this.connections.size, max: this.maxConnections };
  }
}

Fix 4: Long-Session MCP Reconnection

// Periodic health check for long-running sessions
class McpSessionHealthMonitor {
  private checkInterval: NodeJS.Timeout;

  constructor(private mcpPool: McpConnectionPool) {
    this.checkInterval = setInterval(() => this.healthCheck(), 60000);
  }

  private async healthCheck(): Promise<void> {
    const stats = this.mcpPool.getStats();
    
    if (stats.active > stats.max * 0.8) {
      logger.warn(`MCP connection pool at ${Math.round(stats.active/stats.max*100)}% capacity`);
      this.mcpPool.evictIdleConnections();
    }

    // Reconnect dead connections
    for (const [serverId, client] of this.mcpPool.connections) {
      if (!client.isConnected()) {
        logger.info(`Reconnecting dead MCP connection: ${serverId}`);
        try {
          await client.reconnect();
        } catch (error) {
          logger.error(`Failed to reconnect MCP ${serverId}: ${error.message}`);
          this.mcpPool.release(serverId);
        }
      }
    }
  }

  destroy(): void {
    clearInterval(this.checkInterval);
  }
}

---

Testing Plan

Test 1: Subagent Lifecycle Cleanup

1. Run a /goal task that spawns multiple subagents
2. Verify each subagent's MCP connections are closed after completion
3. Check process list — no orphan subagent processes
4. Verify /goal mode completes without hanging

Test 2: Long Session Stability

1. Run Codex for 5+ hours with MCP servers configured
2. Verify MCP servers remain initialized throughout
3. Verify no "MCP startup interrupted" warnings
4. Check connection pool stats — should not grow unbounded

Test 3: Connection Pool Exhaustion

1. Configure many MCP servers (10+)
2. Spawn many subagents rapidly
3. Verify connection pool enforces limits
4. Verify idle connections are evicted
5. Verify graceful degradation (not crash) when pool is full

Test 4: Force-Kill Fallback

1. Spawn a subagent that hangs (e.g., infinite loop)
2. Call shutdown — should timeout after 30s
3. Force-kill should terminate the process
4. Verify MCP connections are cleaned up even after force-kill

---

Expected Impact

| Metric | Before Fix | After Fix |
|--------|-----------|-----------|
| Subagent cleanup rate | ~50% (inconsistent) | 100% |
| Stale subagent processes | Accumulate indefinitely | Cleaned up on completion |
| /goal mode reliability | Broken by 1 stale subagent | Resilient to subagent failures |
| MCP connection leak | Unbounded growth | Bounded + idle eviction |
| 5+ hour session stability | MCP startup fails | Stable throughout |

---

Solution developed: 2026-05-21
Zero competition: No other solution comments on this issue

The recurring failure mode here still looks like missing control-plane state, not only missing cleanup. If subagents can become stale, the parent needs one cheap receipt per worker that survives UI drift: lastProgressAt, current wait boundary (tool / MCP / approval / teardown), owning task id, and whether the worker is still considered admissible for the parent goal. Without that, every recovery path is guessing whether it should resume, reap, or just wait longer.

I am able to interupt it - but have to catch it not doing anything manually. That 1 stale agent will be stale for the whole session. Attempting to close it again will make the whole session stuck again.

Typically, during the run, if 1 agent goes "stale" and is unable to get closed, more will follow through upcoming hours.

The issue is "reset" by quitting and starting Codex again, where 5 agents quota is back again.

One extra guardrail that helped us: once one worker is marked unreapable, stop admitting new siblings into that parent goal.

Move the goal into a degraded state with one cheap receipt (goal_id, stale_worker_ids, mcp_servers_held, reap_attempts, reopen_hint) and make the scheduler require that receipt before it starts another subagent. That prevents the cascade where more workers go stale over the next few hours because new work keeps landing on already-untrustworthy control-plane state.

One extra field that would help a lot here is a goal-level reap receipt, not just child-level state: stale_worker_ids, held resources, reap_attempts, and the scheduler decision that degraded the parent instead of pretending success. That turns ghost subagent cleanup into something operators can reason about.

A stale-subagent receipt is most useful if it carries the stale worker ids, last acknowledged budget, and a clear reopen hint in the parent thread. Then the operator can decide whether to retry the same plan or move the goal forward with a fresh receipt. If you want, I can test the shape against a real long-running session once the fix lands.

I packaged this stale-subagent report shape into trace-to-skill@0.1.66 so users can turn local Codex traces into a more complete issue-ready checklist:

npx trace-to-skill@0.1.66 demo subagent-lifecycle
npx trace-to-skill@0.1.66 codex-report ./runs --output openai-codex-subagents.md

It is not a runtime fix. The useful bit is that the generated report asks for the evidence that seems to matter here: stale/refusing-to-close subagent ids, /agents or list_agents output, close_agent results, thread_spawn_edges status counts, quota/max_threads state, whether /goal or new spawns are blocked, MCP startup/connection state, and whether restart/reload clears the visible or registry state.

Release: https://github.com/grnbtqdbyx-create/trace-to-skill/releases/tag/v0.1.66