MultiAgentV2 spawn_agent can hang indefinitely while evicting a terminal resident thread

Open 💬 4 comments Opened Jul 17, 2026 by surlymochan

What version of the Codex App are you using?

Originally observed with:

  • Codex App 26.707.72221 (build 5307)
  • Bundled CLI 0.144.2

Current installed version:

  • Codex App 26.715.21425 (build 5488)
  • Bundled CLI 0.145.0-alpha.18

The relevant code path is still unchanged in 0.144.5, 0.145.0-alpha.20, and current main.

What subscription do you have?

ChatGPT Pro

What platform is your computer?

Darwin 25.5.0 arm64 arm
macOS 26.5.1 (25F80)

What issue are you seeing?

In a long-lived Codex Desktop session using MultiAgentV2, spawn_agent can hang indefinitely before the child thread is created.

The parent rollout ends with a structured collaboration.spawn_agent function call, but there is no corresponding function-call output, no child session_meta, and no child activity. The parent turn remains active/inProgress indefinitely.

This is different from a slow child and from the normal concurrency-limit path:

  • Successful spawns: median 3.3s, P95 19.8s, maximum observed 34.2s.
  • Explicit agent thread limit reached failures: 0.05–0.83s and always returned a structured error.
  • Three affected calls had no output and created no child session.
  • A fourth near-reproduction was manually aborted after 264.5s and also created no child session.

What steps can reproduce the bug?

The failure is intermittent, but the repeated incidents share this lifecycle:

  1. Use a long-lived Codex Desktop thread with MultiAgentV2.
  2. Spawn enough direct children to reach the resident-agent capacity.
  3. Let one or more children become terminal (Completed, Errored, or Interrupted) while remaining resident.
  4. Call spawn_agent again, causing reserve_slot to evict an LRU terminal resident.
  5. In the affected state, spawn_agent never returns and no child thread is created.

Two independent hangs were observed in the same parent-thread family, followed by another zero-output spawn in a direct child.

What is the expected behavior?

spawn_agent should always complete within a bounded time:

  • create and return the child thread ID;
  • return AgentLimitReached; or
  • return a structured cleanup/shutdown timeout error.

A stuck terminal resident must not wedge the parent turn indefinitely.

Additional information

The likely blocking path is MultiAgentV2 resident eviction:

  • try_unload_one_resident removes the candidate from the LRU resident deque with pop_lru_candidate;
  • it then awaits ensure_rollout_materialized();
  • it then awaits shutdown_and_wait();
  • neither await has a timeout;
  • there is no cancellation guard that restores the popped resident ID if the parent tool future is cancelled.

Source, current alpha:

https://github.com/openai/codex/blob/rust-v0.145.0-alpha.20/codex-rs/core/src/agent/control/residency.rs#L116-L149

The residency.rs implementation is byte-identical between 0.145.0-alpha.18, 0.145.0-alpha.20, and current main. The stable 0.144.5 path contains the same unbounded awaits.

This also creates a possible accounting leak: the candidate is popped before the unbounded awaits, so cancellation can leave the thread loaded but no longer represented in the resident deque.

Related reports cover similar lifecycle symptoms, but not this V2 pre-create eviction path:

  • #24389 — V1 close_agent hangs on an unresponsive child
  • #25426 — V1 termination wait is unbounded
  • #18335 — agent spawn slots leak across persistent turns

Suggested fix:

  1. Add a bounded timeout around rollout materialization and shutdown during eviction.
  2. Add a cancellation-safe guard that restores or quarantines the popped resident.
  3. Return a structured spawn cleanup timed out result rather than hanging.
  4. Add stage telemetry keyed by spawn call ID: reserve → select LRU → materialize → shutdown → remove → fork → child created.
  5. Add regression tests using a terminal resident whose session loop never terminates.

I have omitted local paths, session IDs, prompts, and raw logs from this public report. The redacted call lifecycle and timing evidence can be provided privately if needed.

View original on GitHub ↗

4 Comments

Xopoko · 1 month ago

Independent reproduction that closely matches the pre-create resident-eviction path described here.

Environment:

  • Codex App 26.707.71524 (build 5263)
  • Bundled CLI 0.144.2
  • Darwin 25.5.0 arm64
  • MultiAgentV2, failing call used fork_turns: "none"

Redacted chronology (UTC):

  • 09:43:22.669: a direct child spawn was issued and returned successfully at 09:43:24.116.
  • Immediately before the failing call, the parent retained three direct children: one completed, one interrupted, and one running.
  • 09:43:29.538: the next direct spawn_agent call was emitted.
  • It produced no function-call output, no child session_meta, and no child activity.
  • While that root spawn remained blocked, the already-running child successfully spawned two grandchildren at 09:44:28 and 09:44:37. This shows that the host and multi-agent dispatcher were still making progress elsewhere.
  • The running direct child completed at 09:54:28; its grandchildren completed at 09:55:17 and 09:55:49.
  • The blocked root spawn still did not return after all of those completions.
  • The user interrupted it at 10:02:41, after 1151.8s. No corresponding target child existed after the abort.

The rollout was structurally healthy (no malformed JSONL records), and no shell command, MCP call, or model generation was active in the blocked parent. The parent was monopolized specifically by the unresolved collaboration.spawn_agent tool call.

This rules out a slow child and ordinary active-concurrency exhaustion. The resident state plus the fact that the request did not wake after every other agent completed strongly supports the reserve_slot -> terminal LRU eviction -> unbounded materialize/shutdown hypothesis in this issue. It also provides an independent reproduction on the same bundled CLI generation.

If maintainers confirm the intended timeout and restore-vs-quarantine semantics, I can prepare an invited focused PR with:

  1. bounded eviction-stage deadlines;
  2. a cancellation-safe guard for the popped resident;
  3. a structured cleanup-timeout result;
  4. regression coverage for a terminal resident whose session loop never terminates.

Local paths, thread/call IDs, prompts, and raw logs are intentionally omitted.

psydigital · 1 month ago

Independent reproduction on stable Codex CLI 0.144.5, confirming the same MultiAgentV2 pre-create resident-eviction hang.

Environment:

  • Codex CLI 0.144.5
  • macOS arm64
  • MultiAgentV2 in a long-lived Desktop session

Redacted chronology:

  • The parent had reached its resident-subagent capacity.
  • The LRU resident selected for eviction was an interrupted reviewer thread. It had been interrupted twice and had never produced a normal assistant result.
  • A new direct spawn_agent call was issued at 2026-07-17T13:33:08.071Z.
  • Shutdown activity began against the old resident, including cancellation of several child MCP/session tasks, but the resident did not terminate completely.
  • The requested new child was never registered: there was no child thread ID, child session_meta, or child activity.
  • The parent remained blocked inside the synchronous spawn call until the user manually interrupted it.
  • The tool returned only at 2026-07-17T17:38:37.871Z with aborted by user after 11920.6s (3h 18m 40.6s of recorded handler duration).
  • After the abort, the old interrupted resident and part of its process stack were still present.

This was not a slow child, test execution, or repository command: the new child never existed. The observed shutdown activity against the prior resident also strongly identifies reserve_slot -> try_unload_one_resident -> shutdown_and_wait as the blocking path, rather than request routing or model startup.

This adds two useful data points to the original report:

  1. The failure reproduces on current stable 0.144.5.
  2. The unbounded wait can persist for multiple hours, fully monopolizing the parent turn and preventing it from providing progress or recovery guidance.

The interrupted candidate rollout contained two turn_aborted terminal events but no normal turn completion. A regression fixture covering an interrupted-twice resident whose process/session teardown does not complete would closely match this reproduction.

Redacted rollout excerpts and exact timing evidence are retained and can be provided privately if maintainers need them. No project paths, prompts, task names, thread IDs, or raw transcripts are included here.

Related: #25426 documents the same missing bounded-shutdown guarantee through the V1 explicit-close path; this reproduction confirms the V2 automatic-eviction path described in this issue.

Aaron-1020 · 1 month ago

Additional independent dataset from one long-lived MultiAgentV2 root session.

Environment and scope:

  • codex-tui on macOS arm64.
  • The long-lived root session's session_meta records CLI 0.144.1.
  • The currently installed CLI is 0.144.5, but I am not claiming these six historical incidents are a 0.144.5 reproduction; this comment adds frequency and lifecycle evidence to the existing stable-version reports above.
  • No raw rollout, local path, session/thread/call ID, prompt, or task name is included here.

Across the complete root rollout currently available, I paired 49 structured spawn_agent calls with their outputs:

  • 40 returned a normal ACK containing a canonical task_name.
  • 3 returned agent thread limit reached in approximately 0.069–0.170 seconds.
  • 6 produced no ACK until the user manually interrupted the parent turn.

The six handler-reported blocked durations were:

| Incident | fork_turns | Handler-reported duration |
| --- | --- | ---: |
| A | none | 28,448.6s |
| B | all | 8,858.1s |
| C | 3 | 8,329.1s |
| D | all | 2,343.9s |
| E | 3 | 34,273.2s |
| F | 3 | 4,540.1s |

Total blocked handler time: 86,793.0 seconds (24h 06m 33s).

For all six targets:

  • the only eventual function-call output was aborted by user after ...;
  • there was no matching target agent_path anywhere in the relevant local session tree;
  • there was no matching target sub_agent_activity in the parent rollout;
  • therefore there is no observable child registration or start evidence.

This dataset does not identify the exact internal await stage by itself. It does show that:

  1. the long-blocking path is behaviorally distinct from the normal millisecond-scale capacity rejection;
  2. full-history fork is not a necessary precondition, because the same symptom occurred with fork_turns set to none, all, and 3;
  3. a synchronous spawn_agent call can monopolize the parent turn for hours while exposing no queryable ticket or intermediate state.

This independently supports the issue's requested fixes: bounded eviction/startup deadlines, cancellation-safe resident accounting, structured timeout output, and stage telemetry from reservation through child registration.

Redacted lifecycle rows are retained locally and can be provided privately if maintainers need them.

jimicze · 1 month ago

Additional independent reproduction matching this issue.

  • Platform: macOS, long-lived Codex multi-agent session.
  • The parent already had prior implementer and reviewer agents in its resident tree.
  • A new quality-review agent was requested with fork_turns: "none".
  • The spawn_agent call returned no agent identifier, capacity error, timeout, or recovery information.
  • The parent turn remained blocked until manually interrupted after 18,901 seconds (5h 15m 1s).
  • The only result was: aborted by user after 18901.0s.
  • Because the call never returned, the parent could not provide progress updates or continue verification.

Uploaded thread: 019f78cf-a24c-7f30-80ac-c32b3ccfc887

This further supports adding a bounded scheduling or resident-eviction timeout, structured stage/capacity errors, and idempotent retry behavior.