MultiAgentV2 spawn_agent can hang indefinitely while evicting a terminal resident thread
What version of the Codex App are you using?
Originally observed with:
- Codex App
26.707.72221(build5307) - Bundled CLI
0.144.2
Current installed version:
- Codex App
26.715.21425(build5488) - Bundled CLI
0.145.0-alpha.18
The relevant code path is still unchanged in 0.144.5, 0.145.0-alpha.20, and current main.
What subscription do you have?
ChatGPT Pro
What platform is your computer?
Darwin 25.5.0 arm64 arm
macOS 26.5.1 (25F80)
What issue are you seeing?
In a long-lived Codex Desktop session using MultiAgentV2, spawn_agent can hang indefinitely before the child thread is created.
The parent rollout ends with a structured collaboration.spawn_agent function call, but there is no corresponding function-call output, no child session_meta, and no child activity. The parent turn remains active/inProgress indefinitely.
This is different from a slow child and from the normal concurrency-limit path:
- Successful spawns: median 3.3s, P95 19.8s, maximum observed 34.2s.
- Explicit
agent thread limit reachedfailures: 0.05–0.83s and always returned a structured error. - Three affected calls had no output and created no child session.
- A fourth near-reproduction was manually aborted after 264.5s and also created no child session.
What steps can reproduce the bug?
The failure is intermittent, but the repeated incidents share this lifecycle:
- Use a long-lived Codex Desktop thread with MultiAgentV2.
- Spawn enough direct children to reach the resident-agent capacity.
- Let one or more children become terminal (
Completed,Errored, orInterrupted) while remaining resident. - Call
spawn_agentagain, causingreserve_slotto evict an LRU terminal resident. - In the affected state,
spawn_agentnever returns and no child thread is created.
Two independent hangs were observed in the same parent-thread family, followed by another zero-output spawn in a direct child.
What is the expected behavior?
spawn_agent should always complete within a bounded time:
- create and return the child thread ID;
- return
AgentLimitReached; or - return a structured cleanup/shutdown timeout error.
A stuck terminal resident must not wedge the parent turn indefinitely.
Additional information
The likely blocking path is MultiAgentV2 resident eviction:
try_unload_one_residentremoves the candidate from the LRU resident deque withpop_lru_candidate;- it then awaits
ensure_rollout_materialized(); - it then awaits
shutdown_and_wait(); - neither await has a timeout;
- there is no cancellation guard that restores the popped resident ID if the parent tool future is cancelled.
Source, current alpha:
The residency.rs implementation is byte-identical between 0.145.0-alpha.18, 0.145.0-alpha.20, and current main. The stable 0.144.5 path contains the same unbounded awaits.
This also creates a possible accounting leak: the candidate is popped before the unbounded awaits, so cancellation can leave the thread loaded but no longer represented in the resident deque.
Related reports cover similar lifecycle symptoms, but not this V2 pre-create eviction path:
- #24389 — V1
close_agenthangs on an unresponsive child - #25426 — V1 termination wait is unbounded
- #18335 — agent spawn slots leak across persistent turns
Suggested fix:
- Add a bounded timeout around rollout materialization and shutdown during eviction.
- Add a cancellation-safe guard that restores or quarantines the popped resident.
- Return a structured
spawn cleanup timed outresult rather than hanging. - Add stage telemetry keyed by spawn call ID: reserve → select LRU → materialize → shutdown → remove → fork → child created.
- Add regression tests using a terminal resident whose session loop never terminates.
I have omitted local paths, session IDs, prompts, and raw logs from this public report. The redacted call lifecycle and timing evidence can be provided privately if needed.
4 Comments
Independent reproduction that closely matches the pre-create resident-eviction path described here.
Environment:
26.707.71524(build5263)0.144.2Darwin 25.5.0 arm64fork_turns: "none"Redacted chronology (UTC):
09:43:22.669: a direct child spawn was issued and returned successfully at09:43:24.116.completed, oneinterrupted, and onerunning.09:43:29.538: the next directspawn_agentcall was emitted.session_meta, and no child activity.09:44:28and09:44:37. This shows that the host and multi-agent dispatcher were still making progress elsewhere.09:54:28; its grandchildren completed at09:55:17and09:55:49.10:02:41, after1151.8s. No corresponding target child existed after the abort.The rollout was structurally healthy (no malformed JSONL records), and no shell command, MCP call, or model generation was active in the blocked parent. The parent was monopolized specifically by the unresolved
collaboration.spawn_agenttool call.This rules out a slow child and ordinary active-concurrency exhaustion. The resident state plus the fact that the request did not wake after every other agent completed strongly supports the
reserve_slot -> terminal LRU eviction -> unbounded materialize/shutdownhypothesis in this issue. It also provides an independent reproduction on the same bundled CLI generation.If maintainers confirm the intended timeout and restore-vs-quarantine semantics, I can prepare an invited focused PR with:
Local paths, thread/call IDs, prompts, and raw logs are intentionally omitted.
Independent reproduction on stable Codex CLI
0.144.5, confirming the same MultiAgentV2 pre-create resident-eviction hang.Environment:
0.144.5Redacted chronology:
spawn_agentcall was issued at2026-07-17T13:33:08.071Z.session_meta, or child activity.2026-07-17T17:38:37.871Zwithaborted by user after 11920.6s(3h 18m 40.6s of recorded handler duration).This was not a slow child, test execution, or repository command: the new child never existed. The observed shutdown activity against the prior resident also strongly identifies
reserve_slot -> try_unload_one_resident -> shutdown_and_waitas the blocking path, rather than request routing or model startup.This adds two useful data points to the original report:
0.144.5.The interrupted candidate rollout contained two
turn_abortedterminal events but no normal turn completion. A regression fixture covering an interrupted-twice resident whose process/session teardown does not complete would closely match this reproduction.Redacted rollout excerpts and exact timing evidence are retained and can be provided privately if maintainers need them. No project paths, prompts, task names, thread IDs, or raw transcripts are included here.
Related: #25426 documents the same missing bounded-shutdown guarantee through the V1 explicit-close path; this reproduction confirms the V2 automatic-eviction path described in this issue.
Additional independent dataset from one long-lived MultiAgentV2 root session.
Environment and scope:
codex-tuion macOS arm64.session_metarecords CLI0.144.1.0.144.5, but I am not claiming these six historical incidents are a0.144.5reproduction; this comment adds frequency and lifecycle evidence to the existing stable-version reports above.Across the complete root rollout currently available, I paired 49 structured
spawn_agentcalls with their outputs:task_name.agent thread limit reachedin approximately 0.069–0.170 seconds.The six handler-reported blocked durations were:
| Incident |
fork_turns| Handler-reported duration || --- | --- | ---: |
| A |
none| 28,448.6s || B |
all| 8,858.1s || C |
3| 8,329.1s || D |
all| 2,343.9s || E |
3| 34,273.2s || F |
3| 4,540.1s |Total blocked handler time: 86,793.0 seconds (24h 06m 33s).
For all six targets:
aborted by user after ...;agent_pathanywhere in the relevant local session tree;sub_agent_activityin the parent rollout;This dataset does not identify the exact internal await stage by itself. It does show that:
fork_turnsset tonone,all, and3;spawn_agentcall can monopolize the parent turn for hours while exposing no queryable ticket or intermediate state.This independently supports the issue's requested fixes: bounded eviction/startup deadlines, cancellation-safe resident accounting, structured timeout output, and stage telemetry from reservation through child registration.
Redacted lifecycle rows are retained locally and can be provided privately if maintainers need them.
Additional independent reproduction matching this issue.
fork_turns: "none".spawn_agentcall returned no agent identifier, capacity error, timeout, or recovery information.aborted by user after 18901.0s.Uploaded thread:
019f78cf-a24c-7f30-80ac-c32b3ccfc887This further supports adding a bounded scheduling or resident-eviction timeout, structured stage/capacity errors, and idempotent retry behavior.