MultiAgentV2: queue-only message to a completed agent pins a residency slot
What version of Codex CLI is running?
Observed with codex-cli 0.144.0 bundled in Conductor. I also checked codex-cli 0.144.1 and current main (5c19155cbd93bfa099016e7487259f61669823ff); the relevant v2 residency and message-delivery logic is unchanged.
What platform is your computer?
Darwin 25.5.0 arm64 arm
Surface: Codex app-server / MultiAgentV2, accessed through Conductor 0.74.0.
What issue are you seeing?
A completed MultiAgentV2 subagent can indefinitely occupy a resident thread slot when it has a pending queue-only mailbox message. Once the other resident slots are active, a new spawn_agent fails even though the number of actively running subagents is below the configured limit:
collab spawn failed: agent thread limit reached
This is not cumulative thread exhaustion. In the observed session, spawning succeeded again later after the pinned mailbox was consumed. The same AgentLimitReached error currently represents active concurrency exhaustion and residency exhaustion, so the parent cannot tell which condition occurred or which agent is blocking eviction.
What steps can reproduce the bug?
Use a small v2 capacity so the behavior is visible:
[features.multi_agent_v2]
max_concurrent_threads_per_session = 4
The value includes the root agent, leaving three subagent resident slots.
- Spawn subagent A and let it reach
Completed. - Call
send_messagetargeting A. In v2 this is queue-only and does not trigger a new turn. - Spawn subagents B and C with tasks that remain active.
- Attempt to spawn subagent D.
- D fails with
collab spawn failed: agent thread limit reached, although only B and C are active. - Call
followup_taskon A so it wakes, consumes its mailbox, and completes again. Retry the spawn; it can now succeed.
The production occurrence was the race-shaped variant: the root called send_message near A's completion boundary. A completed about one second later, leaving the queue-only message pending. With B and C active, spawning D failed.
What is the expected behavior?
A queue-only message should not leave a completed agent as a hidden, indefinitely non-evictable resident.
Any of these behaviors would avoid the deadlock:
- reject
send_messageto an already completed agent and tell the caller to usefollowup_task; - automatically trigger a follow-up turn when a queue-only message crosses the target's completion boundary;
- persist the mailbox independently so the completed thread can be unloaded and resumed later;
- expose an explicit close/drain operation for completed v2 agents.
At minimum, AgentLimitReached should distinguish active concurrency from residency pressure and report data such as max, residents, active, and pending_mailbox_blockers with an actionable recovery hint.
Additional information
Current main makes the interaction explicit:
send_messageisQueueOnly, whilefollowup_taskisTriggerTurn.- A completed v2 resident is unloadable only if its mailbox has no pending items.
- If no resident can be unloaded, admission returns the generic
AgentLimitReached.
Related but distinct:
- #22779 covers completed subagents continuing to count against the older/open-thread quota. This report is specifically about v2 residency, where ordinary completed agents are evictable but completed agents with pending queue-only mail are not.
- #23479 requests capacity/status preflight; reporting mailbox blockers would address this failure mode.
- #24508 requests safe cleanup of completed subagents, but v2 currently has no direct close tool in the exposed delegation toolset.
Increasing features.multi_agent_v2.max_concurrent_threads_per_session reduces the frequency but does not fix the pinned-mailbox lifecycle bug.
4 Comments
Independent reproduction from Codex Desktop (2026-07-21)
I reproduced the admission failure with a controlled baseline and a pending-mailbox variant.
Environment
Baseline: ordinary completed agents were reclaimable
A_OK,B_OK, andC_OKand reachcompleted.Result: D's
spawn_agentreturned successfully without a noticeable wait, and D completed withD_OK.This indicates that ordinary completed agents alone did not block a later spawn in this runtime.
Variant: a queue-only message pinned the completed resident
A_OK.STATUS_PROBEto completed A using queue-onlysend_message. It did not trigger a new turn.The G spawn failed immediately with the exact output:
At that moment only E and F were actively running. A was completed, but it had the pending queue-only mailbox item. The controlled difference from the successful baseline was the pending mailbox on A.
I then called
followup_taskon A; A consumed the queued work and returnedA_DRAINED. E and F also completed. A subsequent retry could not be emitted because the isolated test task encountered a separate tool-routing failure, so I am not claiming that the recovery path was independently verified here.What this reproduction supports
completedresidents were evictable/reclaimable in the baseline.completedresident with pending queue-only mail continued to consume residency capacity.AgentLimitReachederror as true active-concurrency exhaustion.This independently supports the pending-mailbox residency mechanism described in the issue and the need for an actionable error that distinguishes active concurrency from residency pressure.
I audited this against current
mainatc38a60ded2ffbad510f8b512e2902f0128e998e8. The reported residency mechanism is still present, and there is an important race constraint for a fix.Why a tool-handler status precheck is insufficient
handle_message_string_toolresolves and loads the receiver, then submits anOp::InterAgentCommunication. The actual mailbox insertion happens later insession::handlers::inter_agent_communication:A
Completedcheck inmessage_tool.rsalone can pass while the target is still running, followed immediately by target completion before the queued op reachesenqueue_mailbox_communication. That recreates the same completed-plus-pending-mailbox blocker.Current task completion calls
maybe_start_turn_for_pending_work, but that helper intentionally starts only fortrigger_turnmail (or durable sleep). Queue-only mail therefore remains pending, andresidency::is_unloadablerejects the resident becausehas_pending_mailbox_items()is true.Invariant a fix should enforce
After the target reaches a terminal state, queue-only mail must not become an indefinite residency blocker. The atomic decision needs to live at the actual mailbox admission/completion boundary, not only in the model-facing tool handler.
Two viable semantics appear to require maintainer direction:
use followup_taskerror. This needs an acknowledgement path from the session handler back to the tool call; the current op submission only confirms that the op was accepted by the session loop.Regression coverage
A focused test should cover both orderings:
send_message(A).AgentLimitReachedcaused by a hidden queue-only blocker.Per the invitation-only contribution policy, I will not open an unsolicited PR. If a maintainer selects the intended delivery semantic and invites a contribution, I can implement the narrow fix and integration test.
This issue also exposes a message-lifecycle boundary that is currently difficult for the caller to observe:
I opened #37744 for the complementary protocol gap: a parent should be able to distinguish
queued,consumed,applied,rejected,superseded, andunknown, rather than treating a completedsend_messagecall as proof of application.At a terminal boundary, the runtime should define one deterministic behavior: reject queue-only delivery and direct the caller to a follow-up turn, promote the message into a new turn, or externalize the pending mailbox so it does not pin residency. Whichever behavior is selected, the resulting delivery and reclaim state should be observable.
Corroborating production reproduction
Observed on Codex Desktop
26.818.61809with bundled CLI0.149.0-alpha.4.3in session01a03411-e72a-73a2-aae2-97872a621dab.The parent called queue-only
send_messageon child A while A was running; A completed shortly afterward. Two fresh siblings then started, but a third fresh spawn failed withagent thread limit reached. At failure, the list showed the root plus the two siblings running and A stillcompleted. Callinginterrupt_agenton A reported its prior completed status but did not free admission, and the third spawn failed again.This closely matches the race-shaped reproduction described here: a message crossing the completion boundary appears to leave the completed child resident and non-evictable. It also explains the broader symptom reported in #39694. The session had previously created 16 fresh children over multiple waves, but the evidence supports residency pressure—not a cumulative 16-child limit.