MultiAgentV2: queue-only message to a completed agent pins a residency slot

Open 💬 4 comments Opened Jul 11, 2026 by zhibisora

What version of Codex CLI is running?

Observed with codex-cli 0.144.0 bundled in Conductor. I also checked codex-cli 0.144.1 and current main (5c19155cbd93bfa099016e7487259f61669823ff); the relevant v2 residency and message-delivery logic is unchanged.

What platform is your computer?

Darwin 25.5.0 arm64 arm

Surface: Codex app-server / MultiAgentV2, accessed through Conductor 0.74.0.

What issue are you seeing?

A completed MultiAgentV2 subagent can indefinitely occupy a resident thread slot when it has a pending queue-only mailbox message. Once the other resident slots are active, a new spawn_agent fails even though the number of actively running subagents is below the configured limit:

collab spawn failed: agent thread limit reached

This is not cumulative thread exhaustion. In the observed session, spawning succeeded again later after the pinned mailbox was consumed. The same AgentLimitReached error currently represents active concurrency exhaustion and residency exhaustion, so the parent cannot tell which condition occurred or which agent is blocking eviction.

What steps can reproduce the bug?

Use a small v2 capacity so the behavior is visible:

[features.multi_agent_v2]
max_concurrent_threads_per_session = 4

The value includes the root agent, leaving three subagent resident slots.

  1. Spawn subagent A and let it reach Completed.
  2. Call send_message targeting A. In v2 this is queue-only and does not trigger a new turn.
  3. Spawn subagents B and C with tasks that remain active.
  4. Attempt to spawn subagent D.
  5. D fails with collab spawn failed: agent thread limit reached, although only B and C are active.
  6. Call followup_task on A so it wakes, consumes its mailbox, and completes again. Retry the spawn; it can now succeed.

The production occurrence was the race-shaped variant: the root called send_message near A's completion boundary. A completed about one second later, leaving the queue-only message pending. With B and C active, spawning D failed.

What is the expected behavior?

A queue-only message should not leave a completed agent as a hidden, indefinitely non-evictable resident.

Any of these behaviors would avoid the deadlock:

  • reject send_message to an already completed agent and tell the caller to use followup_task;
  • automatically trigger a follow-up turn when a queue-only message crosses the target's completion boundary;
  • persist the mailbox independently so the completed thread can be unloaded and resumed later;
  • expose an explicit close/drain operation for completed v2 agents.

At minimum, AgentLimitReached should distinguish active concurrency from residency pressure and report data such as max, residents, active, and pending_mailbox_blockers with an actionable recovery hint.

Additional information

Current main makes the interaction explicit:

Related but distinct:

  • #22779 covers completed subagents continuing to count against the older/open-thread quota. This report is specifically about v2 residency, where ordinary completed agents are evictable but completed agents with pending queue-only mail are not.
  • #23479 requests capacity/status preflight; reporting mailbox blockers would address this failure mode.
  • #24508 requests safe cleanup of completed subagents, but v2 currently has no direct close tool in the exposed delegation toolset.

Increasing features.multi_agent_v2.max_concurrent_threads_per_session reduces the frequency but does not fix the pinned-mailbox lifecycle bug.

View original on GitHub ↗

4 Comments

Yurdos-r · 1 month ago

Independent reproduction from Codex Desktop (2026-07-21)

I reproduced the admission failure with a controlled baseline and a pending-mailbox variant.

Environment
  • Surface: Codex Desktop on Windows
  • App/CLI version: not exposed to the isolated test thread, so I cannot report it reliably
  • Observed capacity: root plus three direct subagent resident slots
  • No files or configuration were changed
Baseline: ordinary completed agents were reclaimable
  1. In a fresh isolated task, spawn direct subagents A, B, and C with clean history.
  2. Let all three return A_OK, B_OK, and C_OK and reach completed.
  3. Spawn a new direct subagent D.

Result: D's spawn_agent returned successfully without a noticeable wait, and D completed with D_OK.

This indicates that ordinary completed agents alone did not block a later spawn in this runtime.

Variant: a queue-only message pinned the completed resident
  1. In a second fresh isolated task, spawn A and let it complete with A_OK.
  2. Send STATUS_PROBE to completed A using queue-only send_message. It did not trigger a new turn.
  3. Spawn E and F with tasks that remain active for about 45 seconds.
  4. While E and F were active, attempt to spawn G.

The G spawn failed immediately with the exact output:

collab spawn failed: agent thread limit reached

At that moment only E and F were actively running. A was completed, but it had the pending queue-only mailbox item. The controlled difference from the successful baseline was the pending mailbox on A.

I then called followup_task on A; A consumed the queued work and returned A_DRAINED. E and F also completed. A subsequent retry could not be emitted because the isolated test task encountered a separate tool-routing failure, so I am not claiming that the recovery path was independently verified here.

What this reproduction supports
  • Normal completed residents were evictable/reclaimable in the baseline.
  • A completed resident with pending queue-only mail continued to consume residency capacity.
  • With that pinned resident plus two active children, a fourth direct child was rejected using the same generic AgentLimitReached error as true active-concurrency exhaustion.
  • The current error gives no indication that a completed agent's mailbox is the blocker.

This independently supports the pending-mailbox residency mechanism described in the issue and the need for an actionable error that distinguishes active concurrency from residency pressure.

KNaiFen · 22 days ago

I audited this against current main at c38a60ded2ffbad510f8b512e2902f0128e998e8. The reported residency mechanism is still present, and there is an important race constraint for a fix.

Why a tool-handler status precheck is insufficient

handle_message_string_tool resolves and loads the receiver, then submits an Op::InterAgentCommunication. The actual mailbox insertion happens later in session::handlers::inter_agent_communication:

message_tool.rs -> AgentControl::send_inter_agent_communication
                -> ThreadManagerState::send_op
                -> session::handlers::inter_agent_communication
                -> InputQueue::enqueue_mailbox_communication

A Completed check in message_tool.rs alone can pass while the target is still running, followed immediately by target completion before the queued op reaches enqueue_mailbox_communication. That recreates the same completed-plus-pending-mailbox blocker.

Current task completion calls maybe_start_turn_for_pending_work, but that helper intentionally starts only for trigger_turn mail (or durable sleep). Queue-only mail therefore remains pending, and residency::is_unloadable rejects the resident because has_pending_mailbox_items() is true.

Invariant a fix should enforce

After the target reaches a terminal state, queue-only mail must not become an indefinite residency blocker. The atomic decision needs to live at the actual mailbox admission/completion boundary, not only in the model-facing tool handler.

Two viable semantics appear to require maintainer direction:

  • Promote queue-only mail that crosses the running-to-terminal boundary into a wake-up turn, preserving the message and draining the resident.
  • Reject terminal queue-only admission and return an actionable use followup_task error. This needs an acknowledgement path from the session handler back to the tool call; the current op submission only confirms that the op was accepted by the session loop.

Regression coverage

A focused test should cover both orderings:

  1. Target A is already completed before send_message(A).
  2. A completes after the tool-side check but before mailbox admission.
  3. With A plus two active residents at capacity 3, spawning D must either succeed after the defined drain behavior or return the explicit delivery error, never generic AgentLimitReached caused by a hidden queue-only blocker.
  4. Assert that A has no stranded queue-only mailbox item after the operation settles.

Per the invitation-only contribution policy, I will not open an unsolicited PR. If a maintainer selects the intended delivery semantic and invites a contribution, I can implement the narrow fix and integration test.

wuyak · 18 days ago

This issue also exposes a message-lifecycle boundary that is currently difficult for the caller to observe:

accepted by the session loop
  != admitted to the mailbox
  != consumed by a child turn
  != applied as steering
  != safe to discard
  != resident thread reclaimable

I opened #37744 for the complementary protocol gap: a parent should be able to distinguish queued, consumed, applied, rejected, superseded, and unknown, rather than treating a completed send_message call as proof of application.

At a terminal boundary, the runtime should define one deterministic behavior: reject queue-only delivery and direct the caller to a follow-up turn, promote the message into a new turn, or externalize the pending mailbox so it does not pin residency. Whichever behavior is selected, the resulting delivery and reclaim state should be observable.

theosp · 3 days ago

Corroborating production reproduction

Observed on Codex Desktop 26.818.61809 with bundled CLI 0.149.0-alpha.4.3 in session 01a03411-e72a-73a2-aae2-97872a621dab.

The parent called queue-only send_message on child A while A was running; A completed shortly afterward. Two fresh siblings then started, but a third fresh spawn failed with agent thread limit reached. At failure, the list showed the root plus the two siblings running and A still completed. Calling interrupt_agent on A reported its prior completed status but did not free admission, and the third spawn failed again.

This closely matches the race-shaped reproduction described here: a message crossing the completion boundary appears to leave the completed child resident and non-evictable. It also explains the broader symptom reported in #39694. The session had previously created 16 fresh children over multiple waves, but the evidence supports residency pressure—not a cumulative 16-child limit.