MultiAgentV2 concurrent follow-up during eviction can leave a removed runtime in residency

Environment

• Upstream commit: 61a44880a85d2fd0d8770908dea5733495e571c8
• Feature mode: MultiAgentV2 enabled
• Reproduction: deterministic codex-core test fixture with V2 resident capacity set to one

Summary

At the V2 residency limit, eviction can select a completed runtime, A, and remove A from the resident LRU while A is still loaded in the thread manager.

During the asynchronous eviction window, a concurrent followup_task can find A through the loaded-runtime path and touch A back into residency. The original eviction can then continue, shut down A, and successfully remove that exact runtime from the thread manager.

The thread-manager removal path does not reconcile the concurrent residency touch. Residency can therefore continue to contain A even though the thread manager no longer does. The next valid slot reservation then returns a false:

AgentLimitReached { max_threads: 1 }

That failed reservation removes the stale entry as a side effect, so an immediate retry succeeds.

This is distinct from #33777: in this reproduction, eviction does not hang. Shutdown and exact thread-manager removal complete successfully, but the concurrent residency touch survives that removal.

It is also distinct from #32353 and #34518: the target is not kept non-unloadable by pending queue-only mail. It passes the unloadability check and is removed, after which stale residency accounting causes the false capacity rejection.

Deterministic reproduction

I added a test-only one-shot synchronization point that pauses the real eviction path after the production unloadability check and before rollout materialization and shutdown.

The controlled interleaving is:

1. Completed runtime A occupies the only V2 residency slot.

2. Eviction selects A and removes it from the resident LRU.

3. Eviction passes the real unloadability check and pauses.

4. A remains present in the thread manager.

5. The real followup_task handler resolves A through the loaded-runtime path.

6. That path touches A back into V2 residency.

7. The test observes mailbox activity and TurnStarted.

8. Eviction resumes, shuts down A, and removes the exact A runtime from the
   thread manager.

9. Residency still contains A.

10. The next reservation returns AgentLimitReached.

11. An immediate retry reserves successfully.

The synchronization only controls scheduling at an existing production transition. It does not construct an otherwise unreachable state, and it does not rely on arbitrary sleeps.

Independent fixture runs create different thread UUIDs, so the result is not tied to a particular ID.

Expected behavior

After a runtime is successfully removed from the thread manager, its old lifecycle must no longer consume V2 residency capacity.

A subsequent valid reservation must not fail because of a stale residency entry for a runtime that no longer exists in the manager.

Actual behavior

Two deterministic assertions fail on the upstream implementation:

• Manager removal succeeds, but residency remains residents = [A] with no pending slot.
• At capacity one, the first reservation returns AgentLimitReached { max_threads: 1 }, while an immediate retry reserves successfully.

This is a false capacity failure: no loaded or legitimately resident runtime occupies the rejected slot.

Root cause

Eviction candidate selection does not retain shared ownership of the runtime's residency lifecycle.

The relevant sequence is effectively:

remove A from residency
→ asynchronously inspect, materialize, and shut down A
→ eventually remove A from the thread manager

Removing A from residency does not remove it from the thread manager. During the asynchronous gap, the manager still exposes A as loaded.

A concurrent delivery can therefore touch A back into residency while the original eviction still holds A and continues toward shutdown and removal.

The manager-removal boundary does not perform final residency reconciliation, so the reinserted entry survives after removal.

The subsequent reservation discovers and removes the stale entry, but the fixed candidate scan ends without retrying admission in the same operation. That reconciliation is therefore incorrectly surfaced as AgentLimitReached.

Proposed implementation

I have a focused implementation that addresses this specific accounting and lifecycle gap:

• Tokenized RAII eviction claims. Selecting a candidate creates a unique in-flight claim rather than leaving its lifecycle unrepresented. The claim continues to consume capacity exactly once and is restored only while its token remains current.

• Centralized residency reconciliation. Successful thread-manager removal synchronously and idempotently removes every matching resident occurrence and eviction claim.

• Exact-runtime removal. Removal verifies the selected Arc<CodexThread> rather than relying only on thread ID, preventing an old eviction operation from removing a newer same-ID replacement.

• Explicit eviction outcomes. Successful eviction and residency reconciliation both retry admission within the same reservation operation. Only a genuine lack of an unloadable candidate becomes AgentLimitReached.

• Deterministic regression coverage. Tests cover the concurrent follow-up race, stale residency, false capacity rejection, cancellation restoration, competing removal, exact-runtime mismatch, idempotent cleanup, and claim/resident capacity accounting.

The implementation is intentionally limited to:

• stale V2 residency after concurrent touch and removal;
• the resulting false first-attempt capacity rejection;
• cancellation-safe and same-ID/ABA-safe accounting in the changed path.

Locking model

The resulting lock order is:

loaded delivery:
thread-manager read lock
→ residency mutex

successful removal:
thread-manager write lock
→ residency mutex

claim and slot operations:
residency mutex only

There is no residency-to-thread-manager lock edge, and the residency mutex is not held across asynchronous shutdown or materialization work.

This also linearizes the demonstrated race:

• If delivery obtains the manager read lock first, removal waits and subsequently reconciles the residency touch.
• If removal obtains the manager write lock first, the later loaded-runtime lookup cannot find A and therefore cannot touch it.

Validation

Final-source validation recorded:

• Focused race tests: 2/2 passed
• Complete residency-module filter: 7/7 passed
• Existing V2 follow-up completion test: 1/1 passed
• Stress validation: 20 iterations and 40/40 race-test executions passed
• Focused run plus stress: 42 successful executions of the two race tests
• No recorded hang
• Formatting and scoped Clippy checks completed successfully

The full codex-core test suite was not run, so I am not claiming full-suite validation.

Related issues

• #33777 covers unbounded waits during V2 eviction materialization or shutdown and adjacent cancellation/accounting concerns. This report demonstrates a different interleaving: concurrent loaded-runtime touch, successful manager removal, stale residency, and a false first-attempt capacity rejection. The proposed implementation does not address all of #33777.

• #32353 reports a similar AgentLimitReached symptom, but in that case pending queue-only mail keeps a completed runtime non-unloadable. In this case, eviction has already selected the runtime, concurrent follow-up delivery reinserts it, and manager removal succeeds.

• #34518 describes completed subagents remaining as hidden resident blockers. Its reported path is likewise based on a runtime remaining non-unloadable rather than successful removal followed by stale residency.

I did not find an existing public report describing this complete sequence:

eviction selects A
→ concurrent follow-up touches A
→ exact manager removal succeeds
→ stale A remains in residency
→ first reservation falsely fails
→ immediate retry succeeds

Scope and remaining risks

This implementation does not establish that:

• A follow-up accepted after the unloadability check prevents shutdown.
• Accepted work cannot be interrupted during shutdown.
• Channel submission, mailbox enqueue, or TurnStarted guarantees durable processing, completion, persistence, or recovery.
• Every timeout and unbounded cleanup path associated with #33777 is addressed.
• Unrelated ID-only shutdown or removal paths are safe.
• Cancellation after runtime shutdown but before manager removal is fully recovered.

This issue is specifically about residency reconciliation and false capacity accounting, not proof of permanent accepted-work loss.

PR invitation request

I have a focused implementation and deterministic regression coverage ready for this stale-residency and false-capacity failure.

The reproduction, root cause, ownership boundary, locking model, cancellation behavior, and remaining scope are documented above. If this approach aligns with the intended MultiAgentV2 lifecycle design, would a maintainer be willing to invite a focused PR for review?

I will wait for explicit maintainer alignment and invitation before opening the PR.