Codex Desktop: thread/resume can return a thread while Shutdown is pending, leaving a dead agent loop

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #33206

Powered by Codex Action

I did a static lifecycle trace on current public main (56395bddaf26eb2829387ca6a417bf9128e5b239). One cross-layer path can produce exactly the reported ordering, and it is not covered by App Server's existing pending_thread_unloads guard.

For a MultiAgentV2 resident subagent, core eviction currently does this:

1. try_unload_one_resident() obtains the candidate from ThreadManager.
2. It calls candidate_thread.shutdown_and_wait() while the thread is still present in the manager.
3. It removes the thread from the manager only after shutdown completes (line 146).

During step 2, App Server can still resolve that same Arc<CodexThread> as running: resume_running_thread() uses get_thread(), then later enqueues a running-thread resume response on its listener (lines 3486-3506). That leaves a window where Shutdown is committed but thread/resume can subscribe and acknowledge the old runtime before its loop exits.

App Server does have two good closing checks—at thread_resume_inner() and immediately before subscription in the listener (lines 598-623)—but both consult the App-Server-local pending_thread_unloads set. Core residency eviction does not populate that set, so the guard cannot see this shutdown source.

This is a hypothesis to validate against the affected thread's source, not a claim that every occurrence has this cause. If the logged thread was a V2 subagent (or another core-owned residency candidate), the ordering is a close match. Checking its persisted/session source would distinguish this path from App Server's own idle-unload path.

The durable fix likely needs a lifecycle primitive shared by core and App Server, rather than another App-Server-local flag:

• atomically transition a thread from Running to Closing before submitting Op::Shutdown;
• make get_thread_for_resume/resume leasing fail or wait when the phase is Closing;
• after termination, cold-resume from the persisted rollout instead of returning the stale runtime;
• make turn/start reject or rehydrate a Closing/Closed runtime so a stale handle cannot remain indefinitely usable in the registry.

A useful deterministic regression would put a barrier between Op::Shutdown submission and session-loop termination, concurrently invoke thread/resume, and assert that no response exposes the old runtime. I would run that against both shutdown owners: App Server idle unload and core V2 residency eviction. Releasing the barrier should allow a cold resume, after which the first turn/start must succeed. Repeated concurrent resume attempts should converge on one live runtime.

For frontier multi-agent systems, resumability is part of the control plane: a thread must be either leasable or closing, never both. Making that state explicit prevents stale agent handles, duplicate runtimes, and recovery loops across every client—not just the Desktop symptom here.

Disclosure: this analysis was produced with significant assistance from OpenAI Codex. It uses only the public openai/codex repository and synthetic concurrency reasoning; it contains no private code, data, prompts, roadmaps, or product-specific examples.

Confirmed the same lifecycle sequence independently on ChatGPT desktop app 26.715.2305.0, app-server 0.145.0-alpha.18, and Windows 10.0.26200.0 x64.

My logs show Shutdown submitted, thread/resume succeeding about 10 seconds later, the agent loop exiting about 5 seconds after that, and subsequent turn/start requests repeatedly failing with -32603: agent loop died unexpectedly.

Restarting the desktop app restored the affected task. I initially filed #34131 before seeing this report; it contains the screenshot, timestamps, and affected session ID.

I now have an isolated public-source implementation for the completed-thread idle-unload race described here, rebased onto openai/codex main at 2deed3fb9c00c74dac3d177ea700d6fb7a94539d:

https://github.com/openai/codex/compare/main...luziyi123448-gif:codex/loop-death-fix?expand=1

The published head is fixed at c6abb714d5351e744669b928aa57521b9540b1db (two commits, 34 files). The implementation covers the full acknowledgement boundary rather than only adding another resume-side check:

• core SessionIo transitions from accepting to closing once Shutdown is successfully enqueued, so later user input is rejected with ThreadClosing before loop termination;
• app-server keeps idle-unload state fenced after the 10-second wait expires and returns stable THREAD_DRAINING errors from resume/listener/turn-start paths until actual termination;
• turn/start is acknowledged only after the durable TurnStarted event is persisted and observed;
• runtimeInstanceId plus cancellation-safe (threadId, clientUserMessageId) deduplication gives retries an exactly-once result within one runtime;
• deterministic regressions cover completed turn -> idle unload barrier -> resume/turn-start, timeout -> eventual cold resume, cancellation, and persistence failure.

Post-rebase validation on Windows:

• 3 core lifecycle/persistence regressions passed;
• 6 app-server lifecycle/idempotency regressions passed (nextest marked the passing tracing test as LEAK because its test process exited late);
• codex-rmcp-client: 109 passed, 5 skipped;
• codex-app-server-protocol, codex-protocol, and codex-analytics: 603 passed;
• scoped Clippy, just fmt-check, and git diff --check passed.

I have not opened an unsolicited PR because the repository contribution policy is invitation-only. If this lifecycle contract matches the intended direction, would a maintainer be willing to invite a PR from the published branch (or indicate any scope changes needed before invitation)?

Confirming the same failure on a newer Windows build, with an additional recovery path that may help isolate the stale-runtime boundary.

Environment

• Codex Desktop Windows MSIX: 26.730.8199.0 (x64)
• Windows: 10.0.26200.0
• Affected thread: ordinary owner-visible top-level project task; the failing resend did not involve closing subagents
• The thread was long-running and image/annotation-heavy (local rollout was approximately 220 MiB before recovery). I do not know whether size was causal or only an amplifier.

Sanitized lifecycle ordering

The affected thread's app-server logs showed:

t+00s  Submission ... op: Shutdown
t+10s  Shutting down Codex instance
t+10s  Agent loop exited
t+12s  composing running thread resume response ...
       active_turn_present=false active_turn_id=None active_turn_status=None

The logged resume response was therefore composed after the loop-exit event in this occurrence. This still looks consistent with the stale runtime registration described here, and suggests the acknowledgement path can complete after termination rather than only while shutdown is pending.

User-visible behavior

The composer contained a normal text prompt plus 25 in-app Browser annotation attachments. The content is omitted here.

Every manual retry produced:

Error creating chat
failed to start turn: internal error; agent loop died unexpectedly

Reopening the task did not recover it. A task-native background send attempt also returned the same error before a user-input submission was created for the dead thread. The UI continued retaining the complete composer payload.

Recovery evidence

A supported Codex task handoff from the local checkout to a fresh Codex worktree succeeded:

1. Handoff created a replacement task/runtime from the completed persisted history.
2. The retained 25-annotation composer attachment was carried into the replacement task.
3. The next Send created a real backend turn.
4. The replacement task became active and began producing agent progress.

This recovered the work without restarting the whole Desktop app. It also indicates that the persisted history and composer payload were healthy; the unusable state was attached to the old runtime/task registration.

A useful product recovery would be to make turn/start perform this cold rehydration automatically when the registered loop is absent, without requiring a worktree handoff or replacement task.

Privacy: thread/request IDs, usernames, local paths, workspace/repository names, task title, account details, prompt text, annotation contents, and unrelated logs have been removed.

Follow-up after rechecking the affected task's archived rollout, collaboration events, and persisted thread state.

My earlier comment said the failing resend did not involve closing subagents. That remains true, but it omitted relevant multi-agent history:

• The affected task was an owner-visible top-level task with substantial prior MultiAgentV2 use and real parent↔child messaging.
• The last verified parent↔child back-and-forth ended about 22 hours 45 minutes before the logged Shutdown; it was not immediately adjacent to the failure.
• Immediately before the final successful completion, the task's live collaboration roster reported only the root agent.
• However, persisted state still contained 140 child edges marked open for that parent task when inspected after the failure. This live-roster/persisted-edge mismatch may be relevant to residency or lifecycle bookkeeping.
• During the final successful owner turn, the affected task sent one task-native coordination message to a separate top-level task. The recipient logged receipt about 52 seconds later. I found no reply delivered back to the affected task before its loop was shut down roughly 27 minutes later.

So the precise close-window sequence was not “child replies, parent replies, immediate crash.” It was: extensive earlier two-way subagent history; apparently root-only live roster but many persisted open child edges; then a one-way cross-task send; later the shutdown/resume race and permanent dead-loop behavior.

This does not establish that inter-agent messaging caused the failure. It does add a concrete state condition worth testing alongside the MultiAgentV2 residency hypothesis already discussed here: a completed top-level task whose live agent roster and persisted spawn-edge state disagree, followed by cross-task coordination and idle/resume lifecycle activity.

Useful regression variants may be:

1. parent task with many completed/evicted children whose persisted edges remain open;
2. live roster returns root-only;
3. parent completes a turn and sends a message to another top-level task;
4. idle unload/resume runs concurrently with unrelated agent-communication traffic;
5. assert the parent is either cold-resumable or explicitly closing, never returned as a dead running runtime.

Privacy: child/task IDs, agent names, prompts, message contents, repository/workspace names, local paths, and unrelated logs are omitted.

Hi @luziyi123448-gif, this subagent persistence/history issue aligns with some boundary anomalies observed in multi-agent rollouts. Codex Rescue Alpha5 provides read-only lifecycle and subagent boundary diagnostics, cleanly separating historical start markers from current live execution state without altering the source rollout.

If you have access to the local session, you can run a non-destructive check:

pip install codex-rescue==0.1.0a5
codex-rescue doctor --latest

No raw session data is required, and please redact private paths if you share any output.