thread/resume hangs forever when codex thread is stuck "active" with no running turn (0.146.0)

What version of Codex CLI is running?

0.146.0

What subscription do you have?

max

Which model were you using?

_No response_

What platform is your computer?

_No response_

What terminal emulator and version are you using (if applicable)?

_No response_

Codex doctor report

What issue are you seeing?

title:
labels: bug, app-server

## Summary

thread/resume against a thread that thread/list reports as active never returns, even though the thread has no in-progress turn. The response for the resume is handed to the thread listener (SendThreadResumeResponse), and that listener never replays it, so the JSON-
RPC request future hangs indefinitely.

A bisect-style probe shows every read-level call (thread/list, thread/read, thread/fork) returns in <0.3 s for the same thread; only thread/resume never returns. The thread's last persisted turn on disk is a normal turn_aborted/turn_stopped, so the in-RAM active flag
is a stale listener state, not a real running turn.

This is the state machine a lot of agent runners will hit on a SIGTERM during an interruptible turn: the turn is aborted and flushed to the rollout, but the thread listener's local running flag is never reset.

## Environment

• codex-cli 0.146.0 (codex app-server, WebSocket transport)
• Linux x86_64, single-node container
• reproducible with a plain WS client (does not require a particular runner)

## How the stuck state is produced

Observed while running a multi-thread agent app-server that gets SIGTERM'd between turns:

1. Many threads are running xhigh reasoning turns at the same time.
2. The runner initiates a graceful drain by sending turn/interrupt for an active turn; codex flushes the abort, the rollout ends with {"type":"turn_aborted","reason":"interrupted"}.
3. Before the listener finishes resetting, the app-server is SIGTERM'd (container --stop-timeout expires) and force-killed.
4. After restart, thread/list for the same thread_id shows it as resident again and status.type == "active" with activeFlags == [].

So on the next thread/resume, codex sees the thread as "running" and routes the response through the listener — which has no turn to finish and never responds.

> The two items below are secondary; they only matter because they make the stuck state reachable. The hang itself reproduces on any thread that ends up in the active + no running turn state, however it got there:
>
> - ShutdownState on SIGTERM only waits for running_turn_count == 0 (graceful drain in lib.rs); it does not submit Op::Interrupt/Op::Shutdown to in-flight turns, so a long reasoning turn can pin a thread until the stop timeout force-kills the process.
> - The per-thread listener's running flag appears to be set on turn-start but not reliably cleared when the turn is aborted out-of-band (e.g. the SIGTERM-during-abort race).

## Reproduction

Given a thread T that is active with no running turn (verified via thread/read):

// thread/read T -> 200, instant
{ "thread": { "id": "<T>",
"status": { "type": "active", "activeFlags": [] },
"turns": [] } }

// thread/resume T -> HANGS FOREVER (no response, no error)
{ "jsonrpc": "2.0", "id": 7, "method": "thread/resume",
"params": { "threadId": "<T>", "cwd": "<cwd>" } }

Bisect matrix (same thread T, same app-server, back-to-back):

method wall time result
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
thread/list 0.11 s 200, lists T as resident, active / activeFlags=[]
─────────────────────────────────── ─────────────────── ──────────────────────────────────────────────────────
thread/read 0.04 s 200, T active, turns=[]
─────────────────────────────────── ─────────────────── ──────────────────────────────────────────────────────
thread/fork 0.17 s 200, returns a new thread id copied from T's rollout
─────────────────────────────────── ─────────────────── ──────────────────────────────────────────────────────
thread/resume (new forked thread) 0.03 s 200 — forked thread is idle, resumes cleanly
─────────────────────────────────── ─────────────────── ──────────────────────────────────────────────────────
thread/resume (T) >30 s (timed out) no response

The forked copy resumes in 0.03 s, so the underlying history/config is fine; the only thing that hangs is resuming a thread the listener still thinks is running.

## Root cause

loaded_thread_status (app-server/src/thread_status.rs, ~L429) returns ThreadStatus::Active whenever runtime.running is true, independent of whether any turn is actually in progress:

fn loaded_thread_status(runtime: &RuntimeFacts) -> ThreadStatus {
if !runtime.is_loaded {
return ThreadStatus::NotLoaded;
}
let mut active_flags = Vec::new();
// ...
if runtime.running || !active_flags.is_empty() {
return ThreadStatus::Active { active_flags };
}
// ...
ThreadStatus::Idle
}

The thread-listener task sets running when a turn starts, but the abort path does not appear to reset the listener-level running flag back to false when the turn was aborted out-of-band.

On thread/resume, resume_running_thread sees the resident thread and routes the response through the listener instead of the fast path — it builds a ThreadListenerCommand::SendThreadResumeResponse and sends it on listener_command_tx, then returns
RunningThreadResumeResult::Handled. The listener is the only thing that can emit the response for that request_id. For a thread that has no turn to finish, that branch never makes progress and the response future is never resolved.

The decision arm reads roughly:

} else if let Ok(existing_thread_id) = ThreadId::from_string(&params.thread_id)
&& let Ok(existing_thread_id) = self.thread_manager.get_thread(existing_thread_id).await
{
// ...
Some((existing_thread_id, existing_thread, source_thread))
}

i.e. "if we still hold this thread in memory, treat the resume as a rejoin" — with no check that the thread is actually idle.

## Impact

Clients stuck on thread/resume block forever with no error, no notification, and no codex-side timeout. In our runner this manifests as:

• the leader waiting indefinitely on the resume response future,
• per-turn silence/timeout watchdogs never firing because the turn never starts (round=0, no turn_id),
• the target looking "healthy" (app-server alive, thread/list works) while 12 workers are frozen until someone restarts the container and the stale threads are GC'd.

The hang is silent and self-perpetuating: as long as the thread remains resident as active, every thread/resume on it will hang.

## Proposed fix

The core bug is that the running flag can outlive the turn it was tracking. Two defenses are reasonable and independent:

1. (Required) The listener's running flag must be cleared whenever the turn reaches a terminal state — including out-of-band abort.

shutdown_session_runtime (core/src/session/handlers.rs ~L597) does the right thing for process shutdown:

async fn shutdown_session_runtime(sess: &Arc<Session>) {
// ...
sess.abort_all_tasks(TurnAbortReason::Interrupted).await;
sess.services.unified_exec_manager.terminate_all_processes().await;
// ...
}

…via Op::Shutdown, but the per-turn abort path (turn_interrupt + TurnAborted) doesn't seem to flip listener running back to false in the active + no running turn window. The cleanest fix is to drive running from ground truth (is there an in-progress turn for this
thread?) rather than a listener-local boolean that multiple transitions can leave stale.

1. (Defense in depth) resume_running_thread must not block forever on a thread that has no in-progress turn.

There is already a guard at ~L3197 (wait_for_thread_shutdown with a 10 s timeout) for the config-mismatch branch. The same shape can be applied to the rejoin branch: when thread/read proves status.type == "active" with activeFlags == [] (no real running turn), thread/
resume should either (a) flip the thread to idle and fall through to the fast resume path, or (b) treat the stale-active thread like the idle-but-loaded case — wait_for_thread_shutdown → remove_thread → resume_thread_from_rollout, so the resume is served from disk
rather than from a wedged listener.

## Before / after (expected behavior)

Before (current):

thread/read T → 200 in 0.04s, status=active activeFlags=[]
thread/resume T → <hangs forever>

The client's resume future is never resolved; thread/fork T works because it reads the rollout and doesn't touch the listener.

After (proposed):

thread/read T → 200, status=active activeFlags=[] (stale)
thread/resume T → 200 in <1s
// codex notices: active but no running turn
// → wait_for_thread_shutdown(T) (10s cap)
// → remove_thread
// → resume_thread_from_rollout
// → response served from disk

Resuming a thread that is merely cached as active but has no real turn should be indistinguishable from resuming an idle thread, because on disk (the rollout) the two are identical.

## Workaround (client side, verified)

Clients can sidestep the hang today by probing before resuming and forking the stuck thread into a fresh one (the forked thread starts life idle and resumes in 0.03 s):

r = await client.thread_read(thread_id=T)
status = r["thread"]["status"]
stuck = status.get("type") == "active" and not status.get("activeFlags", [])

if stuck:
forked = await client.thread_fork(thread_id=T, cwd=cwd, model=model)
T_new = forked["thread"]["id"]
await client.thread_resume(thread_id=T_new, cwd=cwd, model=model) # fast path
else:
await client.thread_resume(thread_id=T, cwd=cwd, model=model)

We have this running in our scheduler; 10/10 previously-stuck threads recovered in ~0.2 s each, while idle threads still take the normal fast resume path with no regression.

## What I'm asking

1. Confirm the diagnosis — is the listener-level running flag indeed the state that goes stale, and is resume_running_thread the right place to add a "no in-progress turn → don't route through listener" guard?
2. If the fix direction looks right, I'm happy to open a PR targeting loaded_thread_status / resume_running_thread; pointers to where the listener resets running today would unblock me.

Happy to attach the standalone WS repro script if useful.

What steps can reproduce the bug?

f

What is the expected behavior?

_No response_

Additional information

_No response_