Codex stale-thinking spinner persists after stream state desync (thread stale detection + targeted recovery results)

Follow-up from production-hardening run:

• Run: /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T221428Z/
• Target threads:
• 019c9a7f-46f2-7e03-aceb-89afe89a581b
• 019c489a-0eb0-7142-8c30-e92ced2cf74a
• Tool classification:
• stale and at-risk respectively (heartbeat > 1h, stream markers present)
• Recovery attempt (automated, thread-safe): thread_not_found for both IDs on turn/interrupt.
• Script now treats this as terminal thread_not_found state and emits operator fallback guidance rather than generic failure.
• Recommendation remains: continue in a fresh thread using compact handoff of last confirmed completed steps.

Artifacts in this run include:

• stale_threads.jsonl
• actions.json
• incident_summary.md
• redacted_session_map.csv
• openai_feedback_draft.md
• recovery_results.json

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #12954

Powered by Codex Action

• Follow-up update from latest run:
• Added phrase-signal escalation path in detector (you are stuck somewhere. stop and replan / Reprise: j’ai relancé un test MLX), and explicit per-thread recovery forcing for --recover-thread targets.
• Latest run: /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T221609Z/
• 019c9ba7-e168-78a1-97f2-a188bc74d44a also now surfaces as targeted recovery candidate (with phrase-signal evidence), but app-server returns thread_not_found when interrupted.
• Recovery outputs remain per-thread only, non-global; no unrelated threads are stopped.

Follow-up production-hardening result (run ):

• Script: [stale-think] stale_threads=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/stale_threads.jsonl

[stale-think] actions=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/actions.json
[stale-think] summary=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/incident_summary.md
[stale-think] map=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/redacted_session_map.csv
[stale-think] openai_feedback=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/openai_feedback_draft.md
[stale-think] github_payload=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/github_issue_payload.json
[stale-think] stale_count=2 total_threads=57

• Targets: , ,
• Detection outcome: / / phrase-linked candidate.
• Recovery outcome: all three return on with required operator path: or (fresh thread + compact handoff).
• Version matrix is now explicitly tracked and this run shows mismatch: vs ().
• Evidence artifacts in including recovery results, stale rows, and final report.
• Automation remains active: (, no global stop actions).

Follow-up production-hardening result (run 20260226T222100Z):

• Script: /Users/jeremie/.codex/scripts/codex_stale_thinking_monitor.py
• Targets: 019c9a7f-46f2-7e03-aceb-89afe89a581b, 019c489a-0eb0-7142-8c30-e92ced2cf74a, 019c9ba7-e168-78a1-97f2-a188bc74d44a
• Detection outcome: stale, at-risk, phrase-linked candidate.
• Recovery outcome: all three return thread_not_found on turn/interrupt with required operator path: resume? if still visible, otherwise start fresh thread with compact handoff.
• Version alignment: codex_cli 0.101.0 vs codex_app 0.105.0 (mismatch).
• Evidence artifacts: /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222100Z/ (recovery_results, stale_threads, openai_feedback_draft, incident_summary).
• Automation remains active: /Users/jeremie/.codex/automations/codex-stale-think-watch/automation.toml (hourly, no global stop actions).

Updated follow-up packet (run 20260226T222239Z):

• Recovery remains blocked for primary stale candidate 019c9a7f-46f2-7e03-aceb-89afe89a581b and at-risk 019c489a-0eb0-7142-8c30-e92ced2cf74a in this execution window.
• Final RC package now in /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222239Z/final_rc_report.md.
• Version mismatch continues: codex_cli 0.101.0 vs codex_app 0.105.0 (version_alignment: mismatch).
• This run adds explicit operator handoff language in tooling: Recovered thread: resume? or new (fresh thread handoff). Automation prompt updated accordingly in automations/codex-stale-think-watch/automation.toml.

Follow-up production-hardening run after CLI/App alignment:\n\n- Aligned CLI to app protocol version via
changed 2 packages in 435ms (now both report ).\n- Re-ran targeted recovery on: and .\n- Result remains terminal for both: app-server returns on ; status in .\n- Operator handoff remains explicit: if thread still visible in UI, otherwise to continue from compact handoff.\n- Artifacts for this run:\n - stale_threads.jsonl\n - actions.json\n - openai_feedback_draft.md\n - recovery_results.json\n - incident_summary.md\n - github_issue_payload.json\n\nCurrent blocker for closure is persisted stream-state artifact for these IDs, likely orphaned run IDs/state-db mismatch independent of protocol version skew.

Follow-up production-hardening run after CLI/App alignment:

• Aligned CLI to app protocol version via npm install -g @openai/codex@0.105.0 (now both report codex-cli 0.105.0).
• Re-ran targeted recovery on thread IDs:
• 019c9a7f-46f2-7e03-aceb-89afe89a581b
• 019c489a-0eb0-7142-8c30-e92ced2cf74a
• Result remains terminal for both: app-server returns thread not found on turn/interrupt; status thread_not_found in /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T230417Z/recovery_results.json.
• Operator handoff remains explicit: Recovered thread: resume? if thread still visible in UI, otherwise new for compact handoff in a fresh thread.
• Artifacts for this run:
• stale_threads.jsonl
• actions.json
• openai_feedback_draft.md
• recovery_results.json
• incident_summary.md
• github_issue_payload.json

Current blocker for closure is persisted stream-state artifacts for these IDs, likely orphaned run IDs/state-db mismatch independent of protocol version skew.

RC-ready summary posted in local packet:
/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T230417Z/final_rc_report.md

This confirms both flagged threads are in final state:

• thread_not_found on targeted recovery attempts
• safe recovery mode required operator handoff only
• explicit prompt available: Recovered thread: resume? or new

CLI/App are now aligned at 0.105.0 (npm list -g @openai/codex --depth=0), so remaining blocker is stale/orphaned thread state in old session artifacts.

Follow-up (2026-02-27, production rollout + evidence):

We implemented and validated an automated stale-think containment path in local production.

What is now automated:
1) Detect stale threads from logs/transcripts using:

• unmatched task_started/task_complete
• inProgress+markedStreaming markers
• turn reconciliation errors ("thread not found", "Turn not found for event")
• heartbeat gap

2) Conservative per-thread recovery first (no global restart).

3) If recovery fails (thread_not_found / still stale), auto-relaunch to a fresh thread in Plan mode with continuity context and explicit ops guidance bootstrap.

4) Crash protection in the same loop:

• detects Chrome runaway memory by total RSS and peak process RSS
• optional peak-process termination to prevent system hangs

Operational evidence (sanitized):

• Source stale threads:
• 019c489a-0eb0-7142-8c30-e92ced2cf74a
• 019c9a7f-46f2-7e03-aceb-89afe89a581b
• Recovery attempts on stale source threads return thread_not_found even while stale indicators remain in historical telemetry.
• Auto-relaunch created fresh continuation threads successfully:
• library continuation: 019c9fee-0556-7173-9509-d6f3b64eaef8 (turn completed)
• openclaw continuation: 019c9fee-0555-77f0-b351-9ba90c013fd0 (turn entered inProgress then continued)

Updated hypothesis:

• stale spinner appears to be a UI/state reconciliation artifact after stream desync.
• old thread IDs can become unreachable to app-server (thread not found) while UI/log-derived stale state still persists for that thread lineage.
• fallback relaunch is effective for continuity, but root cause remains in stream-state reconciliation / orphaned turn handling.

Requested backend/UI focus:
1) hard reconciliation when source thread is not resolvable but UI still marks streaming.
2) idempotent closure for orphaned inProgress turn markers.
3) stronger handling for thread-stream-state-changed + missing turn references (Turn not found for event).

Sanitized artifact roots used for this validation:

• ~/.codex/reports/codex-stale-thinking/2026-02-27/
• ~/.codex/runtime/relaunched_threads.jsonl

If useful, I can post a compact event timeline matrix keyed by (thread_id, timestamp, event_type) without prompt contents.

I presume this is a bot. Closing.

Additional follow-up (same day):

New high-signal anomaly observed while hardening fallback logic:

• thread/read succeeds for affected thread IDs, but turn/interrupt on the same (threadId, turnId) returns thread not found (-32600).
• This was reproducible on previously stale source threads and on some freshly relaunched threads.

Example behavior class (sanitized):
1) thread/read(threadId=X, includeTurns=true) => returns turns incl. inProgress.
2) turn/interrupt(threadId=X, turnId=Y) => thread not found: X.
3) UI can remain in spinner state despite inability to interrupt.

Operational mitigation added:

• If relaunch turn remains stuck inProgress, auto-archive that relaunch thread.
• Create/attempt checkpoint continuation; if checkpoint also hangs, archive it too to avoid phantom spinner accumulation.
• Keep only non-spinning continuation threads visible.

Result from latest validation window:

• originally flagged source IDs no longer present as stale in active queue.
• one unrelated stale lineage remains under monitor.

This seems like a request-routing/state-index inconsistency between read-path and interrupt-path for thread identity resolution.