`experimentalRawEvents` cannot be enabled on `thread/resume` / `thread/fork`, so raw-event mirroring permanently breaks after any resume

Resolved 💬 3 comments Opened Jul 20, 2026 by bmaurer Closed Jul 20, 2026

Summary

thread/start accepts experimentalRawEvents: true, which makes the app-server stream every raw ResponseItem as rawResponseItem/completed (and emit rawResponse/completed). However, there is no way to enable raw events on thread/resume or thread/fork: ThreadResumeParams and ThreadForkParams have no such field, and the app-server hardcodes raw_events_enabled: false when it attaches the listener for a resumed or forked thread. Since the flag lives only in the in-process ThreadState (it is not persisted with the thread), every resume of a non-running thread — including a same-process resume of a thread that was originally started with the flag — silently and permanently disables raw events for all subsequent turns. There is no workaround: we probed the param on thread/resume, config overrides (-c), thread/fork, and the experimental-feature APIs, and found no way to re-enable it.

Observed on codex-cli 0.144.4 via live JSON-RPC probes; the code is unchanged at current main (86102db5a1a7a49ce08e79f900e0897d94aa3770, 2026-07-20).

Use case

We build durable session mirroring on top of the app-server: we start threads with experimentalRawEvents: true, journal each rawResponseItem/completed payload, and if the host running the app-server is lost we resume the thread on another host with thread/resume { threadId, history: [ ...journaled items ] }. This is exactly the "FOR CODEX CLOUD" flow that ThreadResumeParams.history documents. The gap means the mirror only survives one hop: after the first thread/resume, no further raw items are emitted, so the journal can never be brought up to date again and a second host failure loses the session.

Root cause (file/line, rust-v0.144.4 and main)

The flag is per-thread in-process state:

  • codex-rs/app-server/src/thread_state.rs:89ThreadState.experimental_raw_events: bool (default false), set only via set_experimental_raw_events from ThreadStateManager::try_ensure_connection_subscribed(..., experimental_raw_events) (thread_state.rs:497-524 at main).
  • codex-rs/app-server/src/request_processors/thread_lifecycle.rs:314-325 — the listener event loop drops EventMsg::RawResponseItem / RawResponseCompleted unless the thread's flag is set. Core always emits these events (codex-rs/core/src/session/mod.rs:3134-3142); the app-server filter is the only gate, and the flag is never persisted to the rollout.

Who sets the flag:

  • thread/start passes params.experimental_raw_events through to the listener attach — thread_processor.rs:1301-1310 at main (:1285-1290 at rust-v0.144.4). ✅
  • thread/resume (both the history variant and disk resume converge on one attach site) hardcodes falsethread_processor.rs:3245-3255 at main (:2858 at rust-v0.144.4). ❌
  • thread/fork hardcodes falsethread_processor.rs:4179-4189 at main (:3609 at rust-v0.144.4). ❌
  • Spawned child (subagent) threads inherit the flag from their parent thread since #25603 ("Inherit raw events for spawned child listeners") — thread_processor.rs:2981-3012 at main. ✅

So the propagation gap was already recognized and fixed for spawned children in #25603; resume and fork were just not covered. A resume with history forks to a new thread id, and a disk resume of an unloaded thread builds a fresh ThreadState, so in both cases the default false wins and raw events are gone for good.

Reproduction (0.144.4)

  1. thread/start { "experimentalRawEvents": true } → run a turn → rawResponseItem/completed notifications arrive for every response item. ✅
  2. thread/resume { "threadId": "<id>", "history": [ ...items ] } (or plain thread/resume after the thread is unloaded) → run a turn → no rawResponseItem/completed notifications, and no field on ThreadResumeParams accepts the opt-in. ❌

Minimal transcript for step 2's turn (0.144.4): the turn streams item/started / item/completed etc. normally, but zero rawResponseItem/completed, confirming the events are filtered app-server-side rather than not produced (core emits them unconditionally; see above).

Proposed fix

Mirror the thread/start opt-in on the other two thread-construction methods, keeping the experimental naming and semantics:

  • Add #[experimental("thread/resume.experimentalRawEvents")] experimental_raw_events: bool to ThreadResumeParams, and the equivalent thread/fork.experimentalRawEvents to ThreadForkParams.
  • Pass the value through the existing ensure_conversation_listener(..., raw_events_enabled) at the resume attach site (thread_processor.rs:3245) and the fork attach site (thread_processor.rs:4179), replacing the hardcoded false.
  • For the running-thread rejoin path in resume_running_thread, set the flag on the existing ThreadState when requested (same monotonic enable-only semantics as try_ensure_connection_subscribed).

Because both new fields are #[experimental], the vendored (non-experimental) schema fixtures are unchanged, and the wire behavior for clients that don't send the field is identical.

We have this implemented and verified on a branch against current main (86102db5a): two new integration tests, thread_resume_with_history_supports_experimental_raw_events (tests/suite/v2/thread_resume.rs) and thread_fork_supports_experimental_raw_events (tests/suite/v2/thread_fork.rs), exercise resume-with-history and fork followed by a live turn against the mock Responses server and assert a rawResponseItem/completed notification arrives. Both fail with the current attach sites (the notification never arrives) and pass with the param plumbed through; the full thread_resume/thread_fork suites (60 tests) and cargo test -p codex-app-server-protocol (including the schema-fixture checks) pass, and write_schema_fixtures produces no diff. Happy to send the PR if the team wants to invite it, per the contribution policy — or feel free to just take the outline above; the change is small.

An alternative we considered: persisting the opt-in with the thread (rollout metadata) so resume restores it automatically. That is more invasive and changes semantics for existing clients, so the per-call param felt safer and consistent with thread/start.

View original on GitHub ↗

This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗