app-server: compaction completes server-side but `turn/completed` is never delivered to the JSON-RPC client — thread stays "Compacting…" yet remains healthy/resumable

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #27395

Powered by Codex Action

I took a deeper look at this one against current main.

What I see in the app-server path:

• EventMsg::TurnComplete is handled in app-server/src/bespoke_event_handling.rs; after aborting pending server requests and noting the thread watcher completion, it calls handle_turn_complete(...).
• handle_turn_complete(...) clears the per-turn summary and emits ServerNotification::TurnCompleted via emit_turn_completed_with_status(...).
• The thread state tracker also treats EventMsg::TurnComplete as terminal state for the active turn.

I also ran the focused compaction test locally in WSL:

cargo test -p codex-app-server --test all auto_compaction_remote_emits_started_and_completed_items -- --nocapture

Result: passed (1 passed; 0 failed). That test drives the app-server JSON-RPC flow, triggers remote auto-compaction on the third turn, waits for the matching turn/completed, and verifies the compaction request includes the expected turn metadata.

So the current happy-path live-connection case appears covered on main. That makes the original 0.139.0 report look either fixed by the turn-state/compaction work that landed around 0.140, or dependent on a lifecycle edge case rather than the basic compaction completion path.

The remaining suspicious gap I found is delivery across connection lifecycle changes. ThreadScopedOutgoingMessageSender::send_server_notification(...) returns early when there are no connection_ids. For in-process queues, TurnCompleted is explicitly classified as requiring delivery, but the thread-scoped sender still depends on a live connection id at send time. If the managed app-server reset / cleanup mentioned in #23954 drops the client connection during a long compaction, the server can finish the turn and keep the thread resumable while the original JSON-RPC client never observes the terminal turn/completed.

A useful next regression test would simulate disconnect/reconnect or stale connection ids during the compaction window, then assert one of these is true:

1. the terminal turn/completed is replayed to the reconnected client, or
2. thread/read / thread/list exposes enough terminal turn state for the client to reconcile instead of waiting forever.

In short: I cannot reproduce the live-connection path on current main; the higher-value repro now seems to be a connection-lifecycle test around long compaction, not another direct compaction happy-path test.

Confirming your connection-lifecycle hypothesis from the client side. In CodeMantis we see exactly this: during a long compaction the thread stays healthy and resumable — a fresh TUI /model works, thread/resume recovers full context — but our original JSON-RPC client never receives turn/completed, so the UI sits on "Compacting…" indefinitely. It's not the happy path; it's delivery when the connection churns underneath an in-flight turn, matching your send_server_notification early-return on missing connection_ids.

Of your two proposed assertions, (2) thread/read/thread/list exposing terminal turn state is the higher-value one for clients — it lets us reconcile on a timeout without forcing a resume, which is what we do as a workaround today. (Re the duplicate flag for #27395: related but distinct — that's a turn/start timeout, ours is compaction-specific and stays resumable. Same family of "server finishes, client left hanging," different trigger.)

This issue maps closely to LS work on durable lifecycle events, subscriber cursors, terminal receipts, and deterministic reconciliation.

The central distinction is:

server-side operation completed
≠ completion event delivered
≠ client projection updated
≠ client knows the thread is idle

A terminal lifecycle transition should not exist only as a best-effort notification.

A possible authoritative turn state:

{
"schema_version": "turn-lifecycle-state/v0.1",
"thread_id": "thread-123",
"turn_id": "turn-044",
"trajectory_id": "project-alpha",
"state": "COMPLETED",
"terminal_sequence": 1842,
"completed_at": "2026-06-24T19:42:11Z",
"compaction": {
"performed": true,
"status": "COMPLETED",
"result_ref": "compaction-result-771"
}
}

The notification should then be treated as a projection of this durable state:

{
"sequence": 1842,
"event_id": "turn-044:completed",
"event_type": "TURN_COMPLETED",
"thread_id": "thread-123",
"turn_id": "turn-044",
"terminal_state_digest": "sha256:..."
}

A useful invariant would be:

«Every terminal turn state must be queryable and replayable even if its live notification was lost.»

Another invariant:

«A client must not depend on a single transient event to determine whether a thread is still active.»

A client cursor could track:

{
"client_id": "external-jsonrpc-client",
"thread_id": "thread-123",
"last_received_sequence": 1841,
"last_acknowledged_sequence": 1841,
"expected_terminal_turn": "turn-044"
}

If the client sees no terminal event within a bounded interval, it should be able to request reconciliation:

get_turn_state(thread_id, turn_id, after_sequence)

Possible response:

{
"thread_id": "thread-123",
"turn_id": "turn-044",
"authoritative_state": "COMPLETED",
"terminal_sequence": 1842,
"client_missing_sequences": [
1842
],
"replay_events": [
{
"event_type": "TURN_COMPLETED",
"sequence": 1842
}
]
}

This is similar to reliable subscription recovery:

live event stream
→ cursor
→ disconnect or dropped event
→ reconnect
→ replay from last acknowledged sequence
→ rebuild client projection

The client should not need to tear down the session and call "thread/resume" merely to discover that the previous turn already finished.

The compaction lifecycle could also be explicit:

TURN_RUNNING
→ COMPACTION_STARTED
→ COMPACTION_COMPLETED
→ TURN_COMPLETED

with each transition carrying a monotonic sequence number.

A terminal receipt might look like:

{
"schema_version": "turn-terminal-receipt/v0.1",
"thread_id": "thread-123",
"turn_id": "turn-044",
"terminal_state": "COMPLETED",
"sequence": 1842,
"result_digest": "sha256:...",
"delivery_status": {
"published": true,
"acknowledged_clients": [],
"replay_available": true
}
}

The important boundary is:

event published
≠ event acknowledged

A publisher should not assume the client received the terminal state merely because the notification write succeeded locally.

A deterministic conformance fixture could test:

1. compaction completes and "turn/completed" is delivered normally;
2. terminal notification is dropped after server commit;
3. client reconnects from the last acknowledged sequence;
4. missing terminal event is replayed exactly once;
5. duplicate replay does not create duplicate completion;
6. client can query authoritative turn state;
7. thread reports idle even when the client projection is stale;
8. process restart preserves terminal state and sequence;
9. compaction result remains linked to the completed turn;
10. client reconciliation produces the same projection as uninterrupted delivery.

Related LS work:

• recovered continuation state:

https://github.com/safal207/LS/pull/651

• causal audit:

https://github.com/safal207/LS/issues/594

• evidence and authorization gates:

https://github.com/safal207/LS/issues/595

• deterministic replay:

https://github.com/safal207/LS/issues/597

Would a small vendor-neutral "TurnTerminalReceipt", subscriber cursor, and dropped-notification replay fixture help make terminal-event delivery and client reconciliation machine-testable?

Author here — glad to see this framed as a reconciliation problem, since that's exactly where we landed as a client.

One caution on scope, though: for us the minimal fix is much smaller than a full durable-event/cursor/receipt system. What actually unblocks an external client is just two primitives:

1. A monotonic sequence number on turn/* (and thread/*) notifications, so a client can detect it missed a terminal event (the original question #3 here), and
2. A pull for authoritative terminal turn state — e.g. thread/read returning the turn's terminal status, or a get_turn_state(thread_id, turn_id) — so we reconcile on a timeout without forcing a thread/resume (which is the workaround we run today, and the anti-pattern you correctly call out).

That's the higher-value of @MakenRosa's two assertions (terminal state exposed for reconciliation, vs. guaranteed replay), and it looks close to what #27996 ("Send request-scoped turn state over WebSocket") already started. The subscriber cursor, ack-tracking, and TurnTerminalReceipt with delivery status are reasonable as a longer-term reliability layer, but I'd keep this issue focused on the queryable-terminal-state + sequence-number increment so it stays shippable — the root cause @MakenRosa found (send_server_notification early-returning on missing connection_ids when the connection churns mid-compaction) is a concrete delivery bug that a pull-based reconcile path closes directly.

A conformance fixture around disconnect/reconnect during the compaction window would be genuinely useful — happy to validate any candidate fix against CodeMantis on 0.140.

Author here — glad to see this framed as a reconciliation problem, since that's exactly where we landed as a client. One caution on scope, though: for us the minimal fix is much smaller than a full durable-event/cursor/receipt system. What actually unblocks an external client is just two primitives: 1. A monotonic sequence number on turn/* (and thread/*) notifications, so a client can detect it missed a terminal event (the original question move all tests under tests/ #3 here), and 2. A pull for authoritative terminal turn state — e.g. thread/read returning the turn's terminal status, or a get_turn_state(thread_id, turn_id) — so we reconcile on a timeout without forcing a thread/resume (which is the workaround we run today, and the anti-pattern you correctly call out). That's the higher-value of @MakenRosa's two assertions (terminal state exposed for reconciliation, vs. guaranteed replay), and it looks close to what #27996 ("Send request-scoped turn state over WebSocket") already started. The subscriber cursor, ack-tracking, and TurnTerminalReceipt with delivery status are reasonable as a longer-term reliability layer, but I'd keep this issue focused on the queryable-terminal-state + sequence-number increment so it stays shippable — the root cause @MakenRosa found (send_server_notification early-returning on missing connection_ids when the connection churns mid-compaction) is a concrete delivery bug that a pull-based reconcile path closes directly. A conformance fixture around disconnect/reconnect during the compaction window would be genuinely useful — happy to validate any candidate fix against CodeMantis on 0.140.

Thanks — agreed on the scope distinction.

The two primitives you identified look like the right Phase 1 contract:

1. a documented monotonic sequence on "turn/" / "thread/" notifications, with a clearly defined scope and behavior across reconnects; and
2. an authoritative terminal-state query through "thread/read" or a small "get_turn_state(thread_id, turn_id)" endpoint.

That is enough for a client to:

• detect a sequence gap;
• wait for a bounded timeout;
• query the committed server-side state;
• converge to "COMPLETED" / idle without calling "thread/resume".

I would treat subscriber cursors, acknowledgements, durable replay, and "TurnTerminalReceipt" as optional Phase 2 reliability layers rather than requirements for fixing this issue.

A minimal conformance fixture could therefore be:

1. start a turn that triggers compaction;
2. disconnect or remove its connection id before terminal notification delivery;
3. let the server commit the completed turn;
4. reconnect the client;
5. observe the sequence gap or timeout;
6. query authoritative turn state;
7. verify the client converges to the same idle/completed projection as uninterrupted delivery;
8. verify no "thread/resume" is required.

One nuance on #27996: it appears to establish request-scoped model turn state across WebSocket and compact requests, which is adjacent and useful, but unless that state is exposed as authoritative terminal lifecycle state through "thread/read", it may not yet close the external-client reconciliation gap.

Happy to help reduce the earlier proposal into this smaller fixture and protocol contract.

+1 — the Phase 1 (sequence + queryable terminal state) / Phase 2 (cursors, acks, receipts, replay) split is the right line, and your 8-step fixture captures exactly the path we hit. The detect-gap → bounded-timeout → query-committed-state → converge-without-thread/resume loop is precisely the reconcile we want in place of our current resume-based workaround.

Your #27996 nuance is the crux, and it's the one thing I'd want a maintainer to confirm: request-scoped turn state crossing the WebSocket/compact path is necessary but not sufficient — it only closes the external-client gap if that state is pullable as authoritative terminal lifecycle state (via thread/read or a small get_turn_state) after a reconnect. If it stays push-only, a client that missed the live event still has nowhere to query.

Happy to pair on reducing this to the minimal fixture + contract — I can bring the client side (CodeMantis driving app-server on 0.140), so the fixture exercises a real external JSON-RPC consumer rather than only an in-process harness. At this point I think it mainly needs a maintainer to say whether the queryable-terminal-state surface belongs to the #27996 line of work or wants a separate method.

+1 — the Phase 1 (sequence + queryable terminal state) / Phase 2 (cursors, acks, receipts, replay) split is the right line, and your 8-step fixture captures exactly the path we hit. The detect-gap → bounded-timeout → query-committed-state → converge-without-thread/resume loop is precisely the reconcile we want in place of our current resume-based workaround. Your #27996 nuance is the crux, and it's the one thing I'd want a maintainer to confirm: request-scoped turn state crossing the WebSocket/compact path is necessary but not sufficient — it only closes the external-client gap if that state is pullable as authoritative terminal lifecycle state (via thread/read or a small get_turn_state) after a reconnect. If it stays push-only, a client that missed the live event still has nowhere to query. Happy to pair on reducing this to the minimal fixture + contract — I can bring the client side (CodeMantis driving app-server on 0.140), so the fixture exercises a real external JSON-RPC consumer rather than only an in-process harness. At this point I think it mainly needs a maintainer to say whether the queryable-terminal-state surface belongs to the #27996 line of work or wants a separate method.

@HaraldSirius Yes — that’s exactly the boundary I’d want to lock in.

I agree the key acceptance point is not merely “request-scoped turn state survives the compact / reconnect path”, but “a client that missed the live terminal event can still pull the authoritative terminal lifecycle state afterward”.

So the minimal external-client contract I’d want a maintainer to confirm is:

1. gap detection on the notification stream (turn/* / thread/*);
2. authoritative pull of terminal turn state after reconnect (thread/read or a small get_turn_state);
3. convergence to the committed terminal state without requiring thread/resume as a recovery primitive.

That’s the crux for me as well: if terminal state remains push-only, the external JSON-RPC client still has no bounded recovery path after missing turn/completed.

And yes — using a real external consumer in the fixture would be ideal. The useful acceptance row feels like:

• disconnect during compaction window;
• terminal event missed on the live stream;
• reconnect detects a gap;
• client queries authoritative terminal state;
• thread converges to terminal state without resume or duplicate side effects.

If maintainers are open to that shape, I think it would cleanly separate the Phase 1 reliability floor from the broader cursor/receipt/replay work.

Thanks for reporting this problem. Until recently, Codex used a separate endpoint and server-side logic for compaction. We recently switched to a more robust approach where the compaction logic is moved locally using the same endpoint as normal turns. This eliminates "remote compaction" errors and increases the stability and reliability of compaction operations. If you see further problems with compaction, please use /feedback and open a new bug report.

@etraut-openai Thank you for the clear explanation and for closing the loop. Moving compaction onto the same local path and endpoint as normal turns directly addresses the split-brain behavior reported here: server-side completion without the corresponding client-visible terminal event.

The main invariant I’ll watch for in future regressions is that compaction and ordinary turns expose the same terminal-state semantics to the JSON-RPC client. If the UI becomes stuck again while the thread remains resumable, I’ll file a fresh report with /feedback data as requested.