Auto-compaction uses stale token usage after tool output, causing unrecoverable context overflow

Summary

In a long-running app-server turn, a large tool result can push the next sampling request beyond the model context window without triggering auto-compaction. The compaction decision uses the last server-reported token usage, which does not include the newly appended tool output. Once the request fails, small follow-up turns can remain unrecoverable because pre-turn compaction also does not account for incoming context updates and user input.

Observed behavior

From a production app-server rollout (identifiers and prompt contents omitted):

• Effective model context window: 258,400 tokens
• Last server-reported input usage: 242,701 tokens
• The next function_call_output payload was 24,143 JSON bytes
• No context_compacted event occurred
• The next sampling request failed with an HTTP 400
• A tiny follow-up prompt on the resumed thread failed immediately with the same error
• Changing the selected model on resume did not recover the thread

The rollout contained 149 successful MCP tool calls and 164 token-usage updates before the failure, so this was not a model availability or authentication problem.

Suspected cause

Current main computes estimated_token_count after sampling, but compaction is gated only by token_status.token_limit_reached, which is derived from server-reported usage:

https://github.com/openai/codex/blob/bc8222b8d9e44377a3d7c7b7970e32e7c29ec34f/codex-rs/core/src/session/turn.rs#L315-L350

The estimate is currently only included in tracing. It is not used to decide whether the newly appended tool result makes another sampling request unsafe.

Pre-turn compaction has the same blind spot for newly incoming items; the source already calls this out:

https://github.com/openai/codex/blob/bc8222b8d9e44377a3d7c7b7970e32e7c29ec34f/codex-rs/core/src/session/turn.rs#L153-L157

The actual pre-turn gate checks only the persisted token status before recording incoming context/user input:

https://github.com/openai/codex/blob/bc8222b8d9e44377a3d7c7b7970e32e7c29ec34f/codex-rs/core/src/session/turn.rs#L800-L822

Expected behavior

• Before a follow-up sampling request, compaction should consider the estimated model-visible context after newly appended tool outputs.
• Before the first sampling request of a resumed turn, compaction should include pending context reinjection/diffs and the new user input.
• A thread near the context boundary should compact and remain resumable instead of becoming permanently stuck on 400 responses.

Suggested regression test

Use a deliberately small context/auto-compaction limit:

1. Return server token usage just below the compaction threshold.
2. Append a tool result large enough that the estimated next request exceeds the threshold/window.
3. Assert that compaction runs before the next sampling request.
4. Persist/resume the same near-limit thread with a small user input and assert that pre-turn compaction runs before sampling.

Possible fix direction

Use the estimated post-append token count in the mid-turn compaction decision (conservatively taking the maximum of server usage and the local estimate), and estimate pending incoming items in run_pre_sampling_compact before deciding whether to compact.