bytes/4 estimate overwrites measured last_token_usage after compaction, shifting auto-compact decisions for non-ASCII sessions

What version of Codex CLI is running?

main (recompute_token_usage present at codex-rs/core/src/session/mod.rs:3807)

What subscription do you have?

n/a (self-hosted)

Which model were you using?

Any (mechanism is model-independent)

What platform is your computer?

macOS arm64 (uname -mprs: arm64 Darwin)

What terminal emulator and version are you using (if applicable)?

n/a

Codex doctor report

not available

What issue are you seeing?

After every compaction, rollback, or new context window, recompute_token_usage unconditionally replaces the server-measured last_token_usage with a bytes/4 heuristic estimate and zeroes the entire token breakdown. The auto-compact decision (token_limit_reached, evaluated pre-turn) reads this field via get_total_token_usage, so for sessions with significant non-ASCII content (CJK, emoji) the gate runs on an estimate that under-counts tokens.

Mechanism, verified on main:

1. codex-rs/core/src/session/mod.rs:3807-3844 : recompute_token_usage:

``rust
let Some(estimated_total_tokens) = history.estimate_token_count_with_base_instructions(&base_instructions) else { return; };
...
info.last_token_usage = TokenUsage {
input_tokens: 0, cached_input_tokens: 0, cache_write_input_tokens: 0,
output_tokens: 0, reasoning_output_tokens: 0,
total_tokens: estimated_total_tokens.max(0), // estimate; all breakdown fields zeroed
};
``

1. codex-rs/utils/string/src/truncate.rs:71-74 : approx_token_count = byte length / 4 (4 bytes per token).
2. codex-rs/core/src/context_manager/history.rs:298-316 : get_total_token_usage reads last_token_usage.total_tokens and adds per-item local estimates; feeds token_limit_reached in core/src/session/context_window.rs (active_context_tokens = sess.get_total_token_usage(), line 29), evaluated pre-turn at turn.rs:990-993.
3. Call sites: compact.rs:385, compact_remote.rs:295, compact_remote_v2.rs:320, handlers.rs:543 (rollback), start_new_context_window (3660).

Scope detail: in the BodyAfterPrefix scope the estimate error is largely masked, because the prefill baseline is re-derived from the same estimate (history replacement clears the prefill at state/session.rs:122, and set_auto_compact_window_estimated_prefill_for_scope sets an estimated baseline that the same estimate then subtracts from). The Total scope has no baseline: active_context_tokens (estimate-based) is compared directly against the limit, so the error is fully exposed there.

For CJK-heavy content bytes/4 under-counts by roughly 25-50%: CJK characters are typically 1-2 tokens per 3-byte character (tiktoken o200k counts a CJK char as ~1 token), so a session that is mostly CJK comments has a materially lower estimated total than the true token count. Auto-compact then triggers later than it should, increasing context-overflow risk; and the breakdown fields being zeroed makes the TUI token display read 0 for input/output/cache until the next server usage event.

The estimate self-corrects on the next server usage event (record_token_usage_info, mod.rs:3771), so the affected window is the pre-turn check immediately after compaction/rollback.

What steps can reproduce the bug?

Static demonstration (no live run required):

1. Start a session with a CJK-heavy history (e.g. a repo whose code comments are predominantly Chinese).
2. Trigger a compaction (or /compact, or rollback).
3. Observe recompute_token_usage being invoked (call sites above) and last_token_usage replaced with total_tokens = bytes/4, with input/output/cache all 0.
4. Compare the estimated total against a real tokenizer (e.g. tiktoken o200k) on the same history: for CJK content, bytes/4 under-counts.

What is the expected behavior?

The measured server usage for the post-compaction state drives the auto-compact gate; the breakdown fields should not be zeroed when a previous measured sample exists.

Additional information

Suggested fix options, in preference order:

1. Keep the measured last_token_usage when a server-measured sample exists for the current window, and only fall back to the bytes/4 estimate when no measured sample is available yet (the estimate is intended as a fallback, not an overwrite).
2. If an estimate must be written, keep the breakdown fields (input/output/cached) from the last measured sample rather than zeroing them.
3. Consider a better estimator for non-ASCII content (e.g. counting CJK characters as ~1 token per character) instead of a flat bytes/4.