Official paginated resume drops the cumulative token ledger when TokenCount contains a decimal rate-limit percentage
Open 💬 1 comment Opened Aug 12, 2026 by 92645417d9e5c763259dbebc306e3e
What version of Codex CLI is running?
rust-v0.148.0-alpha.1
What subscription do you have?
- ChatGPT Pro 20x
Which model were you using?
gpt-5.6-sol
What terminal emulator and version are you using (if applicable)?
- Alacritty with Codex running inside tmux
Codex doctor report
Unavailable: all log recording is disabled for performance.
What issue are you seeing?
- I discovered this after accidentally launching the official release against a long-running paginated thread that had previously survived multiple restarts with a local development build.
- The last cumulative total before the official launch was
20,501,934,740tokens. The first newly persisted total after official resume was310,740, exactly the first request's usage rather than the previous total plus that usage. - Resume still loaded the conversation, but emitted no initial
thread/tokenUsage/updatednotification. Later totals continued from the corrupted near-zero baseline. - Returning to the local build prevented the same failure on later resumes but could not repair totals already written after the reset.
What steps can reproduce the bug?
- Use an isolated
CODEX_HOMEcontaining a paginated rollout whose latestTokenCounthas non-nullinfoand a rate-limit window containing"used_percent":25.0. - Start
codex app-server, initialize an experimental API client, and callthread/resumefor that thread without supplying replacement history. - Observe that resume succeeds but emits no restored token-usage notification.
- On a copy of the same rollout, change only the latest
used_percentfrom JSON decimal25.0to the numerically equivalent JSON integer25, leaving all token and other rate-limit fields unchanged. - Repeat the official resume. The persisted
63,048,929cumulative total is now emitted immediately.
What is the expected behavior?
- Every valid persisted
TokenCountshould deserialize during paginated reverse scanning, regardless of whether it includes rate-limit data. - Resume should seed the session with the latest persisted cumulative token usage, emit the restored usage notification, and continue accumulation from that value.
Additional information
codex-rs/exec-server-protocol/Cargo.tomlenables theserde_json/arbitrary_precisionfeature for the binary dependency graph.EventMsgis a tagged enum, whileRateLimitWindow.used_percentuses the defaultf64deserializer. Decimal values inside the buffered tagged payload are represented through serde_json's private number map, which the defaultf64path rejects.scan_model_context_from_lineage_blockingtreats thatRolloutLineas malformed and continues, so the enclosingTokenCount, including its otherwise validinfo, is skipped.record_initial_historyconsequently finds no token ledger to restore.- The local development build happened to mask this upstream bug because an unrelated compatibility change for persisted rate-limit snapshots already deserialized
used_percentthroughserde_json::Numberand converted it withas_f64. That change was intended to make rate-limit percentages round-trip, but it also kept the enclosingTokenCount.inforeadable and therefore preserved the ledger. - A narrow upstream fix is the same field-level number conversion, with both a rollout round-trip test and a paginated-resume test that verifies cumulative token restoration when rate limits are present.
- Current upstream
mainat2230d64464488d8847197722fdca09d90095c705retains both thearbitrary_precisionfeature and the unannotatedf64field. - This issue report was written with assistance from Codex CLI.
1 Comment
Your additional-information section is exactly right, and it's worth connecting to #38979, where the same root cause was independently reproduced and traced end-to-end:
exec-server-protocolenablingserde_json/arbitrary_precisionpoisons the entire workspace build via cargo feature unification, and any float field reached through serde's buffered content (tagged/untagged/flattened enums) then fails on decimals withinvalid type: map, expected f64— because the buffered number arrives as the$serde_json::private::Numbermap. In #38979 the victim was MCPannotations.priority: f32(fractional priority → tool calls broken); here it'sRateLimitWindow.used_percent: f64inside the taggedEventMsg→ the wholeTokenCountline is treated as malformed during reverse scan → 20.5B-token ledger silently reset. Scratch-crate demonstration of the exact failure, plus the fix discussion: https://github.com/openai/codex/issues/38979#issuecomment-5315898955Your narrow field-level fix (deserialize via
serde_json::Number+as_f64) is correct for this victim and matches what your local build accidentally proved — but it patches one field of a workspace-wide landmine. The durable fix argued in #38979 is removingarbitrary_precision(or isolatingexec-server-protocolfrom the unified graph); its ownrpc.rsalready special-cases the private-number token for the use case that motivated the feature. Two independent data-corruption bugs from one Cargo feature line is a strong case for prioritizing that removal, with your rollout round-trip test and #38979's stdio repro as the regression pair.