Official paginated resume drops the cumulative token ledger when TokenCount contains a decimal rate-limit percentage

Open 💬 1 comment Opened Aug 12, 2026 by 92645417d9e5c763259dbebc306e3e

What version of Codex CLI is running?

  • rust-v0.148.0-alpha.1

What subscription do you have?

  • ChatGPT Pro 20x

Which model were you using?

  • gpt-5.6-sol

What terminal emulator and version are you using (if applicable)?

  • Alacritty with Codex running inside tmux

Codex doctor report

Unavailable: all log recording is disabled for performance.

What issue are you seeing?

  • I discovered this after accidentally launching the official release against a long-running paginated thread that had previously survived multiple restarts with a local development build.
  • The last cumulative total before the official launch was 20,501,934,740 tokens. The first newly persisted total after official resume was 310,740, exactly the first request's usage rather than the previous total plus that usage.
  • Resume still loaded the conversation, but emitted no initial thread/tokenUsage/updated notification. Later totals continued from the corrupted near-zero baseline.
  • Returning to the local build prevented the same failure on later resumes but could not repair totals already written after the reset.

What steps can reproduce the bug?

  • Use an isolated CODEX_HOME containing a paginated rollout whose latest TokenCount has non-null info and a rate-limit window containing "used_percent":25.0.
  • Start codex app-server, initialize an experimental API client, and call thread/resume for that thread without supplying replacement history.
  • Observe that resume succeeds but emits no restored token-usage notification.
  • On a copy of the same rollout, change only the latest used_percent from JSON decimal 25.0 to the numerically equivalent JSON integer 25, leaving all token and other rate-limit fields unchanged.
  • Repeat the official resume. The persisted 63,048,929 cumulative total is now emitted immediately.

What is the expected behavior?

  • Every valid persisted TokenCount should deserialize during paginated reverse scanning, regardless of whether it includes rate-limit data.
  • Resume should seed the session with the latest persisted cumulative token usage, emit the restored usage notification, and continue accumulation from that value.

Additional information

  • codex-rs/exec-server-protocol/Cargo.toml enables the serde_json/arbitrary_precision feature for the binary dependency graph.
  • EventMsg is a tagged enum, while RateLimitWindow.used_percent uses the default f64 deserializer. Decimal values inside the buffered tagged payload are represented through serde_json's private number map, which the default f64 path rejects.
  • scan_model_context_from_lineage_blocking treats that RolloutLine as malformed and continues, so the enclosing TokenCount, including its otherwise valid info, is skipped. record_initial_history consequently finds no token ledger to restore.
  • The local development build happened to mask this upstream bug because an unrelated compatibility change for persisted rate-limit snapshots already deserialized used_percent through serde_json::Number and converted it with as_f64. That change was intended to make rate-limit percentages round-trip, but it also kept the enclosing TokenCount.info readable and therefore preserved the ledger.
  • A narrow upstream fix is the same field-level number conversion, with both a rollout round-trip test and a paginated-resume test that verifies cumulative token restoration when rate limits are present.
  • Current upstream main at 2230d64464488d8847197722fdca09d90095c705 retains both the arbitrary_precision feature and the unannotated f64 field.
  • This issue report was written with assistance from Codex CLI.

View original on GitHub ↗

1 Comment

jdcodes1 · 9 days ago

Your additional-information section is exactly right, and it's worth connecting to #38979, where the same root cause was independently reproduced and traced end-to-end: exec-server-protocol enabling serde_json/arbitrary_precision poisons the entire workspace build via cargo feature unification, and any float field reached through serde's buffered content (tagged/untagged/flattened enums) then fails on decimals with invalid type: map, expected f64 — because the buffered number arrives as the $serde_json::private::Number map. In #38979 the victim was MCP annotations.priority: f32 (fractional priority → tool calls broken); here it's RateLimitWindow.used_percent: f64 inside the tagged EventMsg → the whole TokenCount line is treated as malformed during reverse scan → 20.5B-token ledger silently reset. Scratch-crate demonstration of the exact failure, plus the fix discussion: https://github.com/openai/codex/issues/38979#issuecomment-5315898955

Your narrow field-level fix (deserialize via serde_json::Number + as_f64) is correct for this victim and matches what your local build accidentally proved — but it patches one field of a workspace-wide landmine. The durable fix argued in #38979 is removing arbitrary_precision (or isolating exec-server-protocol from the unified graph); its own rpc.rs already special-cases the private-number token for the use case that motivated the feature. Two independent data-corruption bugs from one Cargo feature line is a strong case for prioritizing that removal, with your rollout round-trip test and #38979's stdio repro as the regression pair.