Change `TokenCount` to not re-emit `last_token_usage` on rate-limit-only updates
Summary
EventMsg::TokenCount currently serves two different purposes:
- reporting a real token-usage update
- rebroadcasting the latest rate-limit snapshot
When only rate limits change, Codex emits a new TokenCount event with unchanged total_token_usage and the previous nonzero last_token_usage. That makes the JSONL rollout misleading for consumers that interpret last_token_usage as the incremental usage for that event.
I noticed this while comparing my own local usage counter against slopmeter and @ccusage/codex. Both over-reported on my real session logs for this reason.
Why this is a bug
last_token_usage looks like a per-event delta field. Re-emitting it unchanged on a pure rate-limit update makes the same usage chunk appear multiple times in the session log.
This is not just a tooling misunderstanding. There is a nearby comment that says this path should avoid duplicate TokenCount events, but the current implementation still emits them.
Root cause
update_token_usage_info()appends new usage and emitsTokenCount.
update_token_info()stores that usage inTokenUsageInfo, preserving the most recent nonzerolast_token_usage.
update_rate_limits()updates only rate limits, but also emitsTokenCount.
send_token_count_event()republishes the storedtoken_infotogether with the latest rate limits.
So a rate-limit-only update can emit a fresh TokenCount event that repeats the previous nonzero last_token_usage even though total_token_usage did not advance.
There is also a direct comment / implementation mismatch here:
The comment says rate-limit handling should defer sending "to avoid duplicate TokenCount events", but the implementation immediately calls sess.update_rate_limits(&turn_context, snapshot).await, and that path emits TokenCount right away.
Impact
This affects tools that parse session JSONL rollouts for usage accounting. A reasonable implementation is to treat each token_count event as a new usage snapshot and last_token_usage as the incremental usage for that event. With the current behavior, that interpretation overcounts.
This is exactly what happened in my case:
- my own local counter, which keyed off cumulative advancement, produced lower totals
slopmeterand@ccusage/codex, which trustedlast_token_usagemore directly, produced higher totals
Fix direction
Rate-limit-only updates should not emit a TokenCount event that repeats a stale nonzero last_token_usage.
Any of these would resolve the ambiguity:
- do not emit
TokenCountfromupdate_rate_limits()when usage did not change - emit a separate rate-limit event instead
- or ensure a rate-limit-only rebroadcast does not carry a nonzero stale
last_token_usage
This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗