Prompt-cache misses — mostly mid-loop or after silent context pruning — were 94% of a $210 session's uncached cost; counters can't show cached vs uncached

What version of Codex CLI is running?

codex-cli 0.144.4

What subscription do you have?

None — API key auth against Azure OpenAI (metered credits, not a ChatGPT plan).

Which model were you using?

gpt-5.6-sol

What platform is your computer?

Darwin 25.5.0 arm64 arm

What issue are you seeing?

A single 6.5-hour CLI session billed 27.33M input tokens and cost roughly $210 — out of $291 billed against that model on the day. (Day total is exact from Azure Cost Management; the per-session figure is apportioned by token volume, since billing is per-deployment.)

Nothing malfunctioned in the client, and prompt caching mostly worked — 78% of input tokens were served from cache (median 99% cached on cache-hit requests). The problem: the requests that missed the cache carried 94% of all full-price input, and the CLI surfaces nothing that would let a user see it happening.

Mechanics

14 user instructions produced 184 billed API requests (199 attempts recorded; 15 failed/retried attempts were not billed — see note at the end). Context grew from 16K to 254,605 tokens. Every request resends the accumulated context, so cumulative billed input reached 27,333,680 while no single request exceeded ~25% of the context window.

Billed breakdown, from the session's own token_count records — sums reconcile exactly with the session's cumulative usage and with Azure's billed meters:

| | requests | full-price input tokens |
|---|---|---|
| cache HIT | 146 | 383,341 |
| cache MISS | 38 | 5,703,107 |

Where the misses came from (each miss classified by surrounding events):

| Cause | Misses | Full-price tokens | % of session uncached input |
|---|---|---|---|
| Mid-loop: next call seconds to ~2 min later, append-only context growth — should have hit | 24 | 3,486,229 | 57% |
| Context shrank between calls (history pruning/compaction rewrote the prefix) | 6 | 829,086 | 14% |
| New user turn with context slightly smaller than the previous call (prefix mutated) | 4 | 713,814 | 12% |
| First request after a failed/retried request | 3 | 657,816 | 11% |
| First call of session | 1 | 16,162 | 0% |

A second session the same day (9.55M input, 93 billed requests) shows the same distribution: 10 misses — 4 mid-loop, 3 after context shrank, 1 post-failure, 1 turn-start, 1 first-call.

Idle-related cache expiry — the obvious suspect — explains at most 2 of 38 misses (~4% of uncached cost).

Two distinct problems, one visibility gap

1. The mid-loop misses (57% of cost here) occur seconds apart with append-only prefixes — conditions where a miss shouldn't be possible. This looks like the intermittent dropout reported in #30425; this issue adds the cost: ~$90 of a ~$210 session. Possibly related to the prefix-stability discussion in #35300.
2. The context-shrank misses (14%) are client-side: when history is pruned/compacted, the next request's prefix no longer matches and the entire context re-bills at full price. Maybe the right trade-off — but currently invisible.

Either way the user-facing gap is the same: /status and the status line show context % and cumulative token totals, but a single "tokens used" number cannot distinguish a 78%-cached session from one bleeding full-price re-sends. The number that predicts cost is the cached/uncached split, and it isn't shown anywhere.

Suggested fixes (roughly in order of value/effort)

• Split session token counters into cached vs uncached (status line + /status)
• Surface uncached input for the last request, so a large cache miss is visible when it happens (e.g. a marker when cached_input_tokens == 0 on a large request)
• Indicate when history pruning/compaction occurs, since it invalidates the cache by construction
• Cost in currency would help too (#20087), but matters less than the split

Related

• #30425 — intermittent zero-cache-hit requests despite stable prefixes; the mid-loop misses above look like the same phenomenon, with cost attached
• #35300 — prompt_cache_breakpoint / stable-prefix reuse for GPT-5.6
• #34735 — retry loops without usage guard (different trigger, same visibility gap)
• #20087 — /cost

Note on billing accuracy: the session also auto-retried ~2.3M tokens' worth of requests after timeouts/errors. Those failed attempts were not billed — billed-event sums reconcile with Azure's metered totals to within 191K tokens across the whole day, vs ~2.7M retried. This report is about visibility, not incorrect billing.

Note on revisions: substantially revised after filing — the original blamed idle-time cache expiry; per-miss classification showed idle explains ≤4%. The table above is the verified breakdown; edit history has the original.