Every Code client: elevated cache misses

I'm logged in via ChatGPT and can corroborate that the token usage seems to have increased massively: my weekly limit has completely disappeared running gpt-5-codex _MEDIUM_ for some very basic work over a couple of hours.

Yup, that experience is was what lead me to find this. I don't know how OpenAI's usage limits work, but it looks like it's budget based. Given that cached tokens are 1/10 of the price of normal tokens this hugely increases usage. At least 5x and maybe up to 200x in some situations if my math is right.

Thanks for flagging - can you provide some session ids to help us investigate?

@zemaj can you reproduce this on the Codex CLI specifically? Prompt cache can be invalidated by many things, e.g. changes in tool definitions. I'd prefer to isolate this investigation to code that we manage, i.e. openai/codex specifically.

@rickydutton-businessfitness can you provide session ids from /status or codex resume from instances where usage seemed significantly higher than you'd expect?

I felt like the same was going on this past week too. For my first time I hit my PRO limit :( Boring 48 hours here..

Edit: I guess I'm not really seeing this, just had some large repo inputs (repeated over and over sadly) that ate mine up. This is my last set of sessions' final stats. :(

{
"input_tokens": 5248168,
"cached_input_tokens": 5005184,
"percent_cached": 95.37011772488991
}
{
"input_tokens": 3234910,
"cached_input_tokens": 3104384,
"percent_cached": 95.96508094506493
}
{
"input_tokens": 1724316,
"cached_input_tokens": 1625856,
"percent_cached": 94.28990973812225
}
{
"input_tokens": 28639571,
"cached_input_tokens": 28091648,
"percent_cached": 98.08683237608552
}
{
"input_tokens": 9910693,
"cached_input_tokens": 9665792,
"percent_cached": 97.52892153959365
}
{
"input_tokens": 52580,
"cached_input_tokens": 41728,
"percent_cached": 79.3609737542792
}
{
"input_tokens": 8946727,
"cached_input_tokens": 8584576,
"percent_cached": 95.9521398160467
}
{
"input_tokens": 39360050,
"cached_input_tokens": 38575744,
"percent_cached": 98.00735517358336
}
{
"input_tokens": 13327109,
"cached_input_tokens": 12957184,
"percent_cached": 97.22426671831077
}
{
"input_tokens": 3380166,
"cached_input_tokens": 3268480,
"percent_cached": 96.6958427485514
}
{
"input_tokens": 6260352,
"cached_input_tokens": 5866240,
"percent_cached": 93.70463513872703
}
{
"input_tokens": 30300543,
"cached_input_tokens": 29443456,
"percent_cached": 97.17138072410123
}
{
"input_tokens": 1975752,
"cached_input_tokens": 1809792,
"percent_cached": 91.60016034401079
}
{
"input_tokens": 169350,
"cached_input_tokens": 148480,
"percent_cached": 87.67640980218484
}
{
"input_tokens": 3708602,
"cached_input_tokens": 3556480,
"percent_cached": 95.89813088597806
}
{
"input_tokens": 19184139,
"cached_input_tokens": 18741504,
"percent_cached": 97.69270333164287
}
{
"input_tokens": 334078,
"cached_input_tokens": 315136,
"percent_cached": 94.33006663114601
}
{
"input_tokens": 10075678,
"cached_input_tokens": 9666176,
"percent_cached": 95.93573752555412
}
{
"input_tokens": 238029,
"cached_input_tokens": 223488,
"percent_cached": 93.89108049859472
}
{
"input_tokens": 358765,
"cached_input_tokens": 333184,
"percent_cached": 92.86970579627332
}
{
"input_tokens": 19411406,
"cached_input_tokens": 18838016,
"percent_cached": 97.04611814311647
}
{
"input_tokens": 8237601,
"cached_input_tokens": 7972096,
"percent_cached": 96.77691356986094
}
{
"input_tokens": 10914466,
"cached_input_tokens": 10685568,
"percent_cached": 97.90280165790979
}
{
"input_tokens": 120433,
"cached_input_tokens": 96128,
"percent_cached": 79.81865435553378
}
{
"input_tokens": 944886,
"cached_input_tokens": 871424,
"percent_cached": 92.2253054865878
}
{
"input_tokens": 41656587,
"cached_input_tokens": 40803072,
"percent_cached": 97.95106833884398
}
{
"input_tokens": 4769765,
"cached_input_tokens": 4615296,
"percent_cached": 96.76149663557848
}
{
"input_tokens": 9326943,
"cached_input_tokens": 9018880,
"percent_cached": 96.69706355019002
}

Also I went back over my numbers, while watching a certain other LLM CLI (starts with G...) on its ~12th hour of walking in circles to solve a Gradle issue, and it looks like I got hit with a huge unusual usage % dent suddenly.

This only covers local codex CLI, no cloud usage included:

Period (UTC) | Token pattern | Secondary % pattern | Interpretation
Oct 18 03:54 → Oct 20 00:00 | ~0.5–0.8 B tokens/h | 0 → 24 % | Normal proportional rise
Oct 20 00:00 → Oct 21 03:00 | Throughput doubles (~1.4 B tokens/h) | 24 → 29 % | Still proportional
Oct 21 03:04 | ~0.55 B tokens | +46 % jump (29 → 75 %) | Accounting flush; token rise too small     <----------
Oct 21 → Oct 23 02:35 | ~1.0 B tokens total | 75 → 100 % | Continued accumulation at inflated rate

I guess that answers the question I saw on reddit or elsewhere if the Oct 20th date was going to end our fun with the cloud Codex version :(

I am curious, might try to compute it in the near future, if the token rates for cloud vs local are comparable. Sadly I have found the model creating very-verbose tests and doing stupid things a LOT lately, such as running a giant combined set of tests, not finding the failures in the truncation-surviving context, then either dice roll reading tons of log context to find issues OR going down bad paths missing errors/warnings/context (which often turns it into a loop or worse) via both, but the cloud one seems to "give up" a lot more even after just 15 minutes due to some tests taking too long or it doing it's stupid "I will wait 60 seconds" <waits 5 seconds> <runs check again> behaviors.

I would plead to @dylan-hurd-oai or other OAI folks to consider some kind of scheme to where model failures, loops, mistakes, etc do not go against our quotas. With most other API based usage devs are more in-control of the total session/conversation's token payloads, flows, and so forth. But with the nature of codex we do not have much control over the iterations in a single turn. A single request to Codex can end up with millions of tokens we didn't ask for, didn't want, and were from bad feedback loops, mistakes, bugs, timeouts, other issues more unique to Codex that we cannot easily prevent in the blink of an eye. And they snowball. The fact that I can give an exact task to Codex Cloud and it fails/gives up after 15mins and how many millions of tokens into the toilet, but then start an identical session from the same cached overlay with the same repo and same prompt, and it nails it, is one of my bigger personal frustrations lately. I felt like I was at fault with my processes, prompts, etc but I clearly see a lot of 'random' coming into play that feels like "which model did I get this random spin, is it one that will try a few things, find failing tests, and give up or one that will use the tests to debug or at least document some progress or do ANYTHING useful with all those tokens and resources consumed???".

I will say, that _other LLM_ is MUCH MUCH worse in every respect of that, but I think we'll find ourselves questioning our token usages a lot more heavily as of this week. :(

/rant

Ok after going into a deeeeeep dive on this, I found the root cause. Me.

The latest version of codex uses

conversation_id
session_id

in the headers and the fork I was testing on did not use these. Despite the prompt_cache_key being present in the body of the request it was being invalidated and changed to a different key in the response. This meant that basically every request had an almost fresh cache. Adding in these headers and matching the current request format resolved the problem.

I did see some strange behaviour in the live codex version (it looks like the cache gets partly invalidated on the first assistant response) but that's limited in impact and the same across the API and all other locations. This was what lead me to believe that this issue was present in 0.47 but after digging deeper I realised it was not the same issue.

The caching issue with the missing headers likely affects people using earlier version of codex (0.42 and below), but not the latest release.

So closing this out. Sorry!

I'm on various older franken-forks here too...Hmm guess I will have to dig deeper there too. Thanks for sharing!

@techjoec appreciate the detailed feedback re: Codex Cloud. Will share this with the team!

Hi @zemaj - reopening this thread because we're still seeing increased cache misses from the code_cli_rs relative to codex cli. Going to loop @tibo-openai in here as well.

Sure thing - if you need anything from my end let me know. Last time I did a comparison a few days ago latest release was comparable with codex, but hard to compare. Requests are all stable (i.e. no tool/history changes during session).

The codex team has done a deep investigation of all issues related to rate limits and token usage accounting, and we've implemented a number of adjustments and bug fixes. See here for a summary. This includes some changes related to cache miss accounting.

@etraut-openai It's been working great lately! Thank you so much!