Input token usage is multiplied by 18? Tokenizer reports 474, Codex reports 8.47K
What version of Codex is running?
v0.42.0
Which model were you using?
gpt-5-codex (reasoning high, summaries auto)
What platform is your computer?
MacOS
What steps can reproduce the bug?
I started a new session after upgrading to 42. I entered my text via voice controls and got an immediate, very short response. All good so far. But then I noticed a super crazy token count in the status line below. This cant be right. Look at the following screenshots and details.
This is a high issue.
I started the session like this
<img width="1412" height="663" alt="Image" src="https://github.com/user-attachments/assets/16ccac25-4c5f-4525-8a42-7b850d537eea" />
I used the following prompt:
Hey, I would like to extend our meeting summaries here with a new category. I mean, we have highlights, we have decisions, these kind of things. What I would like to have now is another one that is called For the Team. And what it would basically contain, it would contain information from the meeting that might be highly relevant for other team members that are outside of that specific department that just held the meeting. So I give an example. An example would be someone in the VFX department basically says, "Oh, I finished all the spell effects from one-handed swords for mages," something like this. And this is a piece of information that definitely needs to be carried over or communicated towards the rest of the entire company because then in the next playtest, the team can test those spells on one-handed swords. It is something new that they will discover during their gameplay. And it's worth noting, pointing out all these new additions. So it's difficult at this point to understand what is relevant for the remaining team, but maybe we can look at this from a perspective of a playtest. So if someone gets into the playtest and he plays the game and he hasn't played it for several weeks, this would be exactly something that he would see. And it might also be a good idea to just focus on maybe visual things or things that the other users would notice. So for example, if someone says in the meeting, "Hey, I finished refactoring this code block," this would not have an impact on the remaining team. This is really just for code maintenance. So we need to focus on customer facing or in this case, other employee facing changes on a visual spectrum, something like this. Do you understand what I'm talking about? Let me think about another example. Another example would be the art department says, "We have now finished the set dressing for cave XYZ," for example. So then this would be something that could immediately be communicated, propagated to the rest of the team because then the team would know, "Oh, okay, if I now play the latest code, this cave would look much more different," something like this. Can you capture the substance of what I just said? Maybe put it in a plan and let's talk about it. This is going to be a very, very useful addition for remaining team members.
Got a respone:
<img width="2452" height="702" alt="Image" src="https://github.com/user-attachments/assets/2681bfb7-db93-4562-89ee-baa28a8f4e79" />
<img width="1587" height="734" alt="Image" src="https://github.com/user-attachments/assets/04af0ac7-97af-49d5-918d-76989a819fe7" />
I then ran the text over the OpenAI Tokenizer and only saw 474 tokens used:
<img width="1691" height="1751" alt="Image" src="https://github.com/user-attachments/assets/f44f0d30-db47-430a-9b44-8f35c49f4a88" />
What is the expected behavior?
Token count to be around 474 for input.
What do you see instead?
8.47K instead.
Additional information
_No response_
This issue has 2 comments on GitHub. Read the full discussion on GitHub ↗