o4-mini: unusually high token count (71 k) for moderate-sized prompt – guidance?

Codex CLI revision

v0.1.2504161551

Model

o4-mini

Platform

Darwin 24.4.0 arm64 arm

What I did

• Called:

``bash
codex "explain this repo" --files src/**/*.py
``

• We were trying to update a README.md file for the project, documenting:
• Which environment manager to use (UV, Conda, Pip/venv).
• How to activate and deactivate the environments.
• What is Typer and how to use it for the project CLI.
• This involved several automated apply_patch calls that attempted to modify README.md, but many failed due to "Invalid Context" errors, leading to repeated retries and possibly additional hidden token accumulation.

Observed behaviour

• Server reports:

``
Rate limit reached … Limit 200000, Used 136502, Requested 71884
``
→ CLI claims the single request will add 71 884 tokens.

• Session was closed abruptly due to rate limit exhaustion.

My questions

• Why is the request perceived as nearly 72k tokens?
• Is the CLI adding hidden system prompts, patches, retry traces or duplicating context?
• Does o4-mini use a different tokenizer?
• Any official tips to estimate tokens more accurately before sending?
• To optimise prompts/files to stay within rate limits?

Additional information

• The rate-limit error kills the session (see linked bug).
• It would be helpful to receive guidance on:
• How Codex CLI handles token accumulation during retries, patches, and parallel commands.
• Best practices for budgeting tokens when working with --files, patches, and active project editing.