Context remaining/token estimate overcounts MCP image data URLs (base64)
Summary
When an MCP tool returns an image, the client converts it to ContentItems containing InputImage { image_url: "data:...;base64,<payload>" } and stores that in history. Token/context estimation uses the serialized JSON length of ResponseItem (including that base64 payload), so a large image makes the estimated tokens jump and context remaining drops sharply, sometimes triggering auto-compaction.
Code pointers
- codex-rs/protocol/src/models.rs: convert_mcp_content_to_items builds data URLs for MCP image content (type="image").
- codex-rs/core/src/context_manager/history.rs: estimate_response_item_model_visible_bytes falls back to serde_json::to_string(item).len() for most items, then approx_tokens_from_byte_count_i64.
- codex-rs/core/src/codex.rs: Session::recompute_token_usage uses estimate_token_count_with_base_instructions to set last_token_usage.total_tokens.
Why this is wrong
The model consumes the image as an image, not as the base64 text inside the data URL. Counting base64 bytes as model-visible text massively overestimates context usage.
Repro
- Configure any MCP server tool that returns an image result with a large base64 payload (type="image").
- Call the tool from Codex and let it return the image.
- Observe context remaining percentage and estimated tokens jump disproportionately right after the tool output.
Expected
Image size should not dominate token estimation. Estimation should treat InputImage as fixed-cost or capped-cost, or strip the base64 payload when estimating.
Environment
- openai/codex @ ebceb71db6ba3cca89d06e64e82a8bcf8c286724
This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗