Context remaining/token estimate overcounts MCP image data URLs (base64)

Resolved 💬 3 comments Opened Feb 15, 2026 by yiwenlu66 Closed Feb 21, 2026

Summary
When an MCP tool returns an image, the client converts it to ContentItems containing InputImage { image_url: "data:...;base64,<payload>" } and stores that in history. Token/context estimation uses the serialized JSON length of ResponseItem (including that base64 payload), so a large image makes the estimated tokens jump and context remaining drops sharply, sometimes triggering auto-compaction.

Code pointers

  • codex-rs/protocol/src/models.rs: convert_mcp_content_to_items builds data URLs for MCP image content (type="image").
  • codex-rs/core/src/context_manager/history.rs: estimate_response_item_model_visible_bytes falls back to serde_json::to_string(item).len() for most items, then approx_tokens_from_byte_count_i64.
  • codex-rs/core/src/codex.rs: Session::recompute_token_usage uses estimate_token_count_with_base_instructions to set last_token_usage.total_tokens.

Why this is wrong
The model consumes the image as an image, not as the base64 text inside the data URL. Counting base64 bytes as model-visible text massively overestimates context usage.

Repro

  1. Configure any MCP server tool that returns an image result with a large base64 payload (type="image").
  2. Call the tool from Codex and let it return the image.
  3. Observe context remaining percentage and estimated tokens jump disproportionately right after the tool output.

Expected
Image size should not dominate token estimation. Estimation should treat InputImage as fixed-cost or capped-cost, or strip the base64 payload when estimating.

Environment

  • openai/codex @ ebceb71db6ba3cca89d06e64e82a8bcf8c286724

View original on GitHub ↗

This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗