›_ Codex Issues
Codex Issues / #11845

Context remaining/token estimate overcounts MCP image data URLs (base64)

Resolved 💬 3 comments Opened Feb 15, 2026 by yiwenlu66 Closed Feb 21, 2026
💡 Likely answer: A maintainer (miraclebakelaser, contributor) responded on this thread — see the highlighted reply below.

Summary
When an MCP tool returns an image, the client converts it to ContentItems containing InputImage { image_url: "data:...;base64,<payload>" } and stores that in history. Token/context estimation uses the serialized JSON length of ResponseItem (including that base64 payload), so a large image makes the estimated tokens jump and context remaining drops sharply, sometimes triggering auto-compaction.

Code pointers

  • codex-rs/protocol/src/models.rs: convert_mcp_content_to_items builds data URLs for MCP image content (type="image").
  • codex-rs/core/src/context_manager/history.rs: estimate_response_item_model_visible_bytes falls back to serde_json::to_string(item).len() for most items, then approx_tokens_from_byte_count_i64.
  • codex-rs/core/src/codex.rs: Session::recompute_token_usage uses estimate_token_count_with_base_instructions to set last_token_usage.total_tokens.

Why this is wrong
The model consumes the image as an image, not as the base64 text inside the data URL. Counting base64 bytes as model-visible text massively overestimates context usage.

Repro

  1. Configure any MCP server tool that returns an image result with a large base64 payload (type="image").
  2. Call the tool from Codex and let it return the image.
  3. Observe context remaining percentage and estimated tokens jump disproportionately right after the tool output.

Expected
Image size should not dominate token estimation. Estimation should treat InputImage as fixed-cost or capped-cost, or strip the base64 payload when estimating.

Environment

  • openai/codex @ ebceb71db6ba3cca89d06e64e82a8bcf8c286724

View original on GitHub ↗

3 Comments

miraclebakelaser contributor · 6 months ago

I'm guessing this explains https://github.com/openai/codex/issues/11072

lucamorettibuilds · 6 months ago

Opened #12104 with a fix for this. The core issue is in estimate_response_item_model_visible_bytes() — it counts the full serialized JSON length of InputImage items, including the raw base64 payload. A single 100KB screenshot gets estimated at ~25K tokens when the model actually processes it as a fixed-cost vision tile (~85 tokens).

The fix detects data:image/...;base64, URLs inside ContentItem::InputImage and FunctionCallOutputContentItem::InputImage, subtracts the base64 payload size, and adds a fixed estimate (340 bytes ≈ 85 tokens) matching typical vision model tile costs. Added tests for both the message and function call output paths.

miraclebakelaser contributor · 6 months ago

@etraut-openai thank you for fixing this.

One small issue: 340 bytes (~85 tokens) is too low of an estimate.
https://github.com/openai/codex/blob/0679e70bfce35d589363df24c49abd75ff98b10a/codex-rs/core/src/context_manager/history.rs#L422-L426

The gpt-5.2 image cost is determined by this formula[[1]](https://openai.com/api/pricing/)[[2]](https://developers.openai.com/api/docs/guides/images-vision):
image_tokens = ceil(1.2 * min( ceil(resized_width/32) * ceil(resized_height/32), 1536 ))

In other words, gpt-5.2 image inputs are capped at 1844 tokens (1536 * 1.2), which is ~7373 bytes. The gpt-5.3-codex image cost calculation is not published, so I'm assuming it's in the ballpark. Would be glad to be corrected by someone on your team.

Given that these estimates affect total_usage_tokens, which is used in auto-compaction logic, we should use 1844 tokens as our estimate. The downside of using too low of an estimate is that auto-compaction could fail due to insufficient free context in conversations that contain many images.

Weighing the pros and cons of this change, the con is that auto-compaction may trigger marginally earlier. The pro is that you avoid session-poisoning compaction failures.

My proposed change is:

 /// Approximate model-visible byte cost for one image input. 
 /// 
 /// The estimator later converts bytes to tokens using a 4-bytes/token heuristic, 
 /// so 340 bytes is approximately 85 tokens. 
-const IMAGE_BYTES_ESTIMATE: i64 = 340;
+const IMAGE_BYTES_ESTIMATE: i64 = 7373;

Can I make a PR for this?