›_ Codex Issues
Codex Issues / #28316

Codex should not resend large base64 image tool outputs in subsequent context

Open 💬 8 comments Opened Jun 15, 2026 by wang1970
💡 Likely answer: A maintainer (github-actions[bot], contributor) responded on this thread — see the highlighted reply below.

Summary

When a user submits an image to Codex, sending the image in the immediate request is expected. However, after that turn is processed, Codex appears to persist the full base64 image payload in conversation/tool history and resends it in later /v1/responses requests.

This causes unbounded context growth. In one observed case, a later request included multiple base64 image payloads in tool messages, producing an upstream Chat request of about 8.34 MB and about 5.7 million prompt tokens. Several OpenAI-compatible upstream providers then failed with 502/524 or HTTP 200 streams with no valid output.

Expected behavior

  • User-submitted images can be included in the current request.
  • After the image has been processed, Codex should not keep resending the full data:image/*;base64,... payload as ordinary conversation context.
  • Large image payloads should be replaced by a compact reference, digest, placeholder, or omitted from subsequent context unless explicitly required again.
  • Tool outputs containing large binary/base64 data should be aggressively pruned or summarized before being stored in long-lived conversation history.

Actual behavior observed

A later Codex request contained full base64 images inside historical tool messages. The largest tool message was a PNG data URL of about 7.68 million characters. Another JPEG data URL was about 426k characters.

The request shape after being converted to OpenAI Chat format was approximately:

messages = 77
roles:
  system    4
  user      9
  assistant 39
  tool      25
tools = 10
upstream_body ~= 8,343,294 bytes
prompt_tokens ~= 5,722,717
completion_tokens = 0

Largest historical message contents:

tool message[36] ~= 7,681,241 chars
  [{"type":"input_image","image_url":"data:image/png;base64,..."}]

tool message[23] ~= 426,314 chars
  [{"type":"input_image","image_url":"data:image/jpeg;base64,..."}]

This was not caused by the proxy duplicating request data. The actual upstream request body did not include the proxy diagnostic raw request field; the large payload was already present in the conversation messages submitted by Codex.

Failure mode

Observed downstream/upstream behavior included:

I-智枢云桥/gpt-5.5 -> 502 after about 203s
https://vsllm.com/gpt-5.5 -> HTTP 200 stream, first token after about 84.8s, then dropped
prompt_tokens = 5,722,717
completion_tokens = 0

Other OpenAI-compatible providers also failed for similar Codex /v1/responses requests with large historical context, usually as 502/524 or stream dropped/no valid output.

Why this matters

This makes long-running Codex sessions unstable after image/tool interactions:

  • Context can grow by several MB from a single image.
  • The same large image data can be resent repeatedly in later turns.
  • Prompt token counts can explode into millions.
  • Upstream providers may fail, timeout, or return incomplete streams.
  • The failure is confusing because the user is not submitting a new image in the later turn.

Suggested fix

Codex should treat large image/base64 payloads as ephemeral request data rather than durable text/tool history.

Possible mitigations:

  1. Strip or replace data:image/*;base64,... payloads from historical messages after the turn that consumes them.
  2. Store a compact placeholder such as [image omitted: sha256=..., bytes=...] in later context.
  3. Apply max-size limits to tool outputs before persisting them into conversation history.
  4. Avoid serializing image payloads into tool messages that are replayed to the model in subsequent requests.
  5. Add diagnostics/warnings when a request context contains very large base64 payloads.

The key point: submitting an image in the current user turn is fine, but resending the full base64 image in all later turns should be avoided.

View original on GitHub ↗

8 Comments

github-actions[bot] contributor · 1 month ago

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

  • #28150

Powered by Codex Action

wang1970 · 1 month ago

Thanks for the duplicate detection. I reviewed #28150. This issue is related, but I do not think it is an exact duplicate.

#28150 focuses mainly on Codex Desktop thread persistence/recovery problems: large rollout JSONL, screenshots/appshots/local_images poisoning compaction, Electron crash loops, and safe-resume/repair-thread behavior.

This issue focuses on a different downstream effect of the same underlying image/tool-output lifecycle problem: large historical image payloads are being replayed into subsequent model requests. In the observed case, a previous image/tool result was resent inside later /v1/responses context as data:image/*;base64,..., producing an actual upstream request of about 8.34 MB and about 5.7 million prompt tokens. That caused provider-side 502/524 failures or HTTP 200 streams with no valid output.

So the overlap is:

  • image/tool outputs should not be persisted/replayed indefinitely;
  • large image payloads should be stored out-of-line, summarized, or replaced with lightweight references.

The distinction is:

  • #28150: image history poisons Desktop thread persistence / compaction / recovery.
  • This issue: image history leaks into later model request context and causes prompt-token explosion / upstream API failures.

I think keeping this open is useful because it provides concrete evidence that the problem affects not only local session recovery, but also the actual request payload sent to the model in later turns.

wgu9 · 1 month ago

I checked current main for the request-growth path described here. The issue still looks plausible from the current code, and the risky boundary seems to be long-lived history/replay of FunctionCallOutputContentItem::InputImage, not logging.

Relevant code paths I found:

  • codex-rs/core/src/tools/handlers/view_image.rs: ViewImageOutput::log_preview() correctly omits the data URL for logs, but to_response_item() returns FunctionCallOutputContentItem::InputImage { image_url: self.image_url.clone(), ... }. So the model-visible response item still carries the full data URL.
  • codex-rs/core/src/session/mod.rs: record_conversation_items() records the response items into history and persists the same items to the rollout before sending raw response items.
  • codex-rs/utils/output-truncation/src/lib.rs: truncate_function_output_items_with_policy() applies the byte/token budget to text items, but InputImage { image_url, detail } is pushed through unchanged. formatted_truncate_text_content_items_with_policy() also preserves images after text truncation.
  • codex-rs/core/src/context_manager/history.rs: for_prompt() only strips images when the target model does not support image input. For an image-capable target model, historical image content can remain model-visible.
  • codex-rs/core/src/session/turn.rs: replace_last_turn_images("Invalid image") exists, but it is only used as an error recovery path for CodexErr::InvalidImageRequest, not as normal post-turn history pruning.
  • codex-rs/core/src/session/mod.rs: rollout reconstruction can call prepare_response_items() only behind Feature::ResizeAllImages, and the comment explicitly says persisted rollouts are kept unchanged. That does not appear to provide an unconditional sanitizer for historical data:image/*;base64,... payloads.

So I think the minimal fix should be separate from log redaction and separate from current-turn image submission. The safer owner boundary is the durable-history / replay path: after an image has been consumed for the current turn, historical tool-output images should be replaced with a compact placeholder or digest before they can be persisted/replayed into later prompts. Current-turn user images can still be sent normally.

Suggested regression test shape:

  1. Build conversation history containing a FunctionCallOutput content item with InputImage { image_url: "data:image/png;base64,...large..." }.
  2. Submit a later text-only turn to an image-capable model.
  3. Assert the prompt/request items for that later turn do not contain data:image/ and instead contain a bounded placeholder/reference.
  4. Also assert text truncation remains unchanged, because the existing truncation helper already handles text separately.

This is high impact because a single historical image can bypass the existing output truncation budget and be resent repeatedly, which matches the multi-MB / million-token request shape reported above.

100yenadmin · 1 month ago

I published a diagnostics branch that should help quantify this class of image/tool-output replay without changing persisted history yet:

Compare: https://github.com/openai/codex/compare/main...100yenadmin:codex/resume-payload-diagnostics?expand=1
Commit: 0e980ef9588d6dfb3619f31b0a112c193660451a

I could not open the upstream PR directly because GitHub returned GraphQL: 100yenadmin does not have the correct permissions to execute CreatePullRequest, but the branch is pushed to 100yenadmin/codex.

Scope note: this is not the durable-history sanitizer proposed above. It intentionally does not mutate persisted rollout/history or strip model-visible image payloads. It only logs non-content resume payload diagnostics for cold/running thread/resume: turn/item counts, estimated item payload bytes, item class distribution, user-image/image-item counts, MCP/dynamic-tool output counts, command-output counts, and token-usage replay timing.

Why it is relevant here: the active Desktop lockups in #28867 now include small active threads, so image-heavy/full-history replay is a stress signal rather than a proven sole cause. These diagnostics should let maintainers correlate future stalls with whether the resume payload actually contains image/tool-output heavy history, and then decide whether the next guardrail belongs in resume transport, Desktop rendering, token-usage replay, or durable-history image redaction.

100yenadmin · 1 month ago

@tibo-openai

Necmttn · 29 days ago

After first use, large image payloads should be replaced by digest-backed references in replay history.

Persist media type, byte count, hash, producer turn/tool id, and an optional retained blob pointer. Later context assembly can include a compact placeholder plus token/byte diagnostics, instead of resending base64 bodies that already served their turn.

---

_Generated with ax._

aweiu · 19 days ago

any update?

hertzhzzz · 5 days ago

Repro / confirmation: view_image → persistent base64 in session → later turns 400 context_length_exceeded

Environment

  • macOS
  • Codex Desktop 26.707.72221 (CLI 0.144.2)
  • Custom provider through local proxy (CC Switch 3.17.0):
  • client: wire_api = "responses", base http://127.0.0.1:15721/v1
  • model: gpt-5.6-sol, model_reasoning_effort = "xhigh"
  • Model catalog marks gpt-5.6-sol as context_window = 128000 / max_context_window = 128000

What happens

  1. Thread uses built-in view_image (and/or browser screenshots).
  2. Rollout JSONL stores full data:image/...;base64,... under image / tool-related items (input_image counts grow).
  3. Later turns (even when the user is not attaching a new image) send multi‑hundred‑KB to multi‑MB request bodies and fail with:
HTTP 400
Your input exceeds the context window of this model. Please adjust your input and try again.

User-facing (via local proxy) often becomes:
local proxy failed while handling Codex endpoint /responses + the same cause string.

Measured (redacted)
| Artifact | Observation |
|----------|-------------|
| Session JSONL | 26–43 MB after image-heavy work |
| view_image / input_image | present; e.g. input_image=26 with 135 data:image markers in one session |
| Largest single base64 chunk | commonly ~400–660KB (chars), sometimes larger |
| Failing request body size (gateway) | often ~0.95MB, also ~1.5MB → 400 context exceeded |
| Successful text-ish request | ~100KB → 200 |
| Compact | does not reliably recover once history is image-bloated (consistent with other compact+image issues) |

This matches the core claim of this issue: images are expected on the turn they are viewed, but full base64 tool/image payloads appear to remain in history and get re-sent, causing unbounded context growth and permanent thread death.

Cross-links

  • farion1231/cc-switch#4465 — responses-wire/local proxy may further mishandle image parts as text (token inflation); measurements also posted there
  • #30441, #32128, #23257, #29812 — same family (image history / compact duplication / rollout explosion)

Ask / expected behavior

  1. After an image has been consumed for a turn, subsequent requests should not resend full base64 by default (reference path / hash / short placeholder, or hard-drop old images).
  2. Auto-compact / /compact should strip or externalize inline images; otherwise compact itself 400s.
  3. UI error should mention image history size, not only generic context-window / proxy failure.

I can share additional redacted counters (image count vs body_bytes over time) if helpful. No secrets/payloads attached.