Send Azure prompt_cache_retention (default 24h): large-context gpt-5.4 sessions thrash the in-memory prompt cache

Summary

On Azure OpenAI, long Codex sessions whose context grows past ~150K tokens suffer repeated total prompt-cache misses (cached_input_tokens drops to 0), re-billing the entire context at full input price call after call. Root cause: gpt-5.4 (and older) default to in_memory prompt-cache retention, which Azure evicts under memory pressure once the cached KV entry is large. Azure exposes a per-request fix — prompt_cache_retention: "24h" (extended retention, KV offload to GPU-local storage) — but Codex never sends it, and there is no extra_body/passthrough to add it.

Request: let Codex send prompt_cache_retention (config knob; default "24h" for models that support extended retention).

Note: prompt_cache_key is already handled (codex-rs/core/src/client.rs::prompt_cache_key, scoped per thread) — this issue is only about retention.

Evidence (codex 0.135, Azure gpt-5.4, wire_api = responses)

A single agent run cost ~$15.9 / ~20M total tokens. From the main rollout's token_count events (info.last_token_usage), 112 model calls across 4 turns:

• Calls 2–86 cache cleanly — cached_input_tokens tracks input_tokens up to ~160K.
• From ~call 87 (context 159K→216K), 20 calls return cached_input_tokens = 0, re-sending the full ~164K-avg context as fresh — 3.28M fresh tokens = 71% of all fresh input in the run.
• Ruled out with data: TTL/inactivity (inter-call gaps 5–27s, Azure in-memory TTL is 5–10 min); client context-editing (input_tokens never shrinks during the collapse — context intact, only the cache is gone); prefix churn (caching hits fine when it hits — 191K cached at call 108 — so the prefix is stable/cacheable; it's being evicted).

This matches Azure's prompt-caching docs: gpt-5.4 default retention is in_memory ("cleared within 5–10 min of inactivity … and evicted under memory pressure when full"); extended 24h retention "offloads the key/value tensors to GPU-local storage when memory is full, which significantly increases the storage capacity available for caching." Available for gpt-5.4 but not the default and must be set per request. We confirmed against a live Azure deployment that prompt_cache_retention is accepted and server-validated ("24h"/"in_memory" OK; "bogus" → 400 "Supported values are: 'in_memory' and '24h'").

Likely the same family as #20301 (low cache hit on GPT-5.x), #5556 (fleet-wide cached:input collapsing ~20:1 → ~1:1), #4764 (cache-miss cost overshoot).

Suggested implementation

ResponsesApiRequest (codex-rs/codex-api/src/common.rs) already carries service_tier and prompt_cache_key as optional fields; adding prompt_cache_retention: Option<String> alongside them, populating it where the request is built in core/src/client.rs, and exposing a config knob (default "24h" for supported models) would cover it. Happy to open a PR if the approach sounds right.

Impact

For long, tool-heavy sessions this is the dominant cost: ~71% of fresh input in the run above was the same context re-billed ~20×. At ~50K context those misses would have cost ~0.85M instead of ~3.0M fresh — most of the bill.