response.completed usage that fails serde parse is treated as a retryable stream error and re-issues the whole sampling request

What version of Codex CLI is running?

main @ fa5d5ae047d1891a2f816c22d9ed926a0728ba47 (codex-rs; codex-api/src/sse/responses.rs:437-451, protocol/src/error.rs:362-397)

What subscription do you have?

n/a (self-hosted)

Which model were you using?

Any (mechanism is model-independent; reachable with OpenAI-compatible providers/proxies that omit a usage subfield)

What platform is your computer?

macOS arm64

What terminal emulator and version are you using (if applicable)?

n/a

Codex doctor report

not available

What issue are you seeing?

When the usage block of a response.completed event fails serde deserialization, codex classifies the failure as a retryable stream error and re-issues the entire sampling request. Output items from the first (successful) generation were already persisted to session history and tool executions were already queued before the parse failure. Each retry issues a fresh billed request; with the default retry limit the turn performs 6 attempts (1 initial + DEFAULT_STREAM_MAX_RETRIES = 5), and up to 12 when WebSocket transport is in use and the one-time WebSocket-to-HTTPS fallback resets the counter, before failing with a retry-limit error. The first call's usage is never recorded (no completed event), so the metered cost under-counts while the billed cost multiplies. Based on the code path below, each retry's items are recorded into history with no dedup and emitted to the UI as completed turn items, so the expected user-visible result is repeated items followed by a retry-limit error.

This is distinct from #37138 (usage-less response.completed is accepted with token_usage=None and silently skips totals). Here the usage object is present but incomplete, so the whole event parse fails and the stream is treated as a transient transport error.

Mechanism, verified on main @ fa5d5ae0:

1. codex-api/src/sse/responses.rs:123-161 : ResponseCompletedUsage fields input_tokens, output_tokens, total_tokens have no #[serde(default)]; ResponseCompletedInputTokensDetails.cached_tokens (line 153) has no #[serde(default)] (only cache_write_tokens at 154-155 does); ResponseCompletedOutputTokensDetails.reasoning_tokens (line 160) has no #[serde(default)]. A missing or null nested field fails the whole parse. (usage: null / absent at the top level is handled by Option + #[serde(default)] on ResponseCompleted.usage at lines 116-117; a partial usage object with a missing subfield is not.)
2. codex-api/src/sse/responses.rs:437-451 : on parse error: return Err(ResponsesEventError::Api(ApiError::Stream(error))) with message failed to parse ResponseCompleted: ....
3. codex-api/src/api_bridge.rs:31 : ApiError::Stream(msg) => CodexErr::Stream(msg).
4. protocol/src/error.rs:362-397 : is_retryable() puts CodexErrorDetails::Stream(..) in the true branch (line 387).
5. core/src/session/turn.rs:1403-1416 : retryable error -> handle_retryable_response_stream_error -> loop continues -> full request re-issued. Default DEFAULT_STREAM_MAX_RETRIES = 5 (model-provider-info/src/lib.rs:26); on exhausting retries the one-time WebSocket-to-HTTPS fallback can reset the counter (core/src/responses_retry.rs:31-46).
6. core/src/stream_events_utils.rs:288-323 and 351-357 : output items (tool calls, messages, reasoning) are persisted immediately at output_item.done via record_completed_response_item, and tool execution is queued, BEFORE response.completed arrives.

Because the provider deterministically sends the same malformed usage, every retry fails identically, so the turn bills multiple full generations and then fails with a non-retryable retry-limit error.

Note on tools: after the stream loop breaks with Err, drain_in_flight at turn.rs:2708 still runs, so in-flight tool futures complete and their outputs are recorded into history before the retry. Already-started tools are therefore not re-queued by the runtime; the main residual risk is history pollution (each retry records a fresh generation's items with no dedup) and a model that re-issues a call after seeing prior state. Side effects of tools that already ran are not rolled back.

What steps can reproduce the bug?

Minimal unit-level repro against the real SSE parser (temporary test in codex-api tests, or equivalent):

// After an assistant output_item.done, a completed event whose usage
// omits input_tokens_details.cached_tokens fails the whole stream as ApiError::Stream.
let item = json!({
    "type": "response.output_item.done",
    "item": {
        "type": "message",
        "role": "assistant",
        "content": [{"type": "output_text", "text": "Hello"}]
    }
});
let completed = json!({
    "type": "response.completed",
    "response": {
        "id": "resp_cx2",
        "usage": {
            "input_tokens": 42,
            "output_tokens": 7,
            "total_tokens": 49,
            "input_tokens_details": {
                // cached_tokens intentionally omitted
                "cache_write_tokens": 0
            }
        }
    }
});
// process_sse / collect_events yields:
//   Ok(OutputItemDone(Message))
//   Err(ApiError::Stream(
//     "failed to parse ResponseCompleted: missing field `cached_tokens`"
//   ))

Also fails for:

• output_tokens_details: {} (missing reasoning_tokens)
• "cached_tokens": null (null not accepted without #[serde(default)])

End-to-end with a live provider/proxy:

1. Use an OpenAI-compatible provider/proxy whose response.completed includes input_tokens_details (or output_tokens_details) but omits one inner field, e.g. cached_tokens or reasoning_tokens.
2. Run a turn. Expected from the code path:

• the turn performs multiple billed LLM calls (6, or up to 12 with WebSocket transport fallback) before failing,
• session token totals do not include the first call's usage (no completed event was parsed),
• history contains multiple near-duplicate assistant messages (each retry records a fresh generation's items with no dedup),
• the UI shows the items repeated, then a retry-limit error.

Contrast (correct path today): a usage-less completed event (response: { "id": "resp1" }) is accepted with token_usage=None (existing test parses_items_and_completed). Only a present-but-incomplete usage object triggers the retry storm.

What is the expected behavior?

A response.completed whose usage block is incomplete should still be treated as a completed turn: the usage should be parsed leniently (missing subfields default to 0, or usage recorded as unknown) rather than failing the whole event. The turn should complete normally with the output already generated, and should not re-issue a billed sampling request for a deterministic schema mismatch.

Additional information

Suggested fix, in preference order:

1. Add #[serde(default)] to cached_tokens, reasoning_tokens, and the top-level usage counters so a missing subfield defaults to 0 instead of failing the parse. (Note the existing asymmetry: cache_write_tokens already has #[serde(default)].)
2. More robustly: on ResponseCompleted parse failure that concerns usage only, still emit Completed { token_usage: None } (or a best-effort partial usage) instead of ApiError::Stream, so a metering-shape difference never re-issues a billed request.
3. Do not make all Stream errors non-retryable as a blanket fix, since that would abort turns after side effects; the lenient-usage fix addresses the root cause.

Related: #37138 covers the usage-absent / totals-skip path. This issue is the complementary case where partial usage fails the event and amplifies spend via retries.

Stream("failed to parse ResponseCompleted: invalid type: null, expected i64")