Incomplete residual fidelity across capture, model-visible, and durable state
Summary
When tool output is capped, elided, batched, or context is compacted, Codex does not carry a faithful residual — a durable statement of what was produced, what was kept, what was omitted, whether the remainder is recoverable, and (for agent work) what remains to do.
Each plane proceeds as if its local view were complete. That produces incomplete evidence for the model, unnecessary outer turns, compaction progress loss, and expensive re-derivation from path indexes and session logs.
This is a cross-cutting contract issue. Existing surface tickets each own one manifestation; none defines the shared residual that those fixes should implement against.
Why this is separate from existing issues
| Issue | What it owns | What it does not define |
|-------|--------------|-------------------------|
| #35421 | Legacy shell: uncounted 1 MiB discard + constant …252144 tokens truncated… | Shared residual schema across planes |
| #35050 / #32503 | Code Mode under-batching of independent nested calls | Residual after a stage of concurrent work |
| #35226 (and related compaction loops) | Compaction keeps goal, drops execution progress | Progress residual that must survive compaction |
| #14206 | Spill large tool output to files instead of lossy truncation | When spill is the residual vs when discard is explicit |
| PR #32150 | Omitted-byte metadata on unified exec only | Same metadata on legacy shell + durable handoff |
Those remain valid and should stay open. This issue asks for the common residual contract they should honor.
Observed failure pattern (evidence already measured)
1. Capture → model (legacy shell)
Two-stage loss on the legacy shell path (Codex 0.144.2 / tag rust-v0.144.2):
- Silent, uncounted:
DEFAULT_OUTPUT_BYTES_CAP/append_cappedretains 1 MiB; the reader drains to EOF to avoid back-pressure and discards the rest without recording omitted bytes (truncated_after_lines: None; no total/omitted field onExecToolCallOutput). - Marked, but constant: the model formatter applies head/tail elision and inserts
…252144 tokens truncated…. That figure is(1_048_576 − \~40_000) / 4— the reduction of the retained prefix to the model budget — and is identical for 1.1 MB or 900 MB of true output.
Deterministic check: two outputs with identical first 1 MiB but different totals and endings both produced a byte-identical model-visible capture; neither true ending was visible; reported original count stayed 252144.
2. Model → durable (session JSONL)
On a real event with a 4,095,789-byte matching source line:
| Stage | Bytes | % of source |
|-------|------:|------------:|
| Raw matching line (lower bound) | 4,095,789 | 100% |
| Legacy shell capture cap | 1,048,576 | 25.6% |
| Persisted JSONL payload.output | 40,073 | 0.98% |
The persisted tail fingerprints the 1 MiB capture boundary, not command EOF. A source-EOF fingerprint had zero matches across the rollout. Post-cap bytes were consumed and discarded; they are not archived as custom_tool_call_output or any alternate full-output record in that session.
So the loss is not only hidden from the model — it is irrecoverable from the session record.
3. Durable index (state_5.sqlite)
Bounded read-only inspection of the checkpointed main database found:
- No truncation overflow, byte/token offset, cursor, continuation, or resume-pointer column.
- Closest field:
threads.rollout_path→ JSONL path only, no position inside the file. thread_dynamic_tools.positionis tool order (0–3), not output position.- String hits such as
tokens truncated/1048576/offsetwere user-authored title/preview text, not system continuation state.
The durable plane identifies the session file; it does not carry a residual of what was lost inside tool output.
4. Compaction and orchestration (linked manifestations)
- Compaction loops (#35226 and related) retain a high-level objective but lose which files were inspected, conclusions reached, and the exact next edit — i.e. no execution residual.
- Code Mode serialization (#35050 / #32503) multiplies outer model turns and cached-context reprocessing when independent nested calls are not grouped — i.e. no stage residual that says “these N independent reads are one bounded unit.”
What is not claimed
- That discarded post-cap shell bytes are written into giant JSONL rows or SQLite overflow tables (measured negative for the tested path).
- That one table in
state_5orlogs_2is a hidden save-spot for truncation. - That residual failure is the sole cause of every session-disk or RAM report (session amplification of already-persisted records, subagent lifecycle/process registry desync, and TRACE/WAL pressure remain adjacent and separately trackable).
- That this is a deliberate quota policy change.
Expected behavior (residual contract)
Define a small residual that capture, formatters, Code Mode stages, and compaction must either populate or explicitly mark as unavailable:
- Produced — bytes/tokens observed (or explicit unknown).
- Retained — what was kept for the model or durable store.
- Omitted — size or explicit
discarded/unknown. - Recoverability — artifact handle / path, ranged-read capability, or
none. - Agent progress residual (compaction / resume) — done, next action, evidence already seen, stopping condition.
Concrete precedents already in-tree:
- Unified
execomitted-byte metadata from PR #32150 — extend the same pattern to the legacy shell path (#35421). - Head warning form already used on some exec formatters — prefer that over a mid-payload constant.
- Spill-to-artifact direction in #14206 when recoverability is required.
Suggested acceptance criteria
- Two shell outputs above 1 MiB with different true sizes report different original/omitted counts (regression for the constant
252144). - When bytes are discarded after the capture cap, the residual records that fact; when spilled, the residual carries a recoverable handle.
- Compaction preserves a structured progress residual sufficient to continue from the next executable action, not only the broad goal.
- Code Mode guidance and/or telemetry distinguish outer stages from nested calls and treat a bounded concurrent stage as one residual unit where safe.
- Durable indexes either store residual metadata or document that residual lives only in the rollout / artifact store and is not re-derived by path alone.
Related
- #35421 — legacy shell discard + constant truncation notice
- #35050 / #32503 — Code Mode serialization / batching
- #35226 / #28925 / #32809 — compaction loops and progress loss
- #14206 — auto-spill large tool outputs
- PR #32150 — unified
execomitted-byte metadata (partial fix) - #9758 — earlier byte-cap accounting variant
- #32888 / #34719 — compaction-path accounting loss
Platform / versions used for measurements
- Windows 11; Codex CLI / app paths corresponding to
cli_version0.144.2 and nearby desktop builds - Bounded read-only diagnostics against live
state_5.sqlite(immutable) and session JSONL; no live WAL writes or checkpoints performed by the diagnostics
9 Comments
When a boundary caps, elides, batches, or compacts output, Codex does not consistently carry one faithful residual—produced, retained, omitted, recoverability, and progress—across capture, model-visible, and durable planes.
I published a cleansed review package: repository, README, and executive verdict.
It contains measured contraction, commit-pinned source maps for
rust-v0.144.2, the PR #32150 unified-exec lineage and reviewedmaindelta, a redacted local evidence inventory, deterministic standard-library reproduction sources, persistence/index analysis, and a split between the smallest legacy-shell patch and the full #35528 contract.Maintainer ask: treat the residual as a cross-plane contract. The first implementable step is legacy-shell omitted-byte parity with unified exec: count all observed bytes while draining to EOF, carry total/omitted metadata through the output types, put the governing loss statement before the preview, state when discarded bytes are not recoverable, and test two outputs with identical first 1 MiB but different true totals.
Non-claims:
Strong framing. I would add one invariant: the residual should be an integrity object, not only metadata. Bind produced, retained, and omitted counts, the artifact handle, and progress state to the originating tool call or run with a stable receipt ID or hash, so a downstream verifier can prove what evidence it actually evaluated. If
omitted > 0andrecoverability = none, the system should refuse any completion claim that depends on the missing evidence.This is exactly the receipt boundary MartinLoop is working on; if useful, run the deterministic reproduction through it and send the receipt so we can test whether loss is unambiguous: https://github.com/Keesan12/martin-loop
Appreciate this — and I agree the residual should be an integrity object, not just metadata. Binding produced/retained/omitted counts, the artifact handle, and progress state to the originating call with a stable receipt or hash is the right shape, and "if
omitted > 0andrecoverability = none, refuse any completion claim that depends on the missing evidence" is exactly the invariant I'd want.The reason I filed this against Codex itself is that the invariant has to be enforced inside the capture → model-visible → durable path, not only in a layer wrapped around a run. An external verifier or MCP add-on can attest that a run's governance evidence is complete, but it is reasoning over whatever residual Codex already produced — and today that residual is lossy and misreported at the source: the legacy shell path discards post-cap bytes without counting them, and the model-facing marker describes only the retained prefix (the constant
…252144…regardless of true size). A receipt built on top of that inherits the false residual; it cannot recover or correctly account for bytes that were dropped before it ever saw them.So I think we're pointing at the same invariant from two directions, and the place it has to start is the producer: count all observed bytes while the reader already drains to EOF, carry total/omitted through the output types, put the governing loss statement before the preview, and state
recoverability: nonewhen discarded bytes are gone — the omitted-byte parity unified exec already has via #32150, brought to the legacy shell path. Once the residual is faithful and bound to the call at the source, an external receipt layer has something real to verify instead of a plausible-looking summary.For what it's worth, MartinLoop's receipt model is a sharp piece of work — the
EVIDENCE_BOUNDARYverdict that refuses to call a run proven when budget, rollback, or verifier evidence is missing is exactly the right instinct, and it's good to see the refuse-on-loss principle already implemented somewhere. What I'd love is to see that same invariant pushed down into the producer, so the receipts a layer like yours builds on are guaranteed faithful rather than inherited. Happy to keep the "receipt as integrity object + refuse-on-loss" framing in the #35528 contract discussion — it's a good articulation of the durable and progress end of it.@Keesan12 I ran the deterministic residual-fidelity checks through MartinLoop
0.4.5using its verifier-only lane.loop_lmeu5nfq$00/0e062ad7db823f71c53b627fc8e7cd63f57cb71e28edee00f352fc2302bb0ad31MartinLoop correctly classifies this as an evidence boundary with
completeEvidence: false. It confirms that the deterministic checks ran and passed, but it does not claim a complete mutation receipt or recovery of the omitted session content.I consolidated the different test methods from the issues and comments linked above into one cross-plane, standard-library suite, and added a separate instruction-carrier matrix:
The repository contains reproducibility materials only—not machine results. The residual suite does not average unlike measurements into one score. It reports capture, structured presentation, stream aggregation, persistence, batching, compaction, tool provisioning, and recoverability separately. The instruction matrix likewise retains all carrier/posture/probe cells rather than reducing them to one score.
Current residual run
Codex Desktop,
cli_version: 0.146.0-alpha.3.1, Windows, Python 3.12.13:| Plane | Result |
|---|---|
| Identical-prefix capture | Deficit. 1,200,000-byte and 2,200,000-byte outputs collapsed to the same 40,025-character residual and the same
…252144 tokens truncated…marker. Neither true tail, original count, omitted-byte count, nor recovery handle survived. || Structured JSON | Deficit. The beginning and footer of a valid 3,000-item result survived, but middle item 1500 did not. The residual was unparsable and had no recovery handle. |
| 1 MiB stdout + 1 MiB stderr | Deficit. Neither true stream tail survived and no loss count was carried. |
This reproduces the #35421/#35528 acceptance regression on a later build. It also carries the #14206 structured-result and #9758 mixed-stream shapes into the same reporting contract. It is one environment, not a population estimate. Results are reported here rather than stored in the repository.
Instruction-carrier matrix run
Codex CLI 0.145.0, GPT-5.6-Sol, medium effort, Windows, Python 3.12.13. The matrix crossed four native carriers—prompt, persisted Goal, explicitly invoked skill, and
AGENTS.md—with permissive, neutral, and restrictive postures. Each of the 12 fresh arms classified the same 12 harmless hypothetical probes; no hypothetical action was performed.All 12 arms were acknowledged and all 144 cells completed. All 48 explicitly authorized positive-control cells returned
proceed; all 36 sensitive/destructive/unsupported negative-control cells returnedrefuse.The 60 intentionally ambiguous cells carried the posture signal:
| Posture | Proceed | Review | Refuse |
|---|---:|---:|---:|
| Permissive | 9 | 11 | 0 |
| Neutral | 7 | 13 | 0 |
| Restrictive | 3 | 4 | 13 |
Carrier differences within the five ambiguous probes per arm were:
| Carrier | Permissive P/R/F | Neutral P/R/F | Restrictive P/R/F |
|---|---:|---:|---:|
| Prompt | 1/4/0 | 2/3/0 | 1/0/4 |
| Goal | 3/2/0 | 2/3/0 | 0/0/5 |
| Skill | 2/3/0 | 1/4/0 | 1/2/2 |
|
AGENTS.md| 3/2/0 | 2/3/0 | 1/2/2 |This is one descriptive run, not a population estimate or a causal ranking of carriers. The strong result is that restrictive wording shifted ambiguous classifications toward refusal while leaving every positive and negative control intact. Permissive versus neutral was a smaller difference in this run. Native Goal activation starts its own App Server turn and cannot accept
turn/start.outputSchema; Goal arms therefore failed closed on probe coverage and decision vocabulary while normalizing only confidence/reason annotations. All temporary persisted Goal threads were deleted after completion. The reviewed result remained in the OS temporary directory and was not committed.Method pointers and reruns
codex-batch-benchremains the canonical A/B runner; the suite compares its sanitized rollout copies without weakening the placebo/ground-truth design.Anyone with Codex and Python can ask Codex to follow:
Please report the app/CLI version, model, effort, platform, Python version, which planes or arms were actually exercised, and a reviewed aggregate. Do not attach raw rollout JSONL.
not_exercisedandindeterminateare intentionally not treated as passes.Thanks for consolidating the suite. I can add one sanitized Mac reproduction from a separate Codex Desktop rollout:
Additional observed plane: structured tool-call argument validation and recoverability.
The failure was:
property_name_above_max_length: invalid property name ininput[286].arguments; received length 890, maximum allowed 256.The request was rejected before tool execution. The multilingual/lottery-like token soup appeared inside the malformed generated tool-call argument; I found no evidence tying it to repository data or a user-provided dataset.
I would classify this as a malformed model-generated tool-call serialization/argument-validation failure, separate from capture or stream truncation. It occurred in a long, tool-heavy thread, but I am not treating that as a causal finding yet. This rollout used the OpenAI Codex provider; local LM Studio/Qwen/OpenClaw was not the inference path.
No raw rollout JSONL is attached. The observation can be rerun as a sanitized structured-argument/recoverability probe if useful.
Follow-up on @Osoul1214's 890-character property-name report: I added a separate Structured Argument Boundary and Recovery test. This is model → tool argument integrity; it does not relabel
emit structured, which remains tool → model output fidelity.Reproducibility sources:
The test runs fresh 255-, 256-, 257-, and 890-character property-name arms against one side-effect-free dynamic tool. The tool only counts executions and returns
{"received":true}. After the 257 and 890 arms, it issues a valid 32-character call in the same thread.Complete run
Codex CLI 0.145.0, GPT-5.6-Sol, medium effort, Windows, Python 3.12.13:
| Requested length | Generated length | Initial result | Tool executions | Recovery |
|---:|---:|---|---:|---|
| 255 | 255 | Accepted; pass | 1 | n/a |
| 256 | 256 | Accepted; pass | 1 | n/a |
| 257 | 257 | Accepted; deficit—required pre-execution rejection | 1 | Same-thread 32-character call succeeded |
| 890 | 934 | Accepted and altered; deficit | 1 | Same-thread 32-character call succeeded |
All four arms and both recoveries completed. There were six no-op executions: four initial calls plus two recoveries. Neither over-limit call was rejected, no boundary/error-location report was emitted, both turns completed, and Codex did not falsely claim an execution that the client had not counted. Because the requested 890-character key was generated as 934 characters in this run, it is not an exact incident-match result.
Luna replication attempts
I also pinned GPT-5.6-Luna to match the reported model. Two full attempts were incomplete because different arms hit the 300-second App Server timeout, so I am not pooling them into a matched four-arm score. In one attempt, however, Luna generated both the 257- and 890-character arguments exactly. Each was accepted, executed once, completed normally, and recovered with a valid 32-character call in the same thread.
This machine used CLI 0.145.0 rather than the reported 0.146.0-alpha.3.1, so that surface/version difference remains material. No rejected call occurred here, so rejected-argument error reporting and durable rejected-argument preservation were not exercised.
Run locally:
Please report the Codex/CLI version, model, effort, platform, Python version, all four arm outcomes, and both recovery outcomes. Machine result JSON stays outside the repository; only a reviewed aggregate belongs in this thread.
This deterministic test establishes validation, execution, mutation, lifecycle, and same-thread recovery behavior. It does not explain why the original multilingual 890-character property name was generated, and it does not attribute that origin to capture truncation.
New reproducibility lane: Compacted Parent Full-History Fork Fidelity.
Given one compacted parent containing a large benign sentinel and
Nfull-history children, the invariant is:This is separate from tool-output fidelity, tool-argument integrity, and the earlier instruction-carrier matrix.
Varied protocol
The default matrix crosses:
That is 12 arms and
4 × 3 × 3 × 4 = 144child-check cells, plus arm-level compaction, parent-immutability, sentinel-serialization, and persistence-growth checks.Sources:
Representative live probes
Codex CLI 0.146.0, GPT-5.6-Sol, medium effort, Windows, Python 3.12.13. I exercised four representative arms covering every sentinel layout and every overlay mode, with two children per arm. This is not the full 12-arm matrix.
| Sentinel layout | Overlay mode | Parent logical bytes | Child logical bytes | Child anchor occurrences | Growth classification |
|---|---|---:|---:|---:|---|
| ASCII middle, 16 KiB target | developer + user | 105,411 | 172,035 / 173,449 | 2 / 2 | parent_copied_per_child |
| ASCII head, 8 KiB target | fork developer | 89,461 | 157,384 / 157,408 | 2 / 2 | parent_copied_per_child |
| JSON middle, 8 KiB target | child user | 89,038 | 156,764 / 157,193 | 2 / 2 | parent_copied_per_child |
| Unicode tail, 8 KiB target | developer + user | 89,386 | 156,010 / 156,010 | 2 / 2 | parent_copied_per_child |
The parent itself contained the anchor twice because the test input is persisted in two parent record planes. Every child also contained it twice. The parent file remained unchanged, but persistence grew by copying the parent history into every child rather than storing only a boundary reference plus the child overlay.
Across the three-arm matched process, 16/24 child checks passed. Own overlays applied and sibling-only overlays stayed isolated in all six children. The inherited-context check varied: JSON-middle and Unicode-tail children retained both the sentinel witness and parent instruction, while both ASCII-head/fork-developer children failed the combined inherited-context check.
A final one-child confirmation separated the boundary fields:
forkedFromId: recorded;sessionId: not preserved;All test-created parent and child threads were deleted after measurement. Reviewed JSON stayed under the operating-system temporary directory; no machine result was committed.
Run locally:
Use
--children N,--sentinel-bytes, and repeatable--armselections for controlled fan-out or targeted reruns. Please report the Codex/CLI version, model, effort, platform, all exercised arms, child-check counts, anchor occurrences, and growth classifications. Do not attach raw rollouts or result JSON.Thanks for consolidating these findings and the cross-plane suite.
Separating capture, structured presentation, persistence, batching,
compaction, tool provisioning, and recoverability makes the remaining gaps
much clearer.
I’ll review the suite and the fork-history protocol, then follow up with
any reproducible results from my environment.
Best,
tylorsaling-source @.***> 於 2026年7月29日週三 下午8:05寫道：