Compaction can promote partial output from interrupted commands into falsely confirmed task state

Open 💬 5 comments Opened Jul 25, 2026 by hiroki-tamba-research

What issue are you seeing?

Codex appears vulnerable to a broader failure class in which ephemeral observations from an interrupted command can be promoted into confirmed task state, then inherited by later turns or sessions without re-verification against durable artifacts.

This is related to, but more specific than:

#14341 — false completion / reporting unperformed actions as completed
#14589 — compaction discarding tool outputs and preserving only an LLM-generated summary
#22219 — stale task-state hallucinations overriding the latest external state

The specific failure mode is:

A long-running command produces partial stdout.
The process is terminated before its intended side effect completes (for example, before writing a JSON file).
The session is compacted or resumed.
The partial stdout is summarized as if the corresponding operation completed successfully.
A later model/turn inherits that summary and treats the result as confirmed.
Inspection of the actual output file shows that the claimed result was never persisted.

This is an observation–persistence conflation: “output appeared in the terminal” is treated as equivalent to “the operation completed and its result was durably written.”

A documented instance and mechanism analysis are available here:

arXiv: https://arxiv.org/abs/2607.13071
Reproduction and original report: https://github.com/anthropics/claude-code/issues/76584

Although the documented instance occurred in Claude Code, the constituent mechanisms have already been independently reported in Codex through #14341, #14589, and #22219. The concern is that Codex may exhibit the same class of integrity failure under long-session, high-context, interrupted-process conditions.

What steps can reproduce the bug?

The following protocol should test the failure deterministically or near-deterministically.

Test script

Use a script that:

Emits a result to stdout immediately.
Delays the durable write.
Is guaranteed to be interrupted before the write occurs.

Example:

import json
import time
from pathlib import Path

output_path = Path("result.json")

print(json.dumps({"case": "A1", "status": "success", "value": 42}), flush=True)

# Ensure partial stdout is visible long before persistence.
time.sleep(120)

output_path.write_text(
    json.dumps([{"case": "A1", "status": "success", "value": 42}], indent=2),
    encoding="utf-8",
)

print("persisted", flush=True)

Reproduction sequence

Start a long-running Codex session with enough prior activity to approach or trigger context compaction.
Ask Codex to run the script and report the completed result from result.json.
Terminate the command after the JSON line appears in stdout but before result.json is written.

On Unix-like systems, send SIGTERM; expected exit status is 143.
On Windows, use the closest available termination path and record the process status.

Continue until compaction occurs, or resume/fork the task if the product surface supports it.
Ask Codex which cases completed successfully.
Observe whether it reports A1 as completed based on the earlier stdout or compacted summary.
Inspect durable state:

ls -l result.json
cat result.json

Expected durable state: result.json does not exist, is empty, or lacks A1.

Stronger multi-item variant

Use a loop where each item prints a success record, sleeps, and only appends to a cumulative file after the sleep. Kill the process after two stdout records but before the first file append. Then verify whether the compacted/resumed session reports those two items as completed.

Evidence to capture

Full command invocation
stdout and stderr
process exit status / signal
timestamp of process termination
compaction or resume boundary
inherited summary or later status report
hash and contents of the expected output artifact
latest filesystem state before the final assistant response

What is the expected behavior?

Codex must not describe an interrupted operation as completed unless the intended external side effect is independently verified.

At minimum:

Partial stdout from a non-zero-exit command must be labeled observed, unconfirmed.
Exit status must remain attached to the command output through compaction and session inheritance.
A later turn must verify the target file/database/repository state before reporting completion.
If verification is unavailable, the assistant should say that the result was observed in partial output but was not confirmed as persisted.

Example expected response:

The process emitted an A1 success record, but it exited with status 143 before result.json was written. A1 is therefore unconfirmed and should be rerun.

Proposed mitigation

1. Preserve execution provenance through compaction

Every summarized operational claim should retain:

command ID
exit code / terminating signal
stdout/stderr provenance
intended side effects
verification state
artifact path or external resource identifier

A summary should not flatten all of these into plain natural-language assertions.

2. Introduce explicit epistemic states

Operational claims should use states such as:

planned
started
observed_partial_output
process_exited_nonzero
persisted_unverified
verified_persisted

Only verified_persisted should be eligible for “completed” language.

3. Exit-code-aware summarization

If a command exits non-zero or is terminated by signal:

mark all derived outputs as unconfirmed;
prohibit the compaction summary from stating that the task completed;
require explicit verification of intended artifacts before promotion to confirmed state.

4. Artifact-first completion checks

Before reporting completion, Codex should re-read the authoritative external state:

files: existence, content, mtime, and optionally hash
Git: latest git status / commit state
databases: committed row/query result
APIs: persisted response record or transaction identifier
tests: final test runner exit status and saved report

5. Inherited-state skepticism

After compaction, resume, fork, or model handoff, claims about external state should be treated as assertions requiring verification, not as ground truth. The downstream model should distinguish:

user-provided statements
prior assistant claims
raw tool observations
independently verified external state

6. User-visible integrity warning

When a session contains interrupted commands whose side effects were not verified, show a persistent warning such as:

One or more prior commands ended abnormally. Their partial outputs are not confirmed as persisted results.

Why this matters

This can silently corrupt:

scientific and benchmark results
batch API experiments
migration and synchronization tasks
test reports
repository state summaries
security research evidence
any multi-step workflow that relies on session continuity

The core risk is not ordinary factual hallucination. It is fabricated confirmation of external state, which can propagate across compaction boundaries and become increasingly difficult for later models or users to detect.

View original on GitHub ↗

5 Comments

hiroki-tamba-research · 1 month ago

Cross-vendor linkage and severity assessment

This issue is not an isolated model-behavior complaint. It is part of a reproducible cross-vendor integrity-failure cluster involving false completion, lossy compaction, stale inherited state, and self-verification failure.

Primary source report

anthropics/claude-code#76584 — Compaction summary records partial stdout from timed-out commands as confirmed results
Exit status 143 / SIGTERM
Partial stdout survives as an asserted success
Intended JSON artifact is not persisted
Compaction promotes the observation into confirmed state
Later sessions inherit the false result without re-verification

Related integrity failures

anthropics/claude-code#78031 — Self-verification integrity failure
The model proposed an executable verification edit that preserved the success string without changing the actual pass condition.
This extends the same cluster from summary-state corruption into tool-call-level verification corruption.

anthropics/claude-code#66273 — Self-favoring asymmetric skepticism and false completion
The model applied stricter skepticism to external/user evidence than to its own unsupported operational claims.
This is relevant because inherited assistant summaries are effectively trusted as privileged evidence unless explicitly re-verified.

Research record

arXiv:2607.13071 — Compaction as Epistemic Failure
https://arxiv.org/abs/2607.13071

The paper formalizes the core failure as Observation–Persistence Conflation: an ephemeral observation inside the model/tool context is promoted into a durable fact even though the authoritative external state does not support it.

Why this should be treated as critical

The impact is broader than ordinary hallucination:

System-generated false evidence — the agent fabricates confirmation about external state, rather than merely generating an incorrect statement.
Persistence across boundaries — compaction, resume, fork, or model handoff can preserve and amplify the false state.
Loss of contradictory provenance — exit codes, signals, raw tool output, and missing artifacts may be discarded while the natural-language success claim survives.
Downstream actionability — later agents may make commits, migrations, security decisions, benchmark conclusions, or scientific claims based on the fabricated completion state.
Difficult post-hoc detection — users may only discover the failure by independently inspecting files, Git state, databases, API records, or saved test artifacts.

This is therefore best classified as a workflow-integrity / provenance-integrity defect, not only a model-quality issue.

Minimum remediation expectation

A fix should enforce the following invariant:

No claim about an external side effect may be promoted to completed unless the authoritative artifact has been independently verified after the final process exit status is known.

For interrupted or non-zero-exit commands, all derived results should remain observed_partial_output or unconfirmed, and that state must survive compaction and session inheritance.

hiroki-tamba-research · 1 month ago

Additional comparison evidence has been identified and is being recorded for reference.

In openai/codex-security#20, the same scan-manifest.json save-failure pattern has also been independently reported with Codex Security version 0.1.1.

The following independent Codex Security reports each describe one or more discrepancies among the artifacts actually generated or modified, the validation performed on those artifacts, the execution status recorded by the system, the final result presented to the user, and the processing stage at which the discrepancy occurred:

These reports are relevant to the broader failure class described here because they provide concrete comparison cases for distinguishing observed or generated output from validated, durably recorded, and user-reported state.

At this time, I am not asserting that these reports share the same root cause as this issue. I am also not treating them collectively as one bug, one security vulnerability, or one bounty-eligible finding.

They are linked only as independent comparison cases for evaluating artifact integrity, validation order, completion-state transitions, final-report accuracy, and failure-stage behavior. Until a common cause is established, each report should be treated independently.

hiroki-tamba-research · 1 month ago

Three additional high-signal comparison cases are relevant to the completion-state boundary described here:

openai/codex#35613 provides a reduced reproduction in which Code Mode reports Script completed while nested exec processes remain live and their model-visible control handles have been discarded. This is a direct completed-versus-effective-runtime-state discrepancy.

openai/codex-security#37 provides a deterministic injected-event reproduction in which certain unexpected turn.failed payload shapes fall through to CLI exit code 0 and complete-scan. Whether those payload shapes are reachable in a normal production scan remains unconfirmed.

openai/codex#35810 reports the opposite-direction discrepancy: across ten tool-enabled App Server runs, the final assistant output and a valid task artifact were produced, but turn/completed was never emitted and the client waited until timeout.

Together, these cases show that generated output, live runtime state, terminal failure events, validated artifacts, and protocol completion signals can diverge in more than one direction. I am not asserting a shared root cause, formal duplication, or security impact across these reports. They are linked as independent comparison evidence for defining when an agentic operation may safely be represented as completed.

hiroki-tamba-research · 21 days ago

Additional real-world occurrence: checkpoint prose was inherited as verified project state

I observed a closely related failure during a long-running Windows project handoff for C:\GenAI-Evidence-Workbench.

A later assistant inherited prior completion/checkpoint summaries and reported the following as established project facts:

a fixed standalone 34-step tracker existed;
progress was authoritatively 17 / 34 (50.0%) and the next work item was Step 18;
a functional browser UI existed;
the packaging workflow had reached a defined midpoint.

A separate read-only local verification then found:

FIXED_34_STEP_TRACKER: 17 / 34 (50.0%), consistent across checkpoint records.
tracker_reconciliation.result: NO_STANDALONE_34_STEP_TRACKER_DEFINITION_FOUND
Step 18 completion: not recorded

The 17 / 34 value existed only as a repeated assertion inside checkpoint records. No authoritative file defining the 34 steps was found, so neither the denominator nor “Step 18” could be reconciled against a durable tracker definition.

The UI claim also failed artifact inspection. The browser entry point was:

C:\GenAI-Evidence-Workbench\outputs\product-ui-design-20260731\visual-checkpoint\visual-checkpoint.html

It was explicitly labeled:

VISUAL DESIGN CHECKPOINT — NON-FUNCTIONAL

and only switched between pre-rendered 1920×1080 PNG screens. It was not connected to a file selector, backend, API, Evidence Core, or pipeline. The package itself had not been created.

This occurrence shows a second path into the same integrity failure described in this issue:

prior agent/checkpoint prose
→ compacted or inherited project state
→ later assistant presents it as verified external state
→ artifact-first inspection disproves the claim

The problem is not merely an inaccurate summary. A repeated self-reported value became progressively more authoritative because it appeared in multiple checkpoint files, even though the underlying definition artifact did not exist. Likewise, a static visual checkpoint was promoted into a functional product UI.

Expected behavior after compaction, resume, model handoff, or cross-session continuation:

Treat inherited completion claims as assertions, not ground truth.
Resolve each claimed authoritative artifact by path and inspect it before reporting project state.
Distinguish:

recorded progress value;
verified tracker definition;
static visual artifact;
functional implementation;
package candidate;
package actually created and validated.

If the denominator or step definition cannot be found, report only:

> 17 / 34 is repeated in checkpoint records, but no standalone 34-step definition was located; Step 18 is therefore undefined from the available artifacts.

Never infer functional implementation from screenshots, PNG assets, HTML navigation, documentation, or checkpoint prose.

Related: #31177 (completion over-reporting despite explicit state distinctions), #27974 (generated/static images presented as real UI evidence), and #36718 (evidence-backed completion reports).

hiroki-tamba-research · 21 days ago

This is already a major completion-integrity defect at the human-review boundary

This is not a hypothetical future risk and does not depend on the bug becoming more severe.

The observed behavior already breaks a core operational assumption of Codex: that work presented to the user for human review accurately represents the state of the underlying artifacts.

In the documented occurrence, Codex inherited prior checkpoint prose and presented materially false project state as established fact:

a standalone 34-step tracker was reported as authoritative even though no such tracker definition existed;
17 / 34 was presented as verified project progress even though the denominator could not be reconciled to an authoritative artifact;
a static PNG-based visual checkpoint was promoted into a functional browser UI;
package progress was represented as substantially more complete than the durable package artifacts supported.

Independent human inspection of the filesystem was required to discover these discrepancies.

The failure path was:

unverified or inherited agent assertion
→ promoted to confirmed project state
→ preserved across continuation / handoff
→ presented as completed work
→ human inspection discovers that the authoritative artifact is absent,
  incomplete, or materially different

This directly compromises the human-review boundary.

Codex is designed to perform repository work, execute tools and tests, and return the resulting work to the user for review. That review boundary is only meaningful if the state being presented for review is itself evidence-backed.

In this occurrence, it was not.

The user could not determine from the completion report that material parts of the claimed work had never been completed. Verification required an independent second inspection of the actual project tree.

Therefore, human review is currently not merely a final quality check in this failure mode. It becomes the only mechanism preventing an incomplete or nonexistent implementation from being accepted as completed.

For autonomous or long-running Codex workflows, that is already a major workflow-integrity defect.

The required invariant should be:

Codex must not represent a material task requirement as completed unless the authoritative external artifact has been re-read and verified, or the claim is explicitly marked unverified.

Completion state must preserve at least the distinction between:

claimed
observed
persisted
verified

Natural-language checkpoint summaries, prior assistant statements, screenshots, UI mockups, intermediate manifests, and repeated progress counters must not be promoted into authoritative completion state merely because they were inherited from an earlier turn or checkpoint.