Long-running Codex agent lacked a no-progress watchdog after false derived holds (5.345B token-events)

Summary

A long-running Codex session completed clean source-unit intake for a 396-page controlled scanned standard, then spent extreme compute validating a false strict-closeout premise.

The primary defect was in our local project code: locator/reference occurrences were projected as physical tables or continuations, and object-level holds spread to reference-only pages. Codex did not originate that classifier bug. It materially amplified the incident because the run lacked token-versus-outcome visibility, repeated-input/result detection, a no-progress budget, and an automatic context-pressure handoff.

Session/thread ID: 019f6643-b589-75d3-ae45-7da1f5550351

Measured impact

A source-safe streaming audit of rollout metadata (no prompt, message, tool-body, licensed text, or private DB content retained) found:

• 5,345,347,029 cumulative token-events
• 98.29% cached-input ratio
• 191 context compactions
• approximately 3.98B token-events after clean-source intake was already 371/371 complete
• hottest six-hour zero-progress window: 248,568,594 token-events and 1,558 tool calls
• other zero-progress six-hour windows: roughly 196M–213M token-events

“Token-events” are usage metadata including cached traffic; this is not asserted as billed usage or cost.

Reliability near-miss

The first corrective terminal closeout was also stopped before commit: it proved that all 34 held pages had audit coverage but did not bind every page to explicit physical table/continuation dispositions. It mechanically zeroed genuine continuations. A later adversarial review also proved that a hand-shaped, hash-recomputed candidate could bypass the disposition evidence.

The corrected local contract now:

• requires exact per-page machine-readable dispositions
• derives relationship-review scope dynamically
• rebuilds the entire candidate from immutable evidence at validation/acceptance/replay and requires byte equality
• binds every page-decision hash, hierarchy review, raw hold-set count/hash, and continuation relationship review
• rejects forged table, continuation, hierarchy, and hold-ID mutations even after recomputing candidate hashes/IDs
• stops before a third expensive deterministic cycle unless an input, method, evidence item, or hypothesis changes

The corrected run ultimately reached 396/396 with the source DB unchanged.

Product requests

1. Show cumulative total/cached/fresh/output/reasoning tokens plus tokens since the last task-owned outcome change.
2. Detect identical normalized tool inputs with unchanged output/state hashes; after two expensive unchanged cycles, require an explicit changed-condition statement.
3. Provide a configurable no-progress budget based on wall time, token traffic, tool calls, and task-owned counters.
4. Warn when cached-input share is high while completion counters and state hashes are unchanged.
5. Automatically checkpoint and propose a fresh-agent handoff after sustained context pressure or repeated compactions, verifying takeover.
6. Reuse identical deterministic results inside a stable-input epoch, or at minimum flag the duplicate before another expensive orchestration cycle.
7. Add evals for false-premise loops: reference mention vs physical target, adjacent citations vs continuation, and internally consistent gates built on a bad classification.
8. Let /feedback attach a source-safe incident summary while associating backend telemetry by session ID.

Full public-safe report and local corrective design:
https://github.com/Demigodofa/codex-agent-ops/blob/d808559/knowledge/codex_long_run_no_progress_incident_2026-07-29.md

Please use the session ID for server-side telemetry. The raw ~937MB rollout should not be requested publicly because it may contain private tool output and controlled-source evidence.