›_ Codex Issues
Codex Issues / #37673

migrate-rollouts silently drops >16 MiB JSONL records while reporting migration successful

Open 💬 4 comments Opened Aug 9, 2026 by Yasei-no-otoko

What version of Codex CLI is running?

0.148.0-alpha.5

What subscription do you have?

Not applicable (local rollout migration).

Which model were you using?

Not applicable (local rollout migration).

What platform is your computer?

macOS / Darwin 27.0.0 / arm64

What terminal are you using?

zsh

Codex doctor report

Not included: this is a deterministic local storage transformation and source-level reproduction with no auth, network, or model dependency.

What is the issue?

codex migrate-rollouts --apply can silently omit valid legacy JSONL records larger than 16 MiB while reporting the thread as successfully migrated.

The current implementation sets:

const MAX_ROLLOUT_LINE_BYTES: usize = 16 * 1024 * 1024;

When a record is larger than that limit, read_rollout_record() consumes and discards the whole record through its newline and returns no line to the canonicalizer. The record is therefore absent from the staged paginated rollout and is not recoverable from the SQLite projection. Migration can still finish with status = migrated and no warning or message.

The size check happens before JSON parsing and is not restricted to tool output records, so it also affects valid compacted checkpoints.

Dry-run does not detect this. For a legacy rollout it reads session metadata and returns eligible without scanning all records, so the first full-record scan and oversized-record handling occur only during --apply.

The existing migration test also demonstrates the behavior: a valid oversized function_call_output is absent after migration while the outcome remains migrated.

Steps to reproduce

  1. Create a valid legacy rollout JSONL containing normal session metadata and normal events.
  2. Add a syntactically valid JSONL record larger than 16 MiB, for example:
  • a response_item / custom_tool_call_output containing inline image data; or
  • a compacted record whose replacement_history exceeds 16 MiB.
  1. Add another valid event after the oversized record to show that processing continues.
  2. Run a dry-run:

``shell
codex migrate-rollouts --thread <THREAD_UUID> --json
``
The rollout is reported as eligible; the oversized record is not reported.

  1. Back up the test rollout, then run:

``shell
codex migrate-rollouts --apply --thread <THREAD_UUID> --json
``

  1. Compare the migrated rollout with the input.

What did you expect to happen?

Migration should never silently turn a valid rollout into a partial rollout.

Preferably, migration should preserve valid records losslessly. If a finite per-record limit is required, then:

  • dry-run should scan for and report oversized records;
  • apply should fail closed before replacing the source, or require an explicit opt-in to omit data;
  • any partial result should not use the unqualified migrated status;
  • output should report skipped record counts, sizes, and positions;
  • a recoverable backup should exist before source replacement.

What happened instead?

  • The oversized record is consumed through its newline but not passed to canonicalization.
  • Migration continues with later records.
  • The staged file replaces the original rollout without the oversized record.
  • The outcome can be migrated with message = null.
  • There is no skipped-record count, byte count, ordinal, sentinel, or warning.
  • Surviving records are assigned canonical ordinals, so there is no ordinal gap to reveal the loss.
  • Dry-run reports eligible because it does not perform the full scan.

For a dropped custom_tool_call_output, resume may only synthesize an aborted output if the corresponding call remains; the actual output is not restored.

For a dropped compacted { replacement_history } record, resume may replay surviving raw events, but that is not necessarily equivalent to the compacted model-visible history and its checkpoint/baseline state. A newer surviving checkpoint can reduce runtime impact, but it does not restore archival fidelity.

Additional information

I performed a read-only, content-safe metadata audit of three existing legacy rollouts. I did not run --apply on them.

Across those three files, there were:

  • 10 syntactically valid records larger than 16 MiB;
  • 242,416,165 total bytes in those records (about 231 MiB);
  • 7 response_item / custom_tool_call_output records dominated by inline image data;
  • 3 compacted / replacement_history records dominated by image-bearing history.

This shows the limit is reachable in real Codex sessions, not only with artificial terminal output. No session payloads, inline images, thread IDs, workspace paths, or credentials are included in this report.

Relevant implementation and tests:

Related but distinct reports:

Suggested regression coverage:

  • >16 MiB custom_tool_call_output;
  • >16 MiB compacted / replacement_history;
  • a valid record after the oversized record;
  • dry-run/apply consistency;
  • resume/readback fidelity;
  • explicit non-success status whenever any record is omitted.

View original on GitHub ↗

4 Comments

weivwang · 17 days ago

I confirmed this is still present on current main (3c60d4d). read_rollout_record() consumes records larger than 16 MiB but represented them as line: None; canonicalization then skipped them while still publishing the staged rollout as Migrated.

I prepared a focused apply-time fail-closed patch. Oversized records now return an explicit migration error, so staged output is never published and the original legacy rollout remains recoverable. Handling for small malformed records is unchanged.

I also covered the bounded-subagent path. Its reverse scanner could previously skip an oversized record and stop at an older compacted checkpoint, bypassing the forward reader. The scanner now records whether an oversized record was skipped, and bounded migration rejects that rollout without an additional full-file pass.

Regression coverage exercises both an ordinary valid function output and a bounded-subagent compacted checkpoint, with a valid record after the oversized one. Each test verifies:

  • the outcome is Failed with the oversized-record diagnostic;
  • the source rollout remains byte-for-byte unchanged;
  • no migration journal is written; and
  • the SQLite history row remains Legacy.

Validation:

  • just fmt
  • just fix -p codex-rollout -p codex-thread-store
  • just test -p codex-rollout -p codex-thread-store — 325 passed, 0 failed
  • completed fork CI checks for formatting, cargo shear, and the argument-comment-lint package passed: https://github.com/weivwang/codex/actions/runs/31397775268. The overall fork workflow is red only because the macOS job did not start due to fork billing, the Windows job requires the private codex-runners group, and the Linux Bazel job reached its 30-minute runner limit; none reported a code failure.

This deliberately fixes silent data loss during apply. Making dry-run fully scan every rollout is left as a separate follow-up because that changes compressed-rollout I/O and performance semantics.

Patch:

Per the invitation-only contribution policy, I have not opened a PR. If this fail-closed approach matches the intended migration behavior, would a maintainer please invite me to submit the focused PR? I will rebase it onto the latest main before opening it.

weivwang · 15 days ago

Update: I rebased the prepared fix onto current main at e1b7b1ac (including the recent persisted-history envelope refactor) and squashed it to one atomic commit:

The draft PR targets only my personal fork; it is not an unsolicited upstream PR. It provides a standard Files/Checks review surface while I wait for an invitation. At the rebase point the branch was ahead 1 / behind 0, with only the five intended rollout/thread-store files changed.

This is reachable with real rollout data, not only a synthetic boundary case: independent field reports in #24948 measured individual rollout records around 52.1 MiB and 72.75 MB, both above the migration reader 16 MiB limit.

Fresh validation on that exact commit:

  • just test -p codex-rollout -p codex-thread-store: 332 passed, 0 failed, 0 skipped
  • just fix -p codex-rollout -p codex-thread-store: passed with no source changes
  • fork CI: https://github.com/weivwang/codex/actions/runs/31601239203
  • formatting, Rust benchmark smoke, cargo shear, and the argument-comment-lint package all passed
  • the Linux repo-wide Bazel lint was cancelled by its configured 30-minute job timeout without reporting a code failure
  • the Windows/macOS jobs executed no steps: the fork lacks the private codex-runners group, and GitHub blocked the hosted macOS runner for account billing/spending-limit reasons

@owenlin0, since you landed the recent rollout-migration work in #37175, #37191, and #37348, could you confirm whether this fail-closed behavior is the intended fix and, if so, explicitly invite a PR under the repository contribution policy? I will open it immediately and sign the CLA.

If direct integration is easier than granting PR access, the single commit is also ready to cherry-pick as-is; its GitHub author resolves to @weivwang, and I am available for follow-up fixes.

shleder · 14 days ago

The silent loss of valid records above 16 MiB is directly relevant to Codex Rescue’s oversized-record diagnostics. Rescue uses bounded reads, keeps the original rollout untouched, and should fail closed rather than treat a lossy derivative as a verified continuation.

It does not replace migrate-rollouts, restore records already discarded by migration, or rewrite large compaction checkpoints. I’d like to validate its behavior against a real pre-migration backup containing one of these oversized semantic records.

If you still have such a copy, would you try:

pipx install codex-rescue==0.1.0a3
codex-rescue sessions
codex-rescue doctor --latest

Sanitized record counts, sizes, and classifications are sufficient. Please don’t publish the raw rollout, inline images, prompts, repository content, SQLite DBs, credentials, or private paths.

https://github.com/shleder/codex-rescue

jdcodes1 · 9 days ago

Verified on main (1f41cc5d92): exactly as described — an oversize record drains through its newline and comes back as RolloutRecord { line: None, .. }, the canonicalizer never sees it, and nothing counts or reports the drop (https://github.com/openai/codex/blob/1f41cc5d92/codex-rs/thread-store/src/local/rollout_migration.rs#L965-L1000, cap at #L71, same cap wired into the subagent path via subagent.rs#L37). The status = migrated claim is therefore made after unaccounted data loss.

Worth noting this limit is not hypothetical: #37719 documents a real session with a single 19.3 MB custom_tool_call_output (inline base64 images), comfortably over the 16 MiB cap — so long multimodal sessions are precisely the ones that will lose records. And it joins two other silent-loss defects in the same migrator: #38761 (thread name dropped) and #38762 (subagent boundary treated as EOF).

Minimal fix: count discarded records per thread and (a) emit a warning with byte counts, and (b) report a distinct status (migrated_with_losses or leave the legacy file authoritative) instead of plain migrated. Better: write a placeholder record in place of the oversize one — preserves ordinal continuity for the projection (relevant to the #38552/#38317 invariant) and tells future readers what was elided — with the original bytes optionally spilled to a sidecar file rather than parsed in memory, since the stated constraint is memory safety, not storage.