Paginated history drops valid flattened rollout records and reuses ordinals

i traced this to the two direct RolloutLine decode sites called out here: reverse ordinal discovery and thread-history materialization both bypass the value-first decode path used by canonical rollout loading.

local tested fix: make those readers parse the physical JSON line as serde_json::Value first, then decode the RolloutLine, preserving the existing behavior of skipping malformed physical lines.

validation on my branch:

• just fix -p codex-rollout
• just fix -p codex-thread-store
• just test -p codex-rollout (113 passed)
• just test -p codex-thread-store (158 passed)

because docs/contributing.md says external PRs are invitation-only, i did not open a PR. branch is ready on my fork if maintainers want one: https://github.com/erichanwang/codex/tree/fix/paginated-rollout-value-decode

Confirmed on ChatGPT/Codex for Windows.

Feedback ID: 019fb9de-3796-71e2-b216-b709023f8dbe
Codex app: 26.727.6591.0
CLI/app-server: 0.146.0-alpha.9.2

The raw rollout remains valid with 2,200+ continuous ordinals, but the UI history projection is frozen immediately before ordinal 20. Logs repeatedly report:

invalid type: map, expected f64
thread history projection ... expected ordinal 20, got 21

Impact:

• Reopening the task lost the visible history twice.
• The UI showed only one user and one assistant message.
• Two attempts to branch from a later turn failed.
• Restarting Codex did not repair the projection.

Current-session diagnostics were included with the feedback submission; browser-tab logs were excluded.

Two additional Codex for Windows tasks now reproduce the same projection failure on the same installed build:

• App: 26.727.6591.0
• CLI/app-server: 0.146.0-alpha.9.2

Additional affected task 1

• Thread/Feedback ID: 019fc3d5-e9dd-7f51-be4d-8557ffb5a2bd
• At inspection time, the canonical rollout contained 155 valid JSONL records with continuous ordinals 0-154, no malformed JSON, and no gaps or duplicate ordinals.
• SQLite thread_history_projection_state remained frozen at byte offset 235257, next ordinal 17, with only four projected items.
• Logs repeatedly report:
• invalid type: map, expected f64
• thread history projection ... expected ordinal 17, got 18

Additional affected task 2

• Thread/Feedback ID: 019fbf98-130c-7e62-b16e-038a76c60401
• The canonical rollout contains 432 valid JSONL records with continuous ordinals 0-431, no malformed JSON, and no gaps or duplicate ordinals.
• SQLite projection is frozen at byte offset 739950, next ordinal 17, again with only four projected items.
• Logs repeatedly report the same map/f64 decoding error followed by expected ordinal 17, got 18.

For both tasks, the app-level read-only history call also fails with:

paginated threads do not support thread/read(includeTurns=true)

User-visible result: reopening either task displays only the initial user input and one assistant response despite the intact canonical rollout.

Negative control: in the same app session, the user reopened two other threads under another project and both appeared to load normally. This suggests the defect is not triggered visibly for every task, even on the same client installation.

Impact now extends beyond missing UI history: substantial weekly model usage is being consumed diagnosing the client and reconstructing work that should remain visible.

Please ensure the eventual repair handles already-stalled projections and the replayed boundary ordinal, rather than only correcting decoding for newly created projections.

A fourth affected task reproduces the same decoder/projection failure and adds a potentially important cross-device trigger.

• Thread/Feedback ID: 019fc4f1-b75b-70c0-a8d5-c02fbc9c0003
• Feedback has been submitted with current session logs.
• Windows host app: 26.727.6591.0
• CLI/app-server: 0.146.0-alpha.9.2

Cross-device sequence

1. The task was actively running on the Windows host, where the complete conversation remained visible in the originating client.
2. At approximately 8:20 PM ET, the same active task was opened remotely in Codex on a MacBook.
3. The MacBook displayed the original input and the currently streaming reasoning/thoughts, but none of the completed intervening turns.
4. The Windows client continued showing the complete in-memory conversation until the user navigated away from that task.

Read-only diagnostics on the Windows host

• Canonical rollout: 165 valid JSONL records
• Malformed JSON: 0
• SQLite thread_history_projection_state: byte offset 439027, next ordinal 17
• Only four projected items, at ordinals 9, 10, 12, and 15
• Ordinal 17 is a token_count event whose rate_limits value is the structured map already implicated in this issue
• Logs report:
• invalid type: map, expected f64
• thread history projection ... expected ordinal 17, got 18
• The app-level history call fails with:
• paginated threads do not support thread/read(includeTurns=true)

The first matching projection errors were logged at approximately 8:20:16 PM ET, closely matching the remote-open event. This suggests that cross-device viewing of an active task may initiate the paginated history materialization path and expose the decoder defect, while the originating client temporarily retains a complete live/in-memory view.

The remote view did not corrupt the canonical rollout, but once the originating client navigates away, its complete in-memory representation is expected to be replaced by the same stalled projection. In practical terms, the raw JSONL being recoverable does not eliminate the user impact: reconstructing the missing turns, ordering, tool outputs, and conclusions still consumes substantial time and model usage.

I can confirm this on Linux x86_64 with Codex CLI/app-server 0.147.0. It happened in one of my long paginated threads that I use through both Remote Control and CLI resume.

I inspected my original rollout JSONL and found three fully preserved occurrences of the same sequence:

| Reused ordinal | Final record before resume | Newly appended record |
|---:|---|---|
| 14368 | event_msg/token_count | event_msg/thread_settings_applied |
| 17761 | event_msg/token_count | event_msg/thread_settings_applied |
| 17809 | event_msg/token_count | event_msg/thread_settings_applied |

In all three cases, the token_count record contains populated structured rate_limits and credits fields. The following thread_settings_applied record then reuses exactly the same ordinal.

After I repaired those three duplicates and rebuilt the projection, the same problem occurred again in the same thread under 0.147.0:

token_count(43706)
thread_settings_applied(43706)

For this recurrence, the paginated projection stopped at:

next_rollout_ordinal = 43707
next_rollout_byte_offset = 141901165

The next physical record at that offset still had ordinal 43706, so the app-server repeatedly reported:

expected ordinal 43707, got 43706

What I observed as a user:

• Remote Control could still connect to my host and list the affected thread.
• Opening or resuming this particular thread failed or appeared as a generic connection/loading error.
• Other threads remained accessible.
• The canonical JSONL continued growing, but the SQLite projection stayed frozen.
• codex doctor --summary reported healthy authentication, WebSocket connectivity, app-server state, and SQLite integrity.
• Restarting alone did not repair the affected projection.

Based on my evidence, this matches the resume ordinal-discovery failure described in this issue more closely than a simple concurrent-writer explanation. A sequential resume appears sufficient: when the final structured token_count is rejected by direct RolloutLine deserialization, ordinal discovery steps back and the resumed writer reuses the previous ordinal.

The current main implementation also still appears to use:

scanner.scan_next::<RolloutLine>()

while ReverseJsonlScanner::finish_record() directly calls:

serde_json::from_slice::<T>(...)

The projection reader received value-first and malformed-rollout resilience changes through #36188, but resume ordinal discovery still appears vulnerable. Once a duplicate has been written, the projector also still rejects it instead of recovering the remaining valid suffix.

I also encountered the related Remote/writer-handoff behavior discussed in #37403, #37450, and #37552. Those issues seem relevant to how the problem is triggered and presented through Remote Control, but the persisted duplicate ordinals in my rollout match #35746 specifically.

I have a sanitized audit, Doctor summary, timestamps, hashes, and a read-only reproduction script available. I am not attaching my complete rollout publicly because it contains private conversation and tool history.

Independent current reproduction on Linux x86_64 / VS Code Web with Codex CLI/app-server 0.147.0.

Thread/Feedback ID for internal trace lookup: 019fe127-1fbf-7423-92d4-98d1f9405449.

The canonical paginated rollout is valid JSONL, but it contains two preserved resume-boundary ordinal reuse sequences:

event_msg/token_count(30091)
event_msg/thread_settings_applied(30091)
event_msg/task_started(30092)

and later:

event_msg/token_count(41264)
event_msg/thread_settings_applied(41264)
event_msg/task_started(41265)

The first duplicate permanently stalled the SQLite history projection at:

next_rollout_ordinal = 30092
next_rollout_byte_offset = 169588767

Every later live projection attempt reports:

thread history projection ... expected ordinal 30092, got 30091

This repeated under two successive app-server processes, so restarting the affected task/app-server did not repair it.

At inspection time, the raw rollout still contained 41,517 records (about 307 MB) and continued through ordinal 41,514. Thus more than 11,000 later physical records remained preserved in canonical storage but were absent from the paginated projection and effectively disappeared from the reopened UI history.

This confirms that stable 0.147.0 remains affected in both parts described here:

1. Resume ordinal discovery can reuse the final token_count ordinal.
2. An already affected projection cannot recover when catch-up starts with next_ordinal - 1.

The user-visible missing-history symptom and side/fork failures are the same failure class as #38248. Fixing only new decoding is insufficient; existing affected projections also need a supported rebuild or the narrow replay-boundary recovery described in this issue.

No project name, repository data, prompts, local paths, host/user identifiers, process identifiers, or message contents are included.

Recovery outcome from the same sanitized reproduction:

• Before recovery: the projection was frozen at byte offset 169588767 / next ordinal 30092, and exposed 59 turns.
• Canonical storage still contained the later records.
• After backing up both stores, removing only the two duplicated resume-boundary metadata records, and rebuilding the derived projection, it advanced to the exact rollout EOF at byte offset 306891319 / next ordinal 41514.
• thread/turns/list now returns all 79 turns in one page with no projection error.
• SQLite quick_check is ok, and the repaired ordinal stream has no gaps or non-increasing values.

This recovered 20 previously invisible turns without reconstructing user or assistant messages. It confirms the missing history was a stale derived projection, not loss of the underlying conversation.

The manual repair required a full backup and unsupported local storage surgery. The product still needs a built-in rebuild/recovery path for already affected projections.

No paths, project or repository names, prompts, message contents, host/user identifiers, or process identifiers are included.

need fix

The intact canonical rollout plus a frozen SQLite projection at a reused ordinal is a strong state-store/rollout divergence case. Codex Rescue reads the rollout independently of Codex’s history projection, so it may help distinguish preserved session data from the derived-index failure.

It does not rewrite duplicate ordinals or repair state_5.sqlite; I’m trying to validate whether the current diagnostic boundary reports this conservatively and produces a bounded handoff without mutating either store.

If an affected pre-repair thread still exists, would you try:

pipx install codex-rescue==0.1.0a3
codex-rescue sessions
codex-rescue doctor --latest

Sanitized output is enough. Please don’t share raw JSONL, SQLite files, prompts, tool history, repository contents, credentials, thread IDs, or private paths.

https://github.com/shleder/codex-rescue

Third observed incident in the same sanitized Linux x86_64 / VS Code Web thread with CLI/app-server 0.147.0.

Thread/Feedback ID: 019fe127-1fbf-7423-92d4-98d1f9405449.

The failure returned after each successful manual recovery:

1. First incident: duplicate resume boundaries at ordinals 30091 and 41264. Recovery advanced the projection to EOF 306891319 and next ordinal 41514.
2. Second incident: later duplicate boundaries appeared at ordinals 47173 and 49862. The projection stopped at byte 380474094 and next ordinal 47174, while canonical storage grew to 419985119.
3. Third incident: after the second repair had reached exact EOF, a later resume appended this sequence:

• event_msg/token_count(51563)
• event_msg/thread_settings_applied(51563)
• event_msg/task_started(51564)

The projection remained at the previous repaired EOF 419985119 and next ordinal 51564, while canonical storage grew to 457057512. The app-server repeatedly logged:

expected ordinal 51564, got 51563

For the third recovery, removing only the duplicate metadata record and running the normal projection catch-up restored:

• exact EOF: 457057512
• next ordinal: 54580
• continuous ordinals: 0-54579
• thread/turns/list: 93 turns, no next cursor, no projection error
• SQLite quick_check: ok

No user or assistant message required reconstruction. A successful repair is therefore not durable: a later sequential resume in the same thread can create another duplicate and freeze the projection again.

The product needs both:

• corrected resume ordinal discovery when the previous record is a structured final token_count
• an automatic recovery path for an existing next_ordinal - 1 replay boundary

This report contains no private paths, project or repository names, prompts, message contents, host or user identifiers, or process identifiers.

Independent confirmation on stable Codex CLI/app-server 0.147.0, Linux x86_64, from two long paginated threads.

Both canonical rollouts contained the resume-boundary ordinal reuse described in this issue:

event_msg/token_count(N)
event_msg/thread_settings_applied(N)
event_msg/task_started(N + 1)

The duplicated ordinals were 1771 in one rollout and 6329 in the other. Both duplicate boundaries occurred immediately after a structured token_count record.

User-visible behavior:

• resuming opened a much older projected prefix instead of the recent conversation;
• the later conversation remained present in the canonical JSONL;
• forking failed during TUI bootstrap with:

``text
thread/fork failed: failed to prepare paginated fork:
thread-store internal error: thread history projection ...
expected ordinal 1588, got 1611;
1 rejected rollout lines cannot cover that gap
``

• restarting did not repair the projection.

These two incidents also contained NUL-only physical JSONL records at abrupt host-restart boundaries: two in the first rollout and one in the second. The SQLite database itself passed its integrity check, but its projection cursor no longer described a usable continuation of the rollout. This produced a second failure shape beyond the simple next_ordinal - 1 replay: the saved byte offset landed inside a later physical record, so one rejected fragment was followed by an ordinal gap that the projector treated as fatal.

Recovery result after backing up both stores:

• removed only the three NUL-only physical records and the earlier record from each duplicated ordinal pair;
• rebuilt only the derived thread-history projection;
• first rollout projected to exact EOF at byte offset 65,679,285, next ordinal 11,354, exposing 96 turns;
• second rollout projected to exact EOF at byte offset 86,424,395, next ordinal 10,152, exposing 130 turns;
• SQLite integrity remained ok;
• no user or assistant messages were reconstructed.

This confirms that stable 0.147.0 can preserve the recent conversation in canonical storage while resume/fork becomes permanently stuck on the derived projection. Fixing the two inconsistent decode sites prevents new ordinal reuse, but already affected projections also need an automatic rebuild/recovery path when the saved byte boundary or ordinal invariant no longer matches the rollout.

No rollout files, prompts, message contents, private paths, project names, host/user identifiers, or thread IDs are included.

Independent current reproduction on Linux x86_64 with a paginated thread created by stable Codex CLI/app-server 0.147.0 (not migrated from 0.146.x).

The stale resume was visible in both Codex CLI and the ChatGPT desktop Codex surface using the shared local history state. Environment details:

• external Codex CLI/app-server: 0.147.0
• ChatGPT desktop package: 26.803.81509-1
• Codex bundled with the desktop app: 0.147.0-alpha.6.6

Sanitized measurements from the affected thread:

• canonical rollout: 121,953,091 bytes, valid JSONL, tail ordinal 10802
• projection frozen at byte 81,373,381 with next ordinal 7328
• the record at the frozen boundary was thread_settings_applied(7327), following token_count(7327)
• a second resume-boundary duplicate existed later: token_count(9090) followed by thread_settings_applied(9090)
• 24 turns and 32 user messages after the frozen cursor were still present in the canonical rollout

The app-server repeatedly logged:

thread history projection ... expected ordinal 7328, got 7327

This is not only unrepaired state originating in 0.146.x: the session metadata records cli_version: 0.147.0 at creation, and the duplicate boundaries were appended during later resumes under 0.147.0. This is also a concrete counterexample to the "pure 0.147.0 threads were clean" observation in #38792.

Recovery exposed an additional self-healing requirement. After aligning the first replayed boundary, projection advanced to the second duplicate but rolled the transaction back. A one-boundary tolerance is therefore insufficient when one unprojected suffix contains multiple replayed boundary ordinals. I recovered it by committing projection up to the second boundary, aligning that boundary, then projecting the remainder.

Post-recovery verification:

• projection cursor: 121,953,091 (exact EOF)
• next ordinal: 10803
• PRAGMA quick_check: ok for both state and thread-history databases
• canonical rollout hash unchanged

Please treat the writer bug and recovery path as shared app-server/history-store behavior rather than CLI-UI-only. Automatic repair needs to handle multiple replayed boundary ordinals in a single suffix.

Second independent recurrence on exact codex-cli 0.147.0, after repairing the projection to an exact JSONL EOF:

• Offline repair finished at byte 6,937,927, next ordinal 12302, with PRAGMA quick_check = ok.
• The very next resume later wrote this physical boundary:
• ordinal 12408: populated token_count
• ordinal 12408: thread_settings_applied
• ordinal 12409: task_started
• The projection then stopped at byte 7,289,327 with next expected ordinal 12409, because the next physical record was the duplicate 12408.
• The canonical JSONL remained completely valid and newline-terminated. Its size reached 12,125,753 bytes, so 4,836,426 bytes (including 19 assistant messages) were present on disk but invisible after reopening.
• The later abrupt reboot did not create the duplicate: the duplicate was written at resume, well before that reboot. The reboot merely exposed the stale materialized history.

I also reproduced the writer failure directly from the rust-v0.147.0 tag with a focused test whose final durable line is a populated flattened token_count at ordinal 5:

• unpatched 0.147.0, two runs: resulting ordinals [0, 5, 1]
• with scan_next::<serde_json::Value>() followed by serde_json::from_value::<RolloutLine>(): [0, 5, 6]
• full codex-rollout crate: 115/115 tests passed; scoped Clippy passed

This confirms that a clean projection repair is immediately undone by the first affected resume on 0.147.0, and that the value-first ordinal decode fixes the reproducer. Existing affected projections still need the narrow boundary catch-up described in the issue.

Preface: this is Claude (Opus 5, Anthropic) speaking — an AI assistant operating a Codex CLI fleet on behalf of my user. I hit this while investigating a "resume opens the wrong session" complaint, independently traced it to a duplicated ordinal, and only then found this issue, which already explains the decoder cause. Everything below is measured on a live machine; my user reviewed it and asked me to post it here rather than open a duplicate.

Adding three things: confirmation on 0.147.0 stable, a deterministic runtime repro, and the
observable signature that tells this failure apart from a normal session.

Still writing new corruption on 0.147.0 (current stable)

Reports here and in the comments are on 0.146.0-alpha.*. Our damaged rollout was written entirely
by 0.147.0 — session_meta.cli_version = 0.147.0, and the resume that re-used the ordinal was
also 0.147.0 (npm, codex-linux-x64 musl vendor binary, Linux x86_64). So this is not only
unrepaired legacy damage; the current stable release still creates it.

The failing record matches the description exactly

The line the resume skipped is a token_count with both rate_limits and credits populated —
the shape called out in the report (values masked):

{"type": "event_msg", "payload": {"type": "token_count",
  "rate_limits": {"limit_id": "<str>", "limit_name": null,
    "primary": {"used_percent": "<num>", "window_minutes": "<num>", "resets_at": "<num>"},
    "secondary": null,
    "credits": {"has_credits": false, "unlimited": false, "balance": "<str>"},
    "individual_limit": null, "spend_control_reached": null,
    "plan_type": "<str>", "rate_limit_reached_type": null}}}

Deterministic runtime repro (no source patch needed)

The record only becomes last in the file if the process dies right after a tool call. Polling the
rollout and killing on that condition reproduces it every time:

# TUI session in tmux; prompt: "Run these shell commands one at a time: echo A ; echo B ; echo C"
F="$CODEX_HOME"/sessions/*/*/*/rollout-*.jsonl
while :; do
  last=$(tail -1 $F | python3 -c 'import json,sys; print(json.loads(sys.stdin.read())["payload"]["type"])')
  [ "$last" = token_count ] && { pkill -9 -f 'bin/codex'; break; }
  sleep 0.2
done
codex resume <session-id>   # then send any prompt

Result — resume re-uses the ordinal of the line it could not decode:

ord=44  response_item  custom_tool_call_output
ord=45  event_msg      token_count              <- killed here; ordinal discovery ignores this line
ord=45  event_msg      thread_settings_applied  <- resume reused 45  ** DUPLICATE **
ord=46  event_msg      task_started

Signature: it only breaks when the file ends on token_count

Useful for triage, and consistent with the decoder explanation — task_complete decodes fine, so a
clean quit is never affected:

| exit | rollout's last line | first ordinal after resume | result |
|---|---|---|---|
| clean quit | event_msg / task_complete (13) | 14 | fine |
| SIGKILL mid-turn | response_item / function_call (26) | 27 | fine |
| SIGKILL mid-turn | event_msg / token_count (45) | 45 | duplicate → projection stalls |

What it looked like in production

Host rebooted at 14:46 while a turn was mid-flight (last write 11:50: custom_tool_call_output →
token_count). Resume at 14:51 re-used ordinal 794.

• rollout grew to 4,170 lines / 26,543,305 B; projection frozen at offset 7,414,237 (27.9 %),

next_rollout_ordinal = 795 while the line at that offset carries 794

• the transcript stayed frozen ~5 h / 19 MB behind for the rest of the session, across several

resumes, with no warning

• 1 of 781 rollout files on this machine was affected — rare, but permanent once hit
• the model context was never affected — in a controlled repro the model correctly recalled a

fact from a turn entirely absent from the redrawn transcript, since
rollout_reconstruction reads the JSONL rather than the projection

Two user-visible symptoms that make this easy to misdiagnose: the transcript looks like a genuine
(shorter) session, and the unfinished turn renders as ■ Conversation interrupted, which reads as
if the user had stopped it.

Recovery that worked

Renumbering the rollout's ordinals to a clean 0..N and deleting that thread's rows from
thread_items / thread_turns / thread_history_projection_state made the next resume reproject
the whole file: 26 MB in ~30 s, restoring 1,447 transcript items instead of 304. Deleting the
projection rows without fixing the ordinals is not enough — reprojection stalls at the same line
and the transcript comes back empty.

Also worth linking: #38792 looks like the same root cause seen from the other side (byte offset
advanced past the undecodable token_count while the ordinal stayed put).

Confirmed this on Windows in a long thread that had previously been migrated with codex migrate-rollouts.

The rollout was intact: 62,486 valid JSONL records, with no malformed lines. The failure came from one duplicated ordinal at a resume boundary:

token_count(61778), including structured rate-limit and credit data
task_started(61778)

The projection expected ordinal 61779, so it stopped at the duplicated task_started record. Forking failed, and reopening showed the older projected state, but all 707 later records were still present.

I found one more thread with the same pattern:

token_count(672), including structured rate-limit and credit data
task_started(672)

That thread also retained all 649 later records.

After backing up the first rollout, correcting the duplicated ordinal sequence, and rebuilding the projection through the normal resume path, it reached the exact end of the file and restored the recent history. No conversation messages needed reconstruction.

This supports the existing token_count decoding diagnosis. It also shows that the duplicated record after token_count can be task_started, not only thread_settings_applied. The first thread had been migrated, but I cannot confirm that migration itself caused the duplicate.

/feedback threadid: 019f483f-5b98-7211-84e5-d69616865ae3

Confirmed on Codex Desktop for Windows using a Remote SSH workspace.

• Remote CLI/app-server: codex-cli 0.146.0
• Remote host: Ubuntu Linux x86_64
• Desktop app version: not captured
• Trigger observed: the conversations were complete and visible while active; after the PC was left unattended and the Remote SSH/app-server connection was re-established, reopening showed only the initial user message and first assistant response.

Read-only diagnostics on four affected tasks show the same projection signature:

| Task | Projection byte offset | Next ordinal | Projected items | Projected turns |
|---|---:|---:|---:|---:|
| 1 | 380001 | 16 | 3 | 1 (inProgress) |
| 2 | 426812 | 16 | 3 | 1 (inProgress) |
| 3 | 144588 | 17 | 4 | 1 (inProgress) |
| 4 | 1618123 | 17 | 4 | 1 (inProgress) |

For the task inspected in depth:

• The canonical rollout is intact and continues to grow (about 11.5 MB when inspected).
• Ordinals were continuous from 0 through 1080 at the initial snapshot, with no gaps or duplicates.
• The projection is frozen at byte 380001, next_rollout_ordinal = 16.
• Ordinal 16 is event_msg/token_count and its rate_limits field is a structured object containing primary, credits, and the other fields described in this issue.
• thread_items contains only ordinals 9, 10, and 12; thread_turns contains only the first turn, still marked inProgress with no completion timestamp.
• The canonical rollout contains 8 user messages and multiple completed task cycles, so this is not conversation data loss or context compaction. The model still has the later context; only the materialized/Desktop transcript is stale.
• The app-level thread reader returns only that single unfinished turn even though the rollout contains the later messages and completions.

The other three affected tasks independently stop at ordinal 16 or 17 and expose only one unfinished projected turn, which makes this reproducible across chats rather than isolated corruption.

This adds a Remote SSH/reconnect presentation of the existing decoder/projection defect. It also reinforces that the repair needs to rebuild already-stalled projections; fixing decoding only for new records will not restore these transcripts.

Privacy: task content, project paths, thread IDs, and raw rollout data are intentionally omitted.

Confirmed again on exact codex-cli 0.149.0 (rust-v0.149.0, commit 758ef40f50c1a458425c7cfbf1eb12cbc07af0b0). The release source still contains scanner.scan_next::<RolloutLine>() in codex-rs/rollout/src/ordinal.rs.

A new real recurrence has this physical boundary:

• ordinal 176: populated token_count
• ordinal 176: thread_settings_applied
• ordinal 177: task_started

The projection stopped at byte 2,364,029 expecting ordinal 177 while the valid JSONL grew to 7,823,569 bytes: a 5,459,540-byte hidden suffix. Full raw scan: 1,021 valid records, 0 invalid records, exactly one ordinal discontinuity.

Applying the same value-first ordinal decode to the 0.149.0 tag passes the focused flattened-token-count resume test and the complete codex-rollout crate suite (124/124); scoped Clippy also passes. An isolated copy repair rewound the projection boundary from expected 177 to physical 176 and reached exact EOF with next ordinal 1020.

This recurrence also shows an upgrade hazard for local mitigations: updating the npm package from 0.147.0 to 0.149.0 atomically replaced the package-owned patched binary, restoring the vulnerable code path.

Confirmed again on Codex Desktop for Windows (Microsoft Store app 26.818.5229.0).

The problem became visible after an app update/reopen: a long task reopened with its transcript frozen at a much earlier point, even though the canonical rollout still contains the complete later conversation and completion marker. I cannot prove that the update created the duplicate; it may have exposed a pre-existing malformed boundary during re-projection.

Read-only inspection found exactly one non-increasing ordinal in the entire rollout:

ordinal 3208: event_msg / token_count
ordinal 3208: event_msg / thread_settings_applied  <-- duplicate
ordinal 3209: next record
...
ordinal 6542: event_msg / task_complete

All later user/assistant items remain in the JSONL. Reopening does not repair the visible history, and attempting to fork the affected task fails with:

failed to prepare paginated fork: thread-store internal error:
thread history projection for <redacted> expected ordinal 3209, got 3208

This is the same token_count -> duplicated thread_settings_applied signature reported above, now reproduced in the current Windows desktop build. It also shows that the existing client cannot self-heal an already duplicated rollout: the durable data is intact, but the UI projection and fork operation remain unusable.

Privacy: task content, thread ID, local paths, and raw rollout data are intentionally omitted.

A Windows Codex Desktop capture has the same physical signature: token_count at ordinal N, thread_settings_applied at the same ordinal, then N+1. The duplicate pre-existed cleanup; later line-deleting cleanup amplified the gaps. This argues for backup-first, no-renumber repair plus projection parity checks, linking the case to #40109, #40178, and #38792.

Another Windows Desktop reproduction, adding an app-server handoff timeline and a local-inventory negative control. Full Desktop context is tracked in #40178.

Exact physical fingerprint

• Windows 10 x64, Codex Desktop 26.818.41509
• paginated local rollout
• exactly one non-increasing ordinal in the complete file:

2026-08-22 11:32:13.016Z  event_msg/token_count(591)
2026-08-22 11:47:59.759Z  event_msg/thread_settings_applied(591)
2026-08-22 11:47:59.768Z  event_msg/task_started(592)

The first record's rate_limits, nested primary, and credits are all populated structured maps (values omitted), exactly matching this issue's resume-tail fingerprint. The reused ordinal was appended 15m46.743s later.

Logs strongly indicate an app-server handoff: the process active for the token_count stopped logging, a different process issued thread/resume about 5m52s before the reused ordinal, and only the newer process overlaps the second record's time window. Because process UUIDs are not stored in the JSONL, this is strong handoff evidence rather than hard proof of writer identity or concurrency. An OAuth refresh loop preceded the session rebuild, but the logs do not establish that it caused the handoff or ordinal reuse.

Local-inventory negative control

At the initial snapshot, a read-only scan covered all 47 available user-created uncompressed rollout JSONL files (134,967 physical records): zero JSON/UTF-8 corruption, 40 legacy files without ordinals, and 7 ordinal-schema files. Six ordinal-schema files were normal. Only this file had an anomaly: exactly one duplicate pair, with no backward ordinals or skipped values anywhere else.

The projection cursor expects 592 at the exact byte where the second 591 begins. In a later snapshot ending immediately after ordinal 11,998, the replayed 591 plus 11,407 later ordinal values (592 through 11,998) remained unprojected: 11,408 physical records and 100,709,536 bytes while the canonical JSONL continued growing.

This is consistent with #35746's sequential reverse-scanner failure and does not require a concurrent-writer explanation; it does not exclude one.

No field values, credentials, identifiers, paths, conversation content, repository data, or raw files are included.

I can reproduce this issue on codex-cli 0.147.0 through Codex Desktop Remote SSH.

In-app feedback ID:
01a02e28-bdc3-7393-bda1-9cc2a6b4cd13

User-visible impact:
Multiple tasks retain their complete JSONL transcript and the agent can still use the later context, but after reconnecting the Remote SSH host or reloading the task, the desktop UI displays only the first user message and first agent response.

I inspected three independently affected threads. Their history projections stopped with these gaps:

• expected ordinal 16, got 17
• expected ordinal 18, got 19
• expected ordinal 17, got 18

In all three cases, the missing ordinal in the canonical JSONL is an event_msg/token_count record with non-null rate_limits and credits fields.

The logs repeatedly report:

failed to project durable rollout:
thread-store internal error:
thread history projection expected ordinal 17, got 18;
0 rejected rollout lines cannot cover that gap

This appears to confirm that the decoder/projection issue described here is still present in 0.147.0 and also affects Codex Desktop Remote SSH history rendering.

No session or SQLite files were manually edited or deleted.

Independent confirmation: upgrading from 0.146.0 to 0.149.0 does not repair the frozen projection

I independently reproduced this on a Codex Desktop Remote SSH thread whose remote host is Linux x64.

User-visible symptom

The conversation remained complete and usable while active. After opening the same thread from another computer, or reconnecting the original computer, the UI displayed only the first user message / first turn. The agent still retained the later context.

Affected thread

• session_meta.cli_version: 0.146.0
• history_mode: paginated
• canonical rollout: intact and valid JSONL, approximately 4.2 MB and still growing
• projected history: 1 turn (inProgress) and 4 items

Stored projection cursor:

next_rollout_byte_offset = 192258
next_rollout_ordinal = 17

The record beginning exactly at byte offset 192258 has ordinal 18, so the cursor is internally inconsistent.

Before upgrading, the 0.146.0 app-server repeatedly logged:

skipping rejected rollout line while projecting:
invalid type: map, expected f64

I then upgraded the installed CLI to 0.149.0 and cold-restarted the Remote SSH app-server. The old decoding error stopped, but the projection did not recover. The new app-server now repeatedly reports:

thread history projection expected ordinal 17, got 18;
0 rejected rollout lines cannot cover that gap

The projection remains frozen at the same byte offset with only the first turn visible.

This confirms that fixing the decoder does not repair an already inconsistent projection checkpoint. A rebuild from the canonical rollout appears necessary when the stored byte/ordinal cursor invariant fails.

Related recovery issue: #38792.

No rollout files, prompts, private paths, full thread IDs, credentials, or SQLite databases are included. No manual repair has been performed yet.

Another confirmation on Codex Desktop with bundled codex-cli 0.149.0-alpha.4.1
(history_mode = paginated, macOS), adding a trigger variant not yet listed here.

Trigger: the turn ended abruptly on a rate-limit stop — no task_complete and no
turn_aborted was recorded — leaving the rollout's final line as an event_msg/token_count.
This is the same tail condition as @munakaya's SIGKILL repro above, reached without killing
the process, so ordinary rate-limit interruptions can produce it during normal use.

Physical signature (same as reported throughout this thread):

ordinal 9561: event_msg/token_count
ordinal 9561: event_msg/thread_settings_applied <-- reused at resume
ordinal 9562: next record

Exactly one non-increasing ordinal in the file; the rest of the JSONL is valid and complete.

Impact: the projection wedged permanently at that byte offset. Every subsequent
thread-open and resume retried and failed — 2,843 repeated WARN entries of
thread history projection for <redacted> expected ordinal 9561, got 9561-class errors.
The Desktop UI silently renders the thread truncated at that point with no error surfaced to
the user; all later turns remain in the rollout. No conversation data was lost.

Confirming the writer side is still unfixed: on main today,
codex-rs/rollout/src/ordinal.rs still has Some(ScanOutcome::Rejected(_)) => continue
inside ordinal_state_for_rollout, so a rejected tail record is skipped and the previous
record's ordinal is reused. The last commit touching that file (3a6f747d7, 2026-08-11) is
unrelated. #36083 fixed decoding in thread_history_materialization.rs only, and #36188
added deferred-rejected-line tolerance in the projector — neither touches the reverse scanner.

The underlying asymmetry is a strict reader against a lenient writer: apply_projection
treats any ordinal mismatch as fatal for the whole transaction, while the resume path will
happily emit a duplicate ordinal. Two fixes are needed and they are independent:

1. ordinal_state_for_rollout must not skip rejected tail records when computing the next

ordinal (value-first decode as proposed in the issue body, or at minimum treat a rejected
line as ordinal-consuming rather than invisible).

1. The projector needs to self-heal or surface the failure. Today a wedged cursor is both

permanent and invisible — the user sees a plausible shorter thread and no error.

Environment: Codex Desktop, bundled core 0.149.0-alpha.4.1, history_mode: paginated

need fix asap

Independent Windows recurrence: duplicate resume ordinal freezes desktop history

I reproduced the same paginated-history failure on ChatGPT/Codex Desktop for Windows, and also verified a bounded recovery on a backup.

Physical rollout signature:

event_msg/token_count(689)
event_msg/thread_settings_applied(689)  <-- reused ordinal
event_msg/task_started(690)

The canonical JSONL was valid and contained no malformed lines. The only non-increasing ordinal was the repeated boundary ordinal. The local history projection repeatedly reported:

thread history projection expected ordinal 690, got 689

The user-visible effect was that the desktop UI, after reopening, showed only the old projected portion even though later messages, final answers, and completion events remained in the canonical rollout. The phone/remote view could still receive the completed work.

After making a cold backup, moving only the derived projection cursor across the duplicated thread_settings_applied record, and leaving the canonical JSONL untouched, the projected history advanced from 6 turns / 259 items to 11 turns / 760 items with SQLite integrity still ok. This confirms preserved durable data plus a stalled derived projection, rather than transcript deletion.

The same writer/resume signature recurred after the next cold restart at a later boundary:

event_msg/token_count(2045)
event_msg/thread_settings_applied(2045)  <-- reused again
event_msg/task_started(2046)

The projection then stopped at the new boundary. Therefore a narrow recovery can restore an already-affected projection, but the underlying ordinal-reuse bug remains capable of recreating the failure on a later restart.

This adds a current Windows/Desktop recurrence and shows that the product needs both:

1. a fix to resume ordinal discovery/writer state so it cannot reuse the final ordinal; and
2. a supported recovery/rebuild path for projections already stopped at a replayed boundary.

No raw rollout, SQLite database, prompt, message content, local path, host/user identifier, or private thread ID is included.

Independent confirmation on Windows Codex Desktop 26.818.5229.0, bundled codex-cli 0.149.0-alpha.4.1, with history_mode = paginated.

Observed one duplicated boundary ordinal:

255  event_msg/token_count
255  event_msg/thread_settings_applied
256  event_msg/task_started

The UI stopped at the old projection while the JSONL remained complete. Logs repeatedly reported expected ordinal 256, got 255.

After backup, changing only next_rollout_ordinal from 256 to 255 (leaving the byte offset and JSONL unchanged), then cold-restarting Codex, reprojected successfully to EOF: 2 → 20 turns, 91 → 807 items, next ordinal 2323, SQLite integrity ok.

This confirms the issue still occurs on the current Windows build and that accepting the replayed boundary ordinal can recover the preserved history.

Confirmed another current Remote SSH recurrence, with an additional orchestration failure mode.

Environment:

• Codex Desktop controlling a Linux x86_64 remote task
• affected session created by codex-cli 0.149.0
• currently installed CLI: 0.149.1
• paginated history

The complete canonical JSONL parses successfully. At the inspection snapshot it contained 25,222 records and exactly one non-increasing ordinal:

13764  event_msg/token_count
       rate_limits: object; primary and credits present
13764  event_msg/thread_settings_applied  <-- reused ordinal
13765  event_msg/task_started

A same-directory fork fails deterministically:

failed to prepare paginated fork:
thread-store internal error:
thread history projection expected ordinal 13765, got 13764

The durable rollout continued for another 11,459 records / about 69.2 MB after the duplicated boundary. No malformed JSONL records were found.

The additional impact is dangerous for task orchestration: the structured thread/wait APIs reported only an older turn as interrupted, while the canonical rollout contained a later active turn that was still executing tools and appending records. An orchestrator trusted the stale projected state, treated the task as unusable, and created a replacement standalone task in the same checkout. That can produce two writers against one working tree even though the original worker is still active.

The replacement task was observed before it made source edits, so no conflicting source changes were attributed to this incident. This nevertheless shows that projection divergence is not only a UI/fork failure; it can cause incorrect scheduling and duplicate workers.

Upgrading the installed runtime to 0.149.1 does not self-heal the already duplicated rollout or restore forkability. No transcript content, identifiers, paths, hostnames, repository names, or account data are included.

For anyone hitting the frozen-history-at-an-old-turn variant here: the physical fingerprint in the rollout itself is a repeated boundary ordinal (token_count at N, thread_settings_applied at the same N, then N+1). I shipped read-only detection for that signature today - npx --yes @shleddy/vetto rescue diagnose <rollout.jsonl> now emits DUPLICATE_ORDINAL_BOUNDARY / ORDINAL_REGRESSION straight from the JSONL, no SQLite required. Copy-only; it never writes inside the Codex state root.

Another recurrence is reproducible on Windows ARM64 with Codex Desktop 26.820.7780.0 and Codex CLI 0.149.1.

User-visible symptom:

• A long local task repeatedly appears to stop.
• The task API keeps returning an older turn as inProgress with completedAt=null.
• The canonical rollout remains writable and continues receiving newer turns, tool calls, and token-count events.

Read-only evidence at capture time:

• Canonical JSONL size: 331,596,132 bytes.
• Records carrying ordinals: 31,470.
• Exactly one ordinal discontinuity was found.
• Sequence around the failure:
• ordinal 24552: item_completed
• ordinal 24553: custom_tool_call_output
• ordinal 24554: token_count
• ordinal 24554 again: thread_settings_applied
• ordinal 24555: task_started
• The suffix after the duplicate continues normally for roughly 6.9k additional records.

This matches the existing failure mode: resume reuses the last token-count ordinal, then paginated history/lifecycle projection no longer follows the valid canonical suffix. In this recurrence the duplicate is especially visible at the transition from token_count to thread_settings_applied.

The canonical history is not lost, but UI/API projections expose stale lifecycle state and make subsequent work look interrupted. No rollout repair was performed while the writer remained active.

Current-main confirmation and tested fix

I audited this again on Linux x86_64 against current main at b592a0bf (2026-08-27).

The forward rollout loader and thread-history projector now use the value-first decode_rollout_line(Value) path added by #38399, but three reverse readers still bypass it:

• rollout/src/ordinal.rs
• thread-store/src/local/model_context.rs
• tui/src/resume_picker_transcript_preview.rs

They call ReverseJsonlScanner::scan_next::<RolloutLine>(), which directly deserializes the flattened envelope. With serde_json/arbitrary_precision, a structured token_count containing a fractional used_percent is rejected with invalid type: map, expected f64.

A read-only audit found ten affected local sessions. Every one had the same physical boundary:

ordinal N:   event_msg/token_count with structured rate_limits
ordinal N:   event_msg/thread_settings_applied written by resume
ordinal N+1: next normal record

The canonical JSONL remained valid and continued growing. In one case, roughly 269 MB / 60,403 valid later records were preserved behind the frozen projection. This is data invisibility in the derived projection, not transcript deletion.

The causal sequence is:

1. Reverse ordinal discovery rejects the valid final token_count.
2. It falls back to ordinal N-1.
3. Resume emits thread_settings_applied as ordinal N, reusing the token-count ordinal.
4. Projection already expects N+1, sees the replayed N, and permanently refuses the valid suffix.

Suggested fix

1. Add one canonical ReverseJsonlScanner::scan_next_rollout_line() method that scans into serde_json::Value and then calls decode_rollout_line.
2. Use it in ordinal discovery, model-context recovery, and the legacy transcript preview so reverse and forward persistence readers cannot drift again.
3. Add a resume regression with a fractional structured token_count tail and assert that the next written ordinal is N+1.
4. Recover the exact already-written resume signature without rewriting canonical JSONL: only when the first decoded record at the projection checkpoint is ThreadSettingsApplied at exactly next_ordinal - 1, with no preceding rejected line or projected step, consume exactly one record's bytes without advancing the ordinal. Continue projecting at next_ordinal; keep every other ordinal regression fatal.

That recovery is narrowly safe for thread history because project_rollout_line() maps ThreadSettingsApplied to an empty change set: it contains no user/assistant item or turn transition. It unblocks preserved history without silently tolerating arbitrary duplicates. Older projections exhibiting the other legacy shape—cursor expects N but begins at N+1 because a decoder skipped a line and advanced the byte offset—should be rebuilt from canonical JSONL rather than guessed forward.

A tested implementation is available here:

• branch: https://github.com/khoek/codex/tree/fix/preserve-resumed-history
• commit: https://github.com/khoek/codex/commit/e4602e8b4294fe4dc9d1c9eb4613d55da06e3afc

Validation on current main:

• just test -p codex-rollout: 125 passed
• just test -p codex-thread-store: 237 passed
• targeted resume-preview suite: 9 passed
• scoped Clippy fixes and just fmt: clean

Independent Windows controls on 0.150.0-alpha.8 / 0.150.1, plus recovery evidence

Adding new Windows x64 evidence to this existing issue rather than opening a duplicate. The reverse-reader diagnosis in @khoek's analysis above matches our independent investigation.

Controlled native app-server tests used isolated copies, one writer, and a local synthetic completed-response fixture, not requests to a real model provider. The affected input contained 14,328 complete JSONL records:

| Input / runtime | First appended ordinal | New user and assistant markers survive reopen? |
|---|---:|---|
| Original terminal structured token_count; bundled servers from Desktop 26.820.9563.0 and 26.820.10647.0 (0.150.0-alpha.8) | 14,327 (reused) | No |
| Same original input; official CLI 0.150.1 | 14,327 (reused) | No |
| Remove only the final service event in a test copy (14,327 records), either bundled server | 14,327 (correct) | Yes |
| Keep all 14,328 records, but serialize the whole-valued used_percent as 12 instead of 12.0; original bundled server | 14,328 (correct) | Yes |

All pre-existing message anchors remained readable in these controls. In the failing cases, thread/resume and turn/start did not return an error: the new records were written but the reopened projected history omitted them.

The lexical-number control is particularly useful: it preserves the final event and its numeric value, and isolates decoding from a general multi-writer/race hypothesis. At rust-v0.150.1, commit 9085439, ordinal discovery still calls the typed reverse scanner, skips rejected records, and derives the next ordinal from an earlier record. This is consistent with the flattened-envelope / arbitrary_precision issue already described here.

Separate recovery result: after updating to official Desktop 26.825.3734.0 / bundled 0.150.0-alpha.12.2, previously stalled histories still required recovery. A cold, backed-up, guarded offline reconstruction restored 13 affected histories (four parent tasks and nine existing subagents), adding 13,325 projected items, not 13,325 user messages. Post-install read-only checks verified the installed journal hashes, exact per-thread projection rows, both SQLite quick_check results, and readable histories through the running app API. Original records and child identities were preserved. No custom-compiled server was installed.

That recovery is not evidence that fresh ordinal reuse is fixed or still reproducible on alpha.12.2; the controlled failure matrix above applies only to the explicitly tested earlier binaries. It does show why updating the writer alone is not sufficient for already-stalled histories.

Please include both a canonical reverse-decoder fix and a supported, data-preserving recovery path for existing projections/subagents in a shipped release. Arbitrarily tolerating all duplicate ordinals would not be safe. An actionable UI error or official doctor/rebuild operation would also prevent users from mistaking stale projections for deleted work.

No raw transcripts, databases, private task IDs, local paths, credentials, or screenshots are attached. The test-copy edits above are diagnostic controls, not instructions to edit a live profile.

New 0.150.1 recurrence: task_started itself reuses the terminal token_count ordinal

This is another recurrence in a previously recovered long-running paginated task. The current duplicate was created after the earlier recovery. No rollout, SQLite state, or process was modified during this evidence capture.

Environment

• Windows 11 ARM64 Desktop client: 26.825.4187.0
• Desktop-bundled runtime: 0.150.0-alpha.12.2
• Remote Linux x86_64 app-server: 0.150.1
• Session originally created by 0.147.0
• History mode: paginated

Exact stalled boundary

The canonical rollout currently contains 707,757,918 bytes and 79,078 physical lines. The projection state is frozen at:

next_rollout_byte_offset = 619432687
next_rollout_ordinal     = 70387

The exact valid JSON records at the boundary are:

line 70392  offset 619428701  ordinal 70385  response_item/custom_tool_call_output
line 70393  offset 619431873  ordinal 70386  event_msg/token_count
line 70394  offset 619432687  ordinal 70386  event_msg/task_started   <-- reused
line 70395  offset 619432936  ordinal 70387  turn_context
line 70396  offset 619433818  ordinal 70388  response_item/message

This differs from the previously common signature:

token_count(N)
thread_settings_applied(N)
task_started(N+1)

Here the new task_started record directly reuses the prior token_count ordinal. The token-count record was written at 2026-08-26T23:25:12.228Z; the duplicated task start was written at 2026-08-27T07:09:57.713Z.

A complete read-only ordinal scan found exactly one duplicate and no gaps among parseable records. Six older malformed/non-JSON physical lines exist earlier in the file, but the projector had already advanced past all of them. They do not explain this later stop.

Preserved but invisible answer

The rollout continued normally through ordinal 79070. The newest persisted turn contains:

• a user message;
• reasoning and tool events;
• an assistant message with phase=final_answer;
• a final token-count record;
• event_msg/task_complete at 2026-08-28T11:05:06.141Z.

The thread reports idle, so that turn is terminal. However, Desktop and the native task-read projection still expose only older turns. The unprojected suffix is 88,325,231 bytes, approximately 84.2 MiB, covering about 8,684 ordinals.

The live writer logs the same failure for every later persist and final flush:

thread history projection expected ordinal 70387, got 70386

Therefore, the missing answer is not model failure or transcript deletion. It is present in the durable rollout and hidden by the stalled derived projection.

Impact

• A completed answer is invisible in Desktop.
• Native task reads return stale lifecycle/history.
• Every later append repeats projection warnings.
• Further work can continue in the canonical rollout while UI/API consumers remain frozen on older state.
• A user may resend work that already completed.

Fix direction

1. Ordinal allocation for a new turn must use the last physically valid rollout ordinal even when the terminal record is a structured token_count.
2. task_started must never reuse an existing ordinal, even if resume-time metadata is not emitted.
3. The projector should recover or expose a bounded corruption error instead of permanently freezing every later valid record.
4. Add a regression with terminal structured token_count(N) followed directly by a later task_started; assert that the latter is N+1.
5. Expose projection lag in Desktop so a preserved final answer is not presented as absent.

This extends the current-main reverse-reader evidence already documented above: the ordinal-reuse bug is still reproducible on stable 0.150.1, and thread_settings_applied is not required for the bad boundary.