Codex leaves valid rollout files unindexed in state DB and lacks supported reindex repair

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #30347
• #30385
• #30163

Powered by Codex Action

Thanks for the duplicate suggestions. I reviewed #30347, #30385, and #30163.

My current reading is that this issue is related but not an exact duplicate:

• #30347 reports a lost Project conversation after an HTTP 429 where a persisted rollout cannot be reconstructed by the UI.
• #30385 reports Desktop sidebar/search not showing recent local project threads even though session_index.jsonl and direct read_thread can load them.
• #30163 reports recent local threads being unexpectedly archived/hidden after an update, with manual repair by moving archived rollout files and changing archive metadata.

This report is specifically about codex doctor / state.rollout_db_parity finding rollout files that exist on disk but have no matching rows in state_5.sqlite, while all SQLite DB integrity checks pass. In this environment the mismatch includes current/recent Windows Desktop + WSL-agent sessions, and the doctor data shows:

rollout DB active files  166
rollout DB archived files 12
rollout DB rows          121
rollout DB active rows   116
rollout DB archived rows 5
rollout DB missing active rows 50
rollout DB missing archived rows 7
rollout DB stale rows    0

A local read-only reconciliation against state_5.sqlite reproduced the same 57 missing rows exactly:

Rollout files total: 178
Indexed according to DB comparison: 121
Missing according to DB comparison: 57
Missing by kind: active=50, archived=7
Missing by month: 2026-05=13, 2026-06=34, 2026-07=10

So the overlap is local thread inventory drift, but the concrete direction here is: rollout files exist, DB rows are missing. This is distinct from stale DB rows pointing to missing rollout files and from UI/sidebar-only visibility problems. A supported repair/reindex command would likely help all of these cases, but I do not think this issue should be closed as a direct duplicate unless maintainers want to consolidate all state/rollout parity drift under one canonical issue.

Additional impact note: this is not just a cosmetic codex doctor warning in my setup.

When the thread/rollout inventory is in this state:

threads: rollout files are missing from the state DB

Browser Use in the Codex Desktop app is affected and does not work reliably. That makes the issue more severe than a stale-index warning, because Browser Use is a core Desktop App workflow.

The important distinction is that the Doctor warning may look optional/degraded, but the user-visible behavior is loss of a core capability. Please treat this as a functional regression in the Desktop App state/thread inventory path, not only as a cleanup/reporting issue.

Version update: this still reproduces after updating Codex from 0.142.5 to 0.143.0.

So the issue is not limited to the originally reported 0.142.5 install. After the update, codex doctor --summary still reports the thread/rollout inventory warning, and the Browser Use impact remains relevant.

Confirmed on macOS with codex-cli 0.145.0 (upgraded from 0.144.6). This is not Windows/WSL-specific.

codex doctor --json reports:

state.rollout_db_parity: warning
active rollout files: 2429
active DB rows: 2253
missing active rows: 176
missing archived rows: 0
stale rows: 0
duplicate DB paths: 0
duplicate rollout thread IDs: 0
scan errors: 1 (one legacy rollout has no usable header record)
state DB integrity: ok

Before upgrading, a read-only filesystem/SQLite comparison on 0.144.6 found 177 rollout IDs absent from state_5.sqlite, with no DB row pointing to a missing rollout. The one-count difference is the malformed-header rollout that 0.145.0 doctor correctly excludes from the valid inventory.

The user-visible symptom was a burst of identical warnings while the Desktop/App host resolved historical thread IDs:

WARN codex_rollout::list: state db discrepancy during find_thread_path_by_id_str_in_subdir: falling_back

The upstream source explains this precisely: find_thread_path_by_id_str_in_subdir falls back to the rollout filesystem when the DB lookup has no row, then calls read_repair_rollout_path for the accessed thread. This repairs only threads that happen to be resolved; codex doctor still exposes no supported bulk repair/reindex action, so 176 valid active rollouts remain unindexed after the upgrade and doctor run.

For comparison, two isolated long-lived Codex profile homes on the same machine are fully consistent (109/109 and 17/17), which rules out general filesystem permissions or SQLite integrity as the cause.

Requested acceptance criteria remain: either prevent valid rollouts from being left unindexed, or provide a supported idempotent repair/reindex command that reconciles every valid rollout without direct user SQL edits and verifies parity afterwards.

same happened to me, experienced on macos with 0.145 stable and codex desktop 26.715.70719. ~~@atlantho could you pls rename the the issue and put relevant tags?~~

nvm, mine issue is a bit different as I see now when I dug deeper, the log is same though. codex doctor shows no warnings, but notes

Notes
   ⚠ rollouts     877 active files · 1.33 GB on disk

I updated the title to make the scope platform-neutral now that there is a confirmed macOS reproduction with the same rollout DB parity direction.

Current title:

Codex leaves valid rollout files unindexed in state DB and lacks supported reindex repair

I cannot apply repository labels myself, but based on the current evidence I think the issue should probably be labeled with:

• "macOS", because come2ry confirmed the same "state.rollout_db_parity" warning on macOS with valid rollout files missing from "state_5.sqlite".
• "regression", because multiple reports tie the mismatch to upgrades/update-time state drift.

I would keep "windows-os" as well, because the original reproduction remains Windows Desktop + WSL-backed runtime. The issue is now best understood as cross-platform rollout/state DB parity drift rather than Windows-only behavior.

I still do not think #30347, #30385, or #30163 are exact duplicates. They overlap in the local thread/session inventory family, but this report is specifically about valid rollout files existing on disk while the state DB lacks corresponding rows, with SQLite integrity otherwise OK, and no supported bulk repair/reindex command available.

Doctor already computes the repair diff - the missing surface is a fixer, not a detector

Hit a related state.rollout_db_parity: warning on macOS in July 2026. The implementation-cost angle seems under-raised in the thread, so sharing what I found.

codex doctor's state.rollout_db_parity check already tallies the exact diff a repair command would apply - missing_active/missing_archived/stale_rows/duplicates/scan-errors (computed via missing_rollout_paths in thread_inventory.rs:240-241) - but the missing-rows DoctorIssue (:369-382, "rollout files are missing from the state DB") carries .measured/.expected and no .remedy, while the missing-state-db (:196), duplicate (:417), and permission (:215/:433) cases all carry one. run_doctor is contractually read-only (doctor.rs:304, "It does not repair issues"), DoctorCommand (:151) exposes only json/summary/all/no_color/ascii flags, and enum Subcommand (main.rs:123) has no Reindex/Repair/Sync variant. So the work reads less as "build a reconciliation engine" and more as expose the diff doctor already computes to a fixer loop over reconcile_rollout (state_db.rs:486, the per-file upsert primitive). read_repair_rollout_path (:564) already does per-thread lazy read-repair (as @come2ry also observed) - a bulk reindex would be the eager equivalent. The diff is the plan, but not blindly applicable - it inherits doctor's classification gaps (see Scope) - and backfill's terminal mark_backfill_complete (backfill.rs:82) would need a reset/re-arm path for a reindex to revisit files already marked complete.

What the local incident showed (evidence, not a prescription)

My incident was path drift - a CODEX_HOME / session-directory change left rollout_path entries dangling - not corruption (PRAGMA integrity_check was ok throughout). This may be a different trigger than #31433's upgrade-associated missing-rows; I'm citing it only as independent corroboration of the diagnostic and the cost argument.

• integrity_check was ok - the DB was structurally fine; the gap was file/path drift, not corruption. A genuinely-malformed-DB case (#27363) would need a different path.
• Doctor reported stale_rows = 934 (931 active rollout files whose recorded path no longer resolved + 3 genuinely-dead archived rows). Path drift inflates both stale_rows and missing_active in lockstep - the DB row points at a stale location while the file sits at the new one.
• A repair must run against live state and re-verify: an earlier same-day snapshot reported 1,386 stale / 1,383 recoverable; live state at repair time showed 934. Counts drift while sessions run.
• Applying doctor's computed diff cleared the warning: stale_rows 934 -> 0; active files/rows 2218/2218; archived 2/2; missing_active/missing_archived = 0; integrity_check = ok; wal_checkpoint(TRUNCATE) = 0|0|0. The 931 files were copied back to the subpaths their DB rows expected (0 overwrites); the 3 dead archived rows were removed.
• What was not done: no DB file deletion, no bulk row delete, no overwriting of existing active rollouts, no trusting of the earlier snapshot's counts. Backups were taken before any write (ephemeral-directory backups do not survive reboot on macOS - a supported command should write to a durable location).

To be explicit: this was a private one-off and not a recommended fix. Manual per-file reconciliation does not scale and is unsafe for most users. The point is that doctor's diff was the repair plan - the outcome validates the cost argument above, not a methodology to replicate.

A genuinely-dead-row safety check I used (proposed, not a verified invariant - there may be reference tables I didn't check): a row is a deletion candidate only if its file is found at no accessible location and it has 0 references in thread_spawn_edges and 0 in thread_dynamic_tools, both verified zero first.

Two unconfirmed root-cause candidates

I could not attribute #31433 to a specific code change (source search indexes files, not issue/PR histories). Two behaviors confirmed in main as of 2026-07-24 could produce missing-rows under different signatures:

• (a) Backfill is terminal. backfill_sessions_with_lease (metadata.rs:154) early-exits when status == Complete (:174) and retains only files with watermark > last_watermark (:243); mark_backfill_complete (backfill.rs:82) is terminal. Signature: a rollout missing at completion time stays permanently unindexed - rows never appear.
• (b) list_threads_db drops rows. When existing_rollout_path (state_db.rs:427) returns None, it calls ctx.delete_thread (:439), emitting stale_db_path_dropped (:438). Signature: rows appear then disappear. My path-drift incident independently supports this direction - a moved file makes the path lookup fail, so the row gets dropped on the next listing.

These behaviors are confirmed in main; whether either caused #31433's missing-rows is the hypothesis. Line numbers will drift; symbol names are the stable anchors. Static reading only - I could not build/run codex-rs.

Scope

A reindex fixes file<->DB parity. It does not fix DB-vs-UI listing drift (#27159/#25463/#30312) or creation-time materialization failures where a rollout was never written (#32728/#30924; #33120 is archive-time, also out of scope). It is distinct from #31074's stale-rows direction (DB row exists, file gone) - #31433 is missing-rows (file exists, no DB row) - though both are drift directions of the same gap.

A reindex must also correctly ingest legacy pre-session_meta rollouts (#26877 - doctor currently flags valid legacy rollouts as scan errors) and must not act on sandbox-constrained false-corruption signals (#33145), or it inherits doctor's existing classification bugs and becomes unsafe to run. #28068 closes the obvious escape hatch: deleting the state DB regenerates it but does not back-scan historical rollouts - no supported path exists, not even a destructive one.

Happy to expand any of the above as design discussion if useful. A full redacted codex doctor --json from the incident is available privately.

Update: root cause narrowed — structural non-transactional gap, exposed by #22236 + #22899

Following up on my earlier comment, a deeper source trace narrows the root cause. The two candidates I flagged there — terminal mark_backfill_complete (a) and list_threads_db dropping rows via delete_thread/stale_db_path_dropped (b) — are both refuted by the evidence below. The actual mechanism is a non-transactional write gap, amplified by two May-2026 changes.

Mechanism (longstanding, new sessions). thread_start (thread_processor.rs::thread_start_task) never eagerly creates a threads-table row — it only calls thread_watch_manager.upsert_thread_silently, which is in-memory (ThreadWatchState HashMap, zero SQLite). Row creation is deferred to the first turn's append_items (live_thread.rs), which performs two independent, non-transactional durable operations in sequence:

1. persist_appended_items → writes the JSONL rollout file to disk;
2. observe_appended_items → update_thread_metadata → apply_metadata_update → state_db.upsert_thread → creates the row.

A crash/quit between (1) and (2) leaves exactly the reported state: rollout file present, threads-table row absent. read_repair_rollout_path repairs it lazily per-thread (the falling_back path @come2ry's log shows), which is why per-thread reindex works but there is no bulk repair.

Amplifier — #22899 (2026-05-15, "Soften SQLite metadata sync failures"). The first-row patch carries observed/create-time metadata (created_at/source/cli_version/git_info, title/preview/token_usage/cwd), so has_observed_metadata_facts makes require_sqlite_write = sqlite_write_failure_should_block(patch) || paginated evaluate FALSE. Any transient upsert_thread failure (e.g. SQLITE_BUSY under multi-process contention) therefore hits the warn-only swallow arm (update_thread_metadata.rs) → returns Ok(()), and mark_pending_update_applied clears the create-time patch — created_at/source/cli_version/git_info are permanently lost. The swallow is silent: warn! only, no telemetry/counter/retry. This is not a guess about intent: #22899's own root-cause note states the append-derived SQLite writes "now propagated through the live thread flush path, so a corrupted optional state DB could surface as a transcript persistence warning even when JSONL writes still succeeded" — i.e. JSONL (rollout file) succeeds while the SQLite row fails, by design best-effort.

Introducer — #22236 (2026-05-13, "Unify thread metadata updates above store"). This moved "the implicit metadata updates triggered by appending rollout items" out of RolloutRecorder (local-threadstore-specific) into a ThreadMetadataSync helper at the LiveThread layer, with LiveThread calling append_items and update_metadata separately (quoting the PR). The previous RolloutRecorder path ran sync_thread_state_after_write on every resumed flush and (re)created a missing threads-table row from the rollout file's own contents (builder_from_items → touch_thread_updated_at → on miss apply_rollout_items → upsert_thread) — an independent self-healing backstop. Post-#22236, RolloutRecorder is a pure JSONL writer; sync_materialized_rollout_path only UPDATEs existing rows (early-returns Ok when get_thread is None). So nothing recreates the row at resume — a transient one-session miss becomes permanent, accumulating across launches.

Timeline fit. #22236 merged 2026-05-13, #22899 merged 2026-05-15. @atlantho's missing-by-month (57 missing: 2026-05=13, 06=34, 07=10) starts exactly at #22236. @come2ry found 177 missing on 0.144.6 before upgrading — consistent with "upgrade-exposed, not introduced": the structural gap was always present, but #22236 removed the next-launch self-heal that previously masked it, so misses began accumulating permanently from May.

Earlier candidates refuted.

• (a) terminal mark_backfill_complete: backfill is a one-shot startup scan, terminal after claiming, and covers only pre-existing files — it cannot explain same-day current-session loss. The 30s STARTUP_BACKFILL_WAIT_TIMEOUT is also not the cause: try_init returns Err on timeout and the app-server fails to start rather than running with state_db=None.
• (b) list_threads_db dropping rows: stale_db_path_dropped deletes only rows whose rollout file is missing (opposite direction); both reports show stale_rows = 0, and missing threads retain rollout files. falling_back is the non-destructive read_repair UPSERT, not a delete.
• Ephemeral subagents: ephemeral threads produce no rollout file and structurally cannot enter the doctor's missing set — missing_rollout_paths iterates on-disk scan files, so all 57 missing have rollout files by definition.

Caveat. Whether the missing set is dominated by resumed sessions (favors #22236) or new sessions (favors the longstanding non-transactional gap) is not distinguishable from source alone. The closing check needs per-file data: a thread_history DB entry for that thread_id predating the rollout file's mtime ⇒ resumed; or the rollout's first SessionMeta.created_at equal to the file's ctime ⇒ new. Either way the fix surface is the same.

Fix surface (builds on the earlier comment). Doctor already computes missing_rollout_paths; a bulk reindex loop over reconcile_rollout/read_repair_rollout_path is still the eager equivalent of the lazy repair. Two additions the trace suggests:

• a telemetry counter on the warn-only arm to categorize the Internal errors (busy/locked/no-such-column/full/readonly) seen in the field — currently fully silent;
• (harder) make the first-row SQLite upsert transactional with the rollout flush, or re-add a resume-time self-heal equivalent to the pre-#22236 sync_thread_state_after_write.

Static source-level as of 2026-07-24; line numbers will drift, symbol names are the stable anchors. Happy to refine if maintainers can share which upsert error dominates in the wild.

Linux confirmation on Codex 0.146.0

This also reproduces on a long-lived Forge host using the standalone Linux x86_64 build of codex-cli 0.146.0. A fresh, sanitized codex doctor --json run on 2026-07-31 reports:

state.rollout_db_parity: warning
state DB integrity: ok
active rollout files: 1542
active DB rows: 671
missing active rows: 871
archived rollout files: 374
archived DB rows: 236
missing archived rows: 138
stale rows: 0
scan errors: 0
malformed file names: 0
duplicate rollout thread IDs: 220
duplicate DB paths: 0
sources: subagent:thread_spawn=492, vscode=370, exec=45

This environment contains both .jsonl and preserved .jsonl.zst rollouts. Current tasks, the app-server, and SQLite integrity checks still work, so the warning is inventory drift rather than general database corruption.

The duplicate-ID dimension is important. A supported repair cannot only insert the 1,009 missing rows. It also needs deterministic handling for multiple rollout files carrying the same thread ID, while preserving the authoritative transcript and avoiding data loss.

Requested repair behavior:

1. Scan active and archived .jsonl and .jsonl.zst rollouts.
2. Rebuild missing threads rows using the same supported reconciliation logic as lazy read repair.
3. Resolve or explicitly quarantine duplicate thread IDs deterministically.
4. Run transactionally or create a recoverable backup, ideally with a dry-run mode.
5. Re-run parity verification and report the remaining exceptions.
6. Prevent new drift by retrying or self-healing failed metadata upserts.

codex doctor, codex debug, and codex app-server in 0.146.0 still expose no supported repair or reindex command. I did not modify the live SQLite database manually and omitted paths, titles, rollout contents, and sample thread IDs from this public report.

I’m testing Codex Rescue 0.1.0a3 against real local-state/index drift. This looks like a useful case because the rollout files exist and are valid, while state_5.sqlite is missing rows.

Rescue discovers sessions from local rollout data and does not mutate Codex SQLite state, so I’d like to see whether it can still enumerate/diagnose one of the unindexed sessions independently of the broken inventory.

If you still have the affected state, would you be willing to try:

pipx install codex-rescue==0.1.0a3
codex-rescue sessions
codex-rescue doctor --latest

If it finds the missing session, sanitized salvage --fork / verify output would help. Please don’t share raw rollouts, SQLite DBs, prompts, credentials, or private paths.

https://github.com/shleder/codex-rescue

Additional cross-OS reproduction that looks like the same underlying state-DB reconciliation problem.

Environment

• Windows + Ubuntu dual boot
• ChatGPT Linux package: 26.810.52044 (amd64)
• The same physical NTFS volume is used as shared CODEX_HOME
• Separate local CODEX_SQLITE_HOME per OS to avoid SQLite/WAL on NTFS

Example layout:

Windows:
CODEX_HOME=D:\.codex
CODEX_SQLITE_HOME=C:\Users\<user>\.codex-sqlite

Linux:
CODEX_HOME=/run/media/<user>/<shared-volume>/.codex
CODEX_SQLITE_HOME=/home/<user>/.codex-sqlite

Reproduction

1. Create a new Codex thread on Linux and send a message.
2. Reboot into Windows and open Codex.
3. The newly-created Linux thread is not shown in the Windows thread list.
4. Rebuilding/deleting the local Windows SQLite state causes the thread to appear, and the thread contents/messages are correct from the shared rollout files.
5. Create another new thread on Windows and send a message.
6. Reboot into Linux and open Codex.
7. The new Windows thread is not shown in Linux either.

The shared rollout/session files are present and the actual messages are usable once the local state DB is reconciled. The issue is that the already-completed local state DB does not notice newly-added rollout files written by the other OS/client.

Confirmed workaround

On Linux, after switching from Windows, this makes the missing new thread appear without deleting the SQLite files:

sqlite3 ~/.codex-sqlite/state_5.sqlite \
  "UPDATE backfill_state SET status='pending', updated_at=0 WHERE id=1;"

Then launch Codex normally. Keeping last_watermark intact causes an incremental backfill, and the newly-added rollout is indexed. After completion:

SELECT status,last_watermark,last_success_at FROM backfill_state;

returns complete|... and the cross-OS thread appears.

A full local DB rebuild also works, but obviously should not be necessary.

Related startup behavior

While testing a fresh local CODEX_SQLITE_HOME, I also hit:

state db backfill is running at <shared CODEX_HOME>; waiting up to 30s
Codex app-server initialize handshake timed out

After allowing the bundled Codex CLI to complete the initial backfill once, the Desktop app started normally.

Expected behavior

When valid rollout/session files are added to CODEX_HOME by another Codex client / another OS, Codex should automatically reconcile the local state DB on startup (preferably incrementally from the previous watermark).

Alternatively, please expose a supported command such as:

codex reindex
codex doctor --repair-threads

so users do not have to delete SQLite databases or manually modify backfill_state.

This use case makes the underlying issue very visible because the shared CODEX_HOME is intentionally common while the SQLite state is necessarily local to each OS.

The key reframe here: the rollout JSONL is the canonical append-only record and the state DB thread inventory is a derived projection. When codex doctor reports rollouts missing from the index but the files exist on disk, that's projection lag or a missed index write — not data loss. The sessions are intact; only the derived lookup table is stale.

The reason there's no safe reindex today is that rebuilding the projection from rollouts requires deciding which rollout fields map to which index columns (title, preview, project, archive state), and doing it wrong can clobber good rows. A read-only reconciliation pass first — enumerating on-disk rollouts, diffing each against its index row, and reporting exactly which fields diverge — gives you the repair manifest without touching anything.

Full disclosure: I maintain codex-rescue (https://github.com/shleder/codex-rescue). codex-rescue diff --latest reconciles rollout vs SQLite vs session_index read-only and flags exactly this projection divergence, and sessions enumerates on-disk rollouts against the indexed set so you can see the unindexed remainder. It never mutates the state DB, so it's a safe way to produce the reconciliation report that a future supported reindex could consume. Happy to share the divergence output if it helps scope the repair command.

Our Windows cleanup case reinforces this projection boundary: rollout JSONL is the canonical record, while state DB history tables are derived projections. Safe repair should first produce a read-only file/DB/ordinal diff, then mutate only with a backup and exact postconditions; this connects to #35746, #40178, and the agent-state drift in #37042.

Additional Windows evidence for the missing repair path. Codex Desktop 26.818.8289.0 left valid raw rollouts with projection cursors stopped at duplicate ordinal boundaries. There was no supported reindex or repair command, so recovery required a backup-first, narrowly guarded SQLite cursor repair while preserving every raw JSONL file. The repaired database passed integrity checks and the app-server reconnected.

A supported doctor or reindex command should detect and reconcile these ordinal and byte-offset mismatches without requiring manual SQLite edits.

This gap - valid rollouts the index never picks up - is exactly why I added an explicit bounded filesystem walk: vetto rescue --json scan --all reads the session roots directly instead of trusting the provider index, so unindexed-but-intact transcripts show up. Read-only, fails closed on unreadable roots rather than silently skipping them.