Desktop (Windows): corrupt logs_2.sqlite B-tree causes a deterministic, unrecoverable app-server handshake timeout

Open 💬 2 comments Opened Aug 17, 2026 by AS-1-Dev

What version of the Codex App are you using?

26.810.7004.0 — MSIX package OpenAI.Codex_26.810.7004.0_x64
ChatGPT.exe FileVersion 151.0.7922.137, built 2026-08-16

What subscription do you have?

Pro 20x

What platform is your computer?

Microsoft Windows NT 10.0.26200.0 x64

What issue are you seeing?

Codex App failed to launch on every attempt with a blocking modal:

ChatGPT failed to start.
Codex app-server initialize handshake timed out
[ Check for Updates ]  [ Quit ]

Root cause: ~/.codex/logs_2.sqlite had a corrupt B-tree. Not size, not disk space, not a broken install, not lock contention.

Verified by opening the main database file with immutable=1 — WAL bypassed, no locking, no other process participating — so this is not an artifact of the live WAL or of a second app-server holding the file:

PRAGMA quick_check;
*** in database main ***
Tree 5 page 549189 cell 1:  Rowid 341223817 out of order
Tree 5 page 546412 cell 1:  Rowid 341223769 out of order
Tree 5 page 548159 cell 21: Rowid 341224892 out of order

SELECT COUNT(*) FROM logs;  ->  database disk image is malformed

Every other database in CODEX_HOME passed quick_check cleanly, including state_5.sqlite (675 threads). Exactly one file was damaged and it was the non-essential telemetry log — yet it prevented the entire application from starting.

Two log signatures were observed, both from this one cause:

A — majority of launches

initialize_handshake_result durationMs=30012 outcome=failure
  errorMessage="Codex app-server initialize handshake timed out"
app_server_connection.state_changed cause=initialize_handshake_timeout
  connectionError={"code":"restart-required"}
Desktop bootstrap failed to start the main app phase=bootstrap-import-main

B — occasionally, the actionable error surfaced instead

app_server_connection.closed code=1 reason="Error: failed to initialize sqlite
  state runtime under <USERPROFILE>\.codex"
initialize_handshake_result durationMs=5948 outcome=failure

Signature B appeared in only 1 of 6 failed launches. The other five gave no indication a database was involved.

What steps can reproduce the bug?

Preconditions: ~/.codex/logs_2.sqlite has a corrupt B-tree. On this machine it appeared after upgrading OpenAI.Codex 26.707.8479.0 → 26.810.7004.0 (crash dumps from the earlier build are present, dated 2026-07-15). Same origin as #23917.

  1. Launch Codex App.
  2. app-server spawns and begins startup work against the damaged log DB.
  3. At 30.0 s the shell aborts the handshake and shows the modal.
  4. Repeat — identical result, indefinitely.

**The key behaviour — the WAL is regenerated each launch, not replayed.** This is what distinguishes this from the existing "large WAL" reports:

10:00:34   543,872 B          132 frames      (healthy, pre-launch)
10:08:29   Desktop launched
10:08:37   codex.exe spawned
10:09:15   1,129,127,232 B    274,060 frames  (+1.05 GB in ~38s)
10:09:39   stable — work finished

An earlier failed launch that morning produced 274,306 frames. Two independent runs, ±0.1%. This is deterministic work against a fixed damaged structure, not accumulated history.

The timing margin — it misses by ~2 seconds, every time:

14:08:37.768   transport start — 30s timer begins
14:09:07.783   timeout fires        (30.014s)  -> shell aborts, modal shown
14:09:09.860   codex.exe responds   (32.09s)   -> 2.1s too late

The app-server does complete the handshake. It is killed ~2 s short on every attempt. Because the shell aborts mid-work nothing commits, the damage is never addressed, and the next launch performs identical work. This is why the condition is permanent rather than transient — it structurally cannot self-heal.

Confirmed: checkpointing does NOT fix this. Tested inadvertently — an unrelated checkpoint reduced the WAL to 33 frames; the very next launch regenerated 274,060 frames and failed identically. A corrupt B-tree cannot be checkpointed into health.

Any user can self-diagnose in ~30 seconds, non-destructively:

py -3 -c "import sqlite3,os; p=os.path.expanduser(r'~\.codex\logs_2.sqlite'); c=sqlite3.connect('file:'+p.replace('\\','/')+'?immutable=1',uri=True); print(c.execute('PRAGMA quick_check(3)').fetchall())"

immutable=1 matters: it bypasses the WAL and all locking, so the result reflects the file itself and is unaffected by any running app-server.

Workaround: with the app fully exited (see tray note below) and any IDE Codex extension closed, move logs_2.sqlite, logs_2.sqlite-wal and logs_2.sqlite-shm aside together. The app rebuilds an empty log DB.

| | before | after |
|---|---|---|
| logs_2.sqlite | 2,439,012,352 B, corrupt | 49,152 B, healthy |
| logs_2.sqlite-wal | 274,060 frames | ~1,000 frames |
| handshake | 32.1 s vs 30 s → killed | window visible in 1.7 s |

No chats, memories, threads or settings are lost — none live in this file.

What is the expected behavior?

  1. A corrupt or oversized telemetry log should not be fatal to the application. If state_5.sqlite opens successfully, the app should start. Log-database init/maintenance should not be blocking on the launch path. This alone would have prevented the entire outage.
  2. On detecting SQLITE_CORRUPT in the log DB, rotate it — rename aside, create a fresh one — rather than retrying an operation that cannot succeed. The current behaviour is an infinite loop with a 30-second period.
  3. Run log retention/maintenance asynchronously, after the handshake completes.
  4. Surface the underlying SQLite error in the UI. "Handshake timed out" gave no hint a database was corrupt, and the modal's only options were "Check for Updates" and "Quit" — neither of which can resolve this.
  5. Cap and rotate logs_N.sqlite. 2.27 GB of trace logs is not a useful diagnostic artifact.
  6. Add an integrity check to codex doctor covering every database in CODEX_HOME, so users can self-diagnose without hand-writing SQLite probes.

Additional information

Ruled out, with evidence — recorded to save triage time:

| Hypothesis | Verdict | Evidence |
|---|---|---|
| Broken/partial install | No | Package Status: Ok, Store-signed, binaries present |
| Application crash | No | No Event ID 1000 for ChatGPT, no WER report, no crash dump since the prior package version |
| Disk full | No | 230 GB free on an NVMe SSD |
| Permissions / ACL | No | User holds FullControl on .codex |
| WSL2 mode (#36320) | No | Logs show a native Windows spawn from the package path |
| Outdated Codex CLI | No | Desktop spawns its own bundled codex.exe from inside the MSIX; cannot be version-skewed with itself |
| WAL too large to replay | No | The WAL is written, not read — rebuilt from 132 → 274,060 frames during one failed launch |
| WAL checkpoint as a fix | No | Tested; next launch regenerated it and failed identically |
| Concurrent app-server contention (#30105) | No | Tested directly — see below |
| Corrupt user/state data | No | All other DBs quick_check clean |
| Reinstalling the app | Would not help | ~/.codex lives outside the MSIX and survives uninstall entirely |

Contention explicitly tested and excluded. An IDE extension ran its own codex.exe app-server against the same CODEX_HOME throughout. After the fix this was tested in the harsher ordering — IDE app-server started first and already holding the databases, then Desktop launched:

codex pid A — IDE extension's,  started 10:41:51
codex pid B — Desktop's,        started 10:42:59

14:42:59.652  codex.exe spawned
14:43:00.016  Transport start success   +0.36s
14:43:01.223  window ready-to-show      +1.70s

Two app-servers coexisted without incident.

Integrity of every other database (quick_check via immutable=1, none modified at any point):

| Database | Size | Result |
|---|---|---|
| state_5.sqlite | 50.7 MB | OK (675 threads) |
| memories_1.sqlite | 40 KB | OK |
| goals_1.sqlite | 32 KB | OK |
| queue_1.sqlite | 28 KB | OK |
| sqlite/codex-dev.db | 0.3 MB | OK |
| sqlite/codex-thread-summaries-dev.db | 12 KB | OK |
| logs_2.sqlite | 2.27 GB | CORRUPT |

Affected schema (diagnostics only — no user-authored records):

CREATE TABLE logs (
  id INTEGER PRIMARY KEY AUTOINCREMENT, ts INTEGER NOT NULL,
  ts_nanos INTEGER NOT NULL, level TEXT NOT NULL, target TEXT NOT NULL,
  feedback_log_body TEXT, module_path TEXT, file TEXT, line INTEGER,
  thread_id TEXT, process_uuid TEXT, estimated_bytes INTEGER NOT NULL DEFAULT 0
);
-- plus idx_logs_ts, idx_logs_thread_id, idx_logs_thread_id_ts,
--      idx_logs_process_uuid_threadless_ts

File state: 2,439,012,352 B = 595,462 pages × 4096, freelist 539, journal_mode=wal.

Secondary observations

  1. WAL sidecars persist across clean exit. After a proper tray-quit with zero remaining processes, all -wal/-shm files remained on disk. This looks intentional (PERSIST_WAL), but it means the presence of sidecar files is not evidence of an unclean shutdown. Several wrong conclusions were reached during this investigation by assuming otherwise — documenting it so it doesn't mislead others.
  1. The app stays resident after the window is closed. Closing the window left 10 processes and the codex.exe app-server running, still holding every database. "I closed Codex App" and "Codex App is not running" are different states. Anyone attempting a file-level workaround must quit via the system tray or they will be operating on open files.
  1. Startup shell-environment resolution times out — not causal, but it delays every launch by ~7 s:

``
Failed to load shell env caller=startup status=timed_out
timeoutMs=5000 durationMs=7144 pathEntryCount=30
`
PATH` contained several duplicated system entries. This happens before the app-server spawn, so it sits outside the 30 s handshake window — but it is real overhead and would erode the margin on a machine already close to the limit.

Related issues

  • #23917 (closed) — closest match and the likely origin. This machine had just been upgraded 26.707.8479.0 → 26.810.7004.0. Suggest reopening or linking; the upgrade path may be where the corruption is introduced.
  • #27741 (open) — same symptom and same workaround, attributed to file size. This report suggests corruption is the actual mechanism in at least some of those cases — worth asking reporters there for a quick_check before chasing size thresholds.
  • #28997, #24275, #31142 (open) — unbounded WAL growth and high-frequency TRACE/streaming writes. Together, likely why the file grew large enough to be at risk in the first place.
  • #30105 (open) — concurrent app-server contention. Investigated and explicitly excluded here by direct experiment.
  • #22444 (closed) — stale processes retaining deleted WALs. Relevant caution for anyone applying the workaround.

View original on GitHub ↗

2 Comments

MilkyWay008 · 10 days ago

Saw your quick_check(3) evidence...... the rowid-out-of-order cells are textbook B-tree corruption, and you're right that WAL replay / checkpoint / reinstall can't fix it because the main DB itself is malformed. The 30s handshake kill ~2s short of completion on every launch is exactly the "structurally cannot self-heal" pattern. Cleanest workaround for end-users right now: fully exit Codex (tray quit, not just window-close...... IDE extensions holding their own app-server also need to stop), then move ~/.codex/logs_2.sqlite + -wal + -shm aside together; rebuilds in ~1.7s and nothing user-authored lives in that file. Genuinely useful framing for the maintainer side: rotating the log DB on SQLITE_CORRUPT instead of looping against a damaged structure would have prevented every one of these outages.

jdcodes1 · 10 days ago

Excellent forensics — and the code on main @ 1f41cc5d92 confirms both halves of your conclusion: the non-essential telemetry DB really is load-bearing for startup, and there is a corruption-recovery mechanism that your failure mode routes around.

1. logs_2.sqlite failing to open is fatal to the entire state runtime. StateRuntime::init opens the logs pool right after the main state DB, and an error is a hard return Err that tears down the runtime:

https://github.com/openai/codex/blob/1f41cc5d92722748e45cae9cecc6d883a4e7cbb1/codex-rs/state/src/runtime.rs#L145-L155

Notably, the post-open logs maintenance a few lines later is already treated as non-fatal (run_logs_startup_maintenance failure only warns, #L261-L266) — so the codebase half-acknowledges that logs are expendable; the open/migrate step just doesn't get the same treatment.

2. Recovery exists but requires a classified error, not a hang. The app-server wraps init in init_sqlite_state_db_with_fresh_start_on_corruption (app-server/src/lib.rs#L1240+), which — when the error chain classifies as SQLite corruption — moves the damaged file into a backup folder and retries. Your signature B ("failed to initialize sqlite state runtime…", 1 of 6 launches) is init producing an actual error; whether the fresh-start retry then fired for logs_2.sqlite should be visible as backup-folder warnings — if it errored without corruption classification (or attribution), that's a recovery gap worth the log excerpt. But your dominant signature A (5 of 6) never gets there at all: the desktop gives the handshake 30s, and if init is wedged on the corrupt file (SQLite grinding through malformed pages / WAL recovery) rather than failing fast, the process is killed with a generic timeout and the recovery machinery never sees an error to classify. That's why the majority of launches gave "no indication a database was involved" — the diagnosis raced a 30-second clock and usually lost.

Fix outline:

  1. Demote logs to actually-non-essential: on open_logs_db failure (any error, not just classified corruption), back up and recreate logs_2.sqlite inline and continue — the backup helper used by the fresh-start path (backup_runtime_db_for_fresh_start) already does the file surgery. Telemetry loss is strictly better than a bricked app; the same argument applies to the goals DB one block below.
  2. Bound each DB open/migration step with its own deadline (well under the desktop's 30s handshake budget), so a wedged file becomes a classified, attributable error that the existing recovery machinery can act on — converting your 5-of-6 silent timeouts into the 1-of-6 recoverable path, and then into automatic recovery.
  3. Desktop side: when the handshake times out, surface the app-server's last stderr/error in the modal — signature B proves the actionable message exists; the timeout path just discards it.

Regression test shape: write garbage pages into a valid-header logs_2.sqlite (rowid-order corruption like yours, not just truncation — open succeeds, scans fail), init the runtime, and assert (a) startup completes with a fresh logs DB and a backup notice, (b) state_5.sqlite is untouched.