›_ Codex Issues
Codex Issues / #35555

CLI hard-fails at startup when any process holds a write lock on logs_2.sqlite (telemetry DB gates boot; flat 5s busy_timeout, no retry)

Open 💬 9 comments Opened Jul 27, 2026 by ryan-chen-opus

What version of Codex CLI is running?

codex-cli 0.145.0 (Homebrew cask)

What subscription do you have?

ChatGPT sign-in. Not a factor — the failure happens before authentication.

Which model were you using?

N/A — the process aborts before any model call.

What platform is your computer?

macOS 26.5.2 (25F84), Apple Silicon (arm64). Default ~/.codex on the internal APFS volume — no network mount, no WSL.

What terminal emulator and version are you using (if applicable)?

Originally hit in a terminal-multiplexer-managed shell; reproduced below under a plain pty allocated by script(1). Not terminal-specific.

What issue are you seeing?

Codex CLI hard-fails at startup — exits 1 before the TUI appears — whenever any process holds a write lock on $CODEX_HOME/logs_2.sqlite for longer than the 5s busy_timeout:

Codex couldn't start because another Codex process is using its local data.
Quit any other copies of Codex that may still be running, then try again.
Technical details:
  Location: ~/.codex/state_5.sqlite
  Cause: failed to initialize state runtime at ~/.codex: failed to open log DB at
  ~/.codex/logs_2.sqlite: error returned from database: (code: 5) database is locked:
  error returned from database: (code: 5) database is locked: (code: 5) database is locked
ERROR: failed to initialize sqlite local db at ~/.codex/state_5.sqlite: ...

Two things compound here:

  1. logs_2.sqlite is a pure telemetry sink, but it gates startup. StateRuntime::init opens the log DB inside try_init_with_roots_and_backfill_lease (codex-rs/rollout/src/state_db.rs, the failed to initialize state runtime at {} context). If that open returns SQLITE_BUSY, the whole process aborts — there is no degraded mode that runs Codex with the SQLite log sink disabled.
  2. The wait is a flat 5s busy_timeout with no retry/backoff (.busy_timeout(Duration::from_secs(5)) in codex-rs/state/src/sqlite.rs). Against a holder that does not release promptly, this is a permanent wedge rather than a transient stall.

Because $CODEX_HOME is shared by every Codex client on a machine (CLI sessions, the desktop app's bundled app-server, IDE extensions), one stuck process takes down all of them.

How I hit it in the field. A CLI instance had been suspended with SIGSTOP (STAT=T in ps) by a terminal multiplexer, right after an in-place update prompted a restart. A suspended process never releases its SQLite locks, so every subsequent codex invocation failed as above until that PID was killed. kill -CONT alone did not clear it — as a background tty job the process immediately re-stopped on SIGTTIN/SIGTTOU; it took kill -TERM followed by kill -CONT.

Worth stating precisely, because it narrows the bug: suspending an idle CLI is not enough to trigger this. I tried that first and the second instance started fine. The suspended process has to be frozen inside a write transaction. So the probability of hitting this scales with how much of its time the log writer spends holding the write lock — which links this to #28224: the flush loop fires every 2s and the retention DELETE runs in the same transaction as the insert batch (prune_logs_after_insert in codex-rs/state/src/runtime/logs.rs), so a busy process holds that lock a large fraction of the time. On this machine the log DB is still absorbing 20,000–35,000 inserts/min on 0.145.0.

What steps can reproduce the bug?

Deterministic and minimal. Uses sqlite3 as the lock holder so you don't have to catch a Codex process freezing at the right instant:

export CODEX_HOME=$(mktemp -d)

# 1. Let Codex create its DBs, then quit the TUI.
codex

# 2. Hold a write lock on the log DB from an unrelated process.
mkfifo "$CODEX_HOME/f"
sqlite3 "$CODEX_HOME/logs_2.sqlite" < "$CODEX_HOME/f" &
exec 9>"$CODEX_HOME/f"
echo "BEGIN IMMEDIATE;" >&9

# 3. Start Codex -> exits 1 with "database is locked".
codex

# 4. Release the lock -> Codex starts normally.
echo "ROLLBACK;" >&9; exec 9>&-
codex

Results on codex-cli 0.145.0 / macOS 26.5.2 / sqlite3 3.51.0, each arm run under script(1) with a 30–40s timeout:

| arm | lock holder on logs_2.sqlite | outcome |
|---|---|---|
| A (control) | none | starts; TUI runs to timeout; 0 lines matching locked |
| B | BEGIN IMMEDIATE held open | exit 1, error text above |
| C (control) | lock released | starts; TUI runs to timeout; 0 lines matching locked |

The arm-B error text is byte-identical to the field failure, including the misleading Location: .../state_5.sqlite line.

What is the expected behavior?

  1. Failing to open logs_2.sqlite should not be fatal. It is a telemetry sink. Codex should emit a warning and continue with the SQLite log sink disabled. Gating startup on state_5.sqlite is defensible; gating it on the log DB is not.
  2. If it must stay fatal, name the blocker. SQLite can't report the holder, but Codex can — e.g. write the owning PID into a heartbeat/lock row on open, and surface blocked by PID <n> instead of a bare (code: 5) database is locked. As written, the Location: line points at state_5.sqlite while the actual blocker is logs_2.sqlite, which sends people to the wrong file and, in the reports below, to deleting the wrong database.
  3. Consider a bounded retry with backoff instead of a flat 5s busy_timeout (also requested in #20213).

A recovery hint in the message would help too: the fix is to find the stuck process (lsof "$CODEX_HOME/logs_2.sqlite") rather than to delete anything.

Additional information

Related but, I believe, distinct — happy to have this folded in if maintainers disagree:

  • #30105 — same error string, but the trigger there is two live app-servers contending. Here the holder is a single frozen process, so retry/backoff alone would not have helped.
  • #20213 — contention causing freezes during use, plus the missing SQLITE_BUSY retry. This report is about a hard abort at startup, with an isolated repro for it.
  • #28666, #31184 — generic "database is locked" reports without a reproduction.
  • #28224 — the log-write volume that widens the window this failure needs; noted above as a contributing factor, not the cause.

Notable in recovery.rs: sqlite_error_detail_is_lock() deliberately classifies "database is locked" as not corruption, so the automatic backup/reset path correctly refuses to fire. That is the right call — but combined with (1) it means the user gets a dead end with no self-heal and no diagnostic pointer.

View original on GitHub ↗

9 Comments

kip-claw · 25 days ago

Corroborating this from a very different environment, with an additional trigger and a working recovery.

Environment

  • Linux, aarch64 (Raspberry Pi) — not macOS/Windows/WSL, so this is not platform-specific.
  • codex-cli 0.144.3, run headless as an embedded app-server by a host process (OpenClaw's Codex harness). No IDE extension, no second Codex instance, no network mount.
  • A dedicated per-agent CODEX_HOME (not the default ~/.codex).

Same failure, different trigger

The issue here is a write-lock on logs_2.sqlite. We hit the same boot-gating behavior via a different cause: an out-of-band maintenance action that rebuilt logs_2.sqlite in place to cap its size (it had grown to multiple GB). Rebuilding it in place left the _sqlx_migrations state inconsistent with the binary's embedded migrations, and from then on every app-server startup aborted with:

Error: failed to initialize sqlite state runtime under <CODEX_HOME>: failed to initialize state runtime at <CODEX_HOME>

The host retried the spawn and gave up:

[agent/embedded] codex app-server stderr: Error: failed to initialize sqlite state runtime under <CODEX_HOME> ...
[agent/embedded] codex app-server connection closed during startup; restarting app-server and retrying
[agent/embedded] codex app-server connection closed during startup; retries exhausted

So whether logs_2.sqlite is locked (#35555), corrupt (#24001 — (code: 26) file is not a database), has a migration checksum mismatch (#23863), or is externally rebuilt (our case), the outcome is identical: a telemetry/diagnostic DB takes down the entire state runtime and every model turn with it. In our setup that knocked out the primary model path completely; the host only stayed up because it could fall back to a different provider.

Two things worth flagging for whoever fixes this

1. PRAGMA integrity_check is ok — so "it's not corruption" is a red herring. Every DB in CODEX_HOME passed both integrity_check and quick_check while startup kept failing:

state_5:    jmode=wal  quick=ok
memories_1: jmode=wal  quick=ok
goals_1:    jmode=wal  quick=ok
logs_2:     jmode=wal  quick=ok

(Matches the observation in #30105 that both DBs pass integrity_check yet init fails.) SQLite structural integrity does not catch an sqlx migration-state mismatch.

2. The error is effectively undiagnosable from the outside. No Caused by: chain is surfaced through the app-server path, and RUST_BACKTRACE=full produced only <unknown> frames:

Error: failed to initialize sqlite state runtime under <CODEX_HOME>: failed to initialize state runtime at <CODEX_HOME>

Stack backtrace:
   0: <unknown>
   1: <unknown>
   ...

The underlying sqlite reason (lock / not-a-database / migration-modified) never reaches the operator. #24001 only got (code: 26) file is not a database because the remote-control path happened to print it. Surfacing the nested cause on this error would save a lot of guesswork.

Isolation (clean repro of the boot-gating)

  1. CODEX_HOME=$(mktemp -d) codex app-server → initializes fine (reaches the normal "project-local config … until trusted" notice, then serves). Confirms the binary/env are healthy.
  2. Point at the real CODEX_HOME → fails at state-runtime init.
  3. Move only logs_2.sqlite (+ -wal/-shm) aside → app-server recreates a fresh logs_2.sqlite and boots normally. state_5.sqlite, goals_1.sqlite, memories_1.sqlite are untouched and fine.

Step 3 is the key point: a single telemetry DB gates the entire runtime, and simply removing it recovers everything with no data loss beyond diagnostic logs.

To reproduce the migration-mismatch flavor specifically: take a working CODEX_HOME, rebuild logs_2.sqlite in place (dump+reimport, VACUUM INTO a replacement, or truncate-and-recreate) so _sqlx_migrations no longer matches the embedded migrations, then start the app-server.

Recovery / workaround that worked

  • Recover: stop the host, delete logs_2.sqlite* (.sqlite, -wal, -shm), restart — codex rebuilds a clean, migration-valid logs_2 on next start. State/memory/goals are preserved.
  • Prevent: to keep logs_2.sqlite from ballooning, delete the whole file while codex is stopped — never truncate/rebuild/VACUUM it in place, because that is exactly what desyncs the sqlx migration state and bricks startup.

Suggested direction (aligns with #24001 and this issue)

Treat logs_2.sqlite as non-gating: if it can't be opened, locked, migrated, or verified, emit a warning and continue (or transparently recreate it) rather than aborting the entire state runtime. Diagnostics/telemetry should never be able to prevent the app-server from booting.

Related: #23863 (sqlx checksum mismatch on logs_2), #24001 (recover from corrupt logs_2), #30105 (integrity_check passes yet init fails).

colonelpanic8 · 25 days ago

Additional reproduction for this root cause: you don't need a stuck holder or a large logs_2.sqlite — ordinary concurrent startup is enough.

The flat 5s busy_timeout with no retry means that when N codex app-server processes initialize against the same CODEX_HOME at once, some fraction simply lose the race and abort. No suspended process, no in-place update, no multiplexer involved.

Environment: codex-cli 0.146.0, Linux (NixOS, kernel 7.1.3), x86_64, local ext4 — no network mount, no WSL.

Minimal repro. Note CODEX_HOME is a fresh empty directory, recreated for every trial:

for n in 2 4 6 8; do
  for trial in 1 2 3; do
    rm -rf /tmp/vh && mkdir -p /tmp/vh
    for i in $(seq 1 $n); do
      echo '' | CODEX_HOME=/tmp/vh codex app-server > /tmp/m-$i.txt 2>&1 &
    done
    wait
    grep -l 'state runtime' /tmp/m-*.txt | wc -l   # failures this trial
  done
done

Failing processes exit 1 with:

Error: failed to initialize sqlite state runtime under /tmp/vh: failed to initialize state runtime at /tmp/vh

Results — 3 trials per concurrency level, each on a newly created empty CODEX_HOME:

| Concurrent starts | Failed / total |
| --- | --- |
| 2 | 1 / 6 |
| 4 | 3 / 12 |
| 6 | 6 / 18 |
| 8 | 9 / 24 |

Roughly one additional casualty per two added processes; a single start never failed across repeated runs (~2s each).

What the virgin-home detail rules out. This reproduces on a CODEX_HOME that did not exist a moment earlier, so it is not DB size, not accumulated history, not corruption, and not a backfill/migration of existing data. It is purely concurrent initialization contending on the telemetry DB that gates boot — which matches the two compounding causes in the original report. Since every Codex client on a machine shares one CODEX_HOME, anything that launches a few app-server processes together (an IDE integration, a supervisor, a tool that introspects providers) hits this in normal operation.

For what it's worth, the practical impact downstream: I hit this from a tool that spawns short-lived codex app-server processes to enumerate slash commands and skills. Three or four concurrent introspection calls are enough for one to fail, and because the failure surfaces as a dead process rather than a retryable error, it reads to the user as "this provider has no commands."

A bounded retry with backoff around the log-DB open would fix the concurrent-startup case even without decoupling telemetry from boot, though decoupling seems clearly right too.

lightcloud00 · 6 days ago

Downstream 0.149.0 mitigation evidence — macOS arm64

Reproduced on codex-cli 0.149.0, exact source tag rust-v0.149.0 at commit 758ef40f50c1a458425c7cfbf1eb12cbc07af0b0.

As a bounded local mitigation, I built a variant changing only:

-.busy_timeout(Duration::from_secs(5))
+.busy_timeout(Duration::from_secs(60))

No schema, migration, database-content, CLI-flag, or public configuration change was made.

Isolated contention test against logs_2.sqlite, holding BEGIN IMMEDIATE for seven seconds:

  • Stock 0.149.0: exited 1 with SQLite code 5, database is locked (7.99 seconds total wall time including TUI initialization).
  • Patched variant: remained alive through the lock, acquired the database after release, and reached the TUI.

Real shared-state verification:

  • The patched CLI reached the authenticated prompt against the existing shared Codex home and began MCP startup.
  • No process-management command was used. Four long-lived Codex app/helper PIDs retained the same PIDs and start times; one ephemeral app exec worker naturally rotated during the window. Only the newly launched verification process was exited.
  • Read-only PRAGMA quick_check returned ok for both state_5.sqlite and logs_2.sqlite.
  • codex --version remained 0.149.0.
  • Patched binary SHA-256: e2844d5d671973a2731296a00880654d84c31dc61ed1c1b80c9c5decc9518acb.

This corroborates that 0.149.0 still has the flat five-second startup failure and that a longer bounded timeout absorbs transient locks lasting over five seconds. It is a downstream workaround, not the full upstream fix: logs_2.sqlite still gates startup, permanent locks still fail after the larger timeout, and the blocker PID is still not identified in the error.

lightcloud00 · 5 days ago

Source publication update

The one-line timeout change has now been committed and published from current upstream main:

GitHub currently prevents this account from opening the corresponding PR in openai/codex: the compare page reports that repository owners have limited pull-request creation to repository collaborators. The preserved branch is lightcloud00:fix/sqlite-busy-timeout-60s, exactly one commit ahead of upstream main, and is ready for maintainer comparison or cherry-pick.

charle-z · 5 days ago

I traced the current main state runtime (8e649e3a) beyond the existing 5s->60s timeout workaround. The structural reason a lock still gates boot is that StateRuntime requires a concrete logs_pool: Arc<SqlitePool> and StateRuntime::init_inner() returns Err immediately when open_logs_db() fails. The same pool is then used by log insert/query/retention and thread-log cleanup, so simply making logs_pool optional would spread degraded-mode branches across several runtime paths.

A narrower fix for this issue's lock/busy failure is possible without that API churn: keep logs_pool concrete, but fall back to a migrated single-connection in-memory SQLite pool when opening/migrating logs_2.sqlite fails specifically with SQLITE_BUSY / SQLITE_LOCKED.

Why this is preferable to only increasing busy_timeout:

  • a permanently suspended writer no longer bricks startup after any timeout;
  • state/goals/memories/queue remain on their normal persistent DBs;
  • existing insert_logs, query_logs, retention, feedback-log reads, and thread cleanup keep the same SqlitePool contract;
  • the degraded process retains its own current-run diagnostics in memory rather than dropping every log operation;
  • no locked file is deleted or renamed.

There is one important recovery interaction: I would not fall back for every log-DB initialization error inside StateRuntime. App-server already has corruption backup/recreate handling based on RuntimeDbInitError and runtime_db_path_for_corruption_error(). Catching every logs error internally would hide corrupt/not-a-database errors from that recovery path. For #35555 the fallback should be gated on an is_sqlite_lock_error(&anyhow::Error) classifier analogous to the existing corruption classifier (the repository already has sqlite_error_detail_is_lock). Other error classes can continue through the current recovery/error path and be handled separately by #23863/#24001.

The focused regression matrix I would require before merging this rather than the 60s-only workaround is:

  1. normal logs DB -> persistent pool, existing behavior;
  2. BEGIN IMMEDIATE held on logs_2.sqlite past the busy timeout -> StateRuntime::init succeeds and state DB reads/writes still work;
  3. while degraded, insert_logs + query_logs work against the ephemeral pool;
  4. after the blocker is released, a fresh process opens the persistent logs DB normally;
  5. corrupt/not-a-database logs DB still reaches the existing backup/recreate path rather than being silently masked;
  6. a locked state_5.sqlite remains fatal/non-degraded (the fallback is logs-only).

The already-published 60s branch is useful for transient contention, but it cannot satisfy cases 2/6 for a permanent lock. I did not publish a second competing timeout patch; the next meaningful patch here should be the logs-only degraded pool plus the above failure-mode tests.

charle-z · 5 days ago

Implementation + validation update (final source state for this attempt)

I took the non-gating log-store direction to a current-main patch rather than duplicating the existing 5s→60s timeout workaround.

Source branch: charle-z:fix/35555-nongating-locked-logs-db
Final source head: a9b059896fb2a594c415d9df308142aa30b209e7
Exact upstream base: 99660ab3c7b861c916e467581fa9b8723504d66b
Compare: https://github.com/openai/codex/compare/99660ab3c7b861c916e467581fa9b8723504d66b...charle-z:codex:fix/35555-nongating-locked-logs-db

The source diff is limited to two files (codex-rs/state/src/sqlite.rs and sqlite_tests.rs). The patch is intentionally lock-only:

  • open_logs_db keeps the normal persistent path first;
  • only SQLite BUSY / LOCKED errors fall back;
  • fallback is a one-connection :memory: pool (SQLite in-memory DBs are connection-local) with the same log migrations applied;
  • the rest of StateRuntime still receives an ordinary SqlitePool, so insert/query/delete/startup-maintenance paths do not need degraded-mode branches;
  • corruption and other non-lock errors still propagate, preserving the existing per-database backup/rebuild recovery path.

Regression coverage holds an actual BEGIN IMMEDIATE against an unmigrated logs_2.sqlite, verifies fallback produces a migrated log store, then releases the lock and verifies the persistent file was not migrated by the fallback. A second control verifies a corrupt log DB still returns an error recognized by is_sqlite_corruption_error.

Repository-harness validation on the final source tree:

  • just test -p codex-state: 204 passed, 7 skipped
  • just fix -p codex-state: passed
  • independent repository cargo fmt gate: passed
  • additional fast gates observed green on the same patch line: Rust benchmark smoke, cargo shear, cargo-deny, blob-size policy, and v8-canary

Validation run with the repository harness: https://github.com/charle-z/codex/actions/runs/32613114092

The final global just fmt step is not a patch-local failure. On the patch run it proceeds through the non-Rust formatters and then fails in workspace cargo metadata because codex-requirements-schema does not contain feature dummy. I also ran just fmt against the exact untouched upstream base SHA in an isolated comparison job; the base is already unable to complete the global formatter in this validation environment (it fails even earlier because uv is absent for the Python formatters): https://github.com/charle-z/codex/actions/runs/32613483413. In other words, the required crate tests/fix and Rust formatting signal are clean, while the repository-wide formatter is blocked outside this two-file diff.

I also checked the degraded pool against the existing log maintenance shape: PRAGMA wal_checkpoint(PASSIVE) is valid on an in-memory SQLite database, so no follow-on maintenance branch is needed.

I attempted to open an upstream draft PR after validation, but GitHub returned 403 Resource not accessible by integration, consistent with this repository's collaborator-only cross-fork PR restriction. The source branch and exact SHA above are preserved for maintainer comparison/cherry-pick.

charle-z · 2 days ago

Correction to my Aug 23 comments: I am withdrawing the in-memory logs_2.sqlite fallback (fix/35555-nongating-locked-logs-db, old head a9b0598). It is not safe to merge.

The adversarial review found a state invariant it violates: StateRuntime::delete_threads_strict() promises to delete a thread and all associated state. With an in-memory fallback, DELETE FROM logs succeeds against an empty ephemeral DB, then queue/memory/goals/thread rows can be deleted while the real locked logs_2.sqlite still retains that thread's persistent logs. That turns a retriable failure into a false success.

The replacement direction keeps the real log store explicitly unavailable instead of faking persistence: only SQLite BUSY/LOCKED may make logs_pool unavailable; log reads/writes then return a local availability error, while core state/goals/memories/queue continue. Strict thread deletion preflights that availability and refuses before mutating state. Corruption, migration, IO, readonly, etc. remain fatal and stay on the existing recovery path. A lock that clears within the existing busy timeout still opens the persistent DB normally; a longer/permanent lock degrades the process until restart.

Real BEGIN IMMEDIATE tests cover transient lock, beyond-timeout lock/degraded startup + strict-delete preservation + restart recovery, and corrupt-log-DB fatality. The full codex-state suite passed 186/186 + doctest on the audited implementation; the rebased replacement head is still going through fork CI. I will post the final branch/SHA only after those gates finish.

charle-z · 2 days ago

Final validated source update after withdrawing the unsafe in-memory fallback.

Candidate branch: charle-z:fix/35555-nongating-logs-v5
Head: 0bc3a2ac47372392370e240a223adc32c888ff30
Exact upstream base used for the final validation snapshot: 74772623db. A final fetch reached f5420174da; none of the five touched files, Cargo.toml, Cargo.lock, or rust-toolchain.toml changed across that drift.

Semantics:

  • logs_2.sqlite is no longer startup-gating only for SQLite BUSY/LOCKED after the existing bounded wait.
  • The persistent log pool becomes explicitly unavailable; there is no fake/in-memory persistence.
  • Core thread state, goals, memories, and queue can still initialize and operate.
  • Log APIs fail locally while degraded; the process stays degraded until restart, which reopens the real log DB after the lock is gone.
  • Corruption, migration, IO/permission and other non-lock failures remain fatal and stay on the existing recovery path.
  • Strict thread deletion preflights persistent-log availability. app-server performs that preflight before preparing/deleting rollout files, avoiding deterministic partial deletion when startup degraded.

Real SQLite lock coverage:

  • transient BEGIN IMMEDIATE lock clears within the timeout -> persistent logs open normally, no degradation;
  • beyond-timeout lock -> runtime boots degraded, strict delete fails without mutating thread/goal state, restart recovers persistence;
  • corrupt logs_2.sqlite -> remains fatal/recoverable, not degraded.

Final gates on the exact v5 SHA via a public Ubuntu 24.04 harness using Rust 1.95.0:

  • cargo check -p codex-app-server --lib: passed;
  • cargo clippy -p codex-state --tests -- -D warnings: passed;
  • cargo test -p codex-state -- --test-threads=1: 189/189 passed;
  • doctest: 1/1 passed.

The same three focal lock/corruption tests also passed directly on the Edge checkout at v5.

The existing 5s->60s busy-timeout proposal is compatible with this design rather than redundant with it: a longer timeout reduces unnecessary degradation under transient contention; non-gating behavior prevents a permanent writer from bricking startup after whatever timeout is chosen.

charle-z · 1 day ago

Revalidated v5 again against current main (2c4a95736bea64256a50f7b8506bd33c181cc85a). The production patch remains patch-equivalent to the previously validated candidate; the only intervening codex-state drift was unrelated thread metadata.

Current branch/head:

The validated semantics are unchanged: only SQLITE_BUSY/SQLITE_LOCKED on logs_2.sqlite allow degraded startup; no in-memory fallback or silent log dropping; strict deletion refuses before mutation while logs are unavailable; state DB locks and corrupt/non-lock log DB failures remain fatal; restart after lock release restores persistent logs.

I also attempted to open the upstream PR from this exact head after duplicate/base validation, but GitHub returned HTTP 404 at creation time, consistent with the repository's current cross-fork PR restriction.