Multi-terminal codex CLI freezes due to SQLite lock contention with no BUSY retry

Thanks for the report, but I don't think your analysis is correct.

Current code does use shared state_5.sqlite / logs_2.sqlite, WAL mode, and a 5s busy_timeout, and insert_logs does not have an explicit retry loop. It’s also plausible that the persistent log DB can grow quickly because the TUI installs the SQLite log layer at TRACE and SSE processing logs each stream event.

That said, the local SQLite log sink is intentionally asynchronous: tracing events are sent through a bounded background queue, and the background flush currently ignores insert_logs failures. So a BUSY error in logs_2.sqlite should not directly propagate into stream handling or drop the model stream/TUI channel. It may drop log rows or add background I/O pressure, but there is no direct “log insert failure aborts stream consumer” path in current code.

Adding a Windows Desktop data point that looks like the same root cause, with a source-level pointer.

Environment:

• Codex Desktop: 26.429.2026.0
• @openai/codex: 0.128.0
• Platform: Windows
• Auth: ChatGPT account

Local evidence:

• %USERPROFILE%\.codex\logs_2.sqlite grew to about 1.01 GB
• logs table had 318,757 rows
• During the slow UI period, the bundled app-server process averaged about 26.6% CPU and had disk read/write bursts
• Recent slow SQL warnings included:
• PRAGMA wal_checkpoint(TRUNCATE) taking 5.58s during thread/list
• slow batched INSERT INTO logs (...)
• Clearing the log SQLite table dramatically improved Codex Desktop app performance.

The largest local log contributors were:

• codex_api::endpoint::responses_websocket: ~83k rows / ~365 MB body text
• codex_otel.log_only: ~85k rows / ~109 MB body text
• codex_otel.trace_safe: ~84k rows / ~100 MB body text

Source-level signal from current upstream:

• The app-server installs the SQLite log DB layer with Targets::new().with_default(Level::TRACE):

https://github.com/openai/codex/blob/f50c02d7bcd4c06a23173389da8ed2c68c03d81d/codex-rs/app-server/src/lib.rs#L581-L584

• The WebSocket client logs full websocket request payloads at TRACE:

https://github.com/openai/codex/blob/f50c02d7bcd4c06a23173389da8ed2c68c03d81d/codex-rs/codex-api/src/endpoint/responses_websocket.rs#L548-L556

• The WebSocket client also logs full websocket event payloads at TRACE:

https://github.com/openai/codex/blob/f50c02d7bcd4c06a23173389da8ed2c68c03d81d/codex-rs/codex-api/src/endpoint/responses_websocket.rs#L597-L599

So this does not appear to be only multi-terminal contention. In Desktop, normal app-server use can persist very large TRACE rows, including full responses_websocket request/event payloads, into logs_2.sqlite. Once the DB is large enough, checkpoints and inserts become visible as CPU/disk bursts and UI slowness.

Workaround that helped locally: quit Codex, then clear/checkpoint/vacuum only logs_2.sqlite with:

uv run python -c "import sqlite3, os; p=os.path.join(os.environ['USERPROFILE'],'.codex','logs_2.sqlite'); con=sqlite3.connect(p, timeout=60); con.execute('PRAGMA busy_timeout=60000'); con.execute('DELETE FROM logs'); con.commit(); con.execute('PRAGMA wal_checkpoint(TRUNCATE)'); con.execute('VACUUM'); con.close()"

Do not run that against state_5.sqlite; this workaround only clears local log rows.

@etraut-openai yeah, that makes sense for the narrow failure path: I agree a BUSY or failed insert_logs probably should not directly abort the stream if the log sink is queuing and dropping insert errors.

I’m wondering if the Desktop slowdown could still come from the log sink indirectly, though, rather than through error propagation:

responses websocket event
  -> trace!(full payload)
  -> LogDbLayer::on_event on emitting task
  -> format span context + payload
  -> queue for background insert
       -> INSERT / checkpoint / WAL work on logs_2.sqlite

Possible latency sources:
  - CPU/allocation before the queue boundary
  - CPU/disk/SQLite pressure from the background writer
  - slower app-server RPCs like thread/list
  - Desktop UI feels unresponsive

So I’m not thinking “log insert failure aborts stream.” More like:

• the SQLite write is async, but event formatting happens before the queue boundary
• the trace site is on a hot websocket streaming path and includes full request/event payloads
• the async writer still shares the same app-server process/runtime/disk
• once logs_2.sqlite is large, checkpoint/WAL work may become visible latency; I saw PRAGMA wal_checkpoint(TRUNCATE) take 5.58s under a thread/list span

Does that path sound plausible, or is there something about the app-server/runtime split that should prevent this from affecting Desktop RPC latency? The part that keeps me suspicious is that clearing only the logs table plus checkpoint/vacuum was followed by Desktop going from very slow back to normal immediately.

@etraut-openai yeah, that makes sense for the narrow failure path: I agree a BUSY or failed insert_logs probably should not directly abort the stream if the log sink is queuing and dropping insert errors. I’m wondering if the Desktop slowdown could still come from the log sink indirectly, though, rather than through error propagation: `` responses websocket event -> trace!(full payload) -> LogDbLayer::on_event on emitting task -> format span context + payload -> queue for background insert -> INSERT / checkpoint / WAL work on logs_2.sqlite Possible latency sources: - CPU/allocation before the queue boundary - CPU/disk/SQLite pressure from the background writer - slower app-server RPCs like thread/list - Desktop UI feels unresponsive ` So I’m not thinking “log insert failure aborts stream.” More like: * the SQLite write is async, but event formatting happens before the queue boundary * the trace site is on a hot websocket streaming path and includes full request/event payloads * the async writer still shares the same app-server process/runtime/disk * once logs_2.sqlite is large, checkpoint/WAL work may become visible latency; I saw PRAGMA wal_checkpoint(TRUNCATE) take 5.58s under a thread/list span Does that path sound plausible, or is there something about the app-server/runtime split that should prevent this from affecting Desktop RPC latency? The part that keeps me suspicious is that clearing only the logs` table plus checkpoint/vacuum was followed by Desktop going from very slow back to normal immediately.

Thanks, it worked for me too.

The strict five-second timeout is a pain in the neck for users who want to launch ten, twenty, or more Codex CLI instances.

I reproduced a closely related shared-CODEX_HOME contention case in Codex CLI 0.144.1, with an additional misleading corruption symptom. A fully stopped, read-only SQLite check subsequently returned ok, so the evidence points to lock contention and/or transient corruption reporting rather than persistent damage to logs_2.sqlite.

Environment

• Codex CLI 0.144.1, installed from the Homebrew cask
• ChatGPT macOS app 26.707.51957 (build 5175), using its bundled Codex app-server
• macOS 26.5.1 (25F80), Apple Silicon/arm64
• One default Codex state directory shared by the app-server and several explicit-model codex exec workers

Reproduction

1. Keep the ChatGPT/Codex app-server running on a long-lived task.
2. Start several bounded codex exec workers against the same default Codex state directory (this is the workaround for the inability to spawn models other than the parent).
3. Let the shared log database and WAL grow during sustained use.
4. Observe worker/app-server startup and state-runtime initialization.

At the time of failure, the files were:

• logs_2.sqlite: 3,769,585,664 bytes
• logs_2.sqlite-wal: 987,209,712 bytes
• logs_2.sqlite-shm: 1,933,312 bytes

lsof showed the bundled Codex app-server holding the database, WAL, and SHM.

Observed behavior

Within one active-app period, the same database alternated between:

• SQLite code 779: database disk image is malformed
• SQLite code 5: database is locked

Sanitized UTC observations:

• 2026-07-12 21:23:01: code 779 / malformed
• 2026-07-12 21:32:26: code 5 / locked, followed by state-runtime rollout-path fallback
• 2026-07-12 21:32:47, 21:38:40, 21:44:10, and 21:47:15: code 779 / malformed again

Some startup-maintenance operations took roughly 20–32 seconds before failing. The workers could continue through rollout-file fallback, but the repeated initialization delays and contradictory diagnosis added substantial overhead.

Fully stopped integrity result

I then quit every ChatGPT/Codex app, CLI, integration, and app-server. A holder check returned no processes. Before testing, I made byte-for-byte backups of the database, WAL, and SHM and verified their SHA-256 hashes.

Running sqlite3 -readonly logs_2.sqlite 'PRAGMA quick_check;' returned exit code 0 and:

ok

The database and WAL sizes and hashes were identical before and after the check:

• database SHA-256: 243644645850e2a3ceaea7f7fc0287a933e0d386beb965013175d84daec0323f
• WAL SHA-256: 94d6222161f36e9ac627bc31ed50afaa81dd4d2a4f05fa542ebd0b7b12900b6e

The transient SHM file was rebuilt by the read-only WAL access, shrinking from 1,933,312 bytes to 32,768 bytes. That caused my conservative whole-file-set “unchanged” check to report no, but the database and WAL themselves did not change. The stopped-state result therefore does not support persistent database corruption.

Impact and unaffected evidence

• Startup/read-maintenance delay and repeated state-runtime fallback
• Contradictory locked versus malformed reporting for the same database
• Significant multi-worker orchestration overhead during sustained repository work (aka token burn)
• Rollout JSONL fallback remained readable
• Main state telemetry remained queryable, including exact model/effort allocations
• Git state, repository files, and independent test artifacts remained valid

I found no evidence of repository/source corruption, model-allocation corruption, or loss of the rollout records. I am not publishing or uploading the raw database because it may contain private operational data.

Expected behavior

• Busy/locked writes and maintenance should use bounded retry/backoff or fail non-fatally without disrupting model-stream or rollout recovery.
• A transient lock or maintenance failure should not be surfaced as persistent database corruption without a confirming integrity check.
• App-server plus CLI-worker sharing of the default state directory should be supported or explicitly serialized.
• The log sink should have bounded retention/checkpoint behavior so a multi-gigabyte database and near-gigabyte WAL do not amplify contention.
• Diagnostics should distinguish the durable database/WAL from the rebuildable SHM sidecar.

This appears to be the same general contention class as #20213, now reproduced with the macOS app-server and CLI workers rather than only multiple interactive terminals. The stopped-state quick_check: ok is the main reason I am commenting here rather than treating this as the persistent-corruption/recovery case in #24001.

Adding a current Windows Desktop data point and a narrower mitigation request.

Environment:

• Codex Desktop package: 26.707.3748.0 (bundled app-server reports 26.707.31428)
• Windows 11 x64
• Shared default %USERPROFILE%\.codex

Observed:

• logs_2.sqlite reached 1,339,052,032 bytes / 351,361 rows in about 10 days.
• Retaining only the most recent 24 hours and vacuuming still left 210,612,224 bytes (~201 MiB).
• In that retained set, the level distribution was:
• TRACE: 35,258 rows / 158,400,214 estimated bytes
• DEBUG: 8,266 / 12,938,058
• INFO: 8,460 / 11,255,910
• WARN: 1,454 / 992,790
• ERROR: 81 / 125,895
• In other words, WARN+ERROR represented only 1,535 rows and ~1.1 MiB of estimated log content.

Current upstream still makes the database sink independent from RUST_LOG:

• codex-rs/app-server/src/lib.rs applies EnvFilter::from_default_env() only to stderr.
• The SQLite layer is installed separately with log_db::default_filter().
• codex-rs/state/src/log_db.rs::default_filter() uses TRACE as its default.
• [analytics] enabled = false does not gate creation of the SQLite log layer.

Current upstream does add per-thread and per-process partition caps in state/src/runtime/logs.rs, but those do not provide a global database cap when many thread IDs/process UUIDs accumulate. Deleted rows also leave allocated pages until vacuum/checkpoint maintenance.

As a local proof of mitigation, I installed a SQLite BEFORE INSERT filter that ignores levels below WARN, deleted existing TRACE/DEBUG/INFO rows, checkpointed, and vacuumed. The live database fell from 210,731,008 bytes to 1,650,688 bytes, PRAGMA quick_check returned ok, and only WARN/ERROR remained. This is an unsupported local workaround, but it demonstrates that filtering before persistence is effective and avoids write amplification.

Requested product fix:

1. Add a supported config for the SQLite sink level, e.g. [logging] sqlite_level = "warn" (Desktop default should probably be WARN or INFO, not TRACE).
2. Enforce a global retained-byte/row cap in addition to per-partition caps.
3. Avoid formatting full TRACE payloads when the SQLite sink will reject them; filtering should happen at the tracing layer before format_feedback_log_body.
4. Make the default and current retained usage visible in diagnostics/settings.