[Bug] Codex CLI 0.145.0 still persists per-SSE TRACE events to SQLite, causing high-frequency disk writes

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #34291

Powered by Codex Action

I observed the same SQLite TRACE sink behavior on Codex Desktop for macOS.

Environment:

• Codex Desktop: 26.715.72359, build 5718
• macOS: 26.5.2 (25F84)
• Hardware: Apple M5 Pro, 48 GB RAM
• Database: ~/.codex/logs_2.sqlite

Measurements before mitigation:

• Main database size: approximately 1.4 GB
• WAL size: approximately 6.6 MB
• Retained TRACE rows: 79,360
• TRACE rows observed in one minute: 436
• MAX(id) increased from 95,653,006 to 95,653,117 in 15 seconds
• The WAL modification time changed every 4–6 seconds
• Recent rows included codex_api::sse::responses
• lsof confirmed that the writer was the Codex process
• 298,873 of 371,602 database pages were on the freelist, approximately 80.4%

As a temporary mitigation, I installed a BEFORE INSERT trigger using RAISE(IGNORE). Afterward, five samples over 20 seconds showed:

• MAX(id) remained fixed at 95,653,627
• Row count remained fixed
• WAL size and modification time stopped changing
• The main database modification time stopped changing

The trigger has survived subsequent Desktop updates to 26.721.30844 and 26.721.31836. I have not removed it to retest the unmitigated behavior on those builds because doing so would deliberately resume the confirmed write path on my primary device.

This appears to be the same underlying SQLite logging-sink issue across CLI and Desktop rather than a WSL-specific problem. Related reports include #17320, #21134, and #24275.

Analysis (community)

Residual analysis for #35092 (CLI 0.145+ still persists per-SSE TRACE into SQLite / high-frequency disk writes) under docs/contributing.md — invitation-only. No unsolicited PR.

Cross-links (same residual class, do not refile #28224): #17320, #28997, #29532, #34291.

Root-cause hypothesis

The #28224-era mitigations were per-target denylists; default_filter() still defaults the SQLite sink to TRACE and never honors RUST_LOG, so per-SSE full-payload TRACE emitters keep thrashing logs_2.sqlite on 0.146.x.

Residual measurement on 0.146.0-alpha.6 — TRACE is still 69.1% of the logs_2.sqlite sink

Following up on this issue's report that 0.145/0.146 still writes per-SSE-event TRACE into the persistent log DB: I re-measured a live logs_2.sqlite today (read-only; codex-cli 0.146.0-alpha.6; all code anchors below verified on main @ 5dd992acd3, 2026-07-24). This build contains every claimed mitigation — #29432 (91e6da943b), #29457 (e98d43ac37), #29599 (63f8f547c0), #31789–#31792 (acb78a41a0, c8d2db9cc0, bfa3eeb8bf, 538f29ed5d); each verified with git merge-base --is-ancestor <sha> rust-v0.146.0-alpha.6. This is residue on top of that fix series — cross-linking #28224 as the umbrella, not asking to reopen it.

Measurement (captured 2026-07-24T16:03:00Z)

| Metric | Value |
|---|---|
| logs_2.sqlite bytes | 1,778,548,736 (~1.78 GB) |
| -wal bytes | 30,405,632 |
| Rows | 137,174 (max ts 16:02:52Z — writes live) |
| Level mix | TRACE 94,833 (69.1%); DEBUG 21,590; INFO 19,584; WARN 797; ERROR 370 |
| Last 60 s | TRACE 786 / DEBUG 32 / INFO 28 — TRACE is 92.9% of fresh writes |
| page_count / freelist_count | 434,216 / 394,121 — 90.8% of pages are prune-freed and never returned to the OS |
| Processes sharing this sink (last hour) | 7 |

Window statistics (last 60 s, last-5-min targets, process count) are capture-time snapshots: the retention prune keeps deleting rows out of those windows after capture, so re-running the same queries later returns fewer rows — the totals above and the MAX(id)-based rate below are the re-checkable numbers.

Writes are ongoing, not legacy: rows moved only +20 in ~15.4 min (137,154 at 15:47:34Z → 137,174; 137,106 at 15:10:05Z) while db_bytes and wal_bytes stayed byte-identical and max ts advanced +918 s. 786 TRACE rows were inserted in the last 60 s alone. Measured via MAX(id) deltas (which capture pruned rows, unlike the retained count): 126 inserts/min in a quiet 16:34Z window, 642/min during active SSE streaming (separate ~15:20Z capture) — with the retained count and db_bytes flat in both cases. Inserts are balanced nearly 1:1 by retention deletes, so each persisted TRACE event costs one insert + 3–4 index updates + one prune delete, all WAL-framed (codex-rs/state/src/runtime/logs.rs:43-45), while the file stays pinned at its historical peak.

Where this comes from on current main

• Sink filter: default_filter() is .with_default(LevelFilter::TRACE) plus a small per-target denylist (codex-rs/state/src/log_db.rs:53-63); LogDbLayer::on_event has no level gate beyond two hardcoded drops (log_db.rs:204-250). No env var or config key touches it.
• Wiring: installed per-layer at exactly two sites, codex-rs/tui/src/lib.rs:1288-1291 and codex-rs/app-server/src/lib.rs:665-668. The RUST_LOG EnvFilter rides only the stderr fmt layer (app-server/src/lib.rs:654,659), so RUST_LOG=warn silences the console while TRACE rows keep persisting; the adjacent comment "Users can control the log level with RUST_LOG" (lib.rs:646-648) is misleading for the sink.
• Retention: per-partition caps of 10 MiB / 1,000 rows (codex-rs/state/src/runtime.rs:77-78) pruned inside every insert transaction (logs.rs:44); a 10-day sweep plus PRAGMA wal_checkpoint(PASSIVE) at startup (logs.rs:296-310). auto_vacuum=INCREMENTAL (codex-rs/state/src/sqlite.rs:250) but zero PRAGMA incremental_vacuum anywhere in codex-rs, and full VACUUM is declined (sqlite.rs:154-156) — freed pages accumulate on the freelist forever.
• Emitters — exactly the per-SSE family this issue reports: codex_api::sse::responses per SSE event (codex-rs/codex-api/src/sse/responses.rs:531, plus :468), codex_app_server::outgoing_message per notification (codex-rs/app-server/src/outgoing_message.rs:600,637), codex_tui::markdown_stream per streamed delta (codex-rs/tui/src/markdown_stream.rs:78). They are the top three targets of the last 5 minutes: 902 / 633 / 285 rows.

Why the 0.142–0.145 fix series is incomplete

#29432 / #29457 (0.142.0) / #29599 (0.143.0-alpha.6) — the ~85% cut is the issue author's own measurement in the #28224 thread, not an OpenAI figure — and their follow-ups #31789–#31792 (0.145.0) were all per-target: emitter deletions, denylist entries, hardcoded drops. #29457 itself introduced the TRACE-default line above ("Keep TRACE persistence for other targets"), so any non-denylisted target still persists full TRACE; RUST_LOG cannot reach the sink; and nothing bounds table-wide bytes or ever shrinks the file. Notably, a startup PRAGMA incremental_vacuum did exist until #21378 (v0.129.0) removed it on the premise that "all the DBs have been reclaimed by now" — a 90.8%-freelist DB created after that release falsifies the premise. For balance, two narrow paths really are closed on main: #29532's bridged target=="log" churn (hard drop in on_event, #29599, log_db.rs:209-211) and #34291's megabyte rmcp::service dumps (capped to INFO by #31789, log_db.rs:60). The residue is everything else — which was never gated.

Minimal fix directions

1. Lower the sink default from TRACE to INFO (WARN is also defensible; INFO keeps the INFO-level feedback_tags markers /feedback triage keys on — codex-rs/core/src/util.rs:28-34), plus an explicit opt-in (e.g. CODEX_LOG_DB_LEVEL=trace) read once at subscriber init for support-driven deep capture. /feedback survives: the read path is level-agnostic (logs.rs:334-436) and the 4 MiB ring (cap at codex-rs/feedback/src/lib.rs:40) stays full-TRACE (:211-219). Honest tradeoff: the attachment is either/or (feedback/src/lib.rs:550), so threaded uploads become INFO+ durable history while anonymous uploads still carry the full-TRACE ring. The bundled codex-state-logs tool's --level (codex-rs/state/src/bin/logs_client.rs:28-30) would want a help-text update.
2. Correct the comment at app-server/src/lib.rs:646-648 (and mirror at the TUI site) — cheaper than making RUST_LOG gate the sink, which would couple console debugging to disk persistence.
3. Demote or remove the three per-event emitters, per the #29432/#30757 pattern (those deleted; demoting to debug! would preserve opt-in capture under CODEX_LOG_DB_LEVEL=debug — maintainer's call). Note this does not reduce feedback-ring churn (the ring's TRACE default admits debug! equally); the wins are persisted volume under opt-in and console ergonomics.
4. Restore a bounded PRAGMA incremental_vacuum(N) in run_logs_startup_maintenance (logs.rs:296-310) — a self-revert of #21378's removal, sized to respect the sqlite.rs:154-156 startup-contention concern (no-op on legacy DBs created before INCREMENTAL auto_vacuum; this host's DB is INCREMENTAL). While there: let _ = insert_logs(...) at log_db.rs:436 drops batches silently on contention — a warn! would make that observable. A table-wide byte budget deserves a separate follow-up.

Fail-first tests

• With default_filter(), emit trace!/debug!/info!/warn! → only INFO and WARN persist (fails today: log_db_filter_tests.rs:42-54 asserts TRACE retention).
• Opt-in path: CODEX_LOG_DB_LEVEL=trace restores full TRACE persistence end-to-end.
• Insert rows older than the 10-day cutoff, run run_logs_startup_maintenance(), assert freelist_count == 0 and page_count shrank (fails today: nothing vacuums).
• Saturate one partition past 10 MiB through the real write path so prune fires, then assert startup maintenance reclaims the prune-freed freelist (fails today).
• A RUST_LOG=trace console layer must not reopen the sink; query_feedback_logs still returns INFO bodies and no trace bodies under the default.

Caveats

Single host. The 1.78 GB is a historical high-water mark of mostly reusable free pages — db_bytes did not move during the window. The claim here is sustained write amplification and policy residue (insert+delete+index WAL churn per TRACE row; file pinned at peak), not that this one file proves an SSD-endurance/TBW problem — I leave that inference to the maintainers. Also, for honesty: hyper_util TRACE is filtered out of the sink by #31791 (2026-07-09); rows retained from before that have since been swept by retention (0 such rows at re-check), so hyper_util is pre-filter history, not a live emitter.

Happy to provide more measurements, run tests, or bisect. Per the repo's contribution policy I will not open a PR uninvited. Related: #28224 (umbrella, closed), #29532, #17320, #28997.

I'm experiencing this issue on 0.145.0 as well linux codex-cli. For anyone coming across it, I applied this fix periodically via systemd unit

https://github.com/openai/codex/issues/28224#issuecomment-4869140087

~/.codex/logs_2.sqlite — 9 days, 168,646 rows, 65% still TRACE:

TRACE 109,345 | DEBUG 29,134 | INFO 28,877 | WARN 1,228 | ERROR 62

Noisiest targets: codex_api::sse::responses (39,842),
codex_app_server::outgoing_message (16,549), codex_tui::markdown_stream (13,198).

The amplification is the real problem, not the row count. sum(estimated_bytes)
is only 166 MB over 9 days (~18 MB/day), but one codex process measured
638 MB/h of actual host writes via /proc/PID/io — roughly 170x. The DB
sits at a fixed 266 MB with an advancing mtime, so it's rewritten in place; du
shows nothing and the WAL never grows. NAND amplification stacks on top (2.3–3.8x
measured on the Micron drives in this box via SMART attrs 247/248).

The block_log_inserts trigger workaround dropped it to 74 MB/h (~88%) and
froze the row count. Worth noting for anyone applying it: the trigger lives inside
the database, so a logs_2 → logs_3 migration would silently drop it. I re-apply
it from a 15-minute systemd timer rather than trusting it to persist.

And heads up if you use opencode, latest (1.18.5) on linux is also experiencing this.

Local corroboration from an OpenClaw-supervised Codex app-server on macOS Apple Silicon, observed 2026-07-26. This report contains aggregate measurements only; I did not inspect or publish feedback_log_body values.

Environment:

• Active app-server: codex-cli 0.144.3 bundled by @openclaw/codex 2026.7.1-1.
• A separate global codex-cli 0.145.0 installation exists, but it was not the process holding this database. This comment should therefore not be read as a local 0.145.0 reproduction.
• SQLite journal mode: WAL. No insert-blocking trigger was present.

Database snapshot while the app-server was active:

• Main DB: 610,488,320 bytes (582.2 MiB)
• WAL: approximately 4.2 MiB
• PRAGMA quick_check: ok
• page_count: 149,045; page_size: 4,096
• Initial freelist_count: 45,303 = 176.9 MiB reclaimable, 30.4% of pages
• A later read showed freelist_count 41,691 = 162.9 MiB, 28.0%, while the main file remained at the same high-water size. This is consistent with new inserts reusing some free pages without returning file space to the OS.
• Retained rows: 254,707
• SUM(estimated_bytes): 302,662,097

Retained level distribution:

• TRACE: 201,804 rows, approximately 227.6 MiB
• INFO: 36,715 rows, approximately 31.8 MiB
• DEBUG: 15,310 rows, approximately 28.7 MiB

Largest target/level pairs:

• TRACE codex_api::sse::responses: 143,566 rows, approximately 146.1 MiB
• TRACE codex_http_client::transport: 308 rows, approximately 55.1 MiB
• TRACE codex_app_server::outgoing_message: 36,235 rows, approximately 5.9 MiB

A five-minute active-window aggregate contained another 455 codex_api::sse::responses TRACE rows (approximately 0.461 MiB). The earlier filtering fixes appear partially effective here: target=log and the codex_otel mirror targets were no longer among the largest retained sources, but per-SSE and HTTP transport persistence still dominated.

No VACUUM, checkpoint, trigger, process restart, or other database mutation was performed during this investigation because the writer still held the DB. An idle VACUUM can reclaim the freelist high-water space locally, but it does not address the underlying insert/prune churn.