Codex App writes high-volume TRACE logs to logs_2.sqlite despite RUST_LOG=warn

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #30162
• #29814
• #29589
• #29532
• #29674

Powered by Codex Action

I put up a candidate fix on my fork:

https://github.com/openai/codex/compare/main...Nicolas0315:codex/log-db-honors-rust-log?expand=1

What changed:

• The SQLite log DB layer now builds its filter from RUST_LOG instead of using a hard-coded TRACE Targets filter.
• Existing no-RUST_LOG behavior is preserved by falling back to TRACE.
• The previous hard exclusions for log, codex_otel.log_only, and codex_otel.trace_safe are retained.
• Added regression coverage for RUST_LOG=warn suppressing TRACE/INFO rows while retaining WARN, plus coverage that the no-env default still allows TRACE.

Local verification on rtx4090 at commit 6f9f177e43:

• just fmt
• git diff --check
• cargo test --manifest-path codex-rs\Cargo.toml -p codex-state sqlite_sink -- --nocapture
• cargo check --manifest-path codex-rs\Cargo.toml -p codex-state -p codex-app-server -p codex-tui

SSH validation on peer machines:

• cyberagent-mac-m5max: fetched 6f9f177e43; DIFF_CHECK_OK; focused sqlite_sink tests passed; CARGO_CHECK_OK.
• nicolas2025: fetched 6f9f177e43; DIFF_CHECK_OK; focused sqlite_sink tests passed; CARGO_CHECK_OK; no lingering cargo/rustc/link processes.
• nicolas-macbook-pro: fetched 6f9f177e43; DIFF_CHECK_OK; Cargo is unavailable on that host, so this was diff-only.
• rtx5060ti: fetched 6f9f177e43; DIFF_CHECK_OK; Cargo is blocked by the host's known linker setup issue (link: extra operand ..., exit 101) before crate code compiles; no lingering cargo/rustc/link processes.

After validation, upstream main had advanced to 526f495f3a; git merge-tree --write-tree origin/main HEAD is clean, so I left the validated commit unchanged rather than rebasing and invalidating the SSH run.

Adding a Windows + WSL-specific data point because this issue has a second-order impact that is easy to misread in Task Manager.

Environment:

• Product: Codex Desktop on Windows, WSL agent environment enabled
• Desktop package: OpenAI.Codex_26.623.4041.0_x64__2p2nqsd0c76g0
• WSL app-server CLI: codex-cli 0.142.2
• WSL app-server command: codex app-server --analytics-default-enabled
• Process env includes RUST_LOG=warn and LOG_FORMAT=json

Observed behavior:

• The active WSL app-server continued writing/churning logs_2.sqlite while idle or nearly idle.
• Recent retained rows were dominated by TRACE|log / websocket / tokio-tungstenite style records, consistent with the reports above.
• The hot WSL thread during the event was sqlx-sqlite-worker, with stack evidence in the WSL Plan9 filesystem path.
• Windows Task Manager attributed high disk I/O to dllhost.exe, but that dllhost.exe instance mapped to the WSL Plan9FileSystem COM server (vp9fs.dll, with WSL-related modules such as lxutil.dll, vmvirtio.dll, vmprox.dll).
• In a short sample, the WSL app-server showed tens of MB/s disk reads while the corresponding Windows dllhost.exe showed similar read volume. This makes the SQLite TRACE churn show up as dllhost.exe / WSL filesystem activity rather than an obvious Codex process on the Windows side.

Important local configuration detail:

• In this WSL-backed Desktop setup, the app-server was using a Windows-backed SQLite home under the Desktop Codex state directory, so WSL SQLite writes crossed the Windows/WSL Plan9 boundary.
• A local mitigation is to keep Desktop config/auth under the Windows Codex home, but move sqlite_home for the WSL app-server to a WSL-native ext4 path. That removes the Plan9/dllhost amplification, but it does not fix the root cause: persistent high-frequency TRACE logging into SQLite.

Why this matters:

• The same root issue can present as whole-system Windows disk pressure, vmmemWSL CPU/cache activity, and non-descript dllhost.exe I/O.
• Users may chase COM Surrogate, Defender, WSL, or hardware symptoms when the initiating writer is the Codex app-server SQLite log sink.
• RUST_LOG=warn is still misleading here if the persistent SQLite sink continues accepting high-volume TRACE-level records.

Expected fix:

• The persistent SQLite log sink should not default to high-volume TRACE persistence in production.
• It should honor an effective production threshold or expose a documented CODEX_LOG_DB_LEVEL / config equivalent.
• It should have hard WAL/checkpoint/size safeguards so diagnostic logging cannot create sustained disk pressure.

Adding a macOS data point from the current latest release.

Environment:

• Codex app bundle: 26.623.81905 (CFBundleVersion=4598)
• Bundled Codex CLI: codex-cli 0.142.5
• Platform: macOS
• Observed shell env in Codex: RUST_LOG=warn, LOG_FORMAT=json
• The Codex app had already been restarted; the TRACE logging persisted after restart.

Observed behavior:

• ~/.codex/logs_2.sqlite: about 684M
• ~/.codex/logs_2.sqlite-wal: about 4.6-5.0M
• SQLite stats: about 683.6 MiB total, with about 306.0 MiB free pages
• codex doctor --json reports log DB integrity: ok, so this does not appear to be caused by log DB corruption.

Recent 60-second sample after restart:

TRACE|log|955|156.0 KiB
TRACE|codex_api::sse::responses|89|123.9 KiB
TRACE|codex_mcp::connection_manager|30|32.3 KiB

Recent 300-second sample after restart:

TRACE|log|1052|776.8 KiB
TRACE|codex_api::sse::responses|419|572.3 KiB
TRACE|codex_mcp::connection_manager|168|185.3 KiB
TRACE|codex_core::session::turn|29|33.1 KiB
TRACE|codex_app_server::outgoing_message|18|2.9 KiB

Overall level distribution during testing:

TRACE|60050|267.28 MiB
INFO|13065|24.20 MiB
DEBUG|12143|23.30 MiB
WARN|1705|1.43 MiB
ERROR|35|0.04 MiB

Representative TRACE bodies were low-level transport/frame diagnostics such as tokio-tungstenite poll/read messages, parsed headers, Opcode: Data(Text), decompression messages, and Received message {"type":"response..."}. I am not attaching the SQLite DB because it may contain private session text and local paths.

This looks consistent with the candidate fix direction above: the persistent SQLite log sink should honor RUST_LOG=warn or otherwise avoid high-volume TRACE persistence in production builds.

Additional reproduction on macOS after the Codex move into ChatGPT.app.

Environment:

• App bundle: /Applications/ChatGPT.app
• ChatGPT.app version: 26.707.31123, build 5042
• Bundled Codex CLI: codex-cli 0.144.0-alpha.4
• Local DB: ~/.codex/logs_2.sqlite
• Existing local workaround before test: BEFORE INSERT ON logs BEGIN SELECT RAISE(IGNORE); END trigger named codex_block_logs_insert

Controlled test performed on 2026-07-10 Asia/Shanghai:

1. PRAGMA wal_checkpoint(TRUNCATE);
2. Temporarily dropped codex_block_logs_insert
3. PRAGMA wal_checkpoint(TRUNCATE);
4. Sampled max(id), count(*), and logs_2.sqlite-wal size/mtime for 20 seconds while Codex/ChatGPT was active
5. Restored the trigger and truncated WAL again

Observed during the 20 second unblocked window:

• Baseline before unblocking: max(id)=180295896, count(*)=3915, WAL 0B
• Sample 1: max(id)=180296162, count(*)=4181, WAL 782832B
• Sample 10: max(id)=180296274, count(*)=4293, WAL 1582112B
• WAL mtime advanced every sample interval while unblocked

New rows inserted during the test window included TRACE/DEBUG records such as:

• TRACE codex_api::sse::responses: 361 rows, ~378312 estimated bytes
• TRACE codex_app_server::outgoing_message: 161 rows, ~27831 estimated bytes
• TRACE hyper_util::client::legacy::pool: 49 rows
• TRACE codex_mcp::connection_manager: 12 rows
• DEBUG codex_core::stream_events_utils: 10 rows

Compared with earlier Codex.app builds, the very noisy tokio-tungstenite frame/header/payload traces were not present in this latest sample, so part of the original websocket frame logging appears improved. However, the persistent SQLite sink still accepts TRACE/DEBUG runtime records and continues growing WAL when the trigger workaround is removed.

After restoring the trigger:

• trigger_count=1
• WAL returned to 0B
• A 10 second post-restore sample showed max(id)=180296630 stable and WAL not growing

Expected behavior:

• With normal app usage and no explicit trace/debug configuration, the persistent SQLite log sink should not store TRACE/DEBUG records or continuously advance WAL during ordinary conversation activity.
• At minimum, SQLite persistence should respect the effective log level or apply a strict allowlist/level filter for durable logs.

I did not attach raw logs_2.sqlite or full log rows because they may contain local paths and conversation/model-response content.

Additional reproduction on the newer ChatGPT.app-bundled Codex build.

Environment:

• macOS
• ChatGPT.app: 26.715.21425, build 5488
• Bundled Codex: codex-cli 0.145.0-alpha.18
• app-server command: /Applications/ChatGPT.app/Contents/Resources/codex -c features.code_mode_host=true app-server --analytics-default-enabled
• RUST_LOG=warn
• config included:
• [analytics] enabled = false
• [feedback] enabled = false
• sqlite_home moved to an external SSD

Observed:

• After moving sqlite_home, the old ~/.codex/logs_2.sqlite stopped growing, so the config migration worked.
• The new SQLite log DB under the external sqlite_home still persisted high-volume TRACE rows.
• Before local mitigation, a 30-second sample showed:
• MAX(id) advanced by 659
• TRACE continued to advance
• recent rows were dominated by codex_api::sse::responses, codex_app_server::outgoing_message, and codex_mcp::connection_manager

Local workaround applied:

CREATE TRIGGER IF NOT EXISTS codex_block_trace_logs
BEFORE INSERT ON logs
WHEN NEW.level = 'TRACE'
BEGIN
  SELECT RAISE(IGNORE);
END;

Validation after workaround:

• In a 30-second sample, TRACE no longer appeared in recent rows.
• TRACE max(id) stayed stable.
• Overall persisted rows dropped from about +659/30s to +93/30s, roughly an 86% reduction.
• INFO / DEBUG / WARN / ERROR rows were still preserved.

This is only a local workaround. The underlying issue appears to be that the persistent SQLite log sink still accepts TRACE events despite RUST_LOG=warn and analytics/feedback collection being disabled.

Additional reproduction after the latest ChatGPT.app / bundled Codex update.

Environment:

• App bundle: /Applications/ChatGPT.app
• ChatGPT.app version: 26.715.31925, build 5551
• Bundled Codex CLI: codex-cli 0.144.6
• Local DB: ~/.codex/logs_2.sqlite
• Existing local workaround before test: BEFORE INSERT ON logs BEGIN SELECT RAISE(IGNORE); END trigger named codex_block_logs_insert

Controlled test performed on 2026-07-20 Asia/Shanghai:

1. PRAGMA wal_checkpoint(TRUNCATE);
2. Temporarily dropped codex_block_logs_insert
3. PRAGMA wal_checkpoint(TRUNCATE);
4. Sampled max(id), count(*), and logs_2.sqlite-wal size/mtime for 20 seconds while Codex was active
5. Restored the trigger and truncated WAL again

Observed during the 20 second unblocked window:

• Baseline before unblocking: count(*)=0, WAL 0B
• Sample 1: max(id)=180296162, count(*)=229, WAL 593312B
• Sample 10: max(id)=180296901, count(*)=271, WAL 1005312B
• Before restoring the trigger: max(id)=180297221, count(*)=591
• WAL mtime advanced every sample interval while unblocked

Rows inserted during the test window included TRACE/DEBUG records such as:

• TRACE codex_api::sse::responses: 365 rows, ~382504 estimated bytes
• TRACE codex_app_server::outgoing_message: 156 rows, ~26976 estimated bytes
• DEBUG codex_core::stream_events_utils: 10 rows
• TRACE codex_core::session::turn: 3 rows
• TRACE codex_mcp::connection_manager: 3 rows

Compared with older Codex.app builds, I did not see the previous very noisy tokio-tungstenite frame/header/payload records in this sample, so that part appears improved. However, the persistent SQLite sink still stores TRACE/DEBUG records during ordinary conversation activity, and WAL still grows when the trigger workaround is removed.

After restoring the trigger:

• trigger_count=1
• WAL returned to 0B
• A 10 second post-restore sample showed max(id)=180297221 stable and WAL not growing

Expected behavior:

• With normal app usage and no explicit trace/debug configuration, the persistent SQLite log sink should not store TRACE/DEBUG records or continuously advance WAL during ordinary conversation activity.
• SQLite persistence should respect the effective log level, or apply a strict durable-log allowlist/level filter.

I am not attaching raw logs_2.sqlite or full log rows because they may contain local paths and conversation/model-response content.

Additional reproduction after another ChatGPT.app / bundled Codex update.

Environment:

• App bundle: /Applications/ChatGPT.app
• ChatGPT.app version: 26.715.52143, build 5591
• Bundled Codex CLI: codex-cli 0.144.6
• Local DB: ~/.codex/logs_2.sqlite
• Existing local workaround before test: BEFORE INSERT ON logs BEGIN SELECT RAISE(IGNORE); END trigger named codex_block_logs_insert

Controlled test performed on 2026-07-20 Asia/Shanghai:

1. PRAGMA wal_checkpoint(TRUNCATE);
2. Temporarily dropped codex_block_logs_insert
3. PRAGMA wal_checkpoint(TRUNCATE);
4. Sampled max(id), count(*), and logs_2.sqlite-wal size/mtime for 20 seconds while Codex was active
5. Restored the trigger and truncated WAL again

Observed during the 20 second unblocked window:

• Baseline before unblocking: max(id)=180297221, count(*)=591, WAL 0B
• Sample 1: max(id)=180297413, count(*)=783, WAL 626272B
• Sample 10: max(id)=180297495, count(*)=865, WAL 1322552B
• Before restoring the trigger: max(id)=180297815, count(*)=1185
• WAL mtime advanced every sample interval while unblocked

Rows inserted during the test window included TRACE/DEBUG records such as:

• TRACE codex_api::sse::responses: 389 rows, ~416214 estimated bytes
• TRACE codex_app_server::outgoing_message: 146 rows, ~25271 estimated bytes
• DEBUG codex_core::stream_events_utils: 10 rows
• TRACE codex_core::session::turn: 3 rows
• TRACE codex_mcp::connection_manager: 3 rows
• TRACE codex_core::session::world_state: 1 row

I still do not see the older high-volume tokio-tungstenite frame/header/payload records in this sample, so that part appears to remain improved. However, the persistent SQLite sink still stores TRACE/DEBUG records during ordinary conversation activity, and WAL still grows when the trigger workaround is removed.

After restoring the trigger:

• trigger_count=1
• WAL returned to 0B
• A 10 second post-restore sample showed max(id)=180297815 stable and WAL not growing

Expected behavior:

• With normal app usage and no explicit trace/debug configuration, the persistent SQLite log sink should not store TRACE/DEBUG records or continuously advance WAL during ordinary conversation activity.
• SQLite persistence should respect the effective log level, or apply a strict durable-log allowlist/level filter.

I am not attaching raw logs_2.sqlite or full log rows because they may contain local paths and conversation/model-response content.