macOS: Persistent SQLite TRACE target=log churn remains after rust-v0.142.0
Summary
After upgrading to rust-v0.142.0, I can still reproduce persistent SQLite log churn on macOS in ~/.codex/logs_2.sqlite.
This looks like a partial fix:
#29432appears to have helped:codex_api::endpoint::responses_websocketis much lower than before.- However,
#29457does not appear to fully solve the persistent SQLite churn in this app build/runtime. The release note says noisy persistent log targets were filtered, and the PR says bridgedtarget=logevents should be excluded from the SQLite sink, butTRACE target=logis still the dominant persisted target here.
I also posted an earlier data point in the closed umbrella issue #28224 here:
https://github.com/openai/codex/issues/28224#issuecomment-4775086154
Opening a new issue because #28224 is closed and this is post-rust-v0.142.0 macOS data.
Environment
- Platform: macOS 26.5.1, arm64
- Codex app:
26.616.71553, build4265 - Embedded CLI:
codex-cli 0.142.0 - PATH CLI:
codex-cli 0.142.0 - App-server process:
/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled - App-server PID/start at sample time:
7763, startedTue Jun 23 10:17:10 2026 RUST_LOG: empty~/.codex/config.toml:[analytics] enabled = false- SQLite trigger count during the sample:
0 logs_2.sqlitejournal mode:wal
No local block_log_inserts trigger was present during this sample.
Files observed
At the start of the sample:
~/.codex/logs_2.sqlite 745,517,056 bytes
~/.codex/logs_2.sqlite-wal 4,437,272 bytes
~/.codex/logs_2.sqlite-shm 32,768 bytesThe active app-server process was holding the DB/WAL/SHM files open.
Read-only 60 second sample
Sample window:
2026-06-23 10:43:00 +0800 to 2026-06-23 10:44:00 +0800SQLite metadata before/after:
trigger_count: 0 -> 0
row_count: 200589 -> 200589
max_id: 725810463 -> 725811933
sqlite_seq: 725810463 -> 725811933
max_ts: 1782182580 -> 1782182638So the retained row count stayed flat, but sqlite_sequence / max_id advanced by 1,470 in about 60 seconds. This looks like continued insert-and-prune churn.
File mtimes also moved during the sample:
logs_2.sqlite mtime: 1782182580 -> 1782182613
logs_2.sqlite-wal mtime: 1782182580 -> 1782182638The WAL size stayed stable at about 4.4 MB, so file size alone hides the ongoing write activity.
Recent persisted targets
Rows in the last 60 seconds:
TRACE log 989 rows, 526.8 KiB
TRACE codex_api::sse::responses 14 rows, 14.0 KiB
TRACE codex_app_server::outgoing_message 12 rows, 2.0 KiB
TRACE codex_mcp::connection_manager 6 rows, 4.9 KiB
DEBUG codex_core::stream_events_utils 3 rows, 3.3 KiB
INFO feedback_tags 3 rows, 3.0 KiB
INFO codex_core::stream_events_utils 2 rows, 5.5 KiB
TRACE codex_api::endpoint::responses_websocket 1 row, 40.8 KiB
TRACE codex_core::session::turn 1 row, 0.9 KiBTRACE targets in the last 5 minutes:
log 989 rows, 526.8 KiB
codex_api::sse::responses 238 rows, 239.5 KiB
codex_mcp::connection_manager 18 rows, 13.8 KiB
codex_app_server::outgoing_message 12 rows, 2.0 KiB
codex_core::session::turn 2 rows, 1.8 KiB
codex_api::endpoint::responses_websocket 2 rows, 80.0 KiBExpected behavior
After rust-v0.142.0, Codex should not continuously persist high-frequency TRACE target=log rows into logs_2.sqlite by default, especially after #29457.
Actual behavior
Codex app-server continued to insert high-frequency TRACE target=log rows into the persistent SQLite log sink. The retained row count stayed flat, but max(id) / sqlite_sequence kept advancing and the DB/WAL mtimes kept moving, indicating ongoing insert-and-prune churn.
Privacy note
I am intentionally not pasting raw feedback_log_body contents because they may contain private conversation, local path, tool, or response data.
43 Comments
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
Additional macOS reproduction on the current Codex app build.
Environment:
APPLE SSD AP0512Z), SMART statusVerified26.616.71553, build4265codex-cli 0.142.0/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled34793RUST_LOG=infowas set vialaunchctl getenv RUST_LOG, but persisted SQLite rows still includedTRACEandDEBUG.Files observed:
lsofshowed the active Codex app-server process holding open handles to the DB, WAL, and SHM files.60-second quiet sample:
This again suggests insert-and-prune churn: retained row count barely changed, but
max(id)advanced by 1,666 in about a minute.Rows visible in that sample window included:
Total visible
TRACErows in the sample window:940.The latest
TRACE target=logsamples were websocket/tungstenite noise, including:Current visible level distribution at collection time:
Impact: this creates sustained writes to the persistent SQLite log sink during normal Codex app use.
RUST_LOG=infodid not stop these persistedTRACErows, so the user-level mitigation does not appear to work for this path.I have a private evidence package with raw command output, SQLite read-only queries, file metadata, hashes, and the 60-second WAL/SQLite sample. I am not attaching raw
feedback_log_bodycontents publicly because they may include local paths, conversation IDs, command fragments, or response data.Please fix this as soon as possible! This issue is widespread and serious. Thank you.
+1 Please fix this as soon as possible! This issue is widespread and serious. Thank you.
Additional current-build datapoint from macOS Desktop
26.616.71553/ build4265, bundled/globalcodex-cli 0.142.0.During active Desktop use, I temporarily removed a local SQLite mitigation trigger, recorded
max(id)=707460512, waited 15 seconds, then restored the trigger. New retained rows over that window:Rows matching the previously noisy targets (
log,codex_otel.log_only,codex_otel.trace_safe,codex_api::endpoint::responses_websocket,codex_api::sse::responses) were1075in the same 15-second window.This is another confirmation that the current Desktop app-server path is still persisting high-volume TRACE
target=logrows on the shipped 0.142.0 build. The local mitigation trigger was restored after the measurement.+1 OMG🤯
Additional macOS data point from another machine: I can still reproduce persistent SQLite TRACE churn after restarting Codex Desktop and after manually compacting the log DB.
Environment
26.616.71553, build4265codex-cli 0.142.0codex-cli 0.142.0/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabledRUST_LOGobserved in the process environment:warn~/.codex/logs_2.sqlitejournal mode observed earlier:walI am not attaching or pasting raw
feedback_log_bodyrows because they can contain private conversation/tool/local-path details.Before manual cleanup
Before compacting, the local log files were:
The table showed high churn/free space:
A 30 second sample after restarting Codex Desktop showed continued insert-and-prune activity:
Recent 2 minute target distribution during that sample was still dominated by TRACE rows:
After manual cleanup / compaction
I then retained only a small recent window of diagnostic rows and ran checkpoint/VACUUM. Integrity check returned
ok, and the files were reduced to:However, without changing anything else, Codex Desktop quickly started growing the files again. About 80 seconds later:
Rows retained in the last 5 minutes at that point:
Expected
After
rust-v0.142.0/ #29432 / #29457, the persistent SQLite sink should not continue high-frequency insert-and-prune churn dominated byTRACE target=logand related websocket/SSE targets during normal Desktop use.Actual
Even after restart and after compacting the database down to a tiny valid state, the active Desktop app-server began writing again immediately. The on-disk DB/WAL grew from about
1.8M + 0Bto about38M + 37Mwithin roughly 80 seconds, and the retained recent rows were still dominated by TRACE targets.This makes the current behavior look like an incomplete fix or a remaining Desktop-app persistent-log sink path that still persists noisy TRACE rows.
I’m seeing what looks like the same underlying SQLite churn pattern on macOS, though my setup differs slightly: I have not specifically disabled analytics/plugins, and the running process is
codex app-server --analytics-default-enabled.I posted the fuller diagnostic here: https://github.com/openai/codex/issues/29612#issuecomment-4780728496
Summary of my repro:
Over a 5-minute idle sample, with Codex Desktop open in the background but not actively used:
The DB was healthy and compacted beforehand, so this did not appear to be leftover pre-fix bloat:
The exact sampled ID window showed retained rows dominated by
TRACE target=log:All retained rows from that sampled window belonged to the same app-server process.
So this looks like the same post-0.142.0 behavior: file size may remain bounded, but the app-server still continuously inserts and prunes persistent SQLite log rows while idle, mostly
TRACE target=log, despiteRUST_LOG=warn.Additional macOS datapoint after cleanup/compaction, from another affected machine.
I can still reproduce the post-
0.142.0insert-and-prune churn after fully compacting the log DB. The historical bloat was cleaned up successfully, but the current app-server continues to persistTRACE target=logrows.Environment
The active app-server process was holding open handles to
logs_2.sqlite,logs_2.sqlite-wal, andlogs_2.sqlite-shm.Before cleanup
Before manually clearing/compacting the DB, the files had grown to:
SQLite stats at that point showed this was mostly free pages from churn:
After
DELETE FROM logs, resetting the sequence, andVACUUM, the DB compacted successfully:Repro after cleanup / restart
Immediately after cleanup and restart, rows began accumulating again, dominated by
TRACE target=log:A 20-second sample shortly after restart showed much higher active churn:
Top retained rows from that ID window:
A later quieter 30-second sample still showed the same insert-and-prune pattern with file sizes bounded:
Top retained rows from that sampled ID window:
I am intentionally not pasting raw
feedback_log_bodyvalues because they may contain private local paths, conversation, tool, or response data.User-visible impact
Before cleanup, while several Codex tasks were open at the same time, I observed frequent Codex restarts and UI lag. Cleaning/VACUUM reduced the historical DB size from ~811M to ~5.3M, but the high-frequency
TRACE target=loginsert/prune behavior is still present on the current Desktop app-server.So the remaining issue seems to be:
0.142.0reduces/bounds some file-size growth, but the shipped macOS Desktop app-server still persists high-frequencyTRACE target=logrows and continues SQLite churn even withRUST_LOG=warn.Additional datapoint after updating to the newer macOS app build
26.616.81150/ build4306.Environment:
26.616.81150, build4306codex-cli 0.142.0/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled88553, startedWed Jun 24 09:20:21 2026Controlled sample with the trigger disabled:
Rows visible in the sample window:
Total
TRACErows in the 30-second window:453.The latest
TRACE target=logrows are still websocket/tungstenite noise, includingWouldBlock,tokio-tungstenite,WebSocketStream.with_context,Stream.poll_next,Received message, decompression, and<FRAME>parsing lines.So this still reproduces on app build
4306; it does not appear fully fixed by this build.现在macos的codex桌面端更新到了26.622.11653，还有这个问题吗
Additional macOS data point from Codex Desktop showing that the issue still reproduces on my machine, and that
sqlite_homeis currently the only effective local mitigation I found.Environment
diskutil: SMART StatusVerified, TRIM supported)/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabledBROWSER_USE_CODEX_APP_VERSION=26.616.81150codex-cli 0.142.0~/.codex/logs_2.sqlite~/.codex/logs_2.sqlite-wal~/.codex/state_5.sqlite-walConfig tried
~/.codex/config.tomlincluded:The restarted app-server process also inherited:
Confirmed via:
Observed behavior before mitigation
After restarting Codex Desktop with the config above, a 60-second
proc_pid_rusagesample still showed:The new app-server process had already accumulated substantial writes shortly after restart:
In an earlier active diagnostic window, app-server lifetime written bytes reached about:
Recent rows in
logs_2.sqlitestill included persisted TRACE records despiteRUST_LOG=warn:Largest recent targets included:
This confirms the SQLite feedback/log sink is still persisting TRACE-level data even when the process environment has
RUST_LOG=warnand analytics/OTel exporters are disabled.Why file size alone hides the issue
The retained DB size was not enough to estimate real write volume. The file stayed bounded while
logs_2.sqlite-waland SQLite insert/prune churn continued. This matches the pattern described in this issue: current file size can look moderate while SSD writes continue due to insert, WAL write, prune, checkpoint, and rewrite activity.Local mitigation tested
I moved only Codex's SQLite-backed runtime state to a RAM disk:
After restarting Codex Desktop,
lsofconfirmed these were all on the RAM disk:A follow-up 60-second sample still showed process-level writes:
But visible internal SSD growth for the Codex SQLite files dropped to zero because the SQLite writes were on the RAM disk:
A broader 30-second scan of likely Codex internal-SSD paths showed only tiny non-SQLite growth:
So
sqlite_hometo RAM disk mitigates SSD wear, but it does not fix the root cause. It only moves the high-frequency SQLite WAL writes away from the internal SSD.Expected fix
Please make the persistent SQLite log sink honor the same filtering as
RUST_LOG/ runtime log level, or provide a documented config option to disable persistent SQLite feedback logs or cap them aggressively. WithRUST_LOG=warn,[analytics].enabled=false, and all[otel]exporters set tonone, TRACE websocket/SSE/internaltarget=logrecords should not continue to be persisted tologs_2.sqliteduring normal desktop usage.Privacy note
I am not attaching the SQLite DB or raw
feedback_log_bodyrows because they may contain private local paths, tool output, and conversation/session details.Follow-up after the question about macOS Desktop
26.622.11653:I checked the affected macOS machine again today. This machine has not yet updated to
26.622.11653, so I cannot confirm whether that newer build fixes the issue.Current local version on this machine:
On this
26.616.81150 / build 4306install, the issue is still reproducible.Before the latest sample, the log files had already grown again after the previous cleanup/compaction:
A read-only ~30 second sample showed continued insert/prune churn:
Recent 5 minute target distribution was still dominated by TRACE rows:
Recent rows were dominated by the active Desktop app-server process:
So my current data point is:
26.616.81150 / build 4306: still reproduces.26.622.11653: not available on this machine yet, so I cannot confirm either way.I am still not pasting raw
feedback_log_bodycontents because they may contain private conversation/tool/local-path details.Confirming this on
codex-cli 0.142.2, pinned to source tagrust-v0.142.2(390b0d25). It's platform-independent (same as the Windows reports #29799/#29463 and macOS #29532); I just hit it on macOS where the OS flags excessive disk writes and, on a busy volume, the continuous WAL I/O stalled the host. It is disk I/O, not memory (process footprint ~130 MB).Root cause, verbatim at the tag:
state/src/log_db.rs:53-59— persisted sink hardcoded toLevelFilter::TRACE;RUST_LOGonly drives the stderr layer (whyRUST_LOG=warnhas no effect, #29463).state/src/runtime/logs.rs— a window-functionDELETEruns inside every insert txn; capsLOG_PARTITION_SIZE_LIMIT_BYTES = 10 MiB,LOG_PARTITION_ROW_LIMIT = 1000(runtime.rs:97-98).state/src/runtime.rs:362-370— WAL mode with nowal_autocheckpoint/journal_size_limit;wal_checkpoint(PASSIVE)only at startup (runtime/logs.rs:296-310) → unbounded WAL (#28997).app-server/src/lib.rs:665-668/tui/src/lib.rs:1242-1245— sink started unconditionally; no config/env/flag to disable or lower it.Cross-platform fix that fits the existing design: default the persisted sink to INFO with an env/
config.tomloverride (mirroring howRUST_LOGdrives the stderr layer) and a[log] persist = falseoff-switch; batch inserts into one txn per N ms/rows; move the prune off the insert path to a periodic task; and setwal_autocheckpoint+ periodicwal_checkpoint(TRUNCATE)to bound the WAL. Ideally, keep recent logs in a bounded in-memory ring buffer and only materialize to SQLite when/feedbackis invoked, eliminating steady-state disk writes.Follow-up to my analysis above: prepared a fix (configurable level w/ INFO default, off-switch, bounded WAL). Reference branch: https://github.com/denispol/codex/pull/1
Additional macOS datapoint after updating to Codex app
26.623.31443/ build4441.Environment:
26.623.31443, build4441codex-cli 0.142.2/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled18892, startedFri Jun 26 10:23:37 2026RUST_LOG=infoControlled sample with the trigger disabled:
Rows visible in the sample window:
Total rows in the 30-second window:
This still reproduces on
26.623.31443/ build4441/codex-cli 0.142.2. The persisted TRACE/DEBUG path still appears to bypassRUST_LOG=info. The latest retained TRACE/DEBUG rows in the sample were network/proxy/client/poller/platform-detection noise, plus app-server/SSE/outgoing-message entries.After the sample, I restored the local
block_trace_debug_logsmitigation trigger and ranwal_checkpoint(TRUNCATE); the WAL returned to0bytes.Additional macOS datapoint after updating to Codex app
26.623.31921/ build4452.Environment:
26.623.31921, build4452codex-cli 0.142.2/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled51634, startedFri Jun 26 14:05:49 2026RUST_LOG=infoControlled sample with the trigger disabled:
Rows visible in the sample window:
Total TRACE rows in the 30-second window:
351.This still reproduces on
26.623.31921/ build4452/codex-cli 0.142.2, though this particular sample was lower volume than my previous26.623.31443/ build4441sample. The latest retainedTRACE target=logrows are still websocket/tungstenite frame noise includingWouldBlock,WebSocketStream.with_context,Stream.poll_next,Received message, and<FRAME>parsing lines.After the sample, I restored the local
block_trace_debug_logsmitigation trigger and ranwal_checkpoint(TRUNCATE); the WAL returned to0bytes.Follow-up datapoint on the latest macOS app build I currently have installed.
Environment:
26.623.31921, build4452codex-cli 0.142.2codex-cli 0.142.2/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled75427, startedFri Jun 26 14:15:39 2026RUST_LOG=warnImportant note: I did not disable my local SQLite mitigation for this check. Two local triggers are currently installed:
sample_log_inserts: ignores inserts when the latest retained row is newer thanNEW.ts - 10, effectively retaining at most about one row every 10 seconds.trim_sampled_logs: keeps only the newest 500 retained rows.With those triggers left enabled, the persisted rows still show that the source is continuously producing TRACE records after the update:
The latest retained rows from the active app-server process continued at the mitigation cadence, about one retained row every 10 seconds:
The
TRACE target=logentries are still websocket/tungstenite noise such asWebSocketStream.with_contextandStream.poll_next.Current file sizes at the end of the check:
So the local trigger mitigation is keeping disk growth bounded on this machine, but the root issue still appears present on
26.623.31921/ build4452/codex-cli 0.142.2: the persistent SQLite sink is still receiving/storing TRACE-level records even withRUST_LOG=warn. The mitigation reduces retained/committed rows; it does not prove the upstream source has stopped emitting TRACE logs.Additional macOS datapoint on Codex app
26.623.31921/ build4452.Environment:
26.623.31921, build4452codex-cli 0.142.2codex-cli 0.142.2/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled18323sqlite_mastertrigger count: 0)Watchdog sample:
Main writer attribution from the watchdog report:
The report also observed that the monitored hot files did not grow during the window:
This looks consistent with SQLite/WAL insert-prune/checkpoint or page-reuse churn: high process-level writes without corresponding file-size growth.
Current
logs_2.sqlitestate after the report:Largest retained log-body targets currently visible:
This still appears to be dominated by the desktop
app-serverpersistent SQLite log path after the 0.142.x fixes. The watchdog report is useful because it correlates app-server process writes with system disk writes over a fixed window, not just retained SQLite row counts.Could you confirm whether the intended fix is only to reduce retained rows/targets, or whether the persistent SQLite log sink is also expected to stop generating this level of WAL / insert-prune write amplification? A supported temporary config/env switch to disable or lower the persisted SQLite log sink would also help avoid local trigger workarounds.
Follow-up CLI comparison after fully quitting the desktop app-server.
Procedure:
codex app-serverprocess was running.logs_2.sqlitebaseline.codex exectask.logs_2.sqlitestate again.Before
codex exec:After
codex exec:Watchdog report window:
Important detail: the report window still included desktop app shutdown tail activity at the beginning. Breaking down the sampled process writes:
The desktop tail writes were concentrated before the app-server was fully gone. The later
codex exectask only produced small session-file writes, andlogs_2.sqlite/ WAL /max(id)did not change.This suggests the large persistent SQLite log write amplification is primarily tied to the desktop
app-serverpath, not to non-interactivecodex execin this test. It also gives users a practical temporary workaround: quit the Desktop App and usecodex execfor read-only / automation-style work when possible.Additional macOS datapoint on Codex Desktop
26.623.42026/ build4514.Environment:
26.623.42026, build4514codex-cli 0.142.3/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled45836RUST_LOG=warnfeedback_log_bodyvalues are included here because they may contain private local paths, tool output, or conversation/session details.The app-server process was holding open handles to the persistent log DB:
Current file sizes during the sample:
SQLite health/state:
Recent 60-second sample:
Recent 10-minute level distribution:
Top retained targets in the same 10-minute window:
So this still reproduces for me on
26.623.42026/ build4514/ embeddedcodex-cli 0.142.3: the persistent SQLite log sink continues to store high-volumeTRACEandDEBUGrows even though the active app-server process hasRUST_LOG=warn.This looks consistent with the root-cause analysis above: the persistent SQLite sink appears to be filtering independently from
RUST_LOG, so the newer0.142.3build still has the desktop app-server SQLite churn path.Adding a current macOS Desktop
26.623.42026/ CLI0.142.3datapoint.Snapshot time: 2026-06-28 11:26 JST
Environment:
26.623.42026, build45140.142.3codex doctor --json:overallStatus=oklaunchctl getenv RUST_LOG:warnSQLite state:
Persisted row levels:
Top persisted
(level, target)rows:So on this build, TRACE rows are still the dominant persisted level in
logs_2.sqliteeven with launchdRUST_LOG=warn. This is not currently disk-filling on my machine, but it is still large enough to matter for startup/runtime pressure and backup/log maintenance.Omitting raw log rows and local paths beyond the database filename.
This is a management project I developed for codex to overwrite logs. It includes functions for analyzing, mitigating, and deleting redundant logs, and the documentation explains the implementation steps and runtime logic in detail. I tested it on this machine and it works well, give it a try! [https://github.com/RuntianLee/codex-disk-guard](url)
Follow-up from the same affected Mac after updating to the latest build currently available here.
Environment:
26.623.42026, build4514codex-cli 0.142.3codex-cli 0.142.3/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled207, startedMon Jun 29 09:57:24 2026RUST_LOG=warnImportant note: I left my local SQLite mitigation triggers enabled for this check. They are still:
sample_log_inserts: retain at most about one row every 10 seconds.trim_sampled_logs: keep only the newest 500 retained rows.So this is not a raw insert-rate measurement. It is a protected-state check to see whether the current app-server is still producing persisted TRACE records after the 0.142.3 update.
Current retained state:
The retained rows are now dominated by the new active desktop app-server process:
30-second protected sample:
logs_2.sqlite-walmtime advanced with each retained row in that sample. File sizes stayed bounded because of the local mitigation:The latest retained
TRACE target=logrows are still mostly websocket/tungstenite noise such asStream.poll_next/WebSocketStream.with_context, plus app-server dispatch/pool entries. I am still not pasting rawfeedback_log_bodyrows beyond short target/category descriptions because they may contain private local/session details.Conclusion from this machine:
26.623.42026/ build4514/codex-cli 0.142.3still appears to feed TRACE-level events into the persistent SQLite sink despiteRUST_LOG=warn. The local trigger mitigation keeps SSD writes bounded here, but it does not indicate that the upstream persistent TRACE source has stopped.Additional macOS datapoint after updating to Codex app
26.623.70822/ build4559.Environment:
26.623.70822, build4559codex-cli 0.142.4/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled43548, startedTue Jun 30 09:34:09 2026RUST_LOG=infofeedback_log_bodyvalues are included here because they may contain private local paths, tool output, or conversation/session details.Controlled sample with the trigger disabled:
Rows visible in the sample window:
After restoring the local trigger and running
PRAGMA wal_checkpoint(TRUNCATE), the local state was:A follow-up 30-second observation with the trigger enabled showed only retained INFO/WARN rows, while TRACE/DEBUG rows were not persisted by the local mitigation. This suggests build
4559still produces persistent TRACE/DEBUG log churn when the local SQLite mitigation is removed.Follow-up from the same affected Mac after updating again.
Environment:
26.623.70822, build4559codex-cli 0.142.4codex-cli 0.142.4/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled412, startedTue Jun 30 10:09:01 2026RUST_LOG=warncodex_chroniclewas also running (pid 532), but the latest retained SQLite rows below are dominated by the app-server process UUID forpid 412.Important note: I left the local SQLite mitigation triggers enabled for this check:
So this is a protected-state check, not a raw insert-rate measurement. On this machine those triggers retain at most about one row every 10 seconds and keep only the newest 500 retained rows.
Current retained SQLite state:
Recent retained rows are dominated by the active desktop app-server:
30-second protected sample:
logs_2.sqlite-walmtime advanced with these retained rows. File sizes at the end of the check:Top retained targets at the end of the check:
I am intentionally not pasting raw
feedback_log_bodyvalues. One retained non-TRACE row appeared to include a response/token-like payload, which is a useful privacy signal: this persistent SQLite sink can contain local/session-sensitive data, not just harmless category names.Conclusion from this machine:
26.623.70822/ build4559/codex-cli 0.142.4still appears to feed TRACE-level events into the persistent SQLite sink despiteRUST_LOG=warn. The local mitigation keeps retained rows and SSD growth bounded here, but it does not show that the upstream persistent TRACE source has stopped.<img width="2546" height="960" alt="Image" src="https://github.com/user-attachments/assets/5c39b2ae-5dd0-4fae-a141-65f33cd1e12d" />
Already fix?
Additional macOS datapoint: the high-frequency TRACE writes still reproduce after setting
RUST_LOG=errorwithlaunchctland fully restarting the Codex app via Cmd+Q.Environment:
26.6(25G5043d),arm6426.623.70822, build4559codex-cli 0.142.4/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled19397, startedWed Jul 1 09:59:43 2026launchctl print gui/501showsRUST_LOG => errorlogs20-second sample after restart:
Column order:
COUNT(*)|MAX(id)|SUM(level="TRACE")|MAX(trace id)|MAX(ts).MAX(id)andMAX(trace id)both continue to increase; WAL mtime also advances.Rows with
id > before_max_idwere mostly TRACE:This suggests the desktop app-server SQLite feedback log path is still persisting TRACE-level records despite
RUST_LOG=errorbeing present in the GUI launchd environment and the app being restarted.Follow-up from the same affected Mac after updating to a newer Desktop build.
Environment:
27.0(26A5368g),arm6426.623.81905, build4598codex-cli 0.142.5codex-cli 0.142.5/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled39970, startedWed Jul 1 12:33:06 2026sample_log_inserts,trim_sampled_logsImportant note: this is a protected-state check, not a raw insert-rate measurement. I did not remove the local mitigation this time. On this machine those triggers retain at most about one row every 10 seconds and keep only the newest 500 retained rows.
Current retained SQLite state:
30-second protected sample:
Top retained targets after the sample:
File sizes at the end of the check:
Conclusion from this machine:
26.623.81905/ build4598/codex-cli 0.142.5does not look fixed yet. Even with the local sampled-retention trigger active, retained rows are still500/500TRACE andmax_trace_idcontinues to advance. The local mitigation is containing retained rows and disk growth here; it is not evidence that the upstream Desktop app-server stopped feeding TRACE events into the persistent SQLite sink.I am not pasting raw
feedback_log_bodyvalues because they may include local paths, tool output, or conversation/session details.I tested 0.142.5 and it was not fixed+11111111111111
Follow-up after updating to 0.142.5, which includes #30771:
I updated to Codex CLI 0.142.5, which includes the #30771 websocket trace fix, and collected another diagnostic window.
The previous “full Responses WebSocket request payload written to trace logs” symptom appears to be improved: in the new window, the largest log body is only about 53 KB, and the main recent log targets are not dominated by
codex_api::endpoint::responses_websocket.However, I still observe residual write amplification while Codex is mostly idle:
~/.codex/logs_2.sqliterows: 86059 -> 87005 (+946)logs_2.sqlitemax id: 296657192 -> 296822464 (+165272)codex_api::sse::responses,codex_core::spawn,codex_mcp::connection_manager, andlogSo my current understanding is:
#30771 seems to fix one important payload-size source, but there may still be another source of write amplification, possibly from high-frequency log insertion/pruning, WAL/checkpoint behavior, or bridged
logevents. The key signal is thatmax_idincreased by ~165k while the retained row count only increased by ~946 and the retained log body volume was small.One caveat: the single largest spike in this window was attributed to a child tool process, so that specific spike may not be the Codex core logger itself. But even excluding that spike, the Codex core process still produced substantial low-CPU writes over the window.
Please let me know if there is a specific trace/log setting or diagnostic command you would like me to run on 0.142.5.
I can reproduce this on macOS as well.
Environment:
Observed before mitigation:
A local SQLite trigger that ignores only TRACE inserts stopped new TRACE growth:
This looks like the same persistent SQLite TRACE churn described in #29532, still reproducible on Codex app 26.623.70822 / embedded CLI 0.142.4.
Additional macOS datapoint after updating to Codex app
26.623.101652/ build4674.Environment:
26.623.101652, build4674codex-cli 0.142.5/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabledRUST_LOGwas unset in the launchd environment at the time of the check.feedback_log_bodyvalues are included here because they may contain private local paths, tool output, or conversation/session details.Notable process state after a full app restart/update:
Controlled sample with the trigger disabled:
Rows visible in the sample window:
Rows by process UUID in the same sample window:
After restoring the local mitigation trigger and running
PRAGMA wal_checkpoint(TRUNCATE), the local state was:A follow-up 30-second observation with the trigger enabled persisted only INFO/WARN rows, while TRACE/DEBUG rows were not persisted by the local mitigation.
This suggests
0.142.5/ build4674may have improved the large websocket request-payload trace symptom from #30771, but the broader persistent TRACE/DEBUG SQLite churn still reproduces when the local SQLite mitigation is removed. In this run, both active app-server processes contributed rows.Follow-up from the same affected Mac after updating to a newer Desktop build again.
Environment:
27.0(26A5368g),arm6426.623.101652, build4674codex-cli 0.142.5codex-cli 0.142.5/Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled98496, startedMon Jul 6 10:49:08 2026launchctl getenv RUST_LOGreturned empty/unset during this check/Users/ron/.local/bin/codex app-server --listen stdio://, PID3728sample_log_inserts,trim_sampled_logsImportant note: this is another protected-state check, not a raw insert-rate measurement. I did not remove the local mitigation this time. On this machine those triggers retain at most about one row every 10 seconds and keep only the newest 500 retained rows.
Current retained SQLite state after the sample:
30-second protected sample:
File sizes at the end of the check:
Top retained targets after the sample:
The newest retained rows immediately after the sample were still from the active Desktop app-server process UUID:
Conclusion from this machine:
26.623.101652/ build4674/codex-cli 0.142.5still does not look fully fixed. The large websocket request-payload trace symptom may be improved, as others noted for #30771, but the broader persistent SQLite TRACE sink still appears active: retained rows remain500/500TRACE andmax_trace_idcontinues to advance under the local sampled-retention trigger.I am not pasting raw
feedback_log_bodyvalues because they may include local paths, tool output, or conversation/session details.Additional reproduction on ChatGPT-integrated Codex / 0.144.0-alpha.4
I can still reproduce the bridged
TRACE target=logpersistence on a newer macOS build that embeds Codex inside the ChatGPT app.Environment
26.707.31428(build5059)/Applications/ChatGPT.app/Contents/Resources/codexcodex-cli 0.144.0-alpha.4~/.codex/logs_2.sqliteEvidence collected before installing any local workaround
The current app-server process persisted 436 rows with
target=logbetween:Example sanitized messages included:
At the time of inspection, the retained target counts included:
No retained rows were found for
codex_otel.log_onlyorcodex_otel.trace_safe, so that part of the filtering appears effective. However, bridgedtarget=logevents are still reaching SQLite, despite #29599 being intended to reject them inside the sink.This appears to be either:
metadata.target() == "log"guard.Local confirmation/workaround
After the measurement, I installed a SQLite
BEFORE INSERT ... RAISE(IGNORE)trigger for the noisy targets. A test insert was ignored, and normal non-blocked log IDs continued advancing while the blocked-target maximum ID stayed unchanged. This is only a local workaround; filtering before formatting/queueing remains preferable.Raw payloads, thread IDs, local paths, and conversation contents are intentionally omitted.
Follow-up from the same affected Mac after updating to the newer ChatGPT-integrated Codex build.
Environment:
27.0(26A5378j),arm6426.707.61608, build5200/Applications/ChatGPT.app/Contents/Resources/codexcodex-cli 0.144.0-alpha.4command -v codexnow resolves to the ChatGPT embedded binary on this machine/Applications/Codex.app/Contents/Resources/codexbinary was not present during this check/Applications/ChatGPT.app/Contents/Resources/codex -c features.code_mode_host=true app-server --analytics-default-enabled92069, startedMon Jul 13 10:03:30 2026~/.local/bin/codex app-server --listen stdio://, PID89531launchctl getenv RUST_LOGreturned empty/unset during this checksample_log_inserts,trim_sampled_logsImportant note: this is another protected-state check, not a raw insert-rate measurement. I did not remove the local mitigation this time. On this machine those triggers retain at most about one row every 10 seconds and keep only the newest 500 retained rows.
Current retained SQLite state after the sample:
30-second protected sample:
File sizes at the end of the check:
Top retained targets after the sample:
The newest retained rows immediately after the sample included new TRACE rows from both the ChatGPT-integrated app-server and the stdio app-server:
Conclusion from this machine: the old
500/500 TRACEprotected-state symptom has changed in the ChatGPT-integrated0.144.0-alpha.4build: retained rows now include DEBUG/INFO/WARN as well. However, the broader issue still does not look fully fixed here.max_trace_idcontinued to advance during the protected sample, andTRACE target=logremains the largest retained target. This matches the concern in the newer ChatGPT-integrated reports: part of the older filtering appears improved, but bridged TRACE/log-style events still reach the persistent SQLite DB.I am not pasting raw
feedback_log_bodyvalues because they may include local paths, tool output, or conversation/session details.I’ve completed an independent reference implementation and validation effort for the persistent SQLite TRACE/DEBUG issue described here.
Repository:
https://github.com/Freyliu0516/Codex-Log-Doctor
Proposed upstream change
The upstream patch is deliberately small and separate from the user-side diagnostic tool:
TRACEtoWARN;Reference patch:
https://github.com/Freyliu0516/Codex-Log-Doctor/tree/main/upstream
The independent tool does not patch or replace an installed Codex binary. The
upstream/directory exists only as a reviewable reference implementation.Validation
The patched Codex source was tested against the pinned upstream base:
codex-stateprimary tests: 151 passed;The separate Codex Log Doctor repository also passed:
The repository additionally contains 18 deterministic component and failure-mode scenarios covering SQLite/WAL contention, queue saturation, backup and rollback, interrupted operations, unsupported schemas, and report redaction. These are supporting tests, not a claim that all 18 scenarios exercise the complete Codex app-server pipeline.
Public CI evidence is linked from the merged initial implementation PR:
https://github.com/Freyliu0516/Codex-Log-Doctor/pull/1
User-side mitigation
The tool can install a fingerprint-gated SQLite
BEFORE INSERTtrigger that rejects TRACE rows. It requires:--apply;This is intended only as a reversible workaround. It reduces retained TRACE rows,
MAX(id)growth, WAL writes, indexing, and subsequent pruning, but it does not eliminate upstream event construction, formatting, queueing, transactions, or attempted INSERT execution.Early filtering in Codex remains the preferred fix.
Request
Would the Codex team be open to reviewing the persistence-filter approach?
If changing the persistent SQLite default from TRACE to WARN is consistent with the intended feedback and diagnostics behavior, I would appreciate guidance on whether the team would invite a focused upstream PR limited to:
codex-rs/state/src/log_db.rs;I’m also happy to adjust the proposed target-specific policy if there are operational events that the feedback pipeline must continue retaining below WARN.
I can still reproduce significant persistent SQLite TRACE churn on the current ChatGPT-integrated Codex desktop build, and I think this deserves clarification after #28224 was closed.
Environment
/Applications/ChatGPT.app26.707.722215307codex-cli 0.144.2~/.codex/logs_2.sqliteCODEX_HOMEDBs such asclient-hub-codex-home/logs_2.sqliteControlled test
I had been using a local SQLite stopgap trigger:
With the trigger enabled, short samples show:
id_delta=0count_delta=0I then temporarily removed the trigger for a controlled test and restored it immediately afterward.
Result: within roughly a one-minute active Codex window, the global
~/.codex/logs_2.sqliteaccumulated about 2.6k rows and the WAL grew to about 4.15 MiB.Dominant persisted targets were still TRACE/DEBUG-style internal plumbing:
TRACE codex_api::sse::responsesTRACE codex_mcp::connection_managerTRACE codex_app_server::outgoing_messageTRACE hyper_util::client::legacy::*DEBUG codex_core::stream_events_utilsAfter restoring the trigger, another short sample returned to:
id_delta=0count_delta=0Why I am posting
#28224 was closed as fixed, but the current shipping desktop app on my machine still bundles
codex-cli 0.144.2. From the discussion around #28224, it sounds like the more relevant logging fixes (#31789, #31790, #31791, #31792) are expected in the0.145.0-alpha.4+line, not necessarily in this bundled desktop version.So I would like to clarify:
block_log_insertsuntil the bundled desktop CLI reaches the fixed line?I would also encourage other affected users to share comparable measurements with:
MAX(id)deltaFresh measurements on codex-cli 0.144.3 (Linux/ext4), plus a minimal tested patch and an A/B benchmark. TL;DR: most of the remaining churn is flush policy, not log content — batching the sink by bytes/severity instead of every 2 s cuts its physical writes ~5x on a steady-state workload, down to SQLite's WAL+checkpoint double-write floor, with zero change to what
/feedbacksees.Measurements: the writes are flush overhead, not content
Four resumed TUI sessions left idle; write rates from
/proc/<pid>/iodeltas, content attribution from queryinglogs_2.sqlite(read-only) for rows inserted inside the same instrumented window:| metric | value |
|---|---|
| physical writes in a 59 s idle window (2 busiest processes) | 28.2 MB |
| log rows retained in that same window | 630 rows / 774 KB |
| physical-to-retained write amplification | ~36x |
| combined idle write rate, 4 sessions | ~353 KB/s ≈ 30 GB/day |
| one idle session running 9.3 days | 123 GB lifetime writes |
The DB itself stays at ~25 MB — retention caps work. The wear comes from the durability policy:
LOG_FLUSH_INTERVAL = 2smeans ~43,000 transactions/day/process, each paying WAL page images + (at retention-cap steady state, which every long-lived session reaches) the windowed prune DELETE inside the same transaction.Row-level attribution also shows why target muting (#29457, #29599, #31789, #31791) can't converge: after accounting for those mutes, idle traffic is dominated by codex's own targets —
codex_app_server::outgoing_message(TRACE),codex_api::sse::responses(TRACE),codex_core::stream_events_utils(DEBUG),feedback_tags(INFO). Muting those defeats/feedback; keeping them keeps the churn. The reasoning that closed #23275 ("bounding individual call sites is brittle and misses future tracing events") applies equally to per-target filters.Patch: flush by bytes/severity/60 s instead of every 2 s
Branch: https://github.com/friday-james/codex-1/tree/fix/log-db-write-churn — 3 files, +127/−9, mostly tests.
/feedbackalready callslog_db.flush().awaitbefore reading (feedback_processor.rs), so feedback content is byte-identical, even mid-session.LogEntry::estimated_bytes()is factored out and shared by the sink buffer and the insert path, so the write-side threshold and partition budgets agree.A/B benchmark (each phase an isolated process)
Harness: https://github.com/friday-james/codex-1/tree/bench/log-db-write-churn (
state/examples/write_churn_bench.rs). It replays the measured idle profile — 5.3 rows/s, ~1.2 KiB/row, thread partition pre-filled past its retention cap (steady state of a long-lived session) — for 3 minutes per policy, measuring/proc/self/iowrite_bytes. Two runs, results identical to ±0.2%:The new policy's 2.9 MB against 1.17 MB of raw content is SQLite's WAL-then-checkpoint double write plus prune/B-tree overhead — i.e. the patch removes nearly all removable churn; going lower requires persisting less content or relaxing durability guarantees. Note the benchmark is single-process and therefore conservative: live sessions share one
logs_2.sqliteacross processes, where checkpoint contention pushes the old policy 2–3x higher than the bench (and in the limit produces the unbounded-WAL growth reported in #28997).Happy to open this as a PR per the invitation policy in contributing.md, or the approach is free to take as-is. Two further options if there's interest: prune hysteresis (once a partition crosses its cap, delete down to ~75% so steady-state inserts don't re-run the windowed DELETE every batch), and honoring an env override for the sink filter, which several reports expected
RUST_LOGto do (#29463, #31111).cc @jif-oai @charley-oai
Request for clarification: current shipping build still lacks the broader SQLite fix
Following up with the current shipping macOS build:
26.707.91948, build5440codex-cli 0.144.5codex-cli 0.144.30.144.5The
rust-v0.144.5release contains only the dangerous-command detection fix in #33455. Its persistent log filter still uses:Source: https://github.com/openai/codex/blob/rust-v0.144.5/codex-rs/state/src/log_db.rs
The
0.144.4...0.144.5release comparison also does not include #31789, #31790, #31791, or #31792. This appears inconsistent with the statement in #28224 that the series of fixes was already in the latest shipping desktop app, CLI, and IDE extension.My local SQLite database is currently protected by this reversible stopgap:
Protected-state verification on the current build shows
rows=0,seq=0, andmax(id)=0. I did not remove the trigger for another raw sample because the shipping0.144.5source contains no relevant persistent-log change. I am not posting any rawfeedback_log_bodyvalues because they may contain private paths, command fragments, tool output, or conversation context.Could the maintainers please clarify:
LevelFilter::TRACEstill the intended production default for the persistent SQLite feedback log?BEFORE INSERT ... RAISE(IGNORE)trigger?/feedbackdiagnostics?A version-specific answer would help users know when it is safe to remove the trigger and restore normal persistent diagnostic logging.
Follow-up after updating this Mac to the current ChatGPT-integrated Codex build.
Environment now:
27.0(26A5378n), Apple Silicon /arm64/Applications/ChatGPT.app26.715.31925, build5551codexresolves to:/Applications/ChatGPT.app/Contents/Resources/codexcodex-cli 0.145.0-alpha.18/Applications/Codex.appis not present on this machineRUST_LOGis unsetI kept the local SQLite mitigation in place for this check. The active triggers are still:
So this is a protected-state verification only; I did not remove the mitigation for another raw write-churn sample, and I am not posting raw
feedback_log_bodyvalues publicly.Current protected-state sample from
~/.codex/logs_2.sqlite:30-second retained-row sample with the mitigation still enabled:
Newest retained rows in that sample were
DEBUG codex_http_client::default_clientandINFO codex_http_client::custom_ca; the newest retained TRACE row stayed at id240689(TRACE codex_app_server::message_processor). The retained sample still includes TRACE entries fromcodex_app_server::message_processor,codex_api::sse::responses, andcodex_app_server::outgoing_message, but I cannot infer the unmitigated raw insert rate from this protected sample.Given #4988716077 and #4989141674, this updated app/CLI datapoint may help separate two questions:
A version-specific maintainer answer would still be useful: is
26.715.31925/ build5551/codex-cli 0.145.0-alpha.18expected to be safe enough for affected users to remove local SQLite mitigations, or should users keep them until a stable desktop/CLI release with an explicit persistent-log fix ships?Hey, I had a look at this.
The SQLite sink already has a guard that drops bridged events with the
logtarget before they reach the queue. The missing piece was a test that proves the guard still works when the outer tracing filter is permissive, which is the path that allowed this noise through in the first place.I've added that regression test. It sends a bridged
target="log"event alongside a normal Codex state event, flushes the sink, and checks that only the normal event was written to SQLite.Branch: https://github.com/TheJhyeFactor/codex/tree/fix/sqlite-log-bridge-regression-29532
I ran
CARGO_NET_OFFLINE=true cargo test -p codex-statelocally on macOS. All 153 unit tests, 3 binary tests, and 1 doctest passed.I couldn't open the PR because GitHub returned a
CreatePullRequestpermission error for my account, so I've linked the tested branch here for review.