›_ Codex Issues
Codex Issues / #29556

Persistent SQLite TRACE log churn still reproducible on Windows after #29432/#29457

Resolved 💬 8 comments Opened Jun 23, 2026 by Yurdos-r Closed Jun 23, 2026
💡 Likely answer: A maintainer (github-actions[bot], contributor) responded on this thread — see the highlighted reply below.

Summary

openai/codex#28224 was closed as completed after #29432 and #29457, but I can still reproduce persistent SQLite log churn on Windows with a newer Codex Desktop app package.

This looks like the same class of issue: ~/.codex/logs_2.sqlite / WAL is still being written continuously during normal Codex Desktop use, and the retained rows are still dominated by TRACE targets such as log, codex_api::sse::responses, codex_app_server::outgoing_message, codex_mcp::connection_manager, hyper_util::*, and large codex_client::transport rows.

I am intentionally not pasting raw feedback_log_body contents because they may contain private conversation/tool data.

Environment

  • Platform: Windows 11 Pro, 64-bit
  • OS version: 10.0.26200, build 26200
  • Codex Desktop package path:
  • C:\Program Files\WindowsApps\OpenAI.Codex_26.616.9593.0_x64__2p2nqsd0c76g0\app\resources\codex.exe
  • codex.exe last write time: 2026-06-23 04:17:10 UTC
  • Active process:
  • PID: 22992
  • Started: 2026-06-23 12:32:46 local time
  • Command line: codex.exe app-server --analytics-default-enabled
  • I did not find explicit trace/debug/log configuration in ~/.codex/config.toml; searching for trace|TRACE|debug|DEBUG|log|RUST_LOG|verbosity only matched the current project path containing the word logs.
  • SQLite trigger count: 0

Current SQLite files

Observed at about 2026-06-23 12:46 local time:

~/.codex/logs_2.sqlite      2,237,538,304 bytes
~/.codex/logs_2.sqlite-wal     35,971,752 bytes
~/.codex/logs_2.sqlite-shm         98,304 bytes

SQLite metadata:

journal_mode:   wal
page_size:      4096
page_count:     546274
freelist_count: 395880
retained rows:  143604
first row:      2026-06-13 12:56:51 local
last row:       2026-06-23 12:46:09 local
max_id:         79313921

The high freelist_count means the main DB file has a large amount of reusable/free space, but the file remains large on disk.

Read-only 20 second sample

Sample window:

START_LOCAL=2026-06-23 12:47:16
END_LOCAL=2026-06-23 12:47:36
BEFORE=row_count=143679,max_id=79316383,sqlite_sequence=79316383,max_ts=1782190035
AFTER=row_count=143778,max_id=79317522,sqlite_sequence=79317522,max_ts=1782190055

During only about 20 seconds:

row_count       +99
max_id          +1139
sqlite_sequence +1139

WAL mtime continued to move every ~1-2 seconds even though the WAL byte size stayed constant, so file size alone hides ongoing writes. The main DB mtime also updated once during the sample.

Selected sample rows:

sample  time_local    db_bytes     db_mtime      wal_bytes  wal_mtime
0       12:47:16.808  2237538304   12:46:53.768  35971752   12:47:15.737
1       12:47:17.820  2237538304   12:46:53.768  35971752   12:47:17.763
3       12:47:19.824  2237538304   12:46:53.768  35971752   12:47:19.745
5       12:47:21.827  2237538304   12:46:53.768  35971752   12:47:21.740
6       12:47:22.860  2237538304   12:47:22.821  35971752   12:47:22.810
9       12:47:25.866  2237538304   12:47:22.821  35971752   12:47:25.741
13      12:47:29.872  2237538304   12:47:22.821  35971752   12:47:29.746
17      12:47:33.877  2237538304   12:47:22.821  35971752   12:47:33.742
19      12:47:35.878  2237538304   12:47:22.821  35971752   12:47:35.743

Recent persisted targets

Rows in the last 60 seconds at sample time:

TRACE log                                        392 rows   0.062 MiB  max 1040 bytes
TRACE codex_app_server::outgoing_message         277 rows   0.046 MiB  max 181 bytes
TRACE codex_api::sse::responses                  251 rows   0.523 MiB  max 88418 bytes
TRACE hyper_util::client::legacy::pool            47 rows   0.022 MiB  max 1265 bytes
TRACE hyper_util::client::legacy::client          39 rows   0.025 MiB  max 1269 bytes
DEBUG hyper_util::client::legacy::connect::http   31 rows   0.024 MiB  max 1257 bytes
DEBUG codex_core::stream_events_utils             18 rows   0.037 MiB  max 8971 bytes
TRACE codex_client::transport                      7 rows   1.023 MiB  max 519445 bytes

Rows in the last 5 minutes from the active process:

process_uuid = pid:22992:672d5f85-9180-4d8a-a3c0-fdbc10744e29
rows         = 2860
estimated    = 10.05 MiB
first_local  = 2026-06-23 12:42:16
last_local   = 2026-06-23 12:47:15

Top 5-minute targets included:

TRACE log                                        557 rows   0.307 MiB
TRACE codex_mcp::connection_manager              484 rows   0.552 MiB
TRACE codex_app_server::outgoing_message         350 rows   0.057 MiB
TRACE codex_api::sse::responses                  346 rows   0.218 MiB
TRACE codex_client::transport                     72 rows   7.706 MiB  max 506097 bytes

Retained table distribution

Current retained rows by level:

TRACE  69878 rows  393.03 MiB  max 7233161 bytes
INFO   37877 rows   48.23 MiB  max 136004 bytes
DEBUG  33571 rows   45.32 MiB  max 78214 bytes
WARN    2219 rows    1.36 MiB  max 1588 bytes
ERROR    108 rows    0.20 MiB  max 41182 bytes

Largest retained target+level pairs:

TRACE codex_client::transport                    3659 rows  330.25 MiB  max 7233161 bytes
INFO  codex_otel.log_only                       14259 rows   20.42 MiB
DEBUG codex_core::stream_events_utils            9047 rows   15.80 MiB
TRACE hyper_util::client::legacy::client        12413 rows   11.33 MiB
DEBUG hyper_util::client::legacy::connect::http 11920 rows   11.28 MiB
INFO  codex_otel.trace_safe                      8144 rows   10.37 MiB
TRACE codex_mcp::connection_manager             11954 rows   10.23 MiB
TRACE codex_api::sse::responses                  4960 rows    9.66 MiB
TRACE log                                       15546 rows    9.63 MiB
TRACE hyper_util::client::legacy::pool           9876 rows    8.05 MiB

Expected behavior

After #29432 / #29457 and the release that closed #28224, Codex Desktop should not continue to persist high-frequency default TRACE rows into logs_2.sqlite / WAL during normal usage.

In particular, target=log, codex_api::sse::responses, codex_app_server::outgoing_message, codex_mcp::connection_manager, hyper_util::*, and large codex_client::transport rows still look noisy enough to cause ongoing SQLite write churn.

Actual behavior

On this Windows machine, the active Codex app-server process still continuously inserts into the SQLite log DB. The WAL mtime changes every ~1-2 seconds, sqlite_sequence advanced by 1139 in a 20-second read-only sample, and recent rows are still dominated by TRACE.

Could #28224 be reopened, or should this be tracked as a follow-up regression/incomplete fix for the Windows Desktop build?

View original on GitHub ↗

8 Comments

github-actions[bot] contributor · 27 days ago

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

  • #29532
  • #29463
  • #28224

Powered by Codex Action

darlingm contributor · 27 days ago

(Cross post of my comment on #28224)

Codex estimated: "this [#28224] regression plausibly burned low-single-digit millions of dollars of SSD endurance across users during the March-June Window," assuming only 5% of users had the write intensity of the reporter, or: "If affected users averaged 50% of that rate, the estimate rises to the low tens of millions."

@jif-oai Please add a permanent regression test that bounds persistent local writes. Maybe #29432 and #29457 solve this issue, or maybe they don't. But, OpenAI should make sure this never happens again by proactively catching similarly damaging bugs in the future whether it has to do with log writes or anything else.

@jif-oai Please also add regression tests for prioritize reports of runaway CPU/memory/disk usage, across macOS, Windows, and Linux. There are many users experiencing CPU/memory runaway on long ignored bugs, which is unnecessarily burning loads of electricity, degrading battery life, and thermal stress is probably causing some level of hardware damage on some machines unnecessarily constantly running at throttling temperatures.

_Note: Codex's estimate is based on ~ 3 million weekly active Codex users, guessing 50% of them were affected, 95% on SSDs, 75% on the affected version, 5% having the intensity reported here, with $0.13 cost per TBW written to SSDs._

zjt40238 · 27 days ago

I can also reproduce the same class of persistent SQLite log churn on macOS with Codex Desktop. I am adding macOS evidence here because this issue currently documents Windows reproduction.

I am intentionally not pasting raw feedback_log_body contents because they may contain private conversation/tool data.

Environment

  • Platform: macOS
  • macOS version: 26.5.1, build 25F80
  • Codex app version: 26.616.51431
  • Codex bundle version: 4212
  • Running app path: /Users/zjt/Desktop/Codex.app
  • Active process command: /Users/zjt/Desktop/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled
  • Process environment included:
  • RUST_LOG=warn
  • LOG_FORMAT=json
  • CODEX_SHELL=1
  • I did not find explicit TRACE logging configuration in ~/.codex/config.toml.

Current SQLite files

Observed on 2026-06-23 around 13:57 Asia/Shanghai:

~/.codex/logs_2.sqlite      1,819,447,296 bytes
~/.codex/logs_2.sqlite-wal     13,690,792 bytes
~/.codex/logs_2.sqlite-shm         32,768 bytes

SQLite metadata:

retained rows:       186,215
min id:              153,939,515
max id:              192,221,719
sqlite_sequence:     192,221,719
first retained row:  2026-06-14 15:57:32 local
last retained row:   2026-06-23 13:57:35 local
page_count:          444,201
page_size:           4,096
freelist_count:      297,266
live DB pages:       about 573.96 MiB
free pages:          about 1,161.20 MiB
file size:           about 1,735.16 MiB

The high freelist_count suggests the file has a large amount of reusable free space, but the SQLite file remains large on disk. The much larger sqlite_sequence compared with retained rows suggests a high historical insert volume with pruning/cleanup.

Retained rows by level

INFO   100,007 rows  131.36 MiB estimated
TRACE   78,634 rows  184.31 MiB estimated
DEBUG    6,843 rows   34.65 MiB estimated
WARN       729 rows    0.35 MiB estimated
ERROR        2 rows    0.00 MiB estimated

Largest retained target/level pairs

TRACE codex_api::endpoint::responses_websocket  47,855 rows  148.43 MiB
INFO  codex_otel.log_only                       49,080 rows   67.00 MiB
INFO  codex_otel.trace_safe                     48,770 rows   61.97 MiB
DEBUG codex_core::stream_events_utils            1,969 rows   31.86 MiB
TRACE codex_client::transport                       17 rows   15.98 MiB
TRACE codex_api::sse::responses                  8,858 rows    9.00 MiB
TRACE log                                       13,353 rows    4.92 MiB
TRACE codex_mcp::connection_manager              5,198 rows    3.97 MiB

Expected behavior

With RUST_LOG=warn and no explicit TRACE configuration, Codex Desktop should not persist high-frequency default TRACE rows into ~/.codex/logs_2.sqlite / WAL during normal use.

Actual behavior

The active Codex app-server process continues to write to the SQLite log database. The WAL modification time continues to update even when the file size is relatively stable, so file size alone hides ongoing write churn. Retained rows are still dominated by TRACE targets, especially codex_api::endpoint::responses_websocket, codex_api::sse::responses, log, and connection-related targets.

This looks like the same issue is still reproducible on macOS, not only Windows.

Yurdos-r · 27 days ago

One extra data point / temporary mitigation note from my local machine:

After confirming the churn, I tested a local stopgap in this order:

  1. create a backup of ~/.codex/logs_2.sqlite first;
  2. add a SQLite BEFORE INSERT ON logs trigger that ignores new inserts;
  3. run PRAGMA wal_checkpoint(TRUNCATE);;
  4. sample again to verify MAX(id), sqlite_sequence, and the WAL file no longer grow.

On my Windows machine this stopped the immediate log DB/WAL growth during the sample window. However, please do not treat this as a universal copy-paste fix. It is only a mitigation idea and it changes local diagnostic logging behavior. Anyone considering it should test against a backup or sandbox copy of their own Codex DB first, confirm that the database remains healthy and that Codex still works for their setup, and only then decide whether it is appropriate to run on their real machine.

The more important fix should still be upstream: Codex should not persist high-frequency default TRACE/local diagnostic rows into logs_2.sqlite during normal desktop use, and there should ideally be a supported config/escape hatch plus regression coverage for persistent local write volume.

llliii666 · 27 days ago

I can reproduce the same class of persistent SQLite log churn on Windows Codex Desktop after the 0.142.0 / 26.616.9593.0 update. Adding one more read-only data point here.

I am intentionally not pasting raw feedback_log_body contents because they may contain private conversation/tool data. This inspection did not delete, move, VACUUM, or modify any Codex files.

Environment

  • Platform: Windows 11, 64-bit
  • OS version: 10.0.26200, build 26200
  • Codex Desktop AppX: OpenAI.Codex 26.616.9593.0
  • Embedded Codex CLI: codex-cli 0.142.0
  • Global Codex CLI: codex-cli 0.142.0
  • Log DB: ~/.codex/logs_2.sqlite

Local-day read-only summary

Observed on 2026-06-23 around 21:54 local time (UTC+08:00):

~/.codex/logs_2.sqlite      382,251,008 bytes  (~364.543 MiB)
~/.codex/logs_2.sqlite-wal    5,281,872 bytes  (~5.037 MiB)
~/.codex/logs_2.sqlite-shm       32,768 bytes  (~0.031 MiB)

SQLite/log metadata for the local day 2026-06-23:

active usage estimate:             about 2h35m (2:35)
first-to-last retained-log span:    about 20h38m, but this includes long idle gaps and is not the usage duration
retained rows for the day:          9,169
previous max id before local day:   46,123,569
current max id / sqlite_sequence:   48,051,431
local-day id delta:                 1,927,862
estimated pruned/replaced rows:     1,918,693
retained estimated log payload:     ~7.09 MiB
estimated logical log volume:       ~1.456 GiB, based on id delta * retained average estimated_bytes

The important signal is that only 9,169 rows are currently retained for the day, while the id counter advanced by 1,927,862. This looks like continuous insert-and-prune churn: the DB file size can remain stable while the SQLite sink is still doing substantial work.

Retained rows by level for the day

TRACE   6,759 rows  ~5.498 MiB estimated
DEBUG   1,112 rows  ~0.443 MiB estimated
INFO    1,081 rows  ~1.038 MiB estimated
WARN      192 rows  ~0.097 MiB estimated
ERROR      25 rows  ~0.015 MiB estimated

Main retained targets for the day

log                                      2,864 rows  ~0.808 MiB
codex_api::sse::responses               2,434 rows  ~2.442 MiB
hyper_util::client::legacy::pool          528 rows  ~0.151 MiB
codex_mcp::connection_manager             464 rows  ~0.360 MiB
hyper_util::client::legacy::connect::http 452 rows  ~0.141 MiB
codex_otel.trace_safe                     341 rows  ~0.401 MiB
codex_otel.log_only                       341 rows  ~0.430 MiB
codex_api::endpoint::responses_websocket  324 rows  ~1.307 MiB

Burst windows

A few 5-minute windows show the problem clearly:

20:25-20:30 local: 1,000 retained rows, id span 85,497
21:45-21:50 local: 1,001 retained rows, id span 64,560
11:20-11:25 local: 1,007 retained rows, id span 37,197
21:50-21:55 local: 2,000 retained rows, id span 20,312

Again, file size alone hides the churn: the WAL remained only a few MiB, but the id counter and retained-vs-id-span data show high-frequency inserts followed by pruning.

Expected behavior: after #29432 / #29457 and the 0.142.0 release, normal Codex Desktop usage should not continue persisting this many default TRACE / stream / OTel / MCP rows into the persistent SQLite log sink.

Actual behavior: on Windows Desktop 26.616.9593.0, the persistent log DB still shows high id churn during about 2h35m of active use on the local day.

wanghew · 27 days ago

Adding another Windows Desktop reproduction with current Microsoft Store build.

Environment:

  • OS: Windows
  • Codex app package: OpenAI.Codex_26.608.1337.0_x64__2p2nqsd0c76g0
  • Desktop app version shown by Appx: 26.608.1337.0
  • Local app-server binary path includes C:\Users\<user>\AppData\Local\OpenAI\Codex\bin\07133f975a59dbd9\codex.exe
  • User environment variable set before restart: RUST_LOG=warn

Observed behavior after clearing the SQLite log DB with DELETE FROM logs, wal_checkpoint(truncate), and VACUUM, then fully restarting Codex Desktop:

logs_2.sqlite      8,929,280 bytes
logs_2.sqlite-wal  6,567,312 bytes
logs_2.sqlite-shm     32,768 bytes
retained rows:      2,479

Level distribution after restart:

TRACE  1,484
INFO     511
DEBUG    369
WARN     115

Top targets:

codex_app_server::outgoing_message                 434
log                                                 397
codex_otel.log_only                                 326
codex_api::sse::responses                           313
hyper_util::client::legacy::pool                    235
hyper_util::client::legacy::connect::http           135
hyper_util::client::legacy::client                  135
codex_core_skills::loader                            60
codex_client::custom_ca                              47
codex_app_server::message_processor                  43
rmcp::service                                        42
codex_otel.trace_safe                                31

This suggests that, at least for this Windows Desktop build, RUST_LOG=warn is not sufficient to prevent persisted TRACE/DEBUG rows from being written to ~/.codex/logs_2.sqlite / WAL.

I intentionally did not paste raw feedback_log_body contents because they may contain private conversation/tool data. I also did not apply the SQLite trigger workaround, so this is still the default logging behavior after a normal cleanup + restart.

mdbecker · 27 days ago

I’m seeing what looks like the same underlying SQLite churn pattern on macOS, though my setup differs slightly: I have not specifically disabled analytics/plugins, and the running process is codex app-server --analytics-default-enabled.

I posted the fuller diagnostic here: https://github.com/openai/codex/issues/29612#issuecomment-4780728496

Summary of my repro:

Codex Desktop: 26.616.71553 • Released Jun 22, 2026
Bundled CLI: codex-cli 0.142.0
Platform: macOS 15.3, Darwin 24.3.0, arm64
Launch method: macOS GUI / Spotlight
Running process: /Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled
Actual process env includes: RUST_LOG=warn

Over a 5-minute idle sample, with Codex Desktop open in the background but not actively used:

new_insert_ids=2975
retained_row_delta=0
insert_ids_per_sec=9.92

The DB was healthy and compacted beforehand, so this did not appear to be leftover pre-fix bloat:

quick_check=ok
integrity_check=ok
journal_mode=wal
freelist_count=115
logs_2.sqlite≈63M
logs_2.sqlite-wal≈4.1M

The exact sampled ID window showed retained rows dominated by TRACE target=log:

TRACE log                                998 rows
TRACE hyper_util::client::legacy::pool     2 rows

All retained rows from that sampled window belonged to the same app-server process.

So this looks like the same post-0.142.0 behavior: file size may remain bounded, but the app-server still continuously inserts and prunes persistent SQLite log rows while idle, mostly TRACE target=log, despite RUST_LOG=warn.

etraut-openai contributor · 27 days ago

Thanks for reporting. We're using #28224 to track this issue.