logs_2.sqlite-wal grows without bound into tens of GB

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #28224
• #27741

Powered by Codex Action

Correction to the original evidence section: the peak pre-cleanup observation was roughly 65 GB of SQLite log WAL files across the two active Codex profile/session folders. The smaller per-file numbers originally listed there were from a later post-cleanup/regrowth check and should not have been presented as the pre-cleanup peak. I edited the issue body to separate the peak observation from the later regrowth evidence.

Additional observation: a closed/stale Codex work-profile log WAL just measured at 219G / 234,157,858,816 bytes: /home/<user>/.codex-work/logs_2.sqlite-wal. At that point the user's /home filesystem was full: 656G size, 623G used, 40M free, 100% used. The file was not held open by currently running Codex processes, so removing only the closed .codex-work/logs_2.sqlite-wal and .codex-work/logs_2.sqlite-shm restored free space to 219G available. Separately, currently running codext app-server/remotes were attached to .codex-personal/logs_2.sqlite-wal; three remote worker processes each showed roughly 170GB cumulative writes in /proc/<pid>/io while the active personal WAL was about 1.6G at inspection time. This confirms the issue can become disk-filling, not just inconvenient for backups.

Follow-up DB-content scan from the surviving base log DBs found a likely self-amplifying source. The schema is a small log DB: logs(id, ts, ts_nanos, level, target, feedback_log_body, module_path, file, line, thread_id, process_uuid, estimated_bytes), with WAL journaling enabled. The base DBs are only about 111 MB each, but they contain many TRACE rows where Codex logs file watcher events for its own log DB/WAL changes.\n\nExamples from .codex-work/logs_2.sqlite:\n\n- total logs rows: 42,534\n- TRACE rows: 38,537\n- rows with inotify event: 35,571\n- rows mentioning logs_2.sqlite-wal: 26,401\n- repeated recent sample: inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some(\"logs_2.sqlite-wal\") }\n\nExamples from .codex-personal/logs_2.sqlite:\n\n- total logs rows: 48,872\n- TRACE rows: 44,153\n- rows with inotify event: 38,580\n- rows mentioning logs_2.sqlite-wal: 25,397\n- recent samples include repeated inotify event ... name: Some(\"logs_2.sqlite\") rows.\n\nThis looks like a feedback loop: Codex logs that its own log DB/WAL changed; that insert changes the WAL again; the watcher observes another modify event; then another TRACE row is written. Because the DB is in WAL mode and checkpoint/truncation is not keeping the WAL bounded, a small 111 MB base DB can generate a 219 GB logs_2.sqlite-wal. There are also large TRACE payload rows from websocket/frame logging, but the inotify self-logging loop is the clearest disk-filling mechanism found so far.

Additional sanitized findings from a deeper investigation:

The strongest evidence now points to a self-watch / self-log loop around $CODEX_HOME.

Likely loop

1. Codex writes a diagnostic row to logs_2.sqlite.
2. SQLite updates logs_2.sqlite-wal.
3. A Codex inotify watcher sees logs_2.sqlite-wal modified.
4. Codex records a TRACE row for that inotify event into the same logs_2.sqlite.
5. That write modifies logs_2.sqlite-wal again.
6. Repeat.

Important environment note

The newest evidence was collected while the affected session was being run through the codext fork, which vendors/runs a codex binary. That means this may be fork-specific, fork-amplified, or caused by an interaction between upstream Codex code and codext's app-server/remote process model. I am posting this here because the DB/schema/logging/watch behavior is Codex-derived, but I am also filing the same sanitized report against Loongphy/codext.

Observed state

Sanitized paths:

CODEX_HOME=/home/<user>/.codex-work
SQLite DB=/home/<user>/.codex-work/logs_2.sqlite
SQLite WAL=/home/<user>/.codex-work/logs_2.sqlite-wal

The current post-cleanup WAL still grew back to roughly 12 GB:

11,884,873,392  /home/<user>/.codex-work/logs_2.sqlite-wal
115,650,560     /home/<user>/.codex-work/logs_2.sqlite
23,101,440      /home/<user>/.codex-work/logs_2.sqlite-shm

Earlier growth samples during active sessions showed severe WAL growth:

2,492,175,672 -> 2,842,709,392 bytes in 10 seconds
5,953,070,432 -> 6,316,977,672 bytes in 10 seconds
8,921,312,072 -> 9,101,376,672 bytes in 5 seconds

This is after a previous stale/closed WAL reached 219 GB and filled the /home filesystem.

SQLite log-table evidence

Counts from the affected logs_2.sqlite:

total_rows=47,565
TRACE=42,794
INFO=3,533
DEBUG=864
WARN=356
ERROR=18
inotify_rows=39,532
logs_2.sqlite-wal mentions=28,727

Top repeated inotify messages:

28,699  inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some("logs_2.sqlite-wal") }
 8,049  inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some("logs_2.sqlite") }
   147  inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some("state_5.sqlite-wal") }

There are also unrelated file-open watcher events for system files such as ld.so.cache, locale.alias, and passwd, which appear to come from a separate /etc watch. The disk-filling loop is the one involving logs_2.sqlite*.

Kernel inotify evidence

The active affected process had an inotify file descriptor with:

FD 30 anon_inode:inotify
  inotify wd:1 ino:c6860b ...

The inode maps to $CODEX_HOME:

hex c6860b == decimal 13010443
13010443 /home/<user>/.codex-work

The repeated SQLite log rows also use WatchDescriptor { id: 1, ... }, and the row names are logs_2.sqlite-wal / logs_2.sqlite. That ties the logged file events directly to a watcher rooted at $CODEX_HOME, not just to a random project directory.

Open file holders

The affected codext/vendored codex process held open handles to:

/home/<user>/.codex-work/logs_2.sqlite
/home/<user>/.codex-work/logs_2.sqlite-wal
/home/<user>/.codex-work/logs_2.sqlite-shm

Other active codext app-server/remote processes held handles to a separate profile's logs_2.sqlite-wal.

Process model clue

The process tree included:

node .../bin/codext ... app-server --listen ws://127.0.0.1:<port>
.../codex ... app-server --listen ws://127.0.0.1:<port>
node .../bin/codext ... --remote ws://127.0.0.1:<port> -C /home/<user>/Projects/<repo>
.../codex ... --remote ws://127.0.0.1:<port> -C /home/<user>/Projects/<repo>
node .../bin/codext ... --sandbox danger-full-access --ask-for-approval never
.../codex ... --sandbox danger-full-access --ask-for-approval never

The exact project names have been omitted intentionally. Non-Codex dev-server processes running in one project did not hold handles to logs_2.sqlite*; only Codex/codext processes did.

Trust/root clue

The current Git repo root resolved correctly to a project directory under /home/<user>/Projects/<repo>, but the UI reportedly displayed a trust warning for /home/<user>, not the project root. If a broad home directory becomes a trusted/watch surface, it can include $CODEX_HOME and therefore Codex's own SQLite state.

Even without that clue, the kernel inotify evidence above shows a watcher rooted at /home/<user>/.codex-work.

Relevant source areas

These upstream files look relevant:

codex-rs/file-watcher/src/lib.rs
codex-rs/app-server/src/fs_watch.rs
codex-rs/app-server/src/skills_watcher.rs
codex-rs/tui/src/onboarding/onboarding_screen.rs

Specifically:

• file-watcher can watch requested paths and can fall back to the nearest existing ancestor.
• app-server exposes fs/watch.
• skills roots can be watched recursively.
• trust onboarding falls back to cwd if no Git root is resolved.

Expected behavior

Codex should not log file watcher events for its own SQLite diagnostic/state files into the same SQLite log sink.

At minimum, file watcher trace logging should suppress:

logs_2.sqlite
logs_2.sqlite-wal
logs_2.sqlite-shm
state_5.sqlite
state_5.sqlite-wal
state_5.sqlite-shm
goals_1.sqlite*
memories_1.sqlite*

Better fixes:

1. Never watch $CODEX_HOME as a filesystem watch root unless explicitly required.
2. Never log watcher events for Codex's own SQLite files into the SQLite log sink.
3. Respect log-level filtering before inserting TRACE rows into logs_2.sqlite.
4. Add WAL size limits, checkpointing, rotation, or emergency safeguards so diagnostic logs cannot consume hundreds of GB.
5. If a broad home directory trust target is selected, exclude Codex state directories from any watch surfaces.

Local mitigation being tested

The local mitigation is to move SQLite-backed runtime state outside the watched CODEX_HOME path:

sqlite_home = "/home/<user>/.local/share/codex-sqlite/work"

This was added after the issue was observed. Already-running Codex/codext processes can continue holding the old logs_2.sqlite* files until restarted.

Cross-filed the sanitized fork-specific report against Loongphy/codext because the latest evidence was collected from codext/vendored-codex processes: https://github.com/Loongphy/codext/issues/11

This kind of program behavior is disrespectful in general, especially toward users with SSDs.

Follow-up after comparing the official OpenAI repo against the Codext fork and testing a fix:

I now think the concrete trigger I hit is not directly an OpenAI/Codex bug. The direct self-watch loop appears fork-specific in Loongphy/codext: that fork adds an AuthWatch that watches the top-level CODEX_HOME directory for auth.json changes. Since the SQLite log DB normally also lives under CODEX_HOME, WAL writes can be observed by that watcher, and raw notify TRACE logs can be written back into the same SQLite log DB.

Fork-specific issue:

https://github.com/Loongphy/codext/issues/11

Fix PR:

https://github.com/Loongphy/codext/pull/12

That said, I think the underlying risk is still present in OpenAI/Codex's implementation: the SQLite log sink persists TRACE logs by default, and the shared log_db.rs did not filter raw notify/inotify compatibility logs. I did not find an official OpenAI/Codex production path that automatically watches CODEX_HOME itself, so the immediate trigger seems absent upstream. But if any current or future official watcher targets the SQLite directory, or a client asks app-server fs/watch to watch that location, the same class of feedback loop could become user-visible.

So this can probably be closed as "not directly reproduced in upstream OpenAI/Codex" if maintainers agree, but I would still recommend hardening the SQLite log sink so it does not persist raw file-watcher backend TRACE/DEBUG noise in a very damaging leaking fs-notify loop.

Potentially a duplicate of / related to:

• https://github.com/openai/codex/issues/28224

Which should have been fixed by:

• https://github.com/openai/codex/pull/29432
• https://github.com/openai/codex/pull/29457

Which were released in:

• https://github.com/openai/codex/releases/tag/rust-v0.142.0

But there still seem to be issues after that:

• https://github.com/openai/codex/issues/28224#issuecomment-4775086154
• https://github.com/openai/codex/issues/29532

Fix bounds the WAL: journal_size_limit=16MiB + wal_autocheckpoint=400, plus a periodic best-effort TRUNCATE checkpoint (previously only a startup PASSIVE checkpoint): https://github.com/denispol/codex/pull/1

I hit these but my sqlite db size was genuinely nutty 😆

% du -h --apparent-size ~/codex-db-backup/logs_2.sqlite*
1.1G    /home/bcosgrove/codex-db-backup/logs_2.sqlite
1.3G    /home/bcosgrove/codex-db-backup/logs_2.sqlite-shm
1.1T    /home/bcosgrove/codex-db-backup/logs_2.sqlite-wal
````

10s of GB? Pshaw.