logs_2.sqlite-wal grows without bound into tens of GB

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #28224
• #27741

Powered by Codex Action

Correction to the original evidence section: the peak pre-cleanup observation was roughly 65 GB of SQLite log WAL files across the two active Codex profile/session folders. The smaller per-file numbers originally listed there were from a later post-cleanup/regrowth check and should not have been presented as the pre-cleanup peak. I edited the issue body to separate the peak observation from the later regrowth evidence.

Additional observation: a closed/stale Codex work-profile log WAL just measured at 219G / 234,157,858,816 bytes: /home/<user>/.codex-work/logs_2.sqlite-wal. At that point the user's /home filesystem was full: 656G size, 623G used, 40M free, 100% used. The file was not held open by currently running Codex processes, so removing only the closed .codex-work/logs_2.sqlite-wal and .codex-work/logs_2.sqlite-shm restored free space to 219G available. Separately, currently running codext app-server/remotes were attached to .codex-personal/logs_2.sqlite-wal; three remote worker processes each showed roughly 170GB cumulative writes in /proc/<pid>/io while the active personal WAL was about 1.6G at inspection time. This confirms the issue can become disk-filling, not just inconvenient for backups.

Follow-up DB-content scan from the surviving base log DBs found a likely self-amplifying source. The schema is a small log DB: logs(id, ts, ts_nanos, level, target, feedback_log_body, module_path, file, line, thread_id, process_uuid, estimated_bytes), with WAL journaling enabled. The base DBs are only about 111 MB each, but they contain many TRACE rows where Codex logs file watcher events for its own log DB/WAL changes.\n\nExamples from .codex-work/logs_2.sqlite:\n\n- total logs rows: 42,534\n- TRACE rows: 38,537\n- rows with inotify event: 35,571\n- rows mentioning logs_2.sqlite-wal: 26,401\n- repeated recent sample: inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some(\"logs_2.sqlite-wal\") }\n\nExamples from .codex-personal/logs_2.sqlite:\n\n- total logs rows: 48,872\n- TRACE rows: 44,153\n- rows with inotify event: 38,580\n- rows mentioning logs_2.sqlite-wal: 25,397\n- recent samples include repeated inotify event ... name: Some(\"logs_2.sqlite\") rows.\n\nThis looks like a feedback loop: Codex logs that its own log DB/WAL changed; that insert changes the WAL again; the watcher observes another modify event; then another TRACE row is written. Because the DB is in WAL mode and checkpoint/truncation is not keeping the WAL bounded, a small 111 MB base DB can generate a 219 GB logs_2.sqlite-wal. There are also large TRACE payload rows from websocket/frame logging, but the inotify self-logging loop is the clearest disk-filling mechanism found so far.

Additional sanitized findings from a deeper investigation:

The strongest evidence now points to a self-watch / self-log loop around $CODEX_HOME.

Likely loop

1. Codex writes a diagnostic row to logs_2.sqlite.
2. SQLite updates logs_2.sqlite-wal.
3. A Codex inotify watcher sees logs_2.sqlite-wal modified.
4. Codex records a TRACE row for that inotify event into the same logs_2.sqlite.
5. That write modifies logs_2.sqlite-wal again.
6. Repeat.

Important environment note

The newest evidence was collected while the affected session was being run through the codext fork, which vendors/runs a codex binary. That means this may be fork-specific, fork-amplified, or caused by an interaction between upstream Codex code and codext's app-server/remote process model. I am posting this here because the DB/schema/logging/watch behavior is Codex-derived, but I am also filing the same sanitized report against Loongphy/codext.

Observed state

Sanitized paths:

CODEX_HOME=/home/<user>/.codex-work
SQLite DB=/home/<user>/.codex-work/logs_2.sqlite
SQLite WAL=/home/<user>/.codex-work/logs_2.sqlite-wal

The current post-cleanup WAL still grew back to roughly 12 GB:

11,884,873,392  /home/<user>/.codex-work/logs_2.sqlite-wal
115,650,560     /home/<user>/.codex-work/logs_2.sqlite
23,101,440      /home/<user>/.codex-work/logs_2.sqlite-shm

Earlier growth samples during active sessions showed severe WAL growth:

2,492,175,672 -> 2,842,709,392 bytes in 10 seconds
5,953,070,432 -> 6,316,977,672 bytes in 10 seconds
8,921,312,072 -> 9,101,376,672 bytes in 5 seconds

This is after a previous stale/closed WAL reached 219 GB and filled the /home filesystem.

SQLite log-table evidence

Counts from the affected logs_2.sqlite:

total_rows=47,565
TRACE=42,794
INFO=3,533
DEBUG=864
WARN=356
ERROR=18
inotify_rows=39,532
logs_2.sqlite-wal mentions=28,727

Top repeated inotify messages:

28,699  inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some("logs_2.sqlite-wal") }
 8,049  inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some("logs_2.sqlite") }
   147  inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some("state_5.sqlite-wal") }

There are also unrelated file-open watcher events for system files such as ld.so.cache, locale.alias, and passwd, which appear to come from a separate /etc watch. The disk-filling loop is the one involving logs_2.sqlite*.

Kernel inotify evidence

The active affected process had an inotify file descriptor with:

FD 30 anon_inode:inotify
  inotify wd:1 ino:c6860b ...

The inode maps to $CODEX_HOME:

hex c6860b == decimal 13010443
13010443 /home/<user>/.codex-work

The repeated SQLite log rows also use WatchDescriptor { id: 1, ... }, and the row names are logs_2.sqlite-wal / logs_2.sqlite. That ties the logged file events directly to a watcher rooted at $CODEX_HOME, not just to a random project directory.

Open file holders

The affected codext/vendored codex process held open handles to:

/home/<user>/.codex-work/logs_2.sqlite
/home/<user>/.codex-work/logs_2.sqlite-wal
/home/<user>/.codex-work/logs_2.sqlite-shm

Other active codext app-server/remote processes held handles to a separate profile's logs_2.sqlite-wal.

Process model clue

The process tree included:

node .../bin/codext ... app-server --listen ws://127.0.0.1:<port>
.../codex ... app-server --listen ws://127.0.0.1:<port>
node .../bin/codext ... --remote ws://127.0.0.1:<port> -C /home/<user>/Projects/<repo>
.../codex ... --remote ws://127.0.0.1:<port> -C /home/<user>/Projects/<repo>
node .../bin/codext ... --sandbox danger-full-access --ask-for-approval never
.../codex ... --sandbox danger-full-access --ask-for-approval never

The exact project names have been omitted intentionally. Non-Codex dev-server processes running in one project did not hold handles to logs_2.sqlite*; only Codex/codext processes did.

Trust/root clue

The current Git repo root resolved correctly to a project directory under /home/<user>/Projects/<repo>, but the UI reportedly displayed a trust warning for /home/<user>, not the project root. If a broad home directory becomes a trusted/watch surface, it can include $CODEX_HOME and therefore Codex's own SQLite state.

Even without that clue, the kernel inotify evidence above shows a watcher rooted at /home/<user>/.codex-work.

Relevant source areas

These upstream files look relevant:

codex-rs/file-watcher/src/lib.rs
codex-rs/app-server/src/fs_watch.rs
codex-rs/app-server/src/skills_watcher.rs
codex-rs/tui/src/onboarding/onboarding_screen.rs

Specifically:

• file-watcher can watch requested paths and can fall back to the nearest existing ancestor.
• app-server exposes fs/watch.
• skills roots can be watched recursively.
• trust onboarding falls back to cwd if no Git root is resolved.

Expected behavior

Codex should not log file watcher events for its own SQLite diagnostic/state files into the same SQLite log sink.

At minimum, file watcher trace logging should suppress:

logs_2.sqlite
logs_2.sqlite-wal
logs_2.sqlite-shm
state_5.sqlite
state_5.sqlite-wal
state_5.sqlite-shm
goals_1.sqlite*
memories_1.sqlite*

Better fixes:

1. Never watch $CODEX_HOME as a filesystem watch root unless explicitly required.
2. Never log watcher events for Codex's own SQLite files into the SQLite log sink.
3. Respect log-level filtering before inserting TRACE rows into logs_2.sqlite.
4. Add WAL size limits, checkpointing, rotation, or emergency safeguards so diagnostic logs cannot consume hundreds of GB.
5. If a broad home directory trust target is selected, exclude Codex state directories from any watch surfaces.

Local mitigation being tested

The local mitigation is to move SQLite-backed runtime state outside the watched CODEX_HOME path:

sqlite_home = "/home/<user>/.local/share/codex-sqlite/work"

This was added after the issue was observed. Already-running Codex/codext processes can continue holding the old logs_2.sqlite* files until restarted.

Cross-filed the sanitized fork-specific report against Loongphy/codext because the latest evidence was collected from codext/vendored-codex processes: https://github.com/Loongphy/codext/issues/11

This kind of program behavior is disrespectful in general, especially toward users with SSDs.

Follow-up after comparing the official OpenAI repo against the Codext fork and testing a fix:

I now think the concrete trigger I hit is not directly an OpenAI/Codex bug. The direct self-watch loop appears fork-specific in Loongphy/codext: that fork adds an AuthWatch that watches the top-level CODEX_HOME directory for auth.json changes. Since the SQLite log DB normally also lives under CODEX_HOME, WAL writes can be observed by that watcher, and raw notify TRACE logs can be written back into the same SQLite log DB.

Fork-specific issue:

https://github.com/Loongphy/codext/issues/11

Fix PR:

https://github.com/Loongphy/codext/pull/12

That said, I think the underlying risk is still present in OpenAI/Codex's implementation: the SQLite log sink persists TRACE logs by default, and the shared log_db.rs did not filter raw notify/inotify compatibility logs. I did not find an official OpenAI/Codex production path that automatically watches CODEX_HOME itself, so the immediate trigger seems absent upstream. But if any current or future official watcher targets the SQLite directory, or a client asks app-server fs/watch to watch that location, the same class of feedback loop could become user-visible.

So this can probably be closed as "not directly reproduced in upstream OpenAI/Codex" if maintainers agree, but I would still recommend hardening the SQLite log sink so it does not persist raw file-watcher backend TRACE/DEBUG noise in a very damaging leaking fs-notify loop.

Potentially a duplicate of / related to:

• https://github.com/openai/codex/issues/28224

Which should have been fixed by:

• https://github.com/openai/codex/pull/29432
• https://github.com/openai/codex/pull/29457

Which were released in:

• https://github.com/openai/codex/releases/tag/rust-v0.142.0

But there still seem to be issues after that:

• https://github.com/openai/codex/issues/28224#issuecomment-4775086154
• https://github.com/openai/codex/issues/29532

Fix bounds the WAL: journal_size_limit=16MiB + wal_autocheckpoint=400, plus a periodic best-effort TRUNCATE checkpoint (previously only a startup PASSIVE checkpoint): https://github.com/denispol/codex/pull/1

I hit these but my sqlite db size was genuinely nutty 😆

% du -h --apparent-size ~/codex-db-backup/logs_2.sqlite*
1.1G    /home/bcosgrove/codex-db-backup/logs_2.sqlite
1.3G    /home/bcosgrove/codex-db-backup/logs_2.sqlite-shm
1.1T    /home/bcosgrove/codex-db-backup/logs_2.sqlite-wal
````

10s of GB? Pshaw.

Analysis (community)

Thanks for tracking this — attempting a residual-focused analysis aligned with docs/contributing.md (invitation-only: analysis in-thread, no unsolicited PR).

• Issue under review: #28997 — logs_2.sqlite-wal grows without bound into tens of GB
• Theme / radar cluster: logs_2.sqlite-wal multi-GB growth (logs_2.sqlite SSD)
• Reporter posture: static + local evidence where available; not claiming a maintainer invite.

Observed (from issue + public residual signal)

What version of Codex CLI is running? codex-cli 0.140.0 codex doctor reports 0.141.0 is available, but this was observed on 0.140.0. Also observed locally: ``text codex-cli 0.140.0 codex-cli 0.138.0 # codext wrapper ` ### What platform is your computer? Linux x86_64, Ubuntu 24.04 noble. Runtime/install from codex doctor: `text Codex Doctor v0.140.0 · linux-x86_64 runtime: npm Node path: ~/.nvm/versions/node/v24.13.0/... terminal: Alacritty auth: ChatGPT auth `` ### Which mode

Local residual measurement (2026-07-24, this reporter)

On a host running codex-cli 0.146.0-alpha.6 (newer than 0.142–0.145 residual reports):

Signal-14 INTERIM evidence (from qwen_s14 mid-flight)

captured: 2026-07-24T15:10:05Z

• /Users/bing/.codex/logs_2.sqlite size=1778548736 mtime=Fri Jul 24 23:09:38 2026
• /Users/bing/.codex/logs_2.sqlite-wal size=30405632 mtime=Fri Jul 24 23:10:04 2026
• codex version: codex-cli 0.146.0-alpha.6

level counts (ro)

TRACE|94794
DEBUG|21528
INFO|19618
WARN|796
ERROR|370

rows/max_ts: 137106|2026-07-24 23:10:05

This is not a claim that every install still burns TB/year — it is a version-pinned observation that TRACE still lands in the SQLite sink after the #28224-era mitigations on a current alpha, consistent with residual issues still open.

Root-cause hypothesis (to be confirmed on main)

1. Feedback/log pipeline still admits TRACE (or default level too verbose) into logs_2.sqlite.
2. WAL/checkpoint policy does not cap growth under idle or streaming load.
3. Partial cut of TRACE paths may leave high-volume targets writing.

High-level fix outline (not a PR)

• Default sink level: never store TRACE in SQLite unless explicitly enabled.
• Honor RUST_LOG / app log level end-to-end into the SQLite writer.
• Hard caps: max DB size / WAL size / rotation with user-visible warning.
• Fail-first tests: write-rate budget under idle + streaming fixtures.

Tests

• Unit: writer rejects TRACE when configured level is INFO.
• Integration: after N seconds idle with default config, row growth rate < bound.

Questions for maintainers

1. Is the intended long-term policy documented for this area (especially sandbox gitdir / log TRACE defaults)?
2. Preferred residual anchor issue if several open threads describe the same mechanism?
3. If the high-level outline above is roughly aligned, would an invited PR with failing-first tests be welcome later — or is an internal fix preferred?

Happy to refine with more measurements or code pointers. No unsolicited PR from me.

Analysis (community)

Residual analysis for #28997 (logs_2.sqlite-wal unbounded growth / disk-fill) under docs/contributing.md — invitation-only, no unsolicited PR.

This is intentionally not a restatement of the TRACE-default residual already filed on #17320. Focus here is the sink budget / WAL lifecycle gap that lets a multi-GB WAL appear even when the base DB is smaller.

Deterministic evidence (code + runtime)

Code on current main (5dd992acd3):

• SQLite log file constant: codex-rs/state/src/sqlite.rs LOGS_DB_FILENAME = "logs_2.sqlite".
• Async sink knobs in codex-rs/state/src/log_db.rs:
• LOG_QUEUE_CAPACITY = 512
• LOG_BATCH_SIZE = 128
• LOG_FLUSH_INTERVAL = Duration::from_secs(2) (periodic commit pressure)
• LogDbLayer::on_event (≈L204) builds a LogEntry and try_sends into the inserter (≈L151–L152, L249). There is no max-bytes / rotate / hard WAL checkpoint budget adjacent to the flush path.
• default_filter() still defaults to TRACE (L55) — that is the volume source; #28997’s residual is the missing ceiling once volume exists.

Runtime on this host (2026-07-24, codex-cli 0.146.0-alpha.6):

• ~/.codex/logs_2.sqlite size 1778548736 bytes (~1.66 GiB)
• ~/.codex/logs_2.sqlite-wal size 30405632 bytes (~29 MiB still dirty under load)
• Level histogram (sqlite3 RO): TRACE|94907 DEBUG|21547 INFO|19528 WARN|796 ERROR|372 (total rows ≈137150)
• Community on #28997 reported tens–hundreds of GB WAL and a self-watch feedback loop; my local numbers show the sink is still actively dirty on 0.146.x even without that extreme.

Root-cause hypothesis

Two cooperating defects:

1. High event fan-in (TRACE-default + streaming targets) continuously dirties the WAL via 2s batch flushes.
2. No enforced max DB/WAL bytes, rotation, or aggressive checkpoint-on-threshold → pathological growth can fill disks (as reported).

This is a resource-governance residual, not “just lower the log level”.

High-level fix outline (not a PR)

1. Hard-cap logs_2.sqlite + WAL (truncate/rotate + user-visible warning when threshold hit).
2. Checkpoint/truncate WAL when size exceeds N MB; never rely on process exit alone.
3. Optionally suppress file-watcher TRACE for $CODEX_HOME/logs_2.sqlite* to break self-amplifying loops reported on this issue.
4. Fail-first integration: generate load, assert WAL stays under bound; regression if WAL exceeds cap after T minutes.

Questions for maintainers

1. Is there an intentional internal budget for logs_2.sqlite / WAL size today?
2. Should rotation live in the state crate next to LogDbLayer, or in app-server packaging?
3. If the outline aligns, would an invited PR with failing-first size-cap tests be welcome?

Happy to refine measurements. No unsolicited PR from me.

Related data point — even after log volume is addressed, the space is never reclaimed:

On Codex Desktop 26.721.4979.0 / app-server 0.146.0-alpha.3.1 (native Windows), ~/.codex/logs_2.sqlite was 650 MB with 0 rows in the logs table:

PRAGMA page_count → 166,493 pages
PRAGMA freelist_count → 166,280 pages (99.9% empty)
SELECT COUNT(*) FROM logs → 0

So the retention/cleanup did run at some point and deleted the rows, but the file is never VACUUMed, leaving a 650 MB empty shell that still gets opened/checkpointed on every launch.

Additionally, an abandoned 257 MB logs_2.sqlite (35,938 rows) was left behind at the pre-migration location ~/.codex/sqlite/logs_2.sqlite, untouched since the DB moved to ~/.codex/ — the migration didn't remove the old file.

Suggestion: run VACUUM (or PRAGMA incremental_vacuum) after retention deletes, and clean up the old-location DB after migration.