app-server: feedback log sqlite (logs_N.sqlite) grows unbounded — ~0.75 GB/day, no retention/rotation

Open 💬 2 comments Opened Jun 4, 2026 by 2vanm

💡 Likely answer: A maintainer (github-actions[bot], contributor) responded on this thread — see the highlighted reply below.

Summary

The app-server's internal feedback/telemetry log database ($CODEX_HOME/logs_<N>.sqlite) grows without bound — no retention, no rotation, no size cap. On a long-running headless deployment (app-server driven 24/7) it reached 9.8 GB in ~10 days (~0.75 GB/day) before we noticed and manually cleared it.

This is distinct from state_<N>.sqlite (actual thread/session state), which stays small and healthy (~106 MB).

Environment

codex-cli 0.132.0 (codex app-server --listen stdio://)
Linux x86_64 (musl vendor build), driven continuously by an automation host (frequent short sessions: heartbeats, scheduled jobs, chat).
CODEX_HOME set to a persistent dir.

Evidence

logs_<N>.sqlite has a single dominant table logs:

1,420,123 rows, ~9.4 GB used (10.47 GB file, 1.08 GB freelist)
schema: id, ts, ts_nanos, level, target, feedback_log_body, module_path, file, line, thread_id, process_uuid, estimated_bytes

Bytes by column — feedback_log_body dominates at 7.3 GB. By target:

 5.88 GB  n=314,237  codex_api::endpoint::responses_websocket   (TRACE)
 0.44 GB  n=329,301  codex_otel.log_only
 0.39 GB  n=324,112  codex_otel.trace_safe
 0.21 GB  n=215,142  codex_api::sse::responses
 0.21 GB  n=    453  codex_core::client

Level mix: 675,801 INFO + 614,236 TRACE + 73,883 WARN + 23,110 DEBUG + 7 ERROR.

The largest single rows are ~800 KB each — full op.dispatch.compact / op.dispatch.user_input_with_turn_context request/response payloads serialised verbatim into feedback_log_body.

Steady-state growth (one row of the per-day breakdown):

2026-05-26  140,239 rows  0.71 GB
2026-05-27  136,379 rows  0.72 GB
...
2026-06-03  139,209 rows  0.75 GB

The logs_<N> suffix appears to be a schema version, not a size-based rotation — within a version the table grows indefinitely. RUST_LOG does not appear to gate these sqlite writes (no EnvFilter hook found for the feedback-log layer), and the only "bounded" reference in the binary (bounded_feedback_logs AS (...)) is a read-side CTE used when assembling a feedback bundle, not a write-side cap.

Impact

~0.75 GB/day ≈ 22 GB/month, ~270 GB/year of pure telemetry on disk for any long-lived app-server deployment.
Full prompt/response payloads (including large compaction turns) are persisted indefinitely on the local disk — also a data-retention concern.

Expected behaviour

The feedback log should be bounded by default — e.g. a max age and/or max size with automatic pruning/rotation (à la a rolling window), or an explicit CODEX_HOME/config setting to cap or disable it. Ideally TRACE-level full-payload dumps should not be persisted to the on-disk feedback DB by default.

Workaround

logs_<N>.sqlite* is safe to delete while the app-server is stopped — it is recreated empty on next start and only debug/feedback-bundle history is lost. But this requires manual intervention and downtime since the file is held open continuously.

View original on GitHub ↗

2 Comments

github-actions[bot] contributor · 2 months ago

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

#25815

Powered by Codex Action

KZ8589 · 1 month ago

Adding two measurements from a Windows machine, including one that argues against a causal
link I had assumed myself.

Environment

Windows 11 Pro (10.0.26100), 4 logical CPUs
Codex desktop app 0.145.0-alpha.30 (same behaviour on 0.144.0-alpha.4)
Codex CLI 0.144.4 (npm @openai/codex)
No size / rotation / log-volume option found in ~/.codex/config.toml

1. Same unbounded growth, smaller scale

~/.codex/logs_2.sqlite reached 1,097.31 MB (verified by file size on disk) and kept growing
across application restarts. Much smaller than the 9.8 GB in the opening report, but the same
shape: no cap, no rotation, no setting I could find.

The growth here coincided with a single long-running task that spawned 116 subagents directly
over 6 days; agent thread limit reached occurred 25 times, and the database held 468 unclosed
relations. Codex's own internal instrumentation reported log aggregation SQL at 18.7 s, DB
connection acquisition at 8.55 s, and 33 s to write 963 TRACE entries — those three numbers are
self-reported by Codex and I have not independently verified them. The only figure I verified
myself is the file size.

2. Measured: zero growth while idle

I wanted to know whether idle clients drive this. On this machine several processes keep the
Codex runtime open at once: one codex mcp-server per open Claude Code window, plus a codex
process embedded in the ChatGPT desktop app.

Measurement window: 116 minutes. Several idle codex clients at the start; the Codex app and
ChatGPT were closed partway through, leaving one idle codex mcp-server; no tasks issued; no
new session records written. Combined size of logs_2.sqlite, its -wal and its -shm was
6,492,072 bytes at both ends — a delta of exactly 0 over the whole window, which spanned both
configurations. Modification timestamps did advance, so the files are being touched, but no
growth resulted.

Caveat: a WAL reuses already-allocated space, so "size unchanged" is not the same as "no
writes". What I can say is that idling did not grow the database.

3. A/B test: quarantining the log DB did not reproduce a speed-up

I moved logs_2.sqlite / -wal / -shm aside (nothing else changed) and ran the same fixed workload
before and after: disposable working folder, 3 subagents launched in parallel, each asked only
to run echo ok and summarise the result in 1-2 sentences; wall-clock from start until all
three finished; identical prompt and command every run.

Before quarantine (n=3): median 53.76 s — runs 179.75 / 53.76 / 35.51
After quarantine (n=6): median 36.24 s — runs 37.52 / 47.45 / 33.83 / 28.70 / 37.35 / 35.13
No run exceeded 90 s after; before, 1 of 3 took ~180 s

Against bands I registered before measuring (improved = median <= 27 s; same level = 35-55 s;
worse = >= 90 s) this lands on "same level". So on this path I could not demonstrate that the
1.1 GB database was what made things slow, even though the numbers did come down.

Caveats, because they matter for how much weight this deserves:

measured with the CLI (0.144.4) while the symptom I was chasing is in the app (0.145.0-alpha.30)
4 logical CPUs shared with other applications; CPU contention is not separated out
the test load (3 subagents, trivial work, ~40 s) is far lighter than the real symptom (many

subagents, heavy work, long duration), so DB contention under heavy load is not ruled out

median 36.24 s sits close to the 35 s band edge

I mention this because the performance reports (e.g. #29177) and this growth report tend to be
read as one story. The growth is clearly real. The causal step from "large log DB" to "slow" is
the part I could not confirm at light load, and I would rather post the negative result than
leave the assumption unchallenged.

Separate observation (not claimed as a defect)

Each subagent also produces its own session record on disk: one run of the test above (parent
plus 3 subagents) creates 4 records, so the 6 runs created 24, and this machine's sessions
directory now totals about 1.7 GB across 1,285 records. I note it only as context for how
expensive high fan-out is in stored records — that directory is my own work history, not a bug.