Codex app repeatedly writes large SQLite etilqs temp files from internal Guardian thread metadata

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #32371
• #32035

Powered by Codex Action

Analysis (community)

Residual analysis for #32588 (Guardian thread metadata in state_5.sqlite spilling into etilqs_* temp files) under docs/contributing.md — invitation-only. No unsolicited PR.

Root-cause hypothesis

The thread-metadata extraction path stores title / preview / first_user_message with no length bound at any layer, and every Guardian review spawns a subagent thread whose first "user message" is the full Guardian prompt (the parent transcript) — so each review persists a multi-KB-to-81KB metadata row, and the thread-list query that re-reads all 1,338 rows × ~16 MB of duplicated text is what spills into the etilqs_* SQLite temp files you measured. Verified at HEAD 4c43465133:

1. No truncation anywhere in the chain. user_message_preview (codex-rs/protocol/src/protocol.rs:2357-2375) returns Some(message.to_string()) — the entire stripped user message, no cap. apply_user_message (codex-rs/state/src/extract.rs:136-146) then sets first_user_message = preview.clone(), preview = preview, and title = strip_user_message_prefix(...).to_string(), again with no length check. The ThreadMetadata schema fields (state/src/model/thread_metadata.rs:115-124) are unbounded String/Option<String>. Your 784 titles >1,000 chars (max 81,403) is exactly this path.

2. Guardian reviews are the producer. Each Guardian review spawns a subagent session with SubAgentSource::Other(GUARDIAN_REVIEWER_NAME) (core/src/guardian/review_session.rs:680), and the review's initial user message is the assembled Guardian transcript prompt — so the three metadata fields capture the prompt text wholesale. That matches your source={"subagent":{"other":"guardian"}} rows and the 1,193 rows with all three fields byte-identical (same parent transcript at similar points). It also explains why archiving 512 guardian rows cut the write rate 97.6% (21.48 → 0.523 MB/s): the query stopped materializing those rows.

3. Why it thrashes while idle. The thread-list/discovery surface re-reads these rows on a cadence (sidebar refresh, search), and SQLite spills large result sets into etilqs_* temp files under $TMPDIR — your 13 temp files / 285 MB over 314.6 s, repeating every 18–42 s, is the list query re-materializing ~16 MB of duplicated metadata per pass. This is independent of the logs_2.sqlite TRACE path (which you correctly ruled out — logs table empty, WAL 20 KB).

Ranked fix outline (all OSS, codex-rs):

• (a) Bound the fields at extraction: truncate title / preview / first_user_message to a display-grade length (e.g. 256 chars) in apply_user_message (or inside user_message_preview itself, which fixes every consumer at once). Display-only change; no behavioral risk. A schema migration could shrink existing bloated rows on open.
• (b) Stop persisting Guardian-review threads as user-visible list entries at all: when source is the Guardian subagent marker, either auto-archive the thread row on review completion or exclude it from the default thread-list/search queries. This matches your measured workaround (manual archiving) and removes the write amplification rather than just shrinking it.
• (c) If Guardian rows must remain listed, store a synthetic preview (e.g. "Guardian review of <parent title>") instead of the raw prompt — the prompt is already persisted in the thread's own rollout, so the metadata copy is pure duplication.

Fail-first tests I'd want before any merge: (1) an 81k-char Guardian prompt produces metadata fields ≤ the cap (today: stored verbatim); (2) 1,338 Guardian-sourced rows do not appear in the default thread-list query (today: they do); (3) the list query over a bloated DB stays under a temp-file spill threshold (today: ~16 MB re-read per pass). Happy to draft the test patch if a maintainer wants it.

One adjacent OSS note (not part of this issue's fix): the same Guardian lifecycle does a full O(N) collection of the parent transcript on every review (core/src/guardian/prompt.rs:116 → :419-521; the GuardianTranscriptCursor at :73-76 is consulted only after collection for Delta rendering), so K reviews cost O(N·K) CPU/RAM — an incremental cursor over history.raw_items()[already_collected..] would collapse that. Mentioning here only because it shares the Guardian anchor; the etilqs_* mechanism above stands alone. No unsolicited PR from me.

Windows corroboration: 8,551 active Guardian histories and severe Desktop degradation

I observed the same failure class on Windows at a substantially larger scale, with a strong mitigation/recovery A/B.

Environment

• Codex Desktop: 26.820.7780.0
• Codex CLI / bundled core inspected during recovery: 0.149.1
• Windows 11 x64: 10.0.26200
• Approval policy: on-request
• Approval reviewer at the time of the incident: legacy-compatible approvals_reviewer = "guardian_subagent"

Observed state

Read-only inspection before maintenance found:

• 8,551 active histories with the exact source {"subagent":{"other":"guardian"}}
• 8,498 had thread_source = subagent
• 53 had thread_source = guardian_review
• approximately 12,283 active rollout files, totaling more than 16 GB
• more than 70 Codex-related child processes and approximately 3.5 GB aggregate memory in one snapshot
• this process count is not fully role-attributed, so I am not claiming that every process was a Guardian process
• logs_2.sqlite had grown to approximately 7.332 GB
• SQLite integrity checks were ok

User-visible symptoms included:

• severe typing lag and delayed character echo
• thread/sidebar clicks frequently doing nothing or opening a blank conversation pane
• messages remaining unsent for 20 seconds or longer
• skills remaining on Loading skill...
• existing threads failing in CLI with errors including:
• thread ... already has an active writer (code -32600)
• failed to load a bounded thread item page

Mitigation and outcome

1. I disabled Approve for Me by changing the approval reviewer back to the user reviewer and restarted Codex.

• CPU usage and Desktop responsiveness improved immediately.

1. During an offline, backup-first maintenance pass, 9,127 DB-backed internal histories were archived, one orphan rollout was quarantined, and two oversized history items were repaired.
2. The active rollout count fell to approximately 3,074.
3. The log database was compacted from approximately 7.332 GB to 1.204 GB.
4. Post-maintenance SQLite integrity checks remained ok, and codex doctor completed successfully.

Afterward, a previously unusable existing thread opened quickly, loaded skills, accepted a message, and returned a response normally. New threads also remained responsive.

The immediate improvement from disabling the reviewer suggests active review work/lifecycle contributed to the live CPU/UI problem. The larger improvement after archiving thousands of internal histories strongly corroborates this issue's finding that completed Guardian histories remaining active materially degrade the thread-list/history path. The log DB compaction was performed in the same maintenance window, so I cannot attribute every part of the final improvement to one database alone.

Requested product behavior

• Completed Auto-review/Guardian histories should be ephemeral, automatically archived, or excluded from normal active thread-list/search/hydration paths.
• Legacy guardian_subagent state should be migrated safely to the current canonical reviewer behavior without retaining old internal histories as active user history.
• Startup/codex doctor should warn when completed internal approval histories accumulate abnormally.
• A supported cleanup/migration path should archive already accumulated internal review histories without requiring direct SQLite maintenance.
• Approval-review cancellation, timeout, and client disconnect paths should reliably release writers and child processes.

This also appears related to #20864, #21211, #32035, #38144, and #38653.

I am not attaching raw rollout files, SQLite databases, thread IDs, prompts, or local paths because they contain private conversation and repository data. I can provide additional sanitized counts or bounded diagnostics if maintainers request them.