Codex app-server can grow to 30-40 GB when local rollout history is huge

Summary

After updating/restarting Codex Desktop, the codex app-server process grew to tens of GB on a 16 GB macOS machine, causing extreme swap usage and memory pressure. Local evidence points to a pathological persisted rollout/session history interacting with app-server thread listing/resume/read paths.

This report is intentionally sanitized: I am not attaching private transcript contents, only process metrics, file sizes, record counts, crash UI, and public repo code paths.

Environment

• Codex Desktop: 26.616.51431 build 4212
• Codex app Chromium framework: 149.0.7827.115
• macOS: 26.3 (25.3.0), arm64
• Physical memory: 16 GB
• Repo inspected: openai/codex at 5f129a4703ceaf843e1cb6996bbf4a4f21225198

Observed impact

• Activity Monitor initially showed codex at ~22.4 GB with ~15.34 GB swap.
• A later screenshot showed the same codex process at ~39.05 GB with ~32.06 GB swap.
• A live footprint/vmmap capture while the process was bad showed:
• codex [690] physical footprint: ~30 GB
• phys_footprint_peak: ~30 GB
• MALLOC_SMALL: ~30 GB dirty
• vmmap: writable regions total ~30.9 GB; ~28.9 GB swapped out
• OpenClaw/node child processes were much smaller at the time; the growth was in /Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled.

Crash screen

The same failure shape also produced this Codex crash screen:

!Codex crash screen showing SIGKILL after opening pathological thread

Visible crash details:

• Codex crashed with the following error:
• (code=null, signal=SIGKILL)
• Most recent error is a WARN from a slow SQL log insert:
• message: slow statement execution time exceeded alert threshold
• summary: INSERT INTO logs (...) VALUES ...
• rows_affected: 31
• rows_returned: 0
• elapsed: about 1.343s
• slow_threshold: 1s

After a tool-level aborted read, the app-server PID had changed/restarted and was back around a few hundred MB RSS, which is consistent with the UI crash/reload path rather than a clean handled error.

Thread-management API repro

The Codex thread-management API can list both problematic threads, but reading the pathological one aborts/crashes:

• codex_app.list_threads(query="Restart Openclaw") succeeds and returns:
• thread id: 019df821-2a2e-7a01-be45-04b76d73c715
• title: Restart Openclaw
• status: notLoaded
• cwd: /Users/.../LaterAI
• codex_app.list_threads(query="OpenClaw recovery root-fix") succeeds and returns:
• thread id: 019ee577-5cd2-75d1-a997-ffb4bf6eeefd
• status: notLoaded
• codex_app.read_thread(threadId="019ee577-5cd2-75d1-a997-ffb4bf6eeefd", turnLimit=1, includeOutputs=false) succeeds, although it returns/truncates a very large single-turn summary.
• codex_app.read_thread(threadId="019df821-2a2e-7a01-be45-04b76d73c715", turnLimit=1, includeOutputs=false) returned aborted.

This makes the repro more concrete: metadata listing works, a smaller huge recovery thread can be read, but reading the pathological 11.88 GiB Restart Openclaw rollout via the thread-management API aborts/crashes Codex.

Local state correlated with the spike

The local Codex state contains one extreme rollout/thread history outlier:

• Rollout JSONL size: 12,754,101,031 bytes (~11.88 GiB)
• Line count: 84,643
• Thread DB tokens_used for that thread: 1,985,738,719
• Another related recovery thread had tokens_used = 40,431,073
• ~/.codex/sessions total size: ~12 GB, dominated by the single rollout file.
• ~/.codex/logs_2.sqlite: ~222 MB plus ~63 MB WAL at time of inspection.

Content-safe sampling of the huge JSONL:

Head sample, first 20,000 lines:

• response_item: 13,922
• event_msg: 5,526
• turn_context: 343
• session_meta: 175
• compacted: 34
• token_count events: 2,779
• largest sampled line: ~11.1 MB

Tail sample, last 2,000 lines:

• response_item: 1,502
• event_msg: 467
• turn_context: 17
• compacted: 14
• token_count events: 248
• largest sampled line: ~60.9 MB

Suspected code paths

These paths appear capable of loading or scanning large rollout histories without size/record caps.

1. thread/read with turns loads persisted history and builds API turns

• codex-rs/app-server/src/request_processors/thread_processor.rs
• thread_read_response_inner -> read_thread_view -> load_persisted_thread_for_read(... include_turns ...)
• load_persisted_thread_for_read calls thread_store.read_thread(... include_history: include_turns ...)
• When history is returned, it calls build_api_turns_from_rollout_items(&history.items).

1. Thread store include_history loads the entire rollout into memory

• codex-rs/thread-store/src/local/read_thread.rs
• attach_history_if_requested calls load_history_items(&path).
• load_history_items calls RolloutRecorder::load_rollout_items(path) and stores all items in StoredThreadHistory.

1. RolloutRecorder::load_rollout_items is unbounded

• codex-rs/rollout/src/recorder.rs
• It streams lines from open_rollout_line_reader, parses each line, then does items.push(item) into a Vec<RolloutItem> with no file-size cap, record-count cap, byte budget, or cancellation/backpressure noted in the inspected code.
• On an 11.88 GiB rollout with some 10-60 MB JSONL records, this can plausibly explain a 30-40 GB app-server footprint.

1. thread/list normally preserves scan-and-repair behavior unless state DB-only is requested

• codex-rs/app-server/src/request_processors/thread_processor.rs accepts use_state_db_only but defaults it through from the client params.
• codex-rs/thread-store/src/local/list_threads.rs calls RolloutRecorder::list_threads when use_state_db_only is false, and only calls list_threads_from_state_db when true.
• The protocol schema notes that useStateDbOnly skips scanning JSONL rollouts to repair thread metadata, which suggests the default thread list path can still touch rollout files.
• The list summary path appears more bounded than full history loading, but it is still part of the startup/listing hazard for pathological rollouts.

Expected behavior

A pathological local rollout should not be able to make the app-server consume tens of GB. Suggested guardrails:

• Refuse, truncate, quarantine, or summarize-only any rollout above a sane byte/record threshold for UI list/read/resume operations.
• Make thread/list use state DB-only by default in the desktop app when state DB is available, or repair in a bounded background job.
• Make thread/read include_turns=true enforce a byte/record budget and return a clear error or partial/summarized history instead of loading the whole rollout into memory.
• Add memory/size telemetry around rollout replay/list/repair paths.
• Consider clamping or treating impossible token counts (for example ~2B tokens) as corrupt/outlier state.

Reproduction shape

I do not have a minimal synthetic repro yet, but the local state suggests this shape:

1. Have a very large rollout JSONL under ~/.codex/sessions (observed: 11.88 GiB, 84,643 lines, individual lines up to ~60.9 MB).
2. Ensure state DB references the thread and contains an extreme token count.
3. Restart Codex Desktop.
4. Open the thread list and/or read/resume the pathological thread with turns/history included.
5. Observe /Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled memory grow to 30-40 GB and swap climb into tens of GB, or observe the app-server abort/crash with SIGKILL.

What would help users immediately

A small safe-mode/quarantine behavior would be very helpful: if Codex detects a rollout above a threshold during startup/list/read/resume, skip full replay, show a recoverable warning, and avoid loading that file into app-server memory.