›_ Codex Issues
Codex Issues / #29510

Codex app-server can grow to 30-40 GB when local rollout history is huge

Open 💬 7 comments Opened Jun 23, 2026 by ghitafilali
💡 Likely answer: A maintainer (github-actions[bot], contributor) responded on this thread — see the highlighted reply below.

Summary

After updating/restarting Codex Desktop, the codex app-server process grew to tens of GB on a 16 GB macOS machine, causing extreme swap usage and memory pressure. Local evidence points to a pathological persisted rollout/session history interacting with app-server thread listing/resume/read paths.

This report is intentionally sanitized: I am not attaching private transcript contents, only process metrics, file sizes, record counts, crash UI, and public repo code paths.

Environment

  • Codex Desktop: 26.616.51431 build 4212
  • Codex app Chromium framework: 149.0.7827.115
  • macOS: 26.3 (25.3.0), arm64
  • Physical memory: 16 GB
  • Repo inspected: openai/codex at 5f129a4703ceaf843e1cb6996bbf4a4f21225198

Observed impact

  • Activity Monitor initially showed codex at ~22.4 GB with ~15.34 GB swap.
  • A later screenshot showed the same codex process at ~39.05 GB with ~32.06 GB swap.
  • A live footprint/vmmap capture while the process was bad showed:
  • codex [690] physical footprint: ~30 GB
  • phys_footprint_peak: ~30 GB
  • MALLOC_SMALL: ~30 GB dirty
  • vmmap: writable regions total ~30.9 GB; ~28.9 GB swapped out
  • OpenClaw/node child processes were much smaller at the time; the growth was in /Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled.

Crash screen

The same failure shape also produced this Codex crash screen:

!Codex crash screen showing SIGKILL after opening pathological thread

Visible crash details:

  • Codex crashed with the following error:
  • (code=null, signal=SIGKILL)
  • Most recent error is a WARN from a slow SQL log insert:
  • message: slow statement execution time exceeded alert threshold
  • summary: INSERT INTO logs (...) VALUES ...
  • rows_affected: 31
  • rows_returned: 0
  • elapsed: about 1.343s
  • slow_threshold: 1s

After a tool-level aborted read, the app-server PID had changed/restarted and was back around a few hundred MB RSS, which is consistent with the UI crash/reload path rather than a clean handled error.

Thread-management API repro

The Codex thread-management API can list both problematic threads, but reading the pathological one aborts/crashes:

  • codex_app.list_threads(query="Restart Openclaw") succeeds and returns:
  • thread id: 019df821-2a2e-7a01-be45-04b76d73c715
  • title: Restart Openclaw
  • status: notLoaded
  • cwd: /Users/.../LaterAI
  • codex_app.list_threads(query="OpenClaw recovery root-fix") succeeds and returns:
  • thread id: 019ee577-5cd2-75d1-a997-ffb4bf6eeefd
  • status: notLoaded
  • codex_app.read_thread(threadId="019ee577-5cd2-75d1-a997-ffb4bf6eeefd", turnLimit=1, includeOutputs=false) succeeds, although it returns/truncates a very large single-turn summary.
  • codex_app.read_thread(threadId="019df821-2a2e-7a01-be45-04b76d73c715", turnLimit=1, includeOutputs=false) returned aborted.

This makes the repro more concrete: metadata listing works, a smaller huge recovery thread can be read, but reading the pathological 11.88 GiB Restart Openclaw rollout via the thread-management API aborts/crashes Codex.

Local state correlated with the spike

The local Codex state contains one extreme rollout/thread history outlier:

  • Rollout JSONL size: 12,754,101,031 bytes (~11.88 GiB)
  • Line count: 84,643
  • Thread DB tokens_used for that thread: 1,985,738,719
  • Another related recovery thread had tokens_used = 40,431,073
  • ~/.codex/sessions total size: ~12 GB, dominated by the single rollout file.
  • ~/.codex/logs_2.sqlite: ~222 MB plus ~63 MB WAL at time of inspection.

Content-safe sampling of the huge JSONL:

Head sample, first 20,000 lines:

  • response_item: 13,922
  • event_msg: 5,526
  • turn_context: 343
  • session_meta: 175
  • compacted: 34
  • token_count events: 2,779
  • largest sampled line: ~11.1 MB

Tail sample, last 2,000 lines:

  • response_item: 1,502
  • event_msg: 467
  • turn_context: 17
  • compacted: 14
  • token_count events: 248
  • largest sampled line: ~60.9 MB

Suspected code paths

These paths appear capable of loading or scanning large rollout histories without size/record caps.

  1. thread/read with turns loads persisted history and builds API turns
  • codex-rs/app-server/src/request_processors/thread_processor.rs
  • thread_read_response_inner -> read_thread_view -> load_persisted_thread_for_read(... include_turns ...)
  • load_persisted_thread_for_read calls thread_store.read_thread(... include_history: include_turns ...)
  • When history is returned, it calls build_api_turns_from_rollout_items(&history.items).
  1. Thread store include_history loads the entire rollout into memory
  • codex-rs/thread-store/src/local/read_thread.rs
  • attach_history_if_requested calls load_history_items(&path).
  • load_history_items calls RolloutRecorder::load_rollout_items(path) and stores all items in StoredThreadHistory.
  1. RolloutRecorder::load_rollout_items is unbounded
  • codex-rs/rollout/src/recorder.rs
  • It streams lines from open_rollout_line_reader, parses each line, then does items.push(item) into a Vec<RolloutItem> with no file-size cap, record-count cap, byte budget, or cancellation/backpressure noted in the inspected code.
  • On an 11.88 GiB rollout with some 10-60 MB JSONL records, this can plausibly explain a 30-40 GB app-server footprint.
  1. thread/list normally preserves scan-and-repair behavior unless state DB-only is requested
  • codex-rs/app-server/src/request_processors/thread_processor.rs accepts use_state_db_only but defaults it through from the client params.
  • codex-rs/thread-store/src/local/list_threads.rs calls RolloutRecorder::list_threads when use_state_db_only is false, and only calls list_threads_from_state_db when true.
  • The protocol schema notes that useStateDbOnly skips scanning JSONL rollouts to repair thread metadata, which suggests the default thread list path can still touch rollout files.
  • The list summary path appears more bounded than full history loading, but it is still part of the startup/listing hazard for pathological rollouts.

Expected behavior

A pathological local rollout should not be able to make the app-server consume tens of GB. Suggested guardrails:

  • Refuse, truncate, quarantine, or summarize-only any rollout above a sane byte/record threshold for UI list/read/resume operations.
  • Make thread/list use state DB-only by default in the desktop app when state DB is available, or repair in a bounded background job.
  • Make thread/read include_turns=true enforce a byte/record budget and return a clear error or partial/summarized history instead of loading the whole rollout into memory.
  • Add memory/size telemetry around rollout replay/list/repair paths.
  • Consider clamping or treating impossible token counts (for example ~2B tokens) as corrupt/outlier state.

Reproduction shape

I do not have a minimal synthetic repro yet, but the local state suggests this shape:

  1. Have a very large rollout JSONL under ~/.codex/sessions (observed: 11.88 GiB, 84,643 lines, individual lines up to ~60.9 MB).
  2. Ensure state DB references the thread and contains an extreme token count.
  3. Restart Codex Desktop.
  4. Open the thread list and/or read/resume the pathological thread with turns/history included.
  5. Observe /Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled memory grow to 30-40 GB and swap climb into tens of GB, or observe the app-server abort/crash with SIGKILL.

What would help users immediately

A small safe-mode/quarantine behavior would be very helpful: if Codex detects a rollout above a threshold during startup/list/read/resume, skip full replay, show a recoverable warning, and avoid loading that file into app-server memory.

View original on GitHub ↗

7 Comments

github-actions[bot] contributor · 2 months ago

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

  • #29391
  • #28150

Powered by Codex Action

pdh0826 · 1 month ago

Still reproducible on a newer Codex Desktop release, with an even larger local rollout. This appears to be the same failure mode.

Environment

  • Codex Desktop: 26.707.62119
  • Bundled codex-cli: 0.144.2
  • macOS: 15.7.4 (24G517), arm64

Observed behavior

  • The issue has recurred for several days.
  • Fully restarting ChatGPT/Codex only reclaims memory temporarily.
  • After one restart, the bundled codex app-server launched at 10:05:13 and reached 26-27 GB by about 10:14.
  • Activity Monitor showed 27.19 GB; sample reported a 32.7 GB peak.
  • A live footprint capture while memory was high reported about 23 GB physical footprint, including about 21 GB dirty in MALLOC_MEDIUM and 832 MB in MALLOC_LARGE.
  • The app-server's MCP/Node child processes were only a few MB each, so the growth was in the main bundled codex process.
  • System pressure was severe: about 12 GB in the compressor and only roughly 167-300 MB unused during captures.

Correlated local state

I checked file metadata only and did not inspect or upload transcript contents.

  • Largest rollout JSONL: 29,138,057,205 bytes (about 27.1 GiB)
  • The file was originally created under the 2026/07/05 session directory and was last modified on 2026-07-14 10:01:22 +0900.
  • Other large rollouts were approximately 2.17 GB, 1.34 GB, and 1.17 GB.

A 3-second live process sample while the problem was occurring showed a Tokio worker in async file reading plus repeated JSON parsing/copy/allocation paths, including:

  • tokio::fs::file::File as AsyncRead::poll_read
  • serde_json::read::StrRead::parse_str
  • _realloc, _platform_memmove, and vm_copy

This strongly matches the unbounded rollout hydration described in this issue. The especially important regression signal is that the problem remains present in Desktop 26.707.62119 / CLI 0.144.2, and restart alone immediately reproduces it because the oversized rollout is loaded again.

No session contents, usernames, project paths, or private logs are included here.

YoungMoneyInvestments · 1 month ago

Confirming this on macOS 26.5 (M-series), ChatGPT.app 26.707.72221 / codex-cli 0.144.2 — reproduced on two different MacBook Pros, so it's not machine-specific.

Measurements tonight:

  • One codex app-server process, 18h uptime → 77 GB footprint (footprint -p), swap 52/53 GB full, macOS 'out of application memory' dialogs.
  • After force-restarting ChatGPT.app: swap dropped 52 GB → 6.6 GB instantly.
  • A fresh app-server grew ~860 MB in its first 33 minutes under agent workload (~1.5 GB/hr), so the growth starts immediately, not only after long sessions.
  • ~/.codex/sessions held 1,637 rollout .jsonl files (2.1 GB). Closing sessions in the UI frees nothing — everything lives in the single long-lived app-server heap.

Note for anyone diagnosing via Force Quit dialog: ChatGPT and Google Chrome can show identical multi-GB values (we saw both at exactly 17.98 GB) — that's shared Chromium-framework memory double-counted (the Codex renderer maps ~1.9 TB virtual). Check per-process RSS/footprint before trusting it.

Workarounds that held for us:

  1. Prune old rollouts: move ~/.codex/sessions files older than 14 days elsewhere (2.1 GB → 528 MB here).
  2. A cron/launchd watchdog that kills just the codex app-server process when its footprint exceeds ~12 GB — ChatGPT.app stays open and respawns it on next use, so this is far less disruptive than restarting the whole app.

Would love a real fix: cap/stream rollout history loading and tool output instead of accumulating it in the app-server heap.

teamleaderleo · 18 days ago

Confirming this on a current ChatGPT/Codex Desktop build:

  • ChatGPT Desktop 26.803.41515 build 6321
  • bundled codex-cli 0.147.0-alpha.6.5
  • macOS 26.5.1 (25F80) arm64

The affected process was:

/Applications/ChatGPT.app/Contents/Resources/codex \
  -c features.code_mode_host=true \
  app-server --analytics-default-enabled

macOS recorded a 16 GB peak physical footprint for app-server. At capture time it was still around 5 GB physical with about 4.7 GB swapped out.

vmmap / footprint showed multi-GB allocator regions:

MALLOC_SMALL: 6.5 GB virtual
MALLOC_LARGE: 1.8 GB virtual
main malloc zone allocated: ~3.6 GB
main malloc zone fragmentation: ~1.0 GB

A one-minute trace kept app-server around 98–136% CPU. A 5-second sample repeatedly hit:

tokio::fs::file::File::poll_read
serde_json::read::StrRead::parse_str
_realloc
_platform_memmove
hashbrown allocation/rehash paths

~/.codex/sessions was 7.7 GB. The five largest rollout files were:

2,087,802,775 bytes
1,287,719,134 bytes
1,212,192,775 bytes
1,211,706,260 bytes
1,211,567,011 bytes

lsof showed the 2.09 GB and 1.29 GB rollouts open by the affected app-server during the incident, along with a bunch of smaller rollouts.

Direct helper children were only about 148 MiB RSS combined, so the multi-GB memory use was in app-server itself.

This looks like the same oversized-rollout hydration path described here: app-server is reading rollout history and doing heavy JSON parsing/allocation while multi-GB rollout files are open.

adamdmcbride · 15 days ago

Corroborating report from a current macOS Desktop build. This is the same failure class, with a remote MCP-output-heavy rollout rather than an image-heavy rollout.

Environment

  • ChatGPT/Codex Desktop: 26.803.61601 (build 6396)
  • Bundled Codex: 0.147.0-alpha.6.5
  • macOS: 27.0 (26A5388g), arm64
  • Hardware: Apple M3 Max, 64 GB RAM
  • Doctor after recovery: config, state parity, installation, provider/WebSocket reachability, MCP, and update checks all pass

Sanitized evidence

  • A single active rollout reached 1,697,119,031 bytes with 12,298 valid JSONL records and 0 parse errors.
  • It was dominated by persisted remote MCP call results: approximately 1.63 GB of the file.
  • The task had made 2,925 high-volume read calls, including 1,475 calls to one paginated endpoint.
  • This was not image/base64 history and was not a malformed JSONL case.

Desktop repeatedly resumed the same rollout successfully, then the application crashed about 24 seconds later:

01:04:19Z  thread/resume started
01:04:27Z  resumed with 12,298 items; parse errors: 0
01:04:51Z  ChatGPT/Codex crash

01:06:58Z  thread/resume started
01:07:06Z  resumed with 12,298 items; parse errors: 0
01:07:30Z  ChatGPT/Codex crash

There were three app crashes in approximately four minutes. All three macOS reports show:

Exception: EXC_BREAKPOINT / SIGTRAP
Termination: Trace/BPT trap: 5
Faulting thread: V8Worker

Two earlier codex-code-mode-host reports from the same incident window contain these frames:

v8::base::FatalOOM
v8::Utils::ReportOOMFailure
v8::internal::V8::FatalProcessOutOfMemory
v8::internal::Heap::FatalProcessOutOfMemory

Those code-mode-host failures may be secondary, but they confirm V8 memory exhaustion was occurring during the same high-output task.

Recovery that worked

  1. Backed up and validated the rollout.
  2. Archived the oversized task through the supported Desktop thread archive operation so it was no longer under active sessions.
  3. Archived the other unpinned active rollouts over 1 GB.
  4. Added the supported tool_output_token_limit config setting for future model-visible tool output.
  5. Added workflow limits against full pagination and unbounded sequential high-volume reads.

After the oversized active rollout was quarantined, the app remained stable for more than two hours with no new crash report.

Secondary local-state observation

logs_2.sqlite was 4.42 GB, with about 3.72 GB represented by free pages. Because the database uses incremental auto-vacuum, bounded incremental_vacuum plus WAL checkpoints reduced it to about 0.70 GB while PRAGMA integrity_check remained ok. This was not the primary trigger, but it added avoidable recovery pressure.

Expected guardrails

This incident supports the guardrails proposed here and adds one important case: persisted remote MCP results need byte budgets too, not only shell output, images, or model-visible context.

Suggested fixes:

  1. Enforce byte/record budgets on thread/resume and load history incrementally.
  2. Cap, summarize, or externalize persisted MCP tool results before rollout serialization.
  3. Detect and quarantine one pathological rollout instead of repeatedly crashing the full app.
  4. Expose a supported safe-mode/session-repair path.
  5. Automatically checkpoint and incrementally vacuum the diagnostic log database.

I am intentionally not attaching the raw rollout, full app logs, or unredacted crash reports because they contain private prompts, local paths, and remote API payloads. Three app .ips reports and two code-mode-host .ips reports are available for private upload through the in-product feedback/support path if an OpenAI maintainer wants them.

shleder · 14 days ago

The 11.88 GiB rollout that makes thread/read abort and drives app-server to 30–40 GB is an unusually strong oversized-history test, especially with individual persisted records measured up to ~60.9 MB.

Codex Rescue cannot fix Codex’s unbounded hydration path or make Desktop reopen the thread. I’m testing whether its bounded local parser can identify the pathological record boundary without materializing the rollout, while keeping the source byte-identical. It does not replay calls and verification remains fail-closed.

If the affected rollout still exists, could you try:

pipx install codex-rescue==0.1.0a3
codex-rescue sessions
codex-rescue doctor --latest

Sanitized output is enough—please don’t publish raw rollouts, databases, tool contents, prompts, paths, or secrets.

https://github.com/shleder/codex-rescue

oneles · 9 days ago

Same issue here, but at a much lower trigger threshold — this is not just a heavy-user problem.

I hit this on a 16GB M1 Pro during normal daily use (not a power Codex user). A single
106 MB rollout was enough to make the whole system unusable.

Environment

  • ChatGPT Desktop (bundled Codex): 26.814.41407 (build 6720)
  • macOS arm64, Apple M1 Pro, 16 GB RAM
  • Pathological rollout: ~/.codex/sessions/2026/08/03/rollout-2026-08-03T10-56-06-019fc58c-*.jsonl

— 106 MB / 4,613 lines, 6 compactions, 558 tool calls over 15 days
(tiny compared to the 11.88 GiB case, yet triggers the same failure)

Measured impact (12 s sampling while ChatGPT was open)

  • CPU: 99.6–100% for the entire 12 s window (8 logical cores)
  • Memory compressor: +229,769 compressions / +58,152 decompressions in 12 s
  • Swap: grew from ~1.6 GB to ~2.3 GB while ChatGPT was open
  • Free memory dropped to ~4.4 GB with other apps idle
  • codex app-server + ChatGPT renderer consumed ~57% CPU

Key point

The same unbounded loading path (RolloutRecorder::load_rollout_items) triggers at
106 MB, not only at 11.88 GiB. On a 16 GB machine already running VS Code/Edge/etc.,
even one long Codex session is enough to stall the whole system. "Archiving" the thread
in the UI does NOT move the file out of ~/.codex/sessions — the app-server still scans
and loads it on startup (verified: file stays in place after Archive).

Request

Please add the guardrails proposed above: byte/record budget for load_rollout_items,
quarantine oversized rollouts, and state-DB-only thread listing. Happy to provide
more data if useful.