macOS: compaction checkpoints re-embed screenshots until rollout hits 3.9 GB; ingestion then uses 10–20 GB RSS, causing jetsam kills and a kernel watchdog panic on a 16 GB Mac

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #29812
• #30932

Powered by Codex Action

Follow-up: I root-caused this and have a validated patch on a fork branch. Since external PRs are invitation-only per CONTRIBUTING.md, posting the results here — happy to submit it as a PR if the team wants to invite it, or feel free to lift the branch directly.

Branch: chyewon/codex:fix/rollout-memory-compacted-image-bloat (single commit, codex-rs/rollout only; cargo test -p codex-rollout green, clippy clean)

Root cause (confirmed in source + live instrumentation)

1. Disk snowball: compaction persists compacted records whose replacement_history re-embeds all prior inline screenshots (base64 data: URLs inside kept user messages), so every checkpoint re-persists every historical screenshot — 98 checkpoints growing 0.1→53 MB each in my 3.9 GB rollout (mechanism of #23257 / #24948).
2. Memory: rollout loading is eager (load_rollout_items collects everything into a Vec), and state-DB backfill visits every session file through it (extract_metadata_from_rollout). Measured on a real 973 MB production rollout: backfill peaks at ~1.0–1.6 GB; thread/resume via app-server peaks at 4.8 GB (~5× file size — scaled to the 3.9 GB rollout that predicts ~19.6 GB, matching the jetsam/panic forensics above).

The patch

1. Redacts inline data: images >16 KB from persisted compacted checkpoints (schema-valid 1×1 PNG placeholder; in-memory history untouched; small/remote images and non-compacted records unchanged) — stops the snowball at the source.
2. Streams backfill metadata extraction via a new for_each_rollout_item visitor — backfill memory becomes O(largest line) instead of O(file).
3. Deserializes rollout lines from a borrow instead of from_value(v.clone()) — removes a per-line full-DOM copy (transient-only win).

Measured (same 973 MB production rollout)

| Metric | Before | After |
|---|---|---|
| Backfill (state_db::init) peak RSS, same in-process harness | 1,604 MB | 701 MB (bounded by line size, not file size) |
| Rollout size with checkpoint-image redaction | 973 MB | 130 MB (−87%) |
| thread/resume peak RSS on the redacted file | 4,825 MB | 952 MB (−80%) |

End-to-end checks on the redacted file: thread/resume returns the same thread id with zero parse errors, and a live model turn on the resumed session completed normally (provider accepted placeholder data URLs; the model replied).

Includes a regression test (recorder_redacts_large_inline_images_in_persisted_compacted_history) covering raw persisted bytes, placeholder round-trip through load_rollout_items, and non-compacted records passing through unchanged.

---

Update (same day): after an adversarial completeness review, the branch gained a second commit (802e92a) extending the persist-time sanitizer beyond message InputImage content to the other image-bearing shapes in replacement_history: FunctionCallOutput content items (FunctionCallOutputContentItem::InputImage) and ImageGenerationCall.result (bare base64), with call_id pairing preserved. The regression test now also asserts the resume entry point (get_rollout_history) consumes a redacted rollout cleanly. cargo test -p codex-rollout green, clippy clean. Known remaining limitation (intentionally out of scope): the resume path still eagerly materializes the full item Vec; with redaction in place files stop reaching GBs, but streaming resume reconstruction would be a worthwhile follow-up.

Confirming this on macOS, at roughly 10× the scale reported in the opening post. Same mechanism, same log signatures.

Environment

• Codex Desktop: ChatGPT.app 26.721.41059, Codex Framework 150.0.7871.128
• macOS 26 (Darwin 27.0.0), Apple Silicon, 18 cores / 48 GB RAM
• Observed 2026-07-26

Scale

| Metric | This issue | Mine |
|---|---|---|
| Largest single rollout | 3.9 GB | 45.14 GB (48,465,379,354 bytes) |
| ~/.codex total | 10 GB | 463 GB |
| ~/.codex/sessions | — | 434 GB (433 GB from July alone) |
| Rollout files > 100 MB | — | 313 |
| logs_2.sqlite | 721 MB | 1.5 GB + 192 MB un-checkpointed WAL |

state_5.sqlite was an additional 97 MB with a 4 MB WAL. The disk hit 94% full.

Evidence that compaction re-embedding is the driver

After stripping image payloads, the 45 GB rollout is 79,899 lines. The 52,004 base64 PNGs it contained were packed into just 1,018 lines — about 51 images per line. That density is only explicable by snapshots re-embedding the same screenshots, exactly as described here.

Record breakdown of that file:

response_item   48,275     compacted       617
event_msg       28,724     turn_context    764
                           session_meta     52

Head sample of the first 3,000 lines, by payload subtype:

function_call_output   744 records   34,393,039 bytes   (89% of sampled volume)
message                 48 records    2,544,401 bytes
reasoning              452 records      963,614 bytes

Largest single JSON line: 2,274,751 bytes, a function_call_output whose output[0] is {"type":"input_image","image_url":"data:image/png;base64,..."}.

Across the whole corpus, images were 1.6% of lines but 87.3% of bytes.

User-visible impact beyond disk

This is not only a storage problem — it degrades the app:

[AppServerConnection] response_routed method=thread/turns/list durationMs=40139 queueWaitMs=8735
[AppServerConnection] response_routed method=thread/turns/list durationMs=56454 queueWaitMs=15548

Opening the affected thread took 40–56 s. The app-server request queue sat pinned at inFlightRequestCount=6 with background requests expiring (reason=queue-expired), and turn/start timed out at 30 s, surfacing as the "Error submitting message" banner (see #27395).

Result of stripping images

Replacing every data:image/png;base64,… payload with a 1×1 transparent PNG across 754 rollout files:

• 534,529 images removed
• 409.4 GB reclaimed
• Individual files shrink 97–98% (45.14 GB → 0.63 GB for the worst one)
• 0 errors; verified line counts unchanged, all lines still valid JSON, and every non-image line byte-identical

That ratio suggests the actual conversational content of even a very heavy multi-month workload is on the order of a few GB — the rest is duplicated pixels.

Suggested fix

Store each image once in a content-addressed blob directory and reference it by hash in the rollout, rather than inlining base64. Failing that, excluding image payloads from compaction snapshots alone would remove the large majority of the growth, since the re-embedding is what turns linear growth into quadratic.

A checkpoint of logs_2.sqlite/state_5.sqlite WALs on shutdown would also help; mine had 192 MB that never got merged.

Workaround for anyone hitting this now

Rewriting each rollout line and substituting the base64 payload with a valid 1×1 transparent PNG keeps the JSON structure intact and the app reads the files normally. Non-image lines can be passed through byte-for-byte. Important: skip any file currently open by the app (lsof -c codex -c ChatGPT -c Codex) — replacing an open file leaves the descriptor pointing at the old inode, so the space is never released.

Related: #32128, #23257, #25815, #29531, #20864.

The 3.9 GB rollout with 98 compacted records growing to ~53.4 MB each is an excellent oversized-persistence test, and the 10–20 GB ingestion spikes show why a bounded local reader matters.

I’m testing Codex Rescue’s oversized-record diagnosis on real cases like this. It will not deduplicate screenshots, rewrite compaction history, or fix Codex’s app-server ingestion. It is local-first, leaves the source rollout untouched, and does not replay unknown side effects.

If the affected session still exists locally, could you try:

pipx install codex-rescue==0.1.0a3
codex-rescue sessions
codex-rescue doctor --latest

Sanitized diagnostic output is sufficient—please don’t publish raw rollouts, screenshots/base64 data, SQLite databases, prompts, paths, or secrets.

https://github.com/shleder/codex-rescue