›_ Codex Issues
Codex Issues / #34268

Multi-agent V2 full-history forks duplicate historical compaction snapshots and inline images, causing >100 GiB session storage growth

Open 💬 9 comments Opened Jul 20, 2026 by gonzalolarralde
💡 Likely answer: A maintainer (github-actions[bot], contributor) responded on this thread — see the highlighted reply below.

What issue are you seeing?

A long-running Codex Desktop conversation using Ultra reasoning and multi-agent V2 produced approximately 110 GiB of local session data under $CODEX_HOME/sessions.

The growth appears multiplicative rather than being caused by subagents producing unusually large outputs:

  1. Context compaction persists Compacted rollout records containing a complete replacement_history.
  2. These replacement histories can contain inline base64-encoded screenshots.
  3. Older compaction records remain in the append-only parent rollout.
  4. A full-history subagent fork loads the parent’s persisted rollout items.
  5. Those inherited items—including all historical compaction snapshots—are serialized into the child rollout.
  6. Ultra reasoning proactively creates many subagents, multiplying the retained history across many files.

Observed impact from one session tree:

| Measurement | Result |
|---|---:|
| Total rollout files | 300 |
| Total reported size | approximately 110.09 GiB |
| Subagent rollout files | 294 |
| Size attributed to subagent rollouts | approximately 106.56 GiB |
| Files sharing the same root session ID | 295 |
| Files at least 1 GB | 51 |
| Files at least 500 MB | 86 |
| Files at least 100 MB | 155 |

Of those files, 159 were recorded by 0.145.0-alpha.18 with multi_agent_version: "v2" and accounted for approximately 88.18 GiB.

Many individual subagent rollouts were between 1.3 GB and 1.9 GB.

A representative completed child rollout was 1,706,557,797 bytes and contained 23,564 JSONL records. Its composition was:

  • 141 compacted records
  • Approximately 1.55 GiB in compacted records
  • 97.6% of the entire file attributable to compacted records
  • 125 individual JSONL records larger than 10 MB
  • Largest JSONL record approximately 12.93 MiB

One inspected replacement_history contained:

  • 118 history items
  • 19 input_image entries
  • 13,316,621 characters of inline data:image/png;base64,... data
  • Approximately 197,716 characters of input text

The child files are not small records that merely reference their parent. A representative child begins with its own session_meta, followed by a copy of the parent’s original session_meta and inherited parent events. Sampled parent payloads were identical, while outer timestamps were rewritten to the child creation time and some response items received new IDs.

The files have separate inodes and link counts of 1, and their allocated sizes are close to their logical sizes. This appears to be independently serialized history rather than sparse files or parent references.

This can consume enough storage to exhaust a local disk during a single long-running Ultra session.

What steps can reproduce the bug?

The severity depends on having a sufficiently large parent history, multiple compactions, inline images, and multiple full-history forks.

  1. Start a persisted local Codex thread using Codex Desktop or another local surface backed by the shared Codex runtime.
  2. Use the same thread for an extended period.
  3. Add several screenshots or other image inputs to the conversation.
  4. Continue the conversation long enough to trigger multiple context compactions.
  5. Select Ultra reasoning with multi-agent V2 enabled.
  6. Give Codex work that causes it to proactively spawn multiple subagents.
  7. Allow the generated spawn_agent calls to omit fork_turns. In multi-agent V2, the omitted value defaults to "all".
  8. Inspect the newly created child rollouts under $CODEX_HOME/sessions.
  9. Compare their sizes and initial JSONL records with the parent rollout.
  10. Repeat with additional subagents and observe that each full-history child can approach the persisted historical size of the parent before doing significant child-specific work.

Example read-only inspection commands:

du -sh "$CODEX_HOME/sessions"

find "$CODEX_HOME/sessions" -type f -name '*.jsonl' -print0 \
| xargs -0 stat -f '%z %N' \
| sort -nr \
| head -50

For a suspected child file:

head -n 1 "$CHILD_ROLLOUT" \
| jq '.payload | {
id,
session_id,
thread_source,
forked_from_id,
parent_thread_id,
multi_agent_version,
source
}'

The child metadata should identify it as a subagent and point to the parent through parent_thread_id and/or forked_from_id.

Inspecting the first several records should show the child’s new session_meta followed by inherited parent rollout records.

The problem becomes increasingly visible as these factors increase:

  • Number of retained compaction records
  • Number and size of inline images
  • Length of the parent thread
  • Number of spawned subagents
  • Use of full-history/default fork_turns="all" behavior

What is the expected behavior?

Spawning subagents should not cause local persisted session storage to grow in proportion to the parent’s complete append-only rollout history for every child, particularly when the history contains multiple historical compaction snapshots representing earlier versions of substantially the same model context.

A newly spawned child should not routinely approach 1–2 GB before performing meaningful child-specific work.

Disk growth should remain bounded enough that proactive multi-agent behavior cannot unexpectedly create more than 100 GiB of local session data from one conversation tree.

This expectation applies regardless of whether the triggering surface is Codex Desktop, the CLI, or codex exec, since the relevant multi-agent and rollout-persistence behavior appears to be shared.

Additional information

Environment and metadata from the affected session:

  • Platform: macOS
  • Surface: Codex Desktop
  • Reasoning mode: Ultra
  • Root metadata:
  • originator: "Codex Desktop"
  • source: "vscode"
  • history_mode: "legacy"
  • Child metadata:
  • thread_source: "subagent"
  • multi_agent_version: "v2"
  • Codex versions present:
  • 0.144.2
  • 0.144.5
  • 0.145.0-alpha.18

Relevant source locations:

  • codex-rs/core/src/session/multi_agents.rs
  • effective_multi_agent_mode maps Ultra reasoning to proactive multi-agent behavior.
  • The behavior applies to SessionSource::Cli, SessionSource::VSCode, SessionSource::Exec, and other sources.
  • codex-rs/core/src/tools/handlers/multi_agents_v2/spawn.rs
  • SpawnAgentArgs::fork_mode defaults omitted or empty fork_turns to "all".
  • "all" becomes SpawnAgentForkMode::FullHistory.
  • codex-rs/core/src/agent/control/spawn.rs
  • AgentControl::spawn_forked_thread flushes the parent and reads the stored thread with include_history: true.
  • It uses parent_history.items as the forked rollout items.
  • The fork filter retains RolloutItem::Compacted.
  • The inspected Compacted records contain almost all of the affected bytes.
  • codex-rs/core/src/session/mod.rs
  • The InitialHistory::Forked branch in record_initial_history persists the inherited rollout items.
  • The code comment says: If persisting, persist all rollout items as-is (the store filters).
  • codex-rs/core/src/compact.rs
  • Compaction creates a CompactedItem with replacement_history: Some(new_history.clone()).
  • codex-rs/core/src/session/mod.rs
  • replace_compacted_history also persists a CompactedItem containing a cloned replacement history.
  • codex-rs/rollout/src/recorder.rs
  • JsonlWriter::write_rollout_item assigns a new timestamp, serializes the item with serde_json::to_string, and writes it to the child JSONL.
  • codex-rs/app-server/README.md
  • thread/fork is documented as creating a new thread by copying stored history.

Questions that may help confirm whether the observed persistence is intentional:

  • Is a subagent fork expected to persist every historical Compacted record from the source rollout?
  • Is each earlier replacement_history expected to remain independently serialized after later compactions supersede it?
  • Is inline base64 image data expected to be repeated across every compaction snapshot and every full-history child fork?
  • Is there an intended upper bound for inherited persisted bytes when spawning a subagent?
  • Should proactive Ultra subagents have the same persistence behavior as an explicit user-created full thread fork?

I can provide additional sanitized structural evidence if needed, including record-size distributions, parent/child metadata relationships, payload hash comparisons, compaction counts, and encoded image lengths.

I cannot attach the complete affected rollouts because they are individually up to approximately 1.9 GB and contain private conversation and image data.

View original on GitHub ↗

9 Comments

github-actions[bot] contributor · 1 month ago

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

  • #33735
  • #34061

Powered by Codex Action

dorukardahan · 1 month ago

i checked current main after the recent paginated history work. It helps paginated parents, but it does not remove the legacy amplification reported here.

For a legacy parent, load_agent_model_context still calls read_stored_thread with include_history: true and returns the stored history items:

https://github.com/openai/codex/blob/808d3c2702ce8eae007c457aa930e7c3b68dd5f6/codex-rs/core/src/agent/control/spawn.rs#L99-L123

The fork path explicitly selects paginated history only when the parent is already paginated:

https://github.com/openai/codex/blob/808d3c2702ce8eae007c457aa930e7c3b68dd5f6/codex-rs/core/src/agent/control/spawn.rs#L596-L713

Forked history also keeps the source format, so a legacy parent stays legacy:

https://github.com/openai/codex/blob/808d3c2702ce8eae007c457aa930e7c3b68dd5f6/codex-rs/protocol/src/protocol.rs#L2679-L2688

The child then persists the inherited model context instead of sharing an immutable prefix:

https://github.com/openai/codex/blob/808d3c2702ce8eae007c457aa930e7c3b68dd5f6/codex-rs/thread-store/src/live_thread.rs#L110-L143

SessionMeta.history_base exists, but the production recorder currently writes history_base: None. i could not find a production writer that sets it to Some(...):

https://github.com/openai/codex/blob/808d3c2702ce8eae007c457aa930e7c3b68dd5f6/codex-rs/rollout/src/recorder.rs#L820-L843

A separate read-only structural scan of one affected store found about 12.3k compacted records. About 90.5% contained inline images. There were about 156.7k image occurrences but only 452 unique payload hashes. i did not inspect or upload image content, prompts, paths, project names, or thread IDs.

This is consistent with the same media being copied through compaction snapshots and inherited child context. The safe target seems to be an immutable shared ancestor prefix plus child-only events. Legacy parents also need a bounded or sanitized projection before a child is persisted. Parent rollouts should never be rewritten during fork repair.

Yasei-no-otoko · 1 month ago

Urgent reproduction evidence and tested patch

I traced the storage amplification on current main (61a44880a85d2fd0d8770908dea5733495e571c8) and prepared a bounded emergency mitigation as a reviewable two-stage stack:

Sanitized local evidence

A read-only audit found:

  • 1,054 JSONL files totaling 196.56 GiB;
  • 133 JSONL files at least 1 GiB each;
  • one legacy automatic-subagent lineage with 199 files totaling 190.80 GiB;
  • two child rollouts created five minutes apart measured 1,539,137,782 bytes / 67,684 records and 1,539,253,221 bytes / 67,766 records.

The later child added only 115,439 bytes / 82 records of logical delta, but allocated another 1.433 GiB physical file: about 13,333.9x write amplification relative to the incremental payload.

Root cause

Ordinary legacy resume already reopens and appends to its existing rollout. A distinct subagent rollout is also intentional. The defective path is automatic spawn_agent with a full-history fork from a legacy plain JSONL parent: it loads and reserializes the parent's entire accumulated rollout, including superseded compaction checkpoints and repeated inline media, into every child.

Patch boundary

The patch reverse-scans local legacy JSONL and retains only the newest safe compaction checkpoint plus the completed-turn suffix needed for reconstruction. For old checkpoints without window_number, it continues toward the head in count-only mode and normalizes the checkpoint ordinal copied to the child, preserving cold-resume window numbering without rewriting the parent.

Conservative full-history fallbacks remain for unsupported or ambiguous histories, and legacy numeric LastNTurns keeps its existing full-read behavior. The scan also preserves legacy compatibility for appended metadata updates and retired ghost_snapshot records. This removes the multiplicative child-write/storage cost; it intentionally leaves O(parent) read I/O for old legacy ordinal counting.

Duplicate-PR audit

I paginated and audited all 203 currently open PRs across changed files, patches, titles/bodies, linked issues, issue comments, reviews, and inline review comments. No open PR implements this local legacy checkpoint selection or automatic-subagent persistence fix. The closest PRs (#30977, #31175, #30866, and #31052) are complementary or target different persistence paths.

Validation
  • cargo fmt --all -- --check
  • cargo clippy --fix --tests --allow-dirty -p codex-rollout -p codex-thread-store -p codex-core
  • cargo test -p codex-rollout -p codex-thread-store — 112 + 157 passed
  • cargo test -p codex-core spawn_agent_ — 37 unit + 15 integration passed
  • cargo test -p codex-core --test all suite::multi_agent_resume — 2 passed

The test_codex integration performs two real legacy compactions, invokes public collaboration.spawn_agent with fork_turns: "all", verifies the child inference request contains only the newest checkpoint plus suffix, checks the persisted normalized ordinal, and cold-resumes the child.

Review status

Codex review completed on both final heads and reported no major issues:

Earlier review findings were addressed with the public test_codex integration, the under-500-line PR split, portable build_with_auto_env, appended metadata handling, and legacy ghost-snapshot normalization. Both PRs are non-draft, CLEAN, and MERGEABLE at the reviewed commits.

Per the contribution policy, I have not opened an unsolicited upstream PR. Given the disk impact, please triage this urgently and, if the approach is acceptable, explicitly invite an upstream submission so these same reviewed commits can be proposed against openai/codex.

dd3ok · 1 month ago

Adding a current Windows Desktop datapoint from a smaller real-world thread tree.

Environment

  • Codex Desktop: 26.721.4979.0
  • Platform: Windows 11 x64
  • Thread tree: one image-heavy root thread and three child threads

Observed storage

| Rollout | Size |
| --- | ---: |
| Root | 2,086.2 MiB |
| Child 1 | 989.5 MiB |
| Child 2 | 986.6 MiB |
| Child 3 | 525.3 MiB |
| Total | 4,587.7 MiB (4.48 GiB) |

The separately stored generated-image artifacts contained 286 PNG files totaling 476.6 MiB. All 286 files had unique SHA-256 hashes, so the amplification was not caused by duplicate PNG files in generated_images.

Structural inspection showed inherited parent history in the child rollouts, including image-bearing records and compacted history. Across the four rollouts, inline base64 image data accounted for approximately 2,027.5 MiB.

This was an observed production workload rather than a minimal synthetic reproduction. The child structure was consistent with full-history inheritance, but I did not confirm whether fork_turns="all" was explicitly selected or obtained through the default behavior.

Even with only three child threads, the session tree reached 4.48 GiB. This supports the report that inherited history, compacted snapshots, and inline images multiply persisted storage at the child-rollout layer.

The image-persistence side of this observation is tracked in #22603.

Proposed direction

A child rollout should persist only child-specific events plus a bounded reference to an immutable parent checkpoint. It should not independently serialize every historical compaction snapshot or raw image payload inherited from the parent.

Privacy

No raw rollout, prompt, image, local path, username, project name, or thread ID is included. Only aggregate sizes, record structure, and content-hash counts were inspected.

ST4B1L · 26 days ago

Windows confirmation of this issue on a newer Codex Desktop/runtime build.

In-product Codex feedback/session reference: 019f4b7a-e382-7862-aeda-71eb42955f7c

Environment

| Component | Value |
| --- | --- |
| Surface | Codex Desktop |
| Codex Desktop package | 26.727.6591.0 |
| Runtime version recorded in current child metadata | 0.146.0-alpha.9.2 |
| OS | Windows 11 Pro, 10.0.26200 |
| Filesystem | NTFS |
| Volume size | 3725.12 GiB |
| Free space at diagnosis | 67.85 GiB |
| Model / reasoning | GPT-5.6 Sol / Ultra |
| Multi-agent configuration | enabled, max_threads = 9, max_depth = 1 |

Observed impact

%USERPROFILE%\.codex\sessions reached 892.810 GiB across 1,112 rollout files. A metadata-only aggregation grouped 854 child-agent rollouts under one long-lived parent thread. Those child files consume 822.711 GiB, approximately 92.2% of the live session store.

The newest child rollouts appeared at approximately 4.07-4.09 GiB immediately after creation, before meaningful child-specific work. The affected parent rollout was approximately 4.53 GiB during diagnosis.

The first records of the giant child files consistently contain the child's own session_meta, a subagent.thread_spawn.parent_thread_id referring to the same parent, and then inherited parent metadata and history records. An NTFS allocation check confirmed that a representative child file was not sparse and that its complete logical range was physically allocated.

The growth is stepwise at child creation rather than a constant idle leak. During a 20-second idle sample the session store changed by only approximately 0.03 MiB, while new full-history child forks produced multi-gigabyte jumps. With nine concurrent threads, one round of child creation can therefore materialize tens of gigabytes in seconds.

File-size distribution

| Size range | Files | Combined size |
| --- | ---: | ---: |
| At least 4 GiB | 5 | 20.85 GiB |
| 3-4 GiB | 98 | 342.71 GiB |
| 1-3 GiB | 258 | 438.35 GiB |
| 100 MiB-1 GiB | 193 | 80.44 GiB |
| Below 100 MiB | 558 | 10.46 GiB |

This confirms the full-history fork amplification described in this issue on Windows and runtime 0.146.0-alpha.9.2. No additional reproduction was triggered because the system volume was already close to exhaustion.

Expected behavior is that child agents reference or receive a bounded representation of inherited context instead of independently serializing the parent's complete append-only rollout and historical compaction snapshots. Codex should also enforce per-thread and global session-storage limits and warn before rollout storage or free disk space reaches a critical level.

ariccio · 18 days ago

I noticed some similar space-usage-explosion, noticed the size distribution was suspicious, and had my own codex poke around. Here's what it spit out:

I have an exceptionally large and long-running legacy Codex session lineage with unusual disk usage. I asked Codex to perform a bounded, de-identified structural audit and help me understand the amplification. It produced the following:

A separate bounded audit of another active legacy lineage found the same structural shape reported here: 112 large child rollouts with independently copied ancestry. Only de-identified aggregate structure is reported here; no prompt, tool output, image payload, private path, project name, or thread ID is included.

Per-file compression can reduce physical bytes, but it cannot remove those 112 serialized ancestries, reduce replay work, or provide parent/child sharing. This is another confirmation that codec compression and structural fork amplification need separate regression coverage.

For performance/storage sheriffing, a small deterministic fixture should model a legacy parent with multiple compactions and repeated synthetic image payloads, then create at least 100 children. It should assert:

  1. persisted child-specific bytes remain bounded relative to the shared ancestor; and
  2. resume/fork CPU and peak allocation do not scale with children × complete parent history.

The fixture does not need private production data or large payloads; compact synthetic messages, image payloads, and compaction records are sufficient to reproduce the multiplicative shape.

luvs01 · 14 days ago

Adding another de-identified Windows Codex Desktop data point from a bounded, metadata-only snapshot.

Storage impact

  • .codex/sessions: 26,236,757,713 logical bytes (about 26.24 GB / 24.43 GiB) across 1,643 JSONL files at the snapshot time.
  • 54 rollout files were at least 250 MiB: 51 had a terminal task_complete, 2 were active, and 1 ended aborted/unknown.
  • The large child rollouts were hundreds of MiB each and contained inherited parent history plus historical compacted / replacement_history material, rather than only fork-local deltas.
  • Sampled child metadata contained native Codex forked_from_id and subagent.thread_spawn provenance. A separate source audit found that OpenCodex does not create or copy .codex/sessions rollout JSONLs, so this amplification is in the Codex fork/rollout persistence path rather than the model-routing proxy.

Compaction trigger vs. persistence

One recent compaction looked premature if only the visible 282,633-token counter was considered. Under the runtime's default server_reasoning_included=false estimator path, the active context reconstructed as:

282,633 visible total + 47,158 estimated prior encrypted-reasoning tokens + 6,852 estimated local tool-output tail = 336,643 tokens.

That exceeds the configured automatic-compaction threshold of 334,800 by 1,843 tokens. This is consistent with a normal compaction trigger; the defect reported here is the subsequent persistence and fork amplification of inherited/compacted history.

Requested safeguards

  1. Persist child rollouts as a parent checkpoint/reference plus child-local deltas instead of serializing the complete inherited history.
  2. Deduplicate or store replacement_history out of line so historical compaction snapshots are not repeatedly embedded.
  3. Add terminal-child retention/GC controls, storage quotas or warnings, and a recoverable cold-archive path.
  4. Add a bounded, privacy-safe storage diagnostic that reports lineage and amplification without exposing prompts or tool payloads.

This also overlaps the broader storage-safety concerns in #34337 and the parent-reference persistence problem in #35647. No raw rollout, prompt, tool output, username, local path, project name, or thread ID is attached.

shleder · 14 days ago

The 110 GiB multi-agent store and 2.64 GiB child rollouts with 232 compacted snapshots are a strong real case for bounded inspection, especially since the same inline-image history is physically inherited by each fork.

Codex Rescue cannot deduplicate those files, fix compaction, or recover deleted sessions. I’m testing whether it safely diagnoses one preserved affected/control rollout without reading image contents or modifying the source. It never replays calls and verification remains fail-closed.

If the 48.4 MiB control or any untouched affected sample remains, could you run:

pipx install codex-rescue==0.1.0a3
codex-rescue sessions
codex-rescue doctor --latest

Sanitized output only—no raw JSONL, screenshots/base64, databases, prompts, project data, paths, or secrets.

https://github.com/shleder/codex-rescue

fabiobatistella · 10 days ago

Additional reproduction confirming this still occurs on a newer Codex Desktop runtime.

Environment:

  • macOS 26.6.1, arm64
  • Codex Desktop 26.810.52044 (build 6662)
  • bundled codex-cli 0.148.0-alpha.9
  • Observed 2026-08-17

Privacy-safe measurements from one long-running parent tree:

  • Parent rollout: 3,071,671,636 bytes
  • 240 descendant subagent rollouts: 192,826,714,890 bytes total
  • 237 completed/closed descendants eligible for cleanup: 185,461,341,969 bytes
  • 92 rollout files were at least 1 GiB
  • Representative child rollouts were about 2.45 GB each
  • In a representative 2.45 GB child, approximately 76.5% of bytes were repeated compacted message/input_text snapshots and 13.1% were persisted tool outputs
  • The files were physically allocated (allocated size closely matched logical size) and were not APFS clones or sparse placeholders
  • No Time Machine/Data-volume snapshots accounted for the usage

The growth accelerated over five calendar days and pushed the APFS Data volume to about 90% used. The parent thread and active descendants were paused, future subagent creation was disabled temporarily, and only validated closed descendants were removed. That reduced the Codex data directory from about 244 GiB to 74 GiB and increased free Data-volume space to 317 GiB.

This strongly matches the full-history fork / retained compaction amplification described here. Raw JSONL files were not attached because they contain private conversation and tool content. This report contains no task names, session IDs, usernames, or personal filesystem paths.