Follow-up to rollout compression: default zstd misses distant history copies (1.64x vs 21.98x)

Maintainer summary

Hi Codex team — first, thank you for building the cold-rollout foundation in #25087 and #25089. The transparent .jsonl.zst reader, materialize-before-append behavior, and background-worker shape are exactly the right direction.

I found one focused remaining gap while testing that implementation against real, large rollouts: the worker's current zstd level-3 encoding uses the default match window, but compaction snapshots and inherited subagent histories often repeat already-compressed payloads tens of MiB apart. The default window cannot see those copies.

On one real 760,247,640-byte rollout:

| Encoding | Compressed bytes | Ratio |
|---|---:|---:|
| Current level-3/default-window path | 463,794,252 | 1.64x |
| Proposed Rust worker path, bounded at 32 MiB | 34,585,710 | 21.982x |

Both variants were verified by full decompression and byte-exact SHA-256. A separate zstd -3 --long=25 CLI baseline reached 14.37x; I am keeping that out of the decision table because the exact Rust worker path is the relevant comparison.

The smallest safe follow-up is a schema-neutral, two-file codec change: for rollouts at least 8 MiB, set the pledged source size, enable long-distance matching, choose ceil(log2(source_size)) clamped to an 8–32 MiB window, and include a frame checksum. A deterministic test repeats an incompressible 4 MiB block three times, asserts meaningful reuse, and verifies exact decode.

This does not enable the under-development worker, change its timing policy, or alter readers, discovery, resume, materialization, retention, or model-context behavior. Writer/compressor coordination and production activation remain separate review decisions.

One important product boundary: the >10x result applies when the worker actually processes an eligible rollout. The codec-only patch does not by itself give every user an automatic 10x reduction, because local_thread_store_compression is currently under-development/default-off and the merged worker still uses a seven-day cold threshold. To verify that the codec can become a real product improvement, I separately implemented and exercised a dependency-ordered safety/activation stack: eight atomic commits across 17 files (+1,691/-200) covering append-handle coordination, recurring policy, worker ownership, crash recovery, materialization validation, archive/unarchive coordination, default activation, and this codec. I am not proposing that whole stack as one PR; it is supporting evidence that the follow-up path is understood and can stay reviewable.

The focused patch is 55 inserted lines across codex-rs/rollout/src/compression.rs and compression_tests.rs, rebased onto current main; codex-rollout passes 100/100. If this direction aligns with the team's intended rollout-compression work, would you be open to inviting that codec-only PR? I would be happy to own CI, review changes, and release validation.

Thanks for taking a look. I know the original work already solved the difficult representation and compatibility foundation; this proposal is meant to make that investment materially more effective on the workloads that need it most.

<details>
<summary><strong>Extended validation / reviewer reference</strong></summary>

Why this is a follow-up rather than a duplicate of the original report

• #24948 established the general unbounded-growth symptom.
• #25087 added transparent compressed reads and materialize-before-append behavior.
• #25089 added the cold compression worker, initially disabled and limited to archived sessions, while explicitly recording the remaining writer/compressor coordination question.
• This issue isolates a narrower property of the merged codec: its default match window misses the dominant distant duplication in affected real files. The codec improvement can be reviewed independently of activation and coordination.

The user impact is still current. My own store reached 75,412,910,173 bytes (70.23 GiB) after roughly one to two months of heavy use. Other recent reports include about 755 GiB dominated by subagent children in #34061 and a user-reported 1.5 TB store in #24948. These are separate user reports, not extrapolations from my sample.

What is repeating

In the 760,247,640-byte sample:

• compacted records accounted for 652.06 MiB / 89.936%.
• payload.replacement_history[].content[].image_url accounted for 632.39 MiB.
• repeated strings at least 256 bytes long accounted for 651.49 MiB.
• one 977,978-byte image value appeared 138 times.
• exact-line deduplication would save only about 1.10 MiB because timestamps and enclosing metadata differ.

There is also a separate structural parent/child problem. In this child, 99.874% of its large-string bytes already existed in the parent. A current independent Linux reproduction in #34061 found 117,949,758 of 118,285,692 child bytes (99.72%) before the child's first real turn. Compression is therefore a low-risk first layer, not a claim that structural history sharing is unnecessary.

Why the window is capped at 32 MiB

The best raw ratio was not selected. A 128 MiB long-distance window saved only another 4.12 MiB on the large sample while raising measured compression memory to about 141.9 MiB. The 32 MiB cap retains most of the observed storage benefit and keeps two possible background jobs within a much smaller memory envelope. On a second 576,966,356-byte real fixture, the proposed worker produced 30,433,952 bytes (18.958x) with a measured peak RSS of about 39.7 MiB.

Whole-store and product-path validation

The codec was also exercised inside a separate production-activation prototype so the result was not inferred from one file:

• APFS shadow of the complete store; the live sessions tree was never modified.
• 79.613474 GiB of logical JSONL became 7.849757 GiB physically (10.142x).
• 660/660 logical rollouts remained byte-exact.
• the initial backlog completed in 24.399 seconds; this is whole-store background processing, not per-turn latency.
• during a separate 22.534-second worker pass, 273 foreground thread/list(limit=1) calls had p50/p95/p99 of 23.871/26.386/27.969 ms, versus 24.258/26.007/26.381 ms after completion. This sample showed no measurable foreground regression.
• a steady-state scan with no eligible files took 0.5 seconds, made no rollout writes, and peaked at 3.172 MiB RSS.

Validated follow-up map beyond the 55-line codec

The larger local implementation is split along failure and rollback boundaries rather than presented as one oversized contribution:

1. hold a shared representation lock for the complete append-handle lifetime and an exclusive lock while replacing a rollout representation;
2. replace the one-shot seven-day policy with startup plus 30-minute recurrence, an 8 MiB size floor, and a 30-minute cold age;
3. keep one locked worker-marker handle through the scan and persist terminal state in place, closing claim and abnormal-exit ABA windows;
4. make representation installation and retirement directory-durable on Unix/macOS, and fail closed on divergent plain/zstd siblings;
5. validate the exact materialized destination before retiring zstd, including the install-to-retirement race;
6. coordinate archive/unarchive moves with the same representation lock and re-resolve after inode replacement;
7. enable the feature in a separately revertible commit with legacy and paginated public-lifecycle regressions;
8. include the bounded long-distance codec as an independent unit.

The combined branch changes 17 files with 1,691 insertions and 200 deletions. Before required fix/format it passed codex-rollout 117/117, codex-thread-store 117/117, codex-features 29/29, and focused legacy/paginated app-server lifecycle tests. The full shadow-store and foreground-latency results above came from this product path, not from a standalone compression CLI.

The remaining structural problem is also being treated separately. Parent-history references and immutable content/history chunks can avoid writing the duplicated bytes in the first place, but they require ownership, delete/GC, repair, mixed-version, and downgrade semantics. Local shadow-reader/equivalence work therefore continues to retain the inline history as a safety oracle and intentionally claims zero production savings so far. I would first align with the team on the intended history_base lifecycle rather than bundle a new wire format into the codec or activation work.

Compatibility and recovery boundaries

• The output remains a standard zstd frame; there is no JSONL schema or path-semantics change.
• Existing transparent read/search/list/materialize behavior is unchanged.
• Digest-verified macOS arm64 stable binaries 0.137.0, 0.141.0, 0.144.2, and 0.144.6 read the tested compressed representation and materialized it before resume. Versions 0.135.0 and 0.136.0 did not, so any future default activation needs an explicit downgrade-support decision.
• A 3.544552 GiB cold rollout was materialized through the actual RolloutRecorder::resume append path in 2.598 seconds at 37.438 MiB peak RSS, with byte-exact recovery.
• The codec-only patch does not claim to solve the known writer/compressor race. I have deterministic race tests and a separate representation-lock design, but would keep that out of this PR unless the team asks for it.
• Structural deduplication/content addressing has larger theoretical upside, but it adds reference ownership, deletion/GC, repair, mixed-version, and privacy boundaries. It should be discussed separately rather than mixed into this low-risk codec improvement.

Focused validation

• just test -p codex-rollout
• just fix -p codex-rollout
• just fmt
• git diff --check
• deterministic long-distance regression with byte-exact decode
• two real-file frame/decode/SHA-256 checks
• compressed thread/read and resume/materialization checks
• complete shadow-store logical-tree comparison

Detailed prior measurements and source trace: initial analysis, race/activation follow-up, and current review-boundary summary. The raw rollout content is intentionally not attached because it contains private prompts, tool outputs, local paths, and images; the measurements expose only sizes, counts, field paths, timings, and hashes.

</details>