Feature proposal: cache-aware generational compaction with a dedicated low-cost summarizer agent

What feature would you like to see?

I would like Codex to move from a single-model, threshold-triggered history compaction strategy toward a cache-aware generational context manager, with compaction/checkpoint generation delegated to a separate lower-cost model or sub-agent.

This is motivated by long-running coding sessions where repeated automatic compaction gradually loses important engineering state, while frequent mid-history rewrites can also destroy prompt-cache prefix reuse.

Related symptoms already reported include:

• #31659 — auto-compaction during /goal can lose the active goal and revive stale prompts
• #29426 — sessions can compact unexpectedly even when context pressure is low
• #8648 — higher chance of replying to stale context after compaction

The proposal below is intended as an architectural direction rather than another report of those individual symptoms.

---

Current problem

Codex currently performs checkpoint compaction by asking the active model to create a handoff summary and replacing older history with the compacted representation.

This has two important long-session failure modes.

1. Repeated summaries are lossy

A coding session contains details that look small linguistically but are operationally critical:

• exact file paths
• symbols and API signatures
• commit SHAs
• commands
• error strings/status codes
• test names and results
• why a change was made
• rejected approaches and why they were rejected
• user constraints that must remain verbatim

If one checkpoint omits one of these details, a later checkpoint cannot reliably reconstruct it. Repeated summary -> summary -> summary therefore accumulates semantic loss and can lead to re-investigation, repeated mistakes, stale decisions resurfacing, or loss of the current goal.

2. Mid-history mutation conflicts with prompt caching

Prompt caching depends on a stable matching prefix.

If a context manager repeatedly removes or rewrites messages in the middle of a large already-cached prompt, everything after the mutation point may need to be prefetched again even if that later content is byte-identical.

So optimizing only for active_context_tokens can be counterproductive.

For example:

• 150K input tokens, 140K cache-hit tokens
• versus a freshly rewritten 70K-token compacted prompt with little/no prefix reuse

The second request is smaller, but may still be more expensive/slower than keeping the larger stable prefix for another few turns.

---

Proposed architecture: cache-aware generational context

The key invariant should be:

Within an epoch, model-visible history should remain append-only whenever possible. Compaction should happen in batches at epoch boundaries rather than continuously rewriting old history.

Suggested model-visible layout:

IMMUTABLE BASE PREFIX
  system instructions
  stable tool schemas
  stable project/developer instructions

FROZEN EPOCH SNAPSHOT
  current goal
  user constraints
  important decisions
  files/symbols
  known bugs
  implementation state
  validation state
  memory index

APPEND-ONLY ACTIVE LOG
  user messages
  assistant messages
  tool calls/results
  state deltas
  retrieved historical context

During the epoch, state changes should be appended as deltas instead of rewriting the snapshot:

STATE SNAPSHOT v4
...

STATE DELTA #1
- GraphId geometry dependency confirmed as problematic

STATE DELTA #2
- modified src/.../PathGraph.cs

STATE DELTA #3
- regression tests 28/28 pass

At the next rollover, the snapshot + deltas are merged once into STATE SNAPSHOT v5, old disposable context is removed in one batch, and the next epoch starts append-only again.

This deliberately trades occasional large cache invalidations for many turns of high prefix-cache stability.

---

Use a separate low-cost compaction model / sub-agent

The active coding model should not also have to act as the memory garbage collector.

When an epoch approaches rollover, Codex could spawn a dedicated compactor/historian sub-agent using a lower-cost/lower-reasoning model (for example the lowest-cost available GPT-5.6 Codex-capable variant).

The main agent continues working while the compactor prepares the next checkpoint.

The compactor's job would be narrow and deterministic:

1. Read the current frozen snapshot, deltas, and relevant transcript range.
2. Produce a structured next-generation checkpoint.
3. Preserve exact technical identifiers verbatim.
4. Separate confirmed facts, assumptions, unresolved questions, completed work, and proposed work.
5. Preserve user requirements and rejected approaches.
6. Optionally validate the checkpoint against the source range before committing it.

This has several advantages:

• avoids spending the strongest coding model on summarization work
• reduces interruption of the main agent loop
• allows a compaction-specific prompt/model configuration
• makes it possible to generate the next checkpoint before the context limit is critical
• makes compaction behavior easier to benchmark independently from coding quality

A possible two-model flow:

Main coding model
      |
      | append-only work
      v
~60-70% context
      |
      +---- spawn low-cost compactor/historian
      |           |
      |           +--> build candidate next snapshot
      |
      v
main model continues working
      |
~75-85% / phase boundary
      |
      +--> finalize checkpoint
      +--> batch-drop disposable history
      +--> begin next epoch

---

Keep compressed history outside the active prompt

Long-term history should not need to remain fully materialized in every request.

Codex already has durable rollout/session data. A future context manager could maintain an external context store containing:

raw transcript
T1 detailed summaries
T2 distilled summaries
T3 long-term summaries
exact errors
commands
file/symbol references
diffs
important decisions

The important distinction is:

The archive may be freely reorganized; the active model prompt should remain prefix-stable.

Old material can then be retrieved when required.

Retrieval should be tail-appended, not restored in-place

If the current task needs an old decision, the context manager should append something like:

[Historical context retrieval]
Source: block b37
Reason: GraphId compatibility decision
...

at the current tail.

It should avoid reinserting the original block at its historical position, because that again mutates the cached prefix.

---

Pending-drop queue instead of immediate pruning

Large tool outputs are often clearly disposable after use, but deleting them immediately can invalidate a valuable cached prefix.

Codex could mark them first:

read#17        droppable
build#23       droppable
grep#28        droppable
diff#31        keep

and only physically remove them when:

• an epoch rollover is already happening
• context pressure is high enough to justify losing cache reuse
• the task phase has ended
• the expected savings exceed the expected cache invalidation cost

This would make tool-output pruning cache-aware rather than purely token-count-driven.

---

Compaction scheduler should consider cache economics

Instead of triggering only from context usage, the scheduler could consider:

active_context_tokens
cached_input_tokens
cache_hit_ratio
context_growth_rate
pending_droppable_tokens
active_tail_tokens
task_phase
retrieval_pressure

Conceptually, the objective is closer to minimizing:

uncached input cost
+ cached input cost
+ latency
+ compaction/model cost
+ information-loss risk

rather than simply minimizing active prompt length.

This could produce decisions such as:

Context is 73% full and 50K tokens are droppable, but 180K tokens are currently cache-hit and the agent is still in the same debugging loop, so postpone rollover.

Then after tests finish / the phase closes:

Build the next checkpoint and perform one 180K -> 40K rollover.

---

Structured checkpoint format

The compaction model should produce a state-transfer record rather than prose summary. For example:

CURRENT OBJECTIVE
USER REQUIREMENTS
COMPLETED WORK
DECISIONS + REASONS
REJECTED APPROACHES + REASONS
REPOSITORY / BRANCH / COMMIT STATE
FILES AND SYMBOLS
CHANGES MADE
CONFIRMED BUGS / SUSPECTED BUGS / DEFERRED ISSUES
EXACT TECHNICAL DATA
TEST / VALIDATION STATE
OPEN QUESTIONS
NEXT ACTIONS
MEMORY / ARCHIVE REFERENCES

Critical values such as paths, identifiers, hashes, commands, errors, configuration keys, API signatures, and numerical thresholds should be preserved verbatim.

---

Optional reversible hierarchy

A further improvement would be a reversible hierarchy similar to generational storage:

raw history -> T1 blocks -> T2 blocks -> T3 blocks

but these blocks should primarily live outside the active prompt.

The model/context manager could expose internal operations roughly equivalent to:

• search historical context
• retrieve exact source block
• inspect context status
• materialize relevant history at the tail

The goal is not a literal billion-token prompt. It is to keep the active prompt small enough to reason well while retaining recoverable access to arbitrarily large cumulative session history.

---

Suggested rollout strategy

This does not need to replace the current compaction implementation in one step.

A possible incremental path:

1. Add a configurable compaction_model / compaction_reasoning_effort.
2. Allow compaction to run as a dedicated sub-agent.
3. Make checkpoint output structured and validation-oriented.
4. Track cached-input tokens and compaction cache cost in telemetry.
5. Add a pending-drop queue for consumed tool output.
6. Introduce frozen snapshot + appended state deltas.
7. Move to batched/epoch rollover scheduling.
8. Add searchable/retrievable compressed-history blocks later.

Even steps 1-4 would already make long-session compaction cheaper to experiment with and easier to evaluate.

---

Metrics that would be useful

For long-running agent benchmarks, it would be useful to measure more than compression ratio:

active prompt tokens
cached input tokens
cache-hit ratio
uncached input tokens
pending droppable tokens
checkpoint size
number of rollovers
facts/constraints lost across rollover
post-compaction re-investigation rate
latency per turn
usage cost per completed coding task

A context manager that produces a 50% smaller prompt but destroys a 90%-cached prefix every few turns may not actually be an improvement.

---

Why this seems particularly useful for Codex

Long-running coding agents have unusually structured memory requirements. Exact technical facts matter more than narrative conversation fidelity, and tool-heavy sessions produce large amounts of consumed but temporarily cache-valuable history.

Codex already has several building blocks that make this direction plausible: durable session history, compaction lifecycle hooks, multi-agent support, token/cache telemetry, and separate model configuration infrastructure.

A cache-aware generational design plus a dedicated low-cost compactor could improve both sides of the current tradeoff:

• less semantic degradation after repeated compaction
• better prompt-cache reuse between compactions

while allowing the strongest model to spend more of its budget on the actual coding task.