Compaction silently discards all tool outputs and assistant reasoning

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #14347
• #13932
• #13279
• #14120
• #13142

Powered by Codex Action

Follow-up after reading the existing memories/ implementation

After digging into codex-rs/core/src/memories/, I want to be more precise about the gap — because the existing system is actually quite good and covers a lot.

What the existing memory system already handles

The Phase 1/2 pipeline is well-designed for cross-session learning:

• Phase 1 (at startup): scans recent rollouts, extracts durable lessons per thread using a structured prompt, stores in state DB
• Phase 2: consolidates into memory_summary.md, injected into subsequent session system prompts
• MemoryTool (when enabled): model can actively read MEMORY.md + rollout summaries during a turn, and is instructed to write updates back when it detects stale guidance

This handles the "learn across sessions" problem well.

The specific gap: compaction timing

The remaining problem is narrower: the model cannot proactively write to memory before compaction replaces its history.

The flow is:

context fills → compact.rs triggers automatically → history replaced → MemoryTool could help now, but facts are already gone

Phase 1 explicitly excludes exactly the facts that are lost:

"Temporary facts (live metrics, ephemeral outputs) that should be re-queried" "Generic status updates ('ran eval', 'looked at logs') without takeaways"

But these "temporary facts" are precisely what a multi-hour coding session needs to remain coherent within that session:

• which test failed at step 3 and why
• what the file actually contained at line 47 when the decision was made
• the exact patch that was applied and confirmed working
• which API path was already ruled out

These aren't durable cross-session lessons — they're session-local technical state that the MemoryTool is also not designed for (it targets long-term reusable knowledge).

Proposed: pre-compaction session snapshot

One targeted fix: before replace_compacted_history() discards history, run a lightweight pass that writes session-local technical state to a structured, temporary location (e.g., ~/.codex/sessions/<id>/checkpoint.md).

Unlike Phase 1 (which extracts lessons), this checkpoint would:

• preserve verbatim the last N tool results (file contents, test output, diffs)
• not summarize — exact content, not impressions
• auto-expire at session end (not fed into Phase 1)
• be injected back after compaction via the same developer instructions mechanism already used for memory_summary.md

The model would see its own technical context restored after compaction instead of discovering mid-task that it no longer knows what files it changed.

This complements rather than replaces the existing system: Phase 1/2 handles cross-session learning, the session snapshot handles intra-session continuity.

I can prototype this as an extension to the existing run_compact_task_inner() flow if there's appetite for it.

Prototype + evaluation results for the pre-compaction snapshot approach

Built a prototype of extract_session_snapshot() and ran L1/L2 metrics against the same coding session from the original report. Sharing the numbers before writing any Rust.

---

How the snapshot works

Before replace_compacted_history() discards history, scan tool results and classify them:

| Signal | Fact type | Importance |
|---|---|---|
| Diff hunks (---/+++/@@) | PATCH | 1.0 |
| Test output (FAILED/ok/panicked) | TEST_RESULT | 0.95 |
| Source file content (pub fn, impl, etc.) | FILE_CONTENT | 0.80 |
| Assistant messages with decisions/findings | DECISION | 0.70 |
| Other command output | COMMAND_OUTPUT | 0.60 |

Items ranked by importance and packed into a fixed token budget (e.g. 2,000 tokens), then appended to the post-compaction context via the same developer instructions mechanism already used for memory_summary.md.

---

L1 — Weighted Fact Recall

8 ground-truth facts extracted from the session (file paths, line numbers, test results, patch content, decisions):

| Fact | Baseline | + Snapshot |
|---|---|---|
| fix location: src/auth.rs line 47 | ✅ | ✅ |
| root cause: missing cache::invalidate() | ❌ | ✅ |
| test_concurrent_refresh: failed → passed | ❌ | ✅ |
| regression test content (user_42) | ❌ | ✅ |
| mobile.rs:89 also calls refresh_token | ❌ | ✅ |
| exact patch: +cache::invalidate(&decoded.sub) | ❌ | ✅ |
| test count: 14 total, 13+1 originally | ❌ | ✅ |
| web.rs:67 also in scope | ❌ | ✅ |

Weighted recall: 15.2% → 100.0% (+84.8%)

---

L2 — Context Efficiency

• Snapshot uses a fixed 2,000-token budget regardless of session length
• Compression ratio vs original tool results: ~1.5x (snapshot is slightly larger than raw because of structure/labels — tradeoff for readability)
• Context overhead scales inversely with session size: at a realistic 10K-token post-compaction context the overhead is ~20%; at 20K it drops to ~10%

---

L3 — Task continuation quality (planned)

The static metrics above don't capture whether a model actually uses the checkpoint correctly. The next step is a live evaluation:

• 4 questions asked to the model after compaction (with vs without snapshot)
• Scored on a 0–3 rubric (0=hallucinated, 1=vague, 2=partial, 3=exact)
• Hypothesis: baseline avg ~0.8–1.2, snapshot avg ~2.4–2.8

Will run this once there's signal that the direction is worth pursuing — happy to share results here.

---

No code changes proposed yet — wanted to validate the idea empirically before touching compact.rs. Let me know if this is a direction that fits with the roadmap.

Reviewed all five — not duplicates, but #14347 is the closest related issue and worth linking.

#14347 ("Extend compaction prompt") addresses the same symptom (information loss across compactions) but from a different angle: improve the LLM summarization prompt so it captures more. That is a valid fix for reasoning and decisions.

The gap this issue focuses on is complementary and lower in the stack: even a perfect compaction prompt cannot recover tool outputs, because collect_user_messages() structurally discards them before the LLM ever sees them for summarization:

pub(crate) fn collect_user_messages(items: &[ResponseItem]) -> Vec<String> {
    items.iter().filter_map(|item| match parse_turn_item(item) {
        Some(TurnItem::UserMessage(user)) => Some(user.message()),
        _ => None,  // tool_results, assistant_messages: dropped here, not summarized
    }).collect()
}

The compaction model is only given user text as input. File contents, test output, applied patches — they are not passed to the summarizer at all. A better prompt cannot help with data it does not receive.

So the two issues address different root causes:

• #14347 → improve what the LLM extracts from what it does receive (user messages + previous summary)
• This issue → preserve what the LLM never receives (tool results, assistant reasoning)

The pre-compaction snapshot proposal would complement #14347's prompt improvements rather than replace them.

#13279 and #14120 are about the compaction retry loop, which #14563 just fixed — different problem.
#13932 is about plan preservation specifically — a subset of the broader tool output loss described here.
#13142 is a feature request for user-configurable compaction routines — related direction, different mechanism.

Codex is working as intended here, so we don't consider this a bug.