memory_stage1 fails on 31% of sessions: "Codex ran out of room in the model's context window"

Your oversized-rollout population is useful as a codex-rescue scale/control test even though Rescue does not repair Codex’s memory-stage jobs. I’m field-testing whether session discovery and doctor stay bounded and conservative on the same long histories that overflow memory_stage1.

If one of the large failed-memory sessions is still local, could you run:

pipx install codex-rescue
codex-rescue sessions
codex-rescue doctor --latest

The useful result is whether the underlying session itself diagnoses cleanly and how long the scan takes. Please do not salvage a healthy session just because its separate memory-consolidation job failed.

Sanitized output, versions, timing and exit codes only—no raw rollouts, memory/SQLite DBs, prompts, credentials, or private paths. Repo: https://github.com/shleder/codex-rescue

I checked the Stage 1 path at commit 312b62ac9. It no longer sends a completely unbounded rollout. It keeps the head and tail of the rendered conversation. That reduces context window failures, but it can silently discard the middle, which may contain a correction, the root cause, or the test that verified the result.

I built a local reference implementation to test a more reliable approach:

• split oversized rollouts into bounded groups of complete interaction turns;
• repeat the previous turn as context only, so references still make sense without producing duplicate memories;
• checkpoint every completed chunk, so restarting does not resend successful work;
• split only the offending chunk after a context overflow instead of retrying the same payload;
• expose partial and failed coverage, and admit only complete rollouts to Phase 2;
• cap each rollout at 64 model requests and 64 planned leaf chunks.

All 418 targeted tests pass:

• codex-memories-write: 59 of 59
• codex-state: 187 of 187
• codex-api: 172 of 172

The tests cover complete turn boundaries, corrections in the middle, oversized messages and tool output, context overflow splitting, restart recovery, retrying only the failed chunk, privacy handling, request limits, and complete only Phase 2 admission.

These are synthetic tests using mock model streams and temporary Codex homes. I did not run them against real conversations or a real memory database. The prototype also uses Codex's current byte based token estimate rather than the selected model's exact tokenizer, so it does not prove production cost, latency, or installed app behavior.

I kept the implementation local because the Codex contribution policy does not accept external pull requests. It is split into six reviewable commits, and I can share sanitized patches if a maintainer wants to inspect the reference code.

This is deliberately scoped as a reliability fix for lost coverage and silent failure. Semantic deduplication, contradiction handling, confidence, and broader memory quality should remain separate work.

Update: I packaged the design, architecture, test matrix, and proof limits here: https://github.com/artemgetmann/codex-memory-extraction-field-note

I maintain a memory server called Vestige, mentioning that upfront since part of this comment touches on it.

Your diagnosis is right and the evidence backs it up cleanly. memory_stage1 is architected as one all at once batch pass over the full rollout, so any session whose transcript is larger than the model's window is unrecoverable by design rather than by accident, and retrying an oversized input against the same all or nothing path can never succeed, which is exactly what your retry_remaining data shows. The size correlation is convincing too, a roughly ten times larger median rollout on the failing side is consistent with a hard ceiling being crossed rather than a random or transient failure.

The comment from artemgetmann in this thread is a solid direction, keeping head and tail loses the middle of a transcript, which is often exactly where a correction or the actual root cause of a bug lives, so a chunked approach that preserves complete turns end to end is a better fit than any single truncation strategy.

On Vestige, this is the one issue in this batch where the connection is closer to direct rather than incidental, though it is still not a fix for your existing code, only a different architectural choice you could make instead. Vestige ingests information incrementally during a session, gated so that only genuinely new or contradicting content triggers a write, rather than doing one giant catch up summarization pass over an entire transcript after the fact. That structurally avoids ever needing to hold a full multi megabyte transcript in one request, because nothing waits until the end to get processed. It does not retroactively fix the 93 sessions you already lost, and it is a different design than patching the existing stage 1 job, but it is the same category of fix your own suggestion list points at with chunk the rollout and do not wait for one all or nothing pass.

https://github.com/samvallad33/vestige

Your 30.7% figure is alarming, but the part I'd underline most is in your own summary line: nothing surfaces these failures to the user. The 95 errored jobs are sitting right there in jobs with status='error' — the data to know that the memory system has holes already exists, it just never escapes into anything a user sees. Whichever fix lands, coverage as a first-class output is the piece worth demanding first: even something as small as a startup line — "long-term memory covers N of M sessions; K failed permanently" — converts silent holes into something a user can actually act on or at least distrust deliberately.

On failure handling: when we hit this class in our own git-backed vault memory setup, the change that mattered wasn't better retries, it was splitting defects into two severity classes instead of one error bucket. Hygiene issues are untidy but nothing believes something false because of them; breaches are a session silently absent from memory, or two entries contradicting each other — those change what a later session does. A permanently failed consolidation like yours is squarely a breach: every future session will reason from a memory that looks complete. Treating breach-class failures as surfaced hard errors rather than logged-and-continued background noise is cheap to do now and very expensive to retrofit once users have built trust in the store.

One gap in all the proposed repair paths so far (artemgetmann's chunking, head+tail, your rescue runs): they reprocess material that partially succeeded before, and overlapping passes over the same rollout are exactly how you get duplicate or subtly conflicting memories — two passes over the same correction rarely produce byte-identical entries, so equality-based dedup won't catch them. @samvallad33's incremental ingestion sidesteps much of this structurally, which is a real argument for it — but if stage-1 keeps its batch shape and gets chunked instead, dedup needs to be designed in on day one: normalized comparison across entries rather than raw equality, plus a marker of which memories were machine-generated in which pass, so a re-run can't quietly double-count a correction. Cheaper to design for up front than to clean out of a sqlite file after users have accumulated months of overlapping passes.

@samvallad33 I looked through Vestige. The incremental memory approach is genuinely interesting, especially deciding whether to create, update, or replace a memory during the session instead of waiting for one large pass at the end.

One factual correction: current Codex no longer sends the full conversation without a size limit. Stage 1 uses up to 70 percent of the available input space and keeps the beginning and end when the conversation is too large. That reduces context window failures, but it can silently remove the middle.

Your larger point still stands. I previously built an MCP memory system and saw a major weakness of model triggered memory writes firsthand. Different models followed the same instructions differently, and agents often forgot to save memories during long coding sessions.

Incremental memory keeps each request smaller, but it does not guarantee complete coverage unless the runtime decides when to capture memory and records which source events were processed.

Coding agents also produce provisional information. A claimed fix may later fail a test. The user may correct an earlier statement. Tool results may change what actually happened.

I currently see bounded Stage 1 extraction as the reliable repair and recovery path, not necessarily the final memory architecture. A hybrid system may eventually work better, but I would want evidence from real conversations before adding that complexity.

@secondbrainstarter I agree with the integrity point. Silent missing coverage is a serious failure. The user should be able to see exactly what was and was not processed.

One clarification: the proposed restart process does not intentionally process completed sections again. Each completed section is saved using its source identity and content fingerprint. Only the failed section is retried or divided. The previous turn is included only as context, and incomplete conversations cannot continue into Phase 2.

Some duplicate risks still remain. The system must save the extracted output and completion record together. It must prevent context only material from producing memories. It must also handle the same fact appearing in separate sections, full reruns, migrations, and extractor version changes.

I would add source range IDs, extractor version, run and attempt IDs, generation history, and one database transaction that saves both the output and completion record.

I would leave semantic deduplication and contradiction resolution for later, until real conversations show how common those problems are. An incorrect semantic merge could destroy useful evidence or hide a coverage failure.

The next useful test would be a small private replay using a few real historical conversations and a disposable memory database. That would measure coverage, restart behavior, duplicates, cost, latency, and basic memory usefulness without turning this reliability fix into a complete memory system redesign