Add an atomic handoff primitive for token-budget context rollover

This is very closely aligned with a pain point I have been hitting in real long-running Codex workflows.

I built a user-land Codex skill for this pattern here:

https://github.com/pplong-15/thread-reincarnation-skill

It does not replace the host-level atomic primitive proposed in this issue, but it addresses the same continuity problem from the workflow side: when a Codex thread becomes overloaded, repeatedly blocked, or starts forgetting P0 rules, the skill creates a compact successor handoff instead of continuing the old context indefinitely.

The handoff preserves:

• verified critical facts
• P0 safety rules
• external API/key handling methods
• known failure fixes
• output paths and project gates
• an on-demand old-thread index, without loading the full old history

One important piece is that the successor thread must pass a memory check before continuing. In practice, this helps keep the new thread fast while reducing dangerous forgetting.

So I think this issue is pointing at the right host-level primitive. The skill above is a downstream/user-land version of the same idea, and may be useful as a reference for what structured handoff fields and successor validation should look like in real Codex use.

I think the core primitive proposed here is valuable, but the first-principles reason is narrower than “a fresh thread has more room.”

Repeated compaction is recursive lossy compression. If (S_0) is the full operational state and each compaction is a lossy transform (C), then after several passes the model is effectively continuing from (C^n(S_0)). The most damaging loss is often not the broad objective; it is the execution frontier: completed effects, pending work, the latest correction, rejected paths, evidence, and the exact terminal condition. This is consistent with the reports in #29356 and #34095.

A fresh context helps only if it breaks that recursive-compression chain using a checkpoint with higher fidelity than the latest compacted summary. It cannot recover information that was already discarded. This is why “create a visible replacement thread after N compactions” by itself is not sufficient, and it also explains the metadata/UI concerns raised when #22641 was closed.

I would refine the requested core capability as an atomic verified continuation epoch, preferably within the same user-visible thread by default:

1. Checkpoint before loss, at a safe boundary

• Capture from the pre-compaction runtime state, not only from an already-compacted summary.
• Commit the transition only after the active turn and tool effects have reached a terminal boundary.
• If compaction happens mid-turn, mark rollover pending and finalize it at turn completion.

1. Structured continuation envelope

• active objective and acceptance/terminal condition;
• latest user corrections and non-negotiable constraints;
• completed versus pending effects;
• artifact paths plus hashes/receipts for important outputs;
• verification already performed and its result;
• rejected approaches and why they must not be retried;
• blockers, risks, and exact next action;
• active goal/subagent relationships and other runtime-owned state.

1. Restore verification before mutation

• The next context epoch must read back the envelope and validate its schema/version and artifact digests.
• Missing or contradictory state should fail closed or request user input.
• Merely injecting the checkpoint is not enough; a replay test should verify that it changes the next decision. For example, the restored agent should decline a previously rejected approach even when that approach again appears locally attractive.

1. Atomicity and idempotency

• Give the transition a server-owned continuation ID/idempotency key.
• A retry after a lost response must return the already-created epoch, not create a second successor.
• Persist the checkpoint, continuation relationship, and restored epoch as one recoverable state transition.

1. Carry runtime metadata explicitly

• model and reasoning effort;
• cwd/project/worktree or environment identity;
• approval/sandbox or permission profile;
• collaboration mode/personality;
• goal and subagent linkage;
• active instruction sources.

Silent fallback to current defaults should be reported as a mismatch, not described as inheritance.

1. Use health signals, not only a count

• A configurable compaction count can be a fallback trigger, but it is a weak proxy for fidelity.
• Better signals include repeated compaction of an already-compacted frontier, inability to verify the latest checkpoint, contradictory completion state, or repeated execution with no new material effect.
• Expose a stable compaction/continuation event ID, trigger, ordinal, thread ID, turn ID, and context-epoch ID to hooks and app-server clients.

This complements rather than replaces thread context pins (#26889): pins preserve selected durable facts, while an atomic continuation envelope preserves the current execution frontier and proves restoration.

A minimal API could remain close to the proposal here:

checkpoint_and_continue(
  idempotency_key,
  boundary_turn_id,
  structured_handoff,
  settings_revision,
  artifact_receipts
) -> {
  continuation_epoch_id,
  created | replayed,
  restore_status
}

It does not need to create another visible thread. Internally starting a clean context epoch while preserving the same thread identity would obtain the main benefit—stopping recursive summary degradation—without requiring users to manage duplicate conversations.

Suggested acceptance tests:

• repeated-compaction replay retains a late user correction, a rejected path, completed effects, and the exact next action;
• the first post-restore decision consumes those fields rather than merely containing them;
• a lost transition response plus retry produces one continuation epoch;
• active tool/subagent work prevents premature checkpoint finalization;
• model/effort/project/environment/approval/sandbox metadata either matches exactly or restoration blocks visibly;
• corrupt or stale artifact receipts prevent material mutation.

User-land handoff files, hooks, persisted goals, and context pins can reduce the risk today, but they cannot make persistence + restoration + context transition atomic. That is the part that appears to justify a core runtime primitive.