Add an atomic handoff primitive for token-budget context rollover

Add an atomic handoff primitive for token-budget context rollover

Related to #33255.

The new fallback phase gives the model a final opportunity to preserve state before context rollover. However, the model still has to choose an external persistence mechanism, write the handoff there, call new_context, and recover that state in the next window.

Would you consider making this transition atomic by allowing the model to provide a structured handoff as part of the context-window transition itself?

I implemented a downstream prototype here:

https://github.com/0xdeafbeef/codex/commit/542dea6afedd3b64ecace39731768166b28a0095

The prototype introduces:

checkpoint_stage(
    completed_stage,
    handoff,
    next_stage,
)

The host starts a fresh context window after the tool call and injects the model-authored handoff into the new window alongside the normal world state.

The tool is guarded:

• available only after substantial reclaimable context has accumulated;
• unavailable while descendant agents are active;
• usable only once per context window;
• handoff size is bounded;
• the completed and next stages must be distinct.

The handoff is expected to preserve the active objective, decisions and invariants, changed state, verification status, open risks, active skills, and the next completion criterion.

The main difference from the fallback prompt is reliability: preserving and restoring the handoff becomes one host-controlled operation rather than a model-coordinated sequence involving an external storage mechanism.

The exact tool shape is not important to me. For example, this could instead be expressed as:

new_context(handoff: String)

I mainly wanted to share the working prototype as a concrete design reference.

@bolinfest, does this fit the intended direction of the token-budget/context-window work, or is external persistent storage deliberately required here?

I've played with this using sol ultra as the coordinator, and it has already been running for 50 hours. Compaction triggers once every few hours, and the model keeps a compact context and an action ledger in Markdown files.

It looks much more token-efficient than triggering compaction at the end of the context and hoping it saves the stuff that actually matters.