Session JSONL bloat: repeated identical AGENTS-derived user_instructions per turn (turn_context)

Can you describe your use case in more detail? Are you reading the on-disk JSONL rollout files for analysis? That's not really what these were designed for. There are other mechanisms that you may find more appropriate for this — for example, using the app server event stream or Otel event stream.

Session rollout bloat: repeated TurnContext.user_instructions

We saw very large Codex session JSONL growth driven by the same turn_context.user_instructions being written on every RolloutItem::TurnContext line.

Observed on one real rollout JSONL:

• turn_context.user_instructions: 1,759 entries, 1 unique, ~16.0MB total
• Per-blob size ~9KB (dominated by repeated AGENTS / user-instructions content)

This makes rollouts noisy and (when people post/log/ingest them) can lead to misleading input-token accounting.

Proposed fix (low-risk): dedupe repeated user_instructions on write

In the rollout recorder/writer, keep a last_user_instructions (string or hash). When persisting a RolloutItem::TurnContext, only write user_instructions when it changes. Otherwise set it to None so it is omitted from JSON (skip_serializing_if = Option::is_none).

Sketch (Rust):

// in rollout_writer(...)
let mut last_user_instructions: Option<String> = None;

...
if let RolloutItem::TurnContext(turn) = &mut item {
    if turn.user_instructions.is_some() {
        if last_user_instructions.as_ref() == turn.user_instructions.as_ref() {
            turn.user_instructions = None;
        } else {
            last_user_instructions = turn.user_instructions.clone();
        }
    }
}

Why it should be safe

• TurnContextItem.user_instructions is already optional and skipped when None
• reconstruct_history_from_rollout(...) does not replay RolloutItem::TurnContext lines (only response items / compacted / events), so removing redundant fields from those lines should not affect replay

Optional: config knob

log_user_instructions = "always|on_change|never" (default on_change).

If helpful, I can provide a clean patch against current main once invited (per CONTRIBUTING).

Can you describe your use case in more detail? Are you reading the on-disk JSONL rollout files for analysis? That's not really what these were designed for. There are other mechanisms that you may find more appropriate for this — for example, using the app server event stream or Otel event stream.

Thanks — totally fair question.

Our use case (why we read the rollout JSONL):

• We run Codex CLI headless as part of a Discord-first orchestrator (multiple backtest hosts, long sessions).
• After a run, we want an offline, local postmortem artifact we can analyze without needing the live app-server stream / OTEL collector.
• We do not feed the raw JSONL back into the model; we parse it locally to generate a compact “run report” (counts, tool output sizes, errors, cost/tokens sanity checks, etc.).
• The JSONL is currently the only durable per-session artifact we can reliably grab from ephemeral machines and archive.

Why the TurnContext.user_instructions repetition matters for us anyway:

• In one real rollout, turn_context.user_instructions was written 1,759 times with 1 unique value (~16MB). That’s unnecessary disk growth and makes local inspection (jq, our own tool) noisy/slow.
• It also leads to misleading accounting if someone naively sums “input tokens” from repeated snapshots (we hit this internally).

We’re open to switching mechanisms
If there’s a supported/recommended way to export a compact, canonical session history from the new sqlite/state DB or via an event stream after the fact, we’d happily adopt that. But until then, the rollout JSONL is what we have for postmortems.

Ask: would you consider either:
1) Dedupe TurnContext.user_instructions in the rollout writer (only emit on change), or
2) Move static-ish instructions into SessionMeta (written once) and omit from later turn contexts?

Both keep the rollout useful for debugging while preventing ballooning on long sessions.

Thanks for the additional details. Like I mentioned, you're using the JSONLs in a manner that isn't intended, so you're likely to see various impedance mismatches. We're unlikely to make changes specifically to accommodate your use case. The JSONL files are more of an internal implementation detail. It's quite likely that they will change in the future, so I don't recommend building features that depend on their format or structure.

Totally understood re: the rollout JSONL being an internal detail — we’ll treat it as best-effort and won’t build hard deps on the schema.

One clarification though: our primary pain isn’t only disk bloat from the JSONL, it’s API-side input token bloat that appears to correlate with the same “project instructions / AGENTS.md” blob being included repeatedly in the model prompt.

In our sessions, that blob is large and very stable, and we see it echoed as turn_context.user_instructions over and over. If that same content is being resent to the model each turn, it’s a big driver of billed input tokens.

Two questions so we can align with the intended surface:
1) Is Codex expected to resend the full project/user instructions on every model request (stateless prompt), or is there a supported way to avoid re-sending / reference it once per session?
2) If it must be resent, does Codex support OpenAI prompt-caching / “cached input tokens” so repeated stable prefixes (AGENTS + base instructions) don’t repeatedly incur full input cost? If yes, where is that configured / enabled?

If the answer is essentially “keep AGENTS small”, that’s fine — we’ll shrink it. But right now the spike we’re investigating looks like the repeated instruction blob is dominating prompt size.

Separately, if there’s a supported way to export a compact postmortem artifact from the app-server/OTel/sqlite store after the fact, we’d love pointers; that would let us stop touching the JSONL entirely.

We recently posted a blog post that should answer your question.

If you'd like to see new functionality, feel free to open a feature request. It's most useful if you can explain the problem in a general sense. It's also fine to propose solutions, but a problem statement will give us the freedom to consider alternative approaches. We generally prioritize feature requests based on community upvotes.

Thanks — read the post and it helps (stateless agent loop + preserving prefix to maximize prompt caching).

Two practical questions so we can align with the intended surfaces:
1) Is there a supported way to observe caching effectiveness (e.g. cached vs uncached input tokens, cache hit rate) from Codex CLI logs / events / OTEL?
2) For offline postmortems on ephemeral hosts, what’s the recommended way to persist the app-server event stream or OTEL stream to disk (so we’re not touching internal JSONLs)? Any flags / config pointers would be hugely appreciated.

We’re going to open a feature request with a general problem statement (offline export + cache visibility) so others can upvote it.

Many enterprises use network-based LLM proxies like LiteLLM for the sort of monitoring and analysis that you're describing here.

OTEL support does include token stats as part of the response.completed event. Fields include input_token_count and cached_token_count`. Refer to this documentation for some details. If you have more detailed questions, you can clone the sources and ask Codex questions about its implementation.