thread-store: a single BOM-prefixed rollout .jsonl fails the entire thread read and blocks message creation

Adding a small root-cause note in case it helps triage.

This looks like two separable failure modes that may be worth testing independently:

1. read_session_meta_line / rollout metadata parsing should probably accept a UTF-8 BOM before the first JSON object, because external editors on Windows can add it without otherwise changing the JSONL payload.
2. The thread list/read path should not let one unreadable rollout abort the whole inventory. A more resilient behavior would be to surface that rollout as unreadable/quarantined while still allowing other threads and new message creation to continue.

A focused regression test could create two rollout files: one normal file and one whose first bytes are EF BB BF { ...session_meta... }. The expected result would be either successful parsing of the BOM-prefixed file, or at minimum a partial inventory with an error attached only to that rollout, not a global thread-store failure.

This seems especially important because the state DB already has stale/missing-rollout tolerance paths; a BOM-prefixed but otherwise valid rollout feels like the same class of recoverable local-state problem.

Took a closer look at the Rust core for this — sharing a root-cause confirmation, the full set of affected read paths, and a minimal fix with a regression test, in case it helps triage.

Root cause (confirmed)

Every rollout consumer parses lines with serde_json::from_str after str::trim(), e.g. codex-rs/rollout/src/list.rs → read_head_for_summary:

let trimmed = line.trim();
if trimmed.is_empty() { continue; }
if let Ok(rollout_line) = serde_json::from_str::<RolloutLine>(trimmed) { ... }

str::trim() follows the Unicode White_Space property, and U+FEFF (the BOM) is not White_Space — so trim() leaves it in place, serde_json rejects the leading \u{FEFF} ("expected value at line 1 column 1"), and the session_meta record is silently dropped from head. read_session_meta_line then sees a non-meta record first and returns exactly the reported "does not start with session metadata".

Not limited to the summary path

All rollout reads funnel through compression::open_rollout_line_reader, so the same BOM also breaks:

• list.rs::read_head_for_summary / read_session_meta_line — the reported failure
• the summary scan in list.rs (saw_session_meta never set)
• search.rs (session-meta line skipped)
• recorder.rs::load_rollout_items — the resume path, which does serde_json::from_str(&line) directly; a BOM there bumps parse_errors and loses the thread_id.

Minimal fix (failure mode #1 only)

Strip a single leading BOM once, at the reader boundary (RolloutLineReader::next_line), so all four consumers benefit from one change and the on-disk file is left untouched:

/// Removes a single leading UTF-8 BOM (U+FEFF) from `line`, if present.
fn strip_leading_bom(line: String) -> String {
    match line.strip_prefix('\u{feff}') {
        Some(rest) => rest.to_string(),
        None => line,
    }
}

Applied to the first emitted line only (a BOM can only appear at byte 0), tracked with a seen_first_line flag on the reader.

Regression test

Mirrors the existing write_rollout helper, prepends the three BOM bytes (EF BB BF) to the file, then asserts both RolloutRecorder::load_rollout_items and read_session_meta_line recover the thread id. I verified both directions locally:

• Before the change the test fails — the session-meta line is dropped, so load_rollout_items returns thread_id = None and read_session_meta_line errors with "does not start with session metadata".
• After the change it passes.

On failure mode #2

@SylvainWinning's point that one unreadable rollout shouldn't abort the whole inventory is well taken — that's the more robust long-term behavior. But it's a separate, larger change (quarantine / partial-inventory semantics), so I'd keep it out of this fix to stay small and reviewable, and follow up separately.

I have the patch + test ready against main. Since contributions are invitation-only, I haven't opened a PR — but if the team would welcome one for failure mode #1, I'm happy to submit it.

The BOM-prefixed rollout is a clean fsck/discovery case for codex-rescue: one locally malformed session should be isolated and reported, not make the whole inventory unusable. I’m field-testing exactly that fail-closed behavior.

If you still have the affected file or can reproduce it in a disposable Codex home, could you run:

pipx install codex-rescue
codex-rescue sessions
codex-rescue doctor --latest

I’m interested in whether Rescue discovers the rest of the session store, how it classifies the BOM-prefixed rollout, and whether it leaves the file byte-for-byte untouched. No automatic “strip BOM” repair is implied here; a conservative diagnostic is the goal.

Please share only sanitized output + versions/exit codes. No raw rollouts/SQLite, prompts, credentials, or private paths. Repo: https://github.com/shleder/codex-rescue