Backward-compat regression: legacy exec_command_end cwd is rejected as PathUri, amplifying replay into Jetsam
What version of the Codex App are you using (From “About Codex” dialog)?
Codex Desktop build 26.727.40816, bundled app-server
0.146.0-alpha.9.2.
Exact bundled source tag:
rust-v0.146.0-alpha.9.2 at
86cc9f2177cad015befd595286d8767a650f7d13.
What subscription do you have?
Pro.
What platform is your computer?
Darwin 24.6.0 arm64 arm. Apple Silicon Mac with 16 GB RAM.
What issue are you seeing?
Resuming or replaying a large legacy thread causes the native Codex app-server
to eagerly materialize the complete rollout, reject thousands of command
lifecycle events written by an older Codex version, persist one TRACE record
per rejected line, and generate heavy SQLite insert/prune/checkpoint traffic.
The observed app-server reached a peak physical footprint of approximately
12.6 GB and was later named largestProcess in a macOS Jetsam memory-pressure
event.
The deterministic backward-compatibility regression is:
- A real rollout written by app-server 0.118.0-alpha.2 contains 7,498
event_msg/exec_command_end records.
- All 7,498 store cwd as a native absolute filesystem path, for example
/redacted/workspace.
- The current ExecCommandEndEvent schema deserializes cwd as PathUri.
- PathUri accepts only valid file URI strings, so the native path is rejected
with: invalid URI: relative URL without a base.
- The loader reported 7,499 parse errors. The 7,498 native-cwd records explain
all but one; the remaining error is separate.
- 5,789 of the rejected records use ParsedCommand::Unknown and have no parsed
path field at all. This confirms that parsed_cmd.path is not the URI error.
The current rollout loader compounds the compatibility bug:
- load_rollout_items reads every JSONL line;
- parses each line to serde_json::Value;
- converts the Value to RolloutLine;
- stores every accepted RolloutItem in one Vec;
- emits a TRACE event for every rejected line.
The persistent log database has an independent amplification path:
- the SQLite log sink defaults to TRACE;
- inserts are batched in groups of 128;
- each batch can run partition-cap queries and window-function deletes;
- startup maintenance performs a passive WAL checkpoint but does not run
incremental_vacuum.
One read-only snapshot showed:
- approximately 1.1 GB legacy rollout;
- 484,033 accepted rollout items;
- 7,499 parse errors;
- approximately 6.2 GB logs_2.sqlite;
- approximately 4.42 GB of freelist pages;
- approximately 1.76 billion lifetime log inserts;
- approximately 8.59 GB of process writes in 528 seconds in a macOS diagnostic,
with SQLite WAL checkpoint/pwrite in the heavy stack.
Subagents are a workload trigger because forking or reopening inherited legacy
history reaches this path. Their count is not a sufficient root cause.
What steps can reproduce the bug?
Compatibility-only reproduction:
- Use a persisted RolloutLine with the v0.118.0-alpha.2
event_msg/exec_command_end shape.
- Store cwd as a native absolute path rather than a file URI. A reduced
illustration of the relevant fields is:
{
"type": "event_msg",
"payload": {
"type": "exec_command_end",
"cwd": "/redacted/workspace",
"parsed_cmd": [
{
"type": "search",
"cmd": "rg needle src",
"query": "needle",
"path": "src"
}
]
}
}
- Load the line through RolloutRecorder::load_rollout_items in
0.146.0-alpha.9.2.
- Observe that the complete RolloutLine is discarded and parse_errors is
incremented with invalid URI: relative URL without a base.
Resource-amplification reproduction:
- Resume, list turns for, or fork a large thread whose historyMode is legacy
and whose rollout contains many of the older exec_command_end records.
- Observe the full rollout being read and materialized even when the caller
requests only a history page.
- Observe one rollout-recorder TRACE record per rejected row in
logs_2.sqlite.
- Repeat history hydration or leave Desktop background hydration active.
- Observe rising app-server footprint and sustained SQLite/WAL write traffic.
An isolated app-server test confirmed that this bundled server accepts
thread/start with historyMode paginated. Nevertheless, all observed real
Desktop threads, including newly created ones, remain legacy.
What is the expected behavior?
- Rollouts written by older Codex versions remain readable.
- Exec command lifecycle events accept their persisted legacy cwd format
without weakening PathUri validation at public API boundaries.
- Resuming, listing, and forking have bounded memory independent of total
persisted history.
- N malformed historical rows produce O(1) summarized diagnostics, not N
persistent TRACE records.
- Logical log retention also bounds physical database size and disk-write
amplification.
- Under memory or disk pressure, hydration is deferred or rejected with a
retryable error instead of destabilizing the host.
Additional information
The source history points to a narrowly scoped fix:
- PR #28780 changed ExecCommandBeginEvent and ExecCommandEndEvent cwd to
PathUri:
https://github.com/openai/codex/pull/28780
- PR #29158 removed legacy native-path deserialization from PathUri:
https://github.com/openai/codex/pull/29158
- PR #29472 already proposed using LegacyAppPathString for these event fields
and accepting both native paths and file URIs:
https://github.com/openai/codex/pull/29472
- PR #29472 was closed because persisted command lifecycle events were believed
to be limited to a removed experimental feature. The 7,498 real persisted
events above contradict that assumption.
The latest inspected main commit, 5157493c23713ac12034cf250ffb0a8ce0670277,
still has strict PathUri deserialization for these fields and no compatibility
adapter.
Suggested P0 fix:
- Keep PathUri globally strict.
- Make only persisted ExecCommandBeginEvent.cwd and
ExecCommandEndEvent.cwd backward-compatible through LegacyAppPathString or
an equivalent field-local serde adapter.
- Add a real-shape v0.118 RolloutLine regression fixture and assert
parse_errors equals zero.
- Aggregate parse failures per rollout and error class.
- Default supported Desktop clients to paginated history and add a bounded
compatibility/migration path for existing legacy threads.
Related resource issues:
- #29510: large rollout app-server memory growth
https://github.com/openai/codex/issues/29510
- #32198: complete legacy replay plus per-record parse errors and TRACE growth
https://github.com/openai/codex/issues/32198
- #33786: pagination still rereads the complete legacy rollout
https://github.com/openai/codex/issues/33786
- #31542: persistent SQLite logging defaults to TRACE
https://github.com/openai/codex/issues/31542
- #35823: incremental auto-vacuum is configured but never executed
https://github.com/openai/codex/issues/35823
- #32431: macOS disk-write events through SQLite WAL checkpoint/pwrite
https://github.com/openai/codex/issues/32431
No real prompt, command output, thread ID, username, organization, repository,
credential, token, or filesystem path is included. Raw rollouts and databases
are not attached because they contain private data. A fully synthetic fixture
can be prepared if maintainers request it.
1 Comment
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action