›_ Codex Issues
Codex Issues / #39624

app-server fs/watch misses append updates to active rollout JSONL on macOS

Open 💬 3 comments Opened Aug 20, 2026 by gfreezy

What issue are you seeing?

On macOS, app-server fs/watch does not emit fs/changed while an active Codex rollout JSONL is being appended to.

The rollout file size and modification time continue changing as reasoning, tool-call, and output items are written, but the app-server connection receives no corresponding fs/changed notification. A notification may appear around a user-message or turn boundary, but incremental turn items do not reliably trigger one.

This prevents clients that use the file notification as a signal to call thread/read from updating the active turn in real time.

What steps can reproduce the bug?

  1. Start codex app-server over stdio and send initialize.
  2. Start or resume a thread and locate its active rollout JSONL under the Codex sessions directory.
  3. Subscribe to the exact absolute file path:
{ "method": "fs/watch", "id": 44, "params": {
  "watchId": "0195ec6b-1d6f-7c2e-8c7a-56f2c4a8b9d1",
  "path": "/absolute/path/to/rollout.jsonl"
} }
  1. Continue the turn so Codex appends reasoning, tool-call, command-output, and assistant-output records to the same file.
  2. Observe that the file size and mtime change, but no fs/changed notification is emitted for those appends.

Control checks from the same machine:

  • The same app-server watcher emits fs/changed for a newly created file in a watched temporary directory, so the subscription and connection are working.
  • fswatch using its default macOS backend also misses the active rollout appends.
  • fswatch -m kqueue_monitor on the same rollout file reports every append immediately.

The FSEvents/kqueue comparison is diagnostic evidence, not a confirmed root cause.

What is the expected behavior?

After the watcher debounce interval, each modification or batch of modifications to the watched file should emit:

{ "method": "fs/changed", "params": {
  "watchId": "0195ec6b-1d6f-7c2e-8c7a-56f2c4a8b9d1",
  "changedPaths": ["/absolute/path/to/rollout.jsonl"]
} }

This should work for append-only writes while the writer keeps the file open, not only for create, replace, rename, or turn-boundary activity.

Additional information

  • Codex CLI: 0.148.0 (current latest release at time of report)
  • Platform: Darwin 25.5.0 arm64 arm
  • macOS: 26.5.2 (25F84)
  • Architecture: Apple Silicon
  • Installation: Homebrew
  • codex doctor --json: app-server check is OK; the unrelated overall failure is only because the noninteractive diagnostic shell has TERM=dumb.
  • The app-server documentation says watching a file emits fs/changed for that path. The observed behavior differs specifically for sustained append activity on the active rollout file.

View original on GitHub ↗

3 Comments

tsuvic · 7 days ago

Source trace on current main (@67b2c8c) — the codex-side pipeline does not drop Modify events, which points the finger at the watcher backend itself, consistent with your fswatch A/B.

How fs/watch is implemented:

  • FsWatchManager::watch registers the exact requested path non-recursively and forwards events after a debounce (codex-rs/app-server/src/fs_watch.rs:76–148); fs/watch dispatch at codex-rs/app-server/src/message_processor.rs:1102.
  • The watcher is notify::recommended_watcher(...) with Config::default() (codex-rs/file-watcher/src/lib.rs:379–392), notify = "8.2.0" with default features (codex-rs/Cargo.toml:366). With default features, RecommendedWatcher on macOS is the FSEvents backend (macos_fsevent is the default feature; macos_kqueue is opt-in), file-level streams, latency 0.0, NoDefer, since-now.
  • The event filter accepts Create/Modify/Remove (is_mutating_event, file-watcher/src/lib.rs:761–766) — so an ItemModified for an append would pass; there is no create/rename-only filtering. The 200 ms debounce (fs_watch.rs:27) coalesces paths until a deadline but never drops them.
  • Subscribing to an existing file watches that file path itself (per-file, not dir+filter), so this is not a directory-scope mismatch either.

And the writer is exactly the "append through a long-lived open fd" class: open_rollout_for_append keeps one append-mode file handle for the session and each item is write_all + flush on that fd (codex-rs/rollout/src/recorder.rs:1896–1919; production path codex-rs/thread-store/src/local/live_writer.rs:354–362). Normal turns do not rewrite/rename the rollout.

Hypothesis (labelled, but it matches all three of your controls): the FSEvents stream does not reliably deliver kFSEventStreamEventItemModified for appends made through a persistently open writer fd, while create/rename events are delivered — which is why fswatch (FSEvents) misses the appends, fswatch -m kqueue_monitor catches them, and notifications appear around turn boundaries (boundary-adjacent operations FSEvents does report: materialize/rename from compression, writer re-open, mtime touches) rather than for the appends themselves.

Direction that would cover it: switch the macOS backend to kqueue — either enable notify/macos_kqueue in codex-rs/Cargo.toml:366 (then RecommendedWatcher resolves to the kqueue watcher, whose EVFILT_VNODE with NOTE_WRITE/NOTE_EXTEND catches appends on the registered file), or construct it explicitly in FileWatcher::new. The per-file watch design already ref-counts exact paths, so fd cost scales with subscription count. Watching the parent directory instead would not fix it by itself, per your FSEvents control.

Coverage note: the existing fs/watch tests only exercise file creation and atomic replace (app-server/tests/suite/v2/fs.rs:753), so there is no regression coverage for the append-with-open-fd case.

argszero · 3 days ago

I verified the proposed fix direction against notify 8.2.0 and it checks out — happy to implement it.

Details that confirm the approach:

  • notify 8.2.0 declares macOS-gated optional deps kqueue (1.1.1) and mio (1.0, os-ext) under [target."cfg(target_os=\"macos\")".dependencies] (notify/Cargo.toml:103-110), so enabling the macos_kqueue feature is sufficient on macOS — no extra non-macOS crates leak in.
  • RecommendedWatcher resolution: all(target_os = "macos", feature = "macos_kqueue") → KqueueWatcher (notify/src/lib.rs:419-421), overriding the default FSEvents mapping (lib.rs:407-408). Kqueue's EVFILT_VNODE + NOTE_WRITE/NOTE_EXTEND delivers the append-through-open-fd modifications that FSEvents misses.
  • The existing per-file, ref-counted watch design (file-watcher/src/lib.rs) keeps the fd cost proportional to subscriptions; still worth noting that recursive directory watches will also move to kqueue with a global feature switch (kqueue watches each subdirectory individually, unlike FSEvents' single stream) — I'll validate the existing fs/watch suite for regressions.

Planned change (small, one concern):

  1. Enable notify/macos_kqueue for the codex-file-watcher crate.
  2. Add a regression test for the append-through-open-fd case (currently uncovered — the existing tests only cover create and atomic replace).
  3. Run the codex-file-watcher and app-server fs/watch test suites.

I'd like to work on this.

argszero · 3 days ago

I implemented the kqueue switch and have it ready in my fork: https://github.com/argszero/codex/tree/fix/fs-watch-kqueue (commit de27bc4c, based on current main).

What the change does:

  • codex-rs/file-watcher/Cargo.toml: notify = { workspace = true, features = ["macos_kqueue"] }. With this feature, notify::recommended_watcher resolves to KqueueWatcher on macOS (EVFILT_VNODE + NOTE_WRITE/NOTE_EXTEND), which delivers the append-through-open-fd modifications that FSEvents misses. The feature only pulls macOS-gated optional deps (kqueue, mio under cfg(target_os = "macos")) and is a no-op on other platforms; nothing else in the workspace consumes notify.
  • New regression test live_watcher_reports_appends_through_open_fd in codex-rs/file-watcher/src/file_watcher_tests.rs: watches an existing file, appends multiple records through a persistent open fd (mirroring open_rollout_for_append's writer pattern), and asserts the change event is delivered with the watched path — the case that previously had no coverage.

Verified on macOS arm64: cargo test -p codex-file-watcher 22/22 pass (new test included), cargo clippy --all-targets and cargo fmt --check clean. Cargo.lock and MODULE.bazel.lock are unchanged.

I attempted to open this as a PR but GitHub rejected PR creation for this repository from an external account (GraphQL CreatePullRequest permission denied, REST 404), which I understand is the platform enforcement of the invitation-only contribution policy. Since you outlined the direction in this thread, I'd be glad to open the PR if you can invite the contribution — or if you'd prefer to take the patch from here, the branch is ready for you to pull from. Happy to adjust the approach (e.g. explicit KqueueWatcher construction instead of the feature switch, or an app-server-level test) if you'd like it done differently.