Orphaned process pairs burn CPU when terminal is closed (SIGHUP not handled)

Open 💬 2 comments Opened Mar 17, 2026 by mjpensa

💡 Likely answer: A maintainer (github-actions[bot], contributor) responded on this thread — see the highlighted reply below.

Summary

When a terminal tab or window is closed while Codex is running, the Node wrapper (codex.js) and its Rust child process both survive as orphans, with the Rust process entering a tight read(fd=0) → EIO spin loop at ~103K syscalls/sec, burning ~15-19% CPU each. Each terminal close creates one orphan pair (Node + Rust), accumulating indefinitely until manually killed.

Root Cause

Three-layer failure chain:

Layer 1 — Signal suppression: codex.js registers a handler for SIGHUP (line 204) that calls forwardSignal("SIGHUP") but never calls process.exit(). Since Node's default SIGHUP behavior (terminate) is suppressed by the handler, the Node process survives. Because the session leader (Node) survives, the kernel never delivers SIGHUP to the Rust child process — SIGHUP only propagates to the process group when the session leader actually dies.

Layer 2 — PTY I/O spin: When the terminal closes, the PTY file descriptors are revoked. The Rust child's crossterm event reader calls read(fd=0) in a tight loop, each returning EIO immediately (~103K calls/sec). This is a known crossterm bug: crossterm-rs/crossterm#793 (open since June 2024). Codex uses a forked crossterm 0.28.1 from nornagon/crossterm.

Layer 3 — No application-level exit: Even if crossterm's event stream ended gracefully, the Rust process's tokio::select! loop has other active branches that keep it alive.

Reproduction

Minimal reproduction using a Python PTY script:

#!/usr/bin/env python3
"""Reproduce codex orphan bug: spawns codex in a PTY, then closes the PTY."""
import pty, os, time, signal

pid, fd = pty.openpty()
child = os.fork()
if child == 0:
    os.setsid()
    os.dup2(fd, 0); os.dup2(fd, 1); os.dup2(fd, 2)
    os.execvp("codex", ["codex"])
else:
    time.sleep(3)          # let codex start
    os.close(fd)           # revoke PTY (simulates closing terminal)
    time.sleep(2)
    # Check: both Node and Rust processes should be dead, but they're not
    os.system(f"ps -o pid,ppid,%cpu,command -p {child}")
    os.system("ps aux | grep 'codex' | grep -v grep")

Steps:

Run the script above
After 5 seconds, check for orphan processes: ps aux | grep codex | grep -v grep
Observe: both the Node wrapper and Rust child survive with elevated CPU

Evidence

From a system with 6 orphan pairs (12 processes):

# lsof showing revoked PTY file descriptors on orphaned Rust process:
codex  PID  user  0u  CHR  ?,?  0t0  ??? /dev/ttys??? (revoked)
codex  PID  user  1u  CHR  ?,?  0t0  ??? /dev/ttys??? (revoked)

# dtruss showing EIO spin rate:
read(0x0, ...) = -1 Err#5  [~103,000 calls/sec]

# CPU usage per orphan pair: ~15-19% each process (30-38% per pair)
# 6 pairs = 104.7% total CPU consumed by orphans

Suggested Fixes

Priority 1: Fix codex.js SIGHUP handler (Node wrapper)

Replace the current signal registration (around line 204):

// Current (broken):
["SIGINT", "SIGTERM", "SIGHUP"].forEach((sig) => {
  process.on(sig, () => forwardSignal(sig));
});

// Fixed:
["SIGINT", "SIGTERM"].forEach((sig) => {
  process.on(sig, () => forwardSignal(sig));
});

process.on("SIGHUP", () => {
  forwardSignal("SIGHUP");
  setTimeout(() => {
    try { child.kill("SIGKILL"); } catch {}
    process.exit(1);
  }, 3000).unref();
});

The .unref() ensures that if the child exits normally before the timeout, Node proceeds through the existing exit path (lines 213-229) without the timer keeping the event loop alive.

Priority 2: Fix crossterm EIO spin (Rust/crossterm layer)

The forked crossterm (from nornagon/crossterm) should treat EIO on read() as a stream-termination event rather than retrying indefinitely. See crossterm-rs/crossterm#793.

Priority 3: Rust application-level cleanup

The Rust binary should handle SIGHUP explicitly (e.g., via tokio::signal::unix::signal(SignalKind::hangup())) and initiate graceful shutdown.

Priority 4: Defense-in-depth — detect revoked FDs

The Rust binary could periodically check if its stdio FDs are still valid (e.g., fcntl(0, F_GETFD)) and exit if they've been revoked.

Environment

OS: macOS 13.7.8 Ventura (x86_64, Intel)
Codex version: codex-cli 0.115.0
Node: v25.6.0
Shell: zsh

View original on GitHub ↗

2 Comments

github-actions[bot] contributor · 5 months ago

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

#13928

Powered by Codex Action

roazanas · 28 days ago

Adding some verification on top of this, because the situation upstream has changed since March and one of the three layers here no longer applies.

1. The upstream fix now exists, but is incomplete — and is not in the pinned fork

crossterm#1067 (open since 10 Jul 2026) fixes the inner Unix read loops described in Layer 2: Ok(0) returns UnexpectedEof, and the Err arm gains return Err(e) so EIO propagates. It also fixes two sibling loops and adds a try_read regression test.

I checked the three revisions that matter, and the read loop is byte-identical in all of them — the fix is in none:

| crossterm | used by | Ok(0) | non-WouldBlock/Interrupted error |
|---|---|---|---|
| 0.28.1 nornagon/crossterm@87db8bf | rust-v0.145.0, rust-v0.146.0 | falls through | falls through |
| 0.29.0 (crates.io) | upstream latest release | falls through | falls through |
| 0.29.0 openai-oss-forks/crossterm@f69a4a0 | current main | falls through | falls through |

So main's migration from the nornagon fork to openai-oss-forks/crossterm did not pick this up. The pin being a fork under OpenAI's control is the upside here — it can be patched without waiting on upstream review.

But cherry-picking #1067 as it stands would not be enough. I built a minimal tokio + EventStream consumer against the PR head and tore its pty down. The CPU spin goes away (~101% → ~0%), but the error never reaches the consumer: the stream stays pending, and the process sits idle instead of exiting. In a 5 s window after teardown, an instrumented poll_next was called 14 times and returned Ok(false) every time, never Err. Details and the mechanism are in my comment on the PR. Applied alone to Codex, it would therefore likely replace a hot orphan with an idle one rather than triggering the shutdown path. I did not build Codex itself with the patch, so that last part is a prediction from the minimal reproduction, not a measurement.

2. Layer 3 looks already solved

The issue notes that even a graceful stream end might not exit. On rust-v0.146.0 that path is handled: TuiEventStream::poll_crossterm_event maps Some(Err(_)) and None to Poll::Ready(None) (event_stream.rs#L214-L219), and the main loop turns that into handle_exit_mode(ExitMode::ShutdownFirst) with a terminal input stream closed warning (app.rs#L1212-L1221). The Codex side is ready to consume an input error — crossterm has to hand it over first.

3. macOS data point, standalone binary (no Node wrapper)

Worth adding because Layer 1 here is about codex.js, which was not in play for my instance — the parent was already gone (PPID 1) and there was no Node wrapper anywhere in the tree. Codex 0.145.0, aarch64-apple-darwin standalone release:

PID    PPID  TTY  STAT  %CPU  TIME       ELAPSED
51804  1     ??   R     99.1  953:16.65  02-11:01:17

FD 0   (revoked)     <- stdin
FD 1   (revoked)
FD 2w  CHR  /dev/null

sample puts 2305 of 2310 samples in the same crossterm function and offset:

2306 <crossterm::event::source::unix::mio::UnixInternalEventSource
     as crossterm::event::source::EventSource>::try_read + 436
! 2305 read  (in libsystem_kernel.dylib) + 8

I found two hot orphans consuming roughly one core each at the time of measurement; one had been alive for 2d11h and had accumulated 953 CPU minutes, 333 of them system time. RSS stayed around 163 MB across the observations I took, with no concurrent memory growth — the immediate failure mode was CPU saturation. A surviving process also keeps its normal runtime resources open; in the unpatched instance I saw handles to state_5.sqlite, logs_2.sqlite, goals_1.sqlite, memories_1.sqlite and their WAL/SHM files.

On the EOF-vs-EIO question: on macOS man 2 revoke documents that a read() from a revoked terminal device returns zero, as an explicit exception to "subsequent operations fail", while man 2 read documents EIO separately, including for an orphaned process group. I did not trace which one my instance hit, so I can only say both are reachable and #1067 handles both.

4. Reproduction

Two notes, since #16862 stalled on the maintainer not being able to reproduce by closing Terminal.app.

The Python script in this issue does not tear the pty down, so it will produce a false negative. pty.openpty() returns (master_fd, slave_fd), so in pid, fd = pty.openpty() the variable fd is the slave; the parent's os.close(fd) closes that slave copy and leaves the master open, so no hangup occurs. Separately, dup2-ing a slave onto 0/1/2 after setsid() does not make it the controlling terminal on macOS/BSD without TIOCSCTTY.

What did reproduce deterministically for me, on stock 0.146.0: pty.fork() (which is forkpty — it does setsid plus login_tty, so the slave really becomes the controlling terminal), then in the child signal.signal(SIGHUP, SIG_IGN) before execv so the process survives the hangup, then in the parent close the last master fd. CPU measured before and after, so startup cost cannot be mistaken for the loop:

| phase | CPU / 5 s wall | share of a core |
|---|---|---|
| before teardown, TUI idle at the prompt | 0.02 s | 0% |
| after teardown | 5.06 s | 101% |

Same hot stack as the orphan above. Notably the descriptors were not revoked in this harness — fd 0/1 still pointed at the live /dev/ttysNNN — so revoke() is not a precondition; a hung-up pty with a surviving process is enough.

The loop is only reachable if the process survives the teardown, which is probably why manual repro has been unreliable. On my machine, closing a kitty window with a running TUI left no orphan at all in 2 of 2 attempts — the process was simply gone. I did not verify the mechanism, so I can only report the clean exit, not its cause. Warp reproduced it twice.

5. Related reports

Same failure family. #11077, #28315 and #19958 contain direct EOF / deleted-or-revoked-PTY / hot-read evidence; the rest match on lifecycle and CPU symptoms: #16862, #13928, #28794, and closed #21021 — closed by its author on 30 May without a fix being confirmed, so it should not be read as resolved.

One small correction for anyone comparing rates: the /proc counters in #28315 give 243963619706 − 243943917947 = 19,701,759 reads in ~2 s, not the 1,172,137 stated there — about 9.8M/s, consistent with a syscall-rate loop. (syscr is process-wide, so it is strong circumstantial evidence rather than a trace of read(0) specifically.)