Codex Desktop Remote SSH reconnect loop leaks thousands of orphaned app-server processes when hosts share an NFS home

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #34142

Powered by Codex Action

Thanks — I reviewed #34142. There is significant overlap: both involve the macOS Desktop SSH bootstrap, the same app-server --listen unix:// process, and old brokers surviving reconnects.

However, I do not think this is an exact duplicate based on the evidence currently available:

• #34142 isolates a shutdown failure where an already-degraded broker receives SIGTERM, blocks in futex_do_wait, and requires SIGKILL.
• This issue is primarily about an unbounded spawn/lifecycle leak during Remote SSH reconnects: 3,277 orphaned app-server wrappers accumulated across 20 hosts.
• The affected hosts share an NFS-backed $HOME and the same CODEX_HOME/control-socket pathname, which appears relevant to the repeated socket failures and reconnect loop.
• I did not capture evidence that the leaked processes had received SIGTERM or were blocked in the same futex shutdown path.

During cleanup I used SIGTERM followed by SIGKILL after a grace period, but the controlling SSH sessions were interrupted, so I could not reliably determine how many processes required SIGKILL. Therefore, this report does not independently reproduce the specific SIGTERM deadlock from #34142.

The two issues may form a compounding chain:

Remote SSH reconnect leaves the previous broker alive → brokers accumulate and degrade → some degraded brokers may then hit the #34142 SIGTERM deadlock.

I am leaving this issue open to track the Remote SSH reconnect/process-lifecycle and shared-home aspects, but I am happy to consolidate it if maintainers confirm that they share the same root cause.

Analysis (community)

Residual analysis for #35217 (Desktop Remote SSH reconnect loop leaks thousands of orphaned app-server processes when hosts share an NFS home) under docs/contributing.md — invitation-only. No unsolicited PR.

Root-cause hypothesis

The unbounded respawn loop may still be closed Desktop (ensure_remote_app_server), but OSS amplifies the multi-host NFS-home failure with globally shared control-socket/lock names, cross-host “stale” unlink, no last-client shutdown for UnixSocket, and unguarded path removal on drop — matching the orphaned-wrapper / missing shared socket pattern reported here.

Verified at HEAD 4c43465133 (distinct from the closed Windows WMI/taskkill storm):

1. Fixed names under CODEX_HOME — no host/generation scope.
codex-rs/app-server-transport/src/transport/mod.rs: bare unix:// listen resolves to fixed paths under CODEX_HOME/app-server-control/:

• app-server-control.sock
• app-server-startup.lock

No hostname, SSH target, connection ID, or generation enters either path. Shared NFS CODEX_HOME ⇒ every remote host contends on the same directory entries.

2. Startup “stale socket” removal is host-local unsafe on NFS.
transport/unix_socket.rs (prepare path): after a failed connect, a socket-typed path is treated as stale and removed. A Unix domain socket is host-local even when its directory entry is visible through NFS — so host A cannot safely decide that host B’s live listener is “stale.” Cross-host unlink of a visible path is a direct multi-host failure mode.

3. Startup lock is the same global path.
The lock serializes file operations on that shared pathname (unix_socket.rs lock region) but cannot make one host’s Unix listener reachable from another. It orders racy unlinks; it does not provide cross-host ownership.

4. No last-client shutdown for UnixSocket.
App-server enables shutdown_when_no_connections for Stdio, not for UnixSocket (codex-rs/app-server/src/lib.rs listen-mode branch). Losing the owning SSH/WebSocket connection therefore does not terminate a Unix-socket app-server. Acceptor + detached client tasks keep running (unix_socket.rs acceptor loop).

5. Unconditional guard unlink on drop.
ControlSocketFileGuard::drop removes the pathname without inode/owner/generation verification. One host/generation exiting can erase the visible path for another still-running listener — again amplified under shared NFS homes.

6. Issue fit.
#35217’s shared NFS home, missing shared socket, reconnect/creation correlation, and orphan scale match this amplifier chain. Treat Desktop reconnect policy as closed; the OSS residual is socket/lock lifecycle under shared home.

Ranked fix outline (all OSS):

1. Scope socket + lock paths by stable host identity (hostname / machine-id / remote target hash), not only CODEX_HOME.
2. Make unlink ownership-safe (inode + peer credential / generation token); never unlink a path you did not create this generation.
3. Introduce an explicit owner lease / connection-lifetime contract for UnixSocket (last-client or last-owner shutdown).
4. Optional: refuse multi-host shared CODEX_HOME with a clear error when control socket dir is on NFS.

Fail-first tests: (1) two logical hosts with same CODEX_HOME get distinct sock paths; (2) failed connect does not unlink another generation’s live sock; (3) UnixSocket with zero clients eventually shuts down under the chosen lease policy.

Happy to help draft the path-scoping patch. No unsolicited PR.

Corroborating this from a different site, with a few observations that may help narrow it down. Still present with remote codex-cli 0.148.0 (Codex/ChatGPT Desktop version 26.814.41407 (6720) on macOS 26.5.2 (25F84) ↔ Linux 6.8 HPC login node over SSH, NFS-shared $HOME, same ~/.codex/app-server-control/app-server-control.sock path).

Same mechanism at smaller scale — but it bites much earlier when the remote has a per-user thread quota. The login node caps each user's systemd slice at TasksMax=512 (pids.max counts threads). On 2026-08-20 my slice hit 512/512: fork: Resource temporarily unavailable for every process, codex's own TUI panicking in the nucleo threadpool, unrelated Node programs aborting on uv_thread_create — on a day I never actively used Desktop. Background reconnects alone did it.

What the slice contained (cgroup walk, exact numbers):

• 28 logind sessions for my user, 2 online, 26 State=closing, all from my own client IP.
• 14 dead sessions each still held a codex app-server proxy pair (node wrapper + native child; ~22–24 threads combined once they had carried traffic, ~3 threads if they never did). So on Linux the proxy leaks too, not only the app-server --listen unix:// daemon — the proxy does not exit when its sshd parent / transport dies.
• 2 dead sessions held orphaned daemon groups (-c features.code_mode_host=true app-server --listen unix:// wrapper + native child + code-mode child, ~33 threads); one was 8+ h old with no clients.
• ≈ 390 of 512 tasks were dead-session Codex remnants.
• The loop, observed live: two fresh daemons spawned 3 min apart (14:48 and 14:51 JST); minutes later both of their ssh sessions were already State=closing while the processes remained.

Two structural details that may be useful for a fix:

1. A leaked daemon survives reparented to PID 1 but stays in its dead session's cgroup (/proc/<pid>/cgroup → session-N.scope) — that is what pins those sessions in State=closing indefinitely.
2. Because the control socket is one canonical path, superseded daemons are identifiable purely structurally: ss -xlHe lists every listener still bound to …/app-server-control.sock with its vfs ino:, and only one listener's ino: equals stat of the path. Every other listener is a daemon that lost the name and can never receive another client, yet keeps running. On the same node, other users show the same thing: /proc/net/unix has 29 such leaked listeners across 5 users; one user has 25 distinct listener sockets on that single path (≥ 24 permanently unreachable daemons).

Related: #29809 (same accumulation → fork/exec failures), #24542 (proxy respawning unmanaged daemons), #30791 (Windows proxies accumulating).

Expected behaviour, from this angle: the proxy should treat transport EOF as terminal (or set PR_SET_PDEATHSIG); the daemon should exit after a grace period with no clients, or as soon as it is superseded on the socket path; and a node-local socket directory ($XDG_RUNTIME_DIR) would avoid the NFS pathname aliasing this issue already points at.

Analysis (community)

@ishiihiro — thanks for the 2026-08-20 HPC corroboration on #35217 (remote CLI 0.148.0, Desktop 26.814.41407, NFS $HOME, TasksMax=512). The new evidence I want to bind is not the NFS pathname aliasing I posted in July. It is the proxy that does not die with sshd, plus the Unix-socket daemon that does not exit when it has zero clients. Both are still in main.

Root-cause hypothesis (re-checked 2026-08-21 at main 536f86e5c): a Remote SSH session leak is two OSS processes, not one. codex app-server proxy is stdio-to-uds (cli/src/main.rs:1346-1354). That helper try_join!s stdin→UDS and UDS→stdout (stdio-to-uds/src/lib.rs:18-43). stdin EOF only half-closes the write side (:33). The process stays alive until the Unix-socket peer closes. The daemon on --listen unix:// is not single-client: single_client_mode is Stdio-only (app-server/src/lib.rs:728), and ConnectionClosed only breaks the server loop for Stdio (:1036-1038). UnixSocket keeps the acceptor running (unix_socket.rs:53-76). So: sshd dies → proxy stdin hits EOF → proxy hangs in copy_socket_to_stdout → daemon keeps the UDS open → both stay in the dead session cgroup. That is the “proxy pair” you counted, and why PR_SET_PDEATHSIG would have helped: stdio-to-uds has zero PR_SET_PDEATHSIG uses (the only hits in-tree are linux-sandbox / pty / sleep-inhibitor).

Still live amplifiers from the July residual, now at this HEAD:

1. Canonical NFS-visible socket path. Bare unix:// is still $CODEX_HOME/app-server-control/app-server-control.sock. Your ss/ino: split is exactly “many listeners, one pathname.”
2. Host-local stale unlink. prepare_control_socket_path still treats ConnectionRefused as stale and remove_files the path (unix_socket.rs:108-131). A Unix socket is host-local even when the directory entry is on NFS, so host A can still knock out host B’s name.
3. Drop unlinks without inode/generation. ControlSocketFileGuard::drop still remove_files unconditionally (unix_socket.rs:178-189).
4. Daemon setsid. App-server spawn still setsid() (app-server-daemon/src/backend/pid.rs:173-181) with stdin/stdout /dev/null (:166-167). That is why a leaked daemon can reparent to PID 1 and remain in session-N.scope: it is not in sshd’s process group, so SIGHUP on SSH close does not reap it. Combined with (UnixSocket ≠ last-client shutdown), it idles forever.

Your two structural details map onto this: (1) PID-1 + dead-session cgroup = setsid daemon + no last-client exit; (2) extra ino: listeners on one path = NFS-visible name + unguarded unlink/bind. Related already on the issue: #29809, #24542.

No unsolicited PR. Ranked OSS levers if maintainers want them: (a) stdio-to-uds should treat stdin EOF as process-terminal (or PR_SET_PDEATHSIG), not wait on the UDS peer; (b) UnixSocket should get an explicit last-client / superseded-inode lease so a daemon that lost the name exits; (c) socket+lock paths should be host-scoped ($XDG_RUNTIME_DIR / machine-id), not a single NFS-visible CODEX_HOME name. I have not reproduced the HPC TasksMax=512 box here; this is a code read against your cgroup/ss numbers.

I reproduced the same class of leak on a single Linux host with a local ext4 home, so a shared NFS CODEX_HOME is not required.

Environment

• Client: ChatGPT/Codex Desktop on Windows using the built-in SSH remote connection
• Remote host: Debian Linux, 12 vCPU, 32 GiB RAM
• Remote Codex: codex-cli 0.150.1 (re-verified on 2026-08-28)
• Remote home/root filesystem: local ext4 (/dev/vda4), not NFS
• Remote app-server command observed:

node /usr/local/bin/codex -c features.code_mode_host=true app-server --listen unix://

with the native Codex child process beneath it.

Observed impact

After repeated Desktop SSH disconnect/reconnect cycles, stale app-server generations remained reparented to PID 1 inside old systemd session scopes.

At the incident peak:

• host load was approximately 209
• approximately 201 tasks were runnable
• 24 Codex processes / 177 OS threads
• summed Codex CPU was approximately 588–590%
• SSH command latency degraded to roughly 80 seconds

The 177 Codex threads consisted of:

• one oldest app-server generation: 24 threads
• ten additional app-server generations: 14 threads each
• thirteen single-thread Codex worker processes

The oldest orphan had survived for 3 days 17 hours 15 minutes. Nine duplicate app-server scopes had survived for approximately 10h49m–10h57m; another 14-thread generation self-exited after approximately 10h28m. Some associated one-thread workers had lived for approximately 1h50m–10h46m.

The stale session scopes reported State=closing / SubState=abandoned, and the oldest app-server was PPID=1 with no active TCP connection. However, a currently used Desktop-created app-server could also appear as abandoned, so killing every abandoned scope is not a safe workaround.

Recovery evidence

I terminated only the precisely identified stale scopes:

• the first bounded cleanup removed 9 stale scopes / 65 processes
• runnable tasks dropped from approximately 201 to 3
• CPU pressure dropped from approximately 91.8% to 43%
• the oldest app-server attempted to respawn its Code Mode host after SIGTERM, so that exact stale scope required SIGKILL
• after clearing that final scope, CPU pressure fell through 4.21% to 0.47%
• SSH command latency recovered to approximately 2–5 seconds

Production containers remained healthy throughout the targeted cleanup.

Why this appears related

The directly observed lifecycle was:

Desktop SSH connection ends or reconnects
-> old app-server outlives the owning SSH transport
-> wrapper is reparented to PID 1 in a closing/abandoned session scope
-> a new Desktop connection starts another app-server generation
-> generations accumulate and eventually saturate the host

This matches the core behavior reported here, but occurred on one host with a local filesystem and current 0.150.1.

Expected behavior

• Disconnect/reconnect should reuse a healthy app-server or cleanly replace the previous generation.
• An ephemeral Desktop-owned app-server should terminate when its owning proxy/transport is gone.
• Reconnect/bootstrap should be single-flight and bounded.
• Desktop or codex doctor should expose the owning connection and provide a supported cleanup action.
• A stale generation should have an idle timeout and resource limits so it cannot saturate a production host.

No hostnames, IP addresses, usernames, account identifiers, tokens, credentials, environment variables, or raw session transcripts are included in this comment. I can provide additional targeted and redacted process/systemd snapshots if maintainers specify what is useful.