›_ Codex Issues
Codex Issues / #36330

SSH Remote app-server accumulates ~100k unreaped Git children during long-lived multi-client sessions

Open 💬 0 comments Opened Jul 31, 2026 by CubeLander

Version and environment

  • Codex Desktop: 26.721.41059
  • Remote Codex CLI during incident: 0.145.0
  • Desktop client: macOS
  • Remote: openEuler 24.03 container, Linux 5.15, aarch64
  • Container PID 1: sshd -D

The remote CLI has since updated to 0.146.0, but Remote remains disabled and the incident has not been intentionally reproduced.

What happened

After roughly two days of a long-lived Codex Desktop SSH Remote connection, with several threads resumed or controlled from Desktop and Android Remote, the host became almost unusable.

Live inspection of the remote app-server found:

open file descriptors:  ~378,000
child processes:        ~125,000
zombie processes:       ~96,000–99,000
zombie command:         git (nearly all observed zombies)
zombie parent:          codex app-server

Restarting the app-server/container immediately restored normal responsiveness and removed the Codex-owned zombies.

Strongest diagnostic evidence

The structured app-server log shows that Codex Desktop continued issuing process/spawn requests while the server was degraded. It repeatedly launched a repository-state pipeline every few seconds, including:

git config --null --get core.fsmonitor
git ... status --porcelain=v1 -z
git ... diff --cached ... --name-only -z
git ... diff ... --raw --numstat -z
git ... hash-object --stdin-paths
git ... ls-files --others --exclude-standard -z
git ... rev-parse --verify --quiet HEAD^{commit}

Immediately after a clean restart, one directly observed app-server child was:

git -c core.hooksPath=/dev/null -c core.fsmonitor= \
  add --sparse --pathspec-from-file=- --pathspec-file-nul

This does not prove that every zombie had the same arguments, but it establishes both ownership and the active Git repository-state path.

Other contemporaneous signals included:

failed to refresh available models: timeout waiting for child process to exit
sampling stream disconnects and retries
SQLite statements exceeding the slow threshold
ignored world-state patch without a full snapshot during thread resume
shell snapshot command timeout
repeated Remote Control token refresh and WebSocket reconnect attempts

The large zombie/FD count was already present before the final reconnect storm, so the network failures may be an amplifier or consequence rather than the initial trigger.

Likely failure boundary

Confirmed:

  • Codex Desktop repeatedly requested Git subprocesses through app-server process/spawn.
  • Exited Git children accumulated under that app-server instead of being reaped.
  • Process and FD counts grew without a useful bound.

Most likely:

  • A repository refresh/reconciliation loop was duplicated or left alive across multi-client thread resume/reconnect activity.
  • The process cleanup path failed to wait/reap children after timeout, cancellation, disconnect, or superseding refreshes.

The available evidence is sufficient to investigate the process/spawn ownership and repository-refresh lifecycle. We have not attempted to identify the exact source line because the incident was destructive enough to require a container restart.

Expected behavior and suggested fix direction

  1. Every process/spawn child must be waited for and reaped on success, timeout, cancellation, client disconnect, thread resume, and shutdown.
  2. Repository refreshes should be single-flight/coalesced per repository and have a strict concurrency bound.
  3. Reconnecting or resuming the same thread must not duplicate background Git polling/reconciliation tasks.
  4. Add a Linux soak test that repeatedly resumes/disconnects clients and asserts that child count, zombie count, and FD count return to baseline.

Reproduction shape

The exact trigger is intermittent:

  1. Connect Codex Desktop to a Linux host with SSH Remote.
  2. Keep app-server alive for an extended period with several long-running threads.
  3. Resume/control threads from Desktop and a mobile Remote client.
  4. Introduce Remote Control/Responses WebSocket failures or reconnects.
  5. Monitor app-server children, zombies, and FDs over time.

We have not intentionally rerun this workload because the first occurrence produced approximately 100k zombies.

Related issues

This may share a lifecycle defect with #26869, but this report adds a Linux SSH Remote case with approximately 100k Git zombies, a single app-server parent, multi-client resume activity, and retained process/spawn evidence.

View original on GitHub ↗