app-server busy-loops at ~90% CPU and leaks ~4.8 GB while idle (0 connections); ignores SIGTERM

Bug report: codex app-server busy-loops at ~90% CPU and leaks ~4.8 GB while completely idle (0 connections)

Summary

A daemonized codex app-server --listen ws://127.0.0.1:4765 instance ran for 5+ days with zero WebSocket connections while continuously consuming ~90% of one CPU core and growing to 3.3 GB RSS + 1.5 GB swap (~4.8 GB anonymous heap). Per-thread sampling shows the tokio runtime workers spinning; there is no client activity, no file-watcher registrations, and no log writes during the spin.

Environment

| Item | Value |
|---|---|
| codex-cli | 0.144.4 |
| Install | npm global (@openai/codex, vendored codex-linux-x64 musl binary) |
| Node.js (wrapper) | v24.18.0 |
| OS | Linux 6.17.0-35-generic, x86_64 |
| RAM / swap | 30 GiB / 8 GiB |
| Command line | codex app-server --listen ws://127.0.0.1:4765 |

Timeline

• 2026-07-10 08:16:57 — process started (node wrapper PID 839412 → vendored binary PID 839419, reparented to PID 1, i.e. daemonized).
• 2026-07-15 ~16:15 — investigated because the machine was low on memory and swap was 7.8/8.0 GiB full.
• Uptime at capture: 5d 07:58. Cumulative CPU time: 4d 18:21 — i.e. the process averaged ~89% of one core for its entire lifetime.

Evidence

1. Process state (captured 2026-07-15)

    PID    PPID                  STARTED     ELAPSED     TIME %CPU %MEM   RSS STAT COMMAND
 839412       1 Fri Jul 10 08:16:57 2026 5-07:58:27 00:00:00 0.0  0.0  1988 Ssl  node ~/.npm-global/bin/codex app-server --listen ws://127.0.0.1:4765
 839419  839412 Fri Jul 10 08:16:57 2026 5-07:58:27 4-18:21:29 89.3 10.5 3416068 Sl  .../codex-linux-x64/vendor/x86_64-unknown-linux-musl/bin/codex app-server --listen ws://127.0.0.1:4765

Live top sample (3 s):

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 839419 min       20   0 6095180   3.3g  45472 S  97.3  10.6    4d+18h codex

2. Zero client connections

$ ss -tnp | grep 4765
(no output — no established connections to the listen port)

3. Memory: ~4.7 GB anonymous heap while idle

/proc/839419/status and smaps_rollup:

VmRSS:     3416200 kB
VmSwap:    1557496 kB
Threads:   62
Rss:       3320584 kB
Anonymous: 3273328 kB
Swap:      1490424 kB

An idle server holding 3.27 GB of anonymous heap plus 1.49 GB swapped out (~4.8 GB total) suggests unbounded accumulation (sessions/log buffers/history never freed).

4. The CPU burn is in the tokio runtime workers

Per-thread CPU delta over a 5-second window (/proc/<pid>/task/*/stat, utime+stime ticks; 100 ticks ≈ 1 full core-second... i.e. ~500 ticks/5 s = 1 core):

TID      ΔCPU(ticks/5s)  name             wchan
2455915  78              tokio-rt-worker  futex_do_wait
839425   77              tokio-rt-worker  (running)
4164247  74              tokio-rt-worker  poll_schedule_timeout
2455945  74              tokio-rt-worker  futex_do_wait
2455932  73              tokio-rt-worker  poll_schedule_timeout
2455941  52              tokio-rt-worker  futex_do_wait
1325525  45              tokio-rt-worker  futex_do_wait
839486   20              notify-rs inoti  ep_poll
839482   20              notify-rs inoti  ep_poll

Seven tokio workers collectively burn ~95% of one core, continuously, with nothing to do. The two notify-rs inotify threads are also active (~4% each) even though both inotify fds have 0 watches registered:

fd 27: anon_inode:inotify, watches=0
fd 30: anon_inode:inotify, watches=0

This looks like a hot polling loop (timer/futex churn) rather than productive work — consistent with poll_schedule_timeout / futex_do_wait wchans at high CPU.

(strace -c attach was blocked by ptrace_scope on this box, so no syscall histogram.)

5. Log database churn (historical)

The server holds ~/.codex/logs_2.sqlite open (fds 11/13/17/37/38). Current state:

logs_2.sqlite       674,136,064 bytes  (674 MB)
logs_2.sqlite-wal    80,063,992 bytes  ( 80 MB)
logs_2.sqlite-shm       163,840 bytes

Schema/table state (read-only inspection):

tables: _sqlx_migrations, sqlite_sequence, logs
logs:  0 rows
sqlite_sequence('logs') = 6,932,082,655

The logs table is empty (retention pruning works) but the AUTOINCREMENT sequence shows ~6.9 billion rows have been inserted into it since the DB was created on 2026-04-18 (~900 inserts/sec sustained average across all codex processes on this machine). The file itself has bloated to 674 MB despite holding 0 rows — it is never VACUUMed. A 10-second live sample during the idle spin showed 0 inserts/sec, so the current CPU burn is not log writing; the DB bloat is a secondary issue.

• stdout/stderr of the binary go to a pipe held by the node wrapper; no on-disk text log was produced (~/.codex/log/ only contains an old codex-login.log).

Expected behavior

An app-server with no connected clients should sit at ~0% CPU (epoll wait) and hold a bounded, small heap. Also, logs_*.sqlite should not grow to 674 MB while containing 0 rows (needs periodic VACUUM/truncate or size cap).

Actual behavior

1. Busy loop: ~90% of one core, sustained for 5+ days, with 0 connections (≈115 core-hours wasted).
2. Memory leak: ~4.8 GB anonymous heap on an idle server, pushing the host into swap exhaustion.
3. Secondary: logs_2.sqlite bloated to 674 MB with 0 live rows (no vacuum/size cap after retention pruning).

Repro / notes

• Not deterministic to reproduce on demand; this instance was launched on 2026-07-10 (likely by an IDE/agent integration) and orphaned (PPID 1). Multiple other short-lived codex mcp-server instances run on the same machine without this symptom.
• The process ignored SIGTERM (both the node wrapper and the vendored binary were still running 2+ seconds after kill); it had to be terminated with SIGKILL. Killing it immediately returned ~3 GB RSS and ~1.3 GB of swap to the system.