CLI Bug — musl allocator causes futex storm and TUI unresponsiveness on many-core Linux

What version of Codex CLI is running?

0.150.1

What subscription do you have?

Pro x20

Which model were you using?

gpt-5.6-sol (xhigh reasoning)

What platform is your computer?

Linux 7.0.0-28-generic x86_64 x86_64 — Ubuntu 24.04.4 LTS, AMD Ryzen Threadripper PRO 3975WX (32 cores / 64 threads), 125 GB RAM

What terminal emulator and version are you using (if applicable)?

Ghostty, with zellij 0.45.0 as multiplexer

Codex doctor report

What issue are you seeing?

On a 64-thread Linux machine the CLI periodically stops responding to keyboard input for seconds to minutes at a time. While unresponsive the process burns 100–450 % CPU, of which roughly half is kernel time. It happens on fresh sessions (within four minutes of start) as well as long-running ones, with and without background terminals or subagents.

Root cause: the published Linux binary is a statically linked musl build
with no custom global allocator. This is not a niche configuration — in the
rust-v0.150.1 release every single Linux artifact is musl
(codex, codex-app-server, codex-code-mode-host, codex-package, bwrap,
codex-responses-api-proxy, all x86_64- and aarch64-unknown-linux-musl).
The only -gnu asset in the whole release is the internal argument-comment-lint
tool. So every Linux user is on the affected build and there is no official
alternative. musl's bundled allocator serialises concurrent allocation; on a many-core host this becomes a futex storm, and worker threads spend most of their time in the kernel handing the lock around instead of running tasks. The input task is queued behind that, so it never gets to read(0) — the TUI looks frozen while the process is busy.

Related: this is very likely the root cause behind #23702 ("CLI/TUI stops reading stdin mid-session"), where the reporter captured the input reactor thread stuck in futex_wait_queue while a healthy sibling process sat in epoll_pwait, and observed the same oversized logs_2.sqlite / WAL. Filing separately because the defect is broader than that one symptom: it affects every many-core Linux host regardless of whether the visible failure is a frozen TUI, slow session load, or general sluggishness.

perf record -F 299 -p <pid> -g -- sleep 20 during a stall (11 564 samples, 0 lost) shows the lock path rather than application work:

 6.26 %  tokio-rt-worker  codex  [.] 0x000000000cdfa17c
 1.92 %  tokio-rt-worker  codex  [.] 0x000000000cdfa184
 1.53 %  tokio-rt-worker  codex  [.] 0x000000000cdfa311
 1.46 %  tokio-rt-worker  [kernel.kallsyms]  [k] futex_wait_setup
 1.33 %  tokio-rt-worker  [kernel.kallsyms]  [k] futex_hash
 1.21 %  tokio-rt-worker  [kernel.kallsyms]  [k] native_queued_spin_lock_slowpath

with the call graph

native_queued_spin_lock_slowpath
  _raw_spin_lock
    raw_spin_rq_lock_nested
      ___task_rq_lock
        try_to_wake_up
          wake_up_q
            futex_wake
              do_futex
                __x64_sys_futex

The hot userspace addresses cluster into two tight groups — six samples spanning 544 bytes, and three spanning 10 bytes — i.e. a compare-and-swap retry loop, not application logic. (The shipped binary is stripped, so no symbols.)

What steps can reproduce the bug?

The pathology scales with TOKIO_WORKER_THREADS, which makes it easy to demonstrate. Measured on one active session, sampling /proc/<pid>/stat fields 14/15 over 8-second windows:

| TOKIO_WORKER_THREADS | CPU | share of time in kernel |
| --- | --- | --- |
| 4 (or unset → 64) | 90 % × 3–4 threads | 48 % |
| 2 | 190–270 % | 25–45 % |
| 1 | 100 % | 4 % |

Minimal standalone reproduction

The same behaviour reproduces without Codex at all, in ~40 lines of Rust — which isolates it to the allocator rather than anything in this codebase:

use std::thread;
use std::time::{Duration, Instant};

fn main() {
    let n: usize = std::env::args().nth(1).and_then(|s| s.parse().ok()).unwrap_or(4);
    let secs: u64 = std::env::args().nth(2).and_then(|s| s.parse().ok()).unwrap_or(5);
    let deadline = Instant::now() + Duration::from_secs(secs);
    let mut handles = Vec::new();
    for _ in 0..n {
        handles.push(thread::spawn(move || {
            let mut sink: u64 = 0;
            let mut bufs: Vec<Vec<u8>> = Vec::with_capacity(64);
            while Instant::now() < deadline {
                for i in 0..64usize {
                    let sz = 32 + (i * 37) % 4096;
                    let mut v = vec![0u8; sz];
                    v[0] = i as u8;
                    sink = sink.wrapping_add(v[0] as u64);
                    bufs.push(v);
                }
                bufs.clear();
            }
            sink
        }));
    }
    let mut total: u64 = 0;
    for h in handles { total = total.wrapping_add(h.join().unwrap()); }
    println!("{total}");
}

Build the same source four ways and run /usr/bin/time -f "%U %S" ./alloctest 8 5:

| build | useful work (user) | kernel | share in kernel |
| --- | --- | --- | --- |
| x86_64-unknown-linux-musl | 7.20 s | 22.70 s | 75 % |
| musl + mimalloc | 39.98 s | 0.00 s | 0 % |
| x86_64-unknown-linux-gnu | 39.98 s | 0.00 s | 0 % |
| gnu + mimalloc | 39.98 s | 0.00 s | 0 % |

Syscall counts over 3 seconds at 2 threads (strace -c -f):

musl:   175 618 syscalls (4.70 s in kernel, 26 066 errors)
glibc:      150 syscalls (0.003 s)

1170× difference. The same ratio holds at 1, 2, 4 and 8 threa

What is the expected behavior?

The TUI should stay responsive to keyboard input, and worker threads should spend their time in userspace rather than in futex_wake / spin_lock. Performance should not degrade as core count rises

Additional information

Two verified fixes

Both were built and run on the machine above. In live sessions — including large plans and subagents over ~20 minutes — kernel share stays at 4–5 %, where the musl build reached 41–48 %.

1. Build for x86_64-unknown-linux-gnu.
2. Keep musl and link mimalloc — preferable, since the release platform does not change. Three lines:

--- a/codex-rs/Cargo.toml
+++ b/codex-rs/Cargo.toml
 memchr = "2.7.6"
+mimalloc = "0.1.52"
 mime_guess = "2.0.5"

--- a/codex-rs/cli/Cargo.toml
+++ b/codex-rs/cli/Cargo.toml
+[target.'cfg(target_env = "musl")'.dependencies]
+mimalloc = { workspace = true }

--- a/codex-rs/cli/src/main.rs
+++ b/codex-rs/cli/src/main.rs
+#[cfg(target_env = "musl")]
+#[global_allocator]
+static GLOBAL_ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;

CI note: building mimalloc for the musl target needs a musl-targeting C compiler (musl-gcc from musl-tools on Debian/Ubuntu), with CC_x86_64_unknown_linux_musl=musl-gcc.

The SQLite correlation is a red herring

Worth stating explicitly, since #23702 suspected it. The oversized databases are real (logs_2.sqlite 312 MB, -wal 86 MB here; #23702 reported 479 MB / 102 MB) but they are not the cause:

• After vacuuming everything (logs_2 down to 288 KB, WAL to 1.7 MB), a freshly started session was pegged again within four minutes.
• During a stall, per-thread accounting shows sqlx-sqlite-worker threads at 0 % while codex-main and tokio-rt-worker threads burn 90 % each.
• Subagent processes sit at 0 % and hold no database connections.

The WAL growth is a real separate issue, but it is a symptom of the same slowdown rather than its cause.

Notes for anyone reproducing this from source

Three things cost me time and are worth stating:

• Build the tag matching your installed version, not main. main may carry

features whose helper binaries the release does not have locally buildable —
e.g. codex-code-mode-host embeds V8 and its build fetches a prebuilt archive
that currently 404s (librusty_v8_ptrcomp_sandbox_release_x86_64-unknown-linux-gnu.a.gz).

• A source build of --bin codex alone is not a drop-in replacement: the helper

binaries must sit next to it. They can be taken from the release assets
(codex-code-mode-host-x86_64-unknown-linux-musl.tar.gz) rather than rebuilt.

• On Ubuntu 24.04 the linux-sandbox crate needs libcap-dev and libseccomp-dev.

Also, since codex-symbols-x86_64-unknown-linux-musl.tar.gz is published with each
release, the addresses above can be resolved to symbol names on your side — the
shipped binary itself is stripped and carries no build-id, so perf cannot match
them automatically.

Why this may not have surfaced internally

Severity scales with core count, and macOS builds use the system allocator — so the problem is invisible on a typical developer laptop and acute on many-core Linux hosts, which is exactly where people run several concurrent sessions.

Workaround for other users meanwhile

TOKIO_WORKER_THREADS=1 takes kernel share from ~48 % to ~4 %, but a single-lane runtime blocks the UI during any CPU-bound task; 2 is a reasonable middle ground. Neither eliminates the bursts — only the allocator change does.