codex app-server fails at startup when getcwd(2) returns (unreachable)/...

codex app-server fails at startup when getcwd(2) returns (unreachable)/…

Summary

codex app-server exits immediately with

Error: error loading default config after config error:
No such file or directory (os error 2)

whenever it's spawned in a process whose kernel-level getcwd(2) returns a path prefixed with (unreachable) — a state that occurs routinely inside nested Linux mount namespaces (systemd services with PrivateMounts=yes, bwrap, LXC-style dev containers, etc.). glibc-linked applications never observe this because glibc's getcwd() wrapper silently rewrites the result. codex is built with musl static-PIE, and musl passes the raw syscall result through unchanged.

Result: codex sees env::current_dir() → "(unreachable)/home/user/src", tries to read config paths joined onto that, hits ENOENT on every one, and dies.

Environment

• codex: codex-cli 0.146.0-alpha.3.1, VS Code extension openai.chatgpt-26.721.41059-linux-x64
• Binary: static-pie linked, stripped, target x86_64-unknown-linux-musl (confirmed via embedded compiler line: --target=x86_64-unknown-linux-musl)
• Host: Ubuntu 26.04, kernel 7.0.0-28-generic, glibc 2.43
• Launch context: VS Code running inside a systemd service unit with PrivateMounts=yes + ProtectHome=tmpfs + BindPaths=…, entered via nsenter --mount --pid --root --wd=/

Reproduction

Any environment where the process's fs_struct produces (unreachable)-prefixed getcwd will trigger it. Minimal repro without a full ns:

# 1. Enter a private mount ns and chroot into it, then verify kernel says (unreachable)
sudo unshare --mount --propagation private bash -c '
    mount --bind /home /home    # detach /home from the outer tree
    umount -l /home             # sever the propagation link
    cd /home/$USER
    # this syscall returns "(unreachable)/home/user" from the kernel:
    strace -e getcwd true 2>&1 | grep getcwd
'
# 2. In the same context, run codex — it dies with the config-not-found error above.

Simpler observational reproduction inside our real setup:

$ strace -e getcwd /path/to/codex app-server 2>&1 | grep getcwd
getcwd("(unreachable)/home/user/src", 512) = 27
$ echo "codex stderr:" ; /path/to/codex app-server 2>&1 <&- | head -1
Error: error loading default config after config error: No such file or directory (os error 2)

Meanwhile, in the exact same process context, glibc-linked binaries see a clean path:

| Binary | Kernel getcwd(2) return | Userspace value |
|---|---|---|
| Dynamic C, glibc 2.43 | (unreachable)/home/user/src | /home/user/src |
| Static C, glibc-static | (unreachable)/home/user/src | /home/user/src |
| Python 3 (dynamic glibc) | (unreachable)/home/user/src | /home/user/src |
| codex (static musl) | (unreachable)/home/user/src | (unreachable)/home/user/src |

Root cause

glibc's getcwd() wrapper, since 2.28, detects the (unreachable) prefix returned by the getcwd(2) syscall and falls back to readlink("/proc/self/cwd") to obtain a canonicalized absolute path. musl's getcwd() (src/unistd/getcwd.c) does not implement this workaround — it returns whatever the syscall wrote to buf. Rust's std::env::current_dir() on the x86_64-unknown-linux-musl target calls libc::getcwd, which is musl's, so the (unreachable) prefix ends up in the returned PathBuf. codex then joins its config filenames onto it and every subsequent read yields ENOENT.

The kernel returns the prefix legitimately: it means the process's cwd dentry can't be walked back to the process's root dentry through the current mount namespace's mount tree. It's the kernel signalling "I can't render this path such that opening it would land you in the same place." An application can either accept the fact and use /proc/self/cwd as an alternative anchor, or treat it as a real error. Silently building further paths on it is the one thing that never works.

Suggested fix

Two possible layers:

1. In codex (most local, fastest to ship): after env::current_dir(), detect a leading (unreachable) component and either fall back to std::fs::read_link("/proc/self/cwd") (mirroring glibc's behavior) or surface a clear, actionable error message instead of the current cascade through the config loader.
2. In rust-lang/rust (broader fix that helps other musl-static Rust apps): normalize std::env::current_dir() on musl targets to strip the (unreachable) prefix or perform the /proc/self/cwd fallback in the stdlib. There's precedent — glibc already does this transparently — and the current asymmetry means Rust apps that work fine on glibc silently break when rebuilt against musl in nested-ns environments.

Either fix would eliminate the class of failures. Doing (1) unblocks users immediately; (2) is the durable fix.

Local workaround

For anyone hitting this before a fix ships: ensure the codex process's cwd is set by an explicit userspace chdir() after the mount-namespace entry, not just inherited. In our systemd + nsenter case, changing

exec nsenter --target "$pid" --mount --pid --root --wd=/home/user -- \
    env HOME=/home/user "$@"

to

exec nsenter --target "$pid" --mount --pid --root -- \
    env HOME=/home/user \
    /bin/bash -c 'cd /home/user && exec "$@"' -- "$@"

fixes it: bash's chdir() (unlike nsenter --wd) produces a cwd dentry that the kernel can walk back to the fs_struct's root inside the child namespace, so getcwd(2) returns a clean path. As a per-app alternative, a two-line wrapper set as chatgpt.cliExecutable that does cd "$CODEX_HOME" before execing the real binary works identically.