Always-on LD_LIBRARY_PATH stripping causes 11x performance regression for CUDA/MKL users

Resolved 💬 10 comments Opened Jan 8, 2026 by johnzfitch Closed Jan 9, 2026

💡 Likely answer: A maintainer (etraut-openai, contributor) responded on this thread — see the highlighted reply below.

What version of Codex is running?

0.00

What subscription do you have?

Pro

Which model were you using?

gpt-5.2

What platform is your computer?

Linux 6.17.9-arch1-1 x86_64

What issue are you seeing?

Pre-main hardening in release builds strips LD_LIBRARY_PATH unconditionally, causing severe performance regressions and broken functionality for:

CUDA workloads: 11-300x slower (GPU libraries not found, fallback to CPU)
Intel MKL workloads: 11x slower (optimized BLAS not found, slow fallback)
Conda environments: Legacy installations relying on LD_LIBRARY_PATH fail
Enterprise deployments: Custom DB drivers, Oracle clients fail to load
HPC clusters: Module systems and custom toolchains broken

This regression was introduced in PR #4521 (commit b8e1fe60c) which made codex_process_hardening::pre_main_hardening() always-on in release builds.

The hardening strips all LD_* environment variables before main(), and because Codex spawns child processes with the parent environment, every child inherits the stripped values.

What steps can reproduce the bug?

Install Codex release build (non-debug)
Set LD_LIBRARY_PATH to custom CUDA installation:

export LD_LIBRARY_PATH=/opt/cuda/lib64:$LD_LIBRARY_PATH

Run a CUDA workload:

codex exec "python3 -c 'import torch; print(torch.cuda.is_available())'"

Observe: Returns False (CUDA libraries not found)
Verify LD_LIBRARY_PATH is stripped:

codex exec "echo \$LD_LIBRARY_PATH"

Output: (empty or different from parent)

Performance regression can be measured with:
time codex exec "python3 -c 'import numpy as np; np.dot(np.random.rand(1000,1000), np.random.rand(1000,1000))'"

What is the expected behavior?

LD_LIBRARY_PATH should be preserved by default to support legitimate use cases (CUDA, MKL, custom libraries).

Users who need maximum security can opt-in via environment variable:
CODEX_SECURE_MODE=1 codex exec "..."

This approach:

Restores performance and functionality for CUDA/MKL users
Maintains strong security through existing Landlock/Seatbelt boundaries
Allows opt-in maximum hardening when truly needed
Has zero breaking changes for existing users

Additional information

I have a tested fix ready for PR submission:

Makes pre_main_hardening() opt-in via CODEX_SECURE_MODE=1 environment variable
Default mode: Preserves LD_LIBRARY_PATH (fast, compatible)
Secure mode: Strips LD_* variables (maximum hardening)
Only 2 files modified: cli/src/main.rs and responses-api-proxy/src/main.rs
Security maintained through Landlock/Seatbelt (primary boundaries)

This affects users in ML/AI research, scientific computing, HPC, and enterprise environments for the most part.

---
TL;DR: The pre-main hardening regression is literally a "ghost" - it executes before main(), strips the environment, and then disappears, leaving no trace except confused users reporting slowness.

Reference documentation:

GITHUB_ISSUE_DETAILED.md (comprehensive analysis)
Ghost in the Codex Machine (Hypothesis & Theory)

View original on GitHub ↗

10 Comments

etraut-openai contributor · 6 months ago

Thanks for the bug report and the detailed analysis. Very helpful.

We want to be secure by default, so making "secure mode" opt-in isn't a very palatable solution. There are other potential solutions we could consider though. An opt-out approach is one. Or maybe we could filter LD_LIBRARY_PATH based on known-good paths.

bolinfest collaborator · 6 months ago

@johnzfitch Thanks for the write-up: this is certainly a surprising result.

Does this mean that you are using codex-aarch64-unknown-linux-gnu instead of codex-aarch64-unknown-linux-musl? Because LD_LIBRARY_PATH should have no effect on musl binaries, correct?

bolinfest collaborator · 6 months ago

@viyatb-oai curious to get your thoughts on this one and https://github.com/openai/codex/issues/8472

johnzfitch · 6 months ago

@bolinfest

Does this mean that you are using codex-aarch64-unknown-linux-gnu instead of codex-aarch64-unknown-linux-musl? Because LD_LIBRARY_PATH should have no effect on musl binaries, correct?

Yes, that is correct.

Official Linux release artifacts are *-unknown-linux-musl (source builds typically default to *-unknown-linux-gnu unless you pass a musl target), and you're right: LD_LIBRARY_PATH doesn't affect a static musl Codex binary itself.

The regression is that we strip all LD_* vars from Codex's own process environment before main(), and that stripped environment is what every child process inherits.

Those children (Python/Conda/NumPy/PyTorch, often glibc-linked) can genuinely depend on LD_LIBRARY_PATH for CUDA/MKL/non-RPATH setups, so things break-and it's 100% silent.
***
@etraut-openai

Thanks for the bug report and the detailed analysis. Very helpful. We want to be secure by default, so making "secure mode" opt-in isn't a very palatable solution. There are other potential solutions we could consider though. An opt-out approach is one. Or maybe we could filter LD_LIBRARY_PATH based on known-good paths.

On "secure by default": agreed.

A good middle ground is to always strip the true injection knobs (LD_PRELOAD/LD_AUDIT/DYLD_INSERT_LIBRARIES, etc.) but preserve LD_LIBRARY_PATH by default (or restore it only for tool-spawned children), and reserve full LD_* stripping for an explicit secure mode.

Filtering LD_LIBRARY_PATH to "known-good" paths is possible, but "good" is highly user/enterprise-specific, so we'd need to be careful and explicit if we modify it.

johnzfitch · 6 months ago

My theory is that this is the ghost behind the Ghost in the Codex Machine investigation last October.

Timeline Correlation

2025-09-30: PR #4521 merges (premainhardening() enabled in release)
2025-10-01: rust-v0.43.0 ships (first affected release)
2025-10-06: first "painfully slow" regression report
2025-10-01 → 2025-10-29: spike in env/PATH inheritance issues (multiple distinct reports)
2025-10-29: emergency PATH/environment fix lands
2025-10 → 2025-11: continued slowness / reconnecting / hang complaints
2025-11-05: "re-connecting" behavior fix lands

Platform Distribution (consistent with the env stripping issues)

macOS: #6012, #5679, #5339, #6243, #6218

pre_main_hardening() strips DYLD_* (macOS analog to LD_*)
Can break/alter dynamic linking and toolchains depending on user setup

Linux / WSL2: #4843, #3891, #6200, #5837, #6263

pre_main_hardening() strips LD_* (including LD_LIBRARY_PATH)
Can break CUDA/MKL/Conda/custom libs (esp. non‑RPATH installs), often manifesting as silent performance regressions

Why this was so hard to catch

Runs pre-main() (via #[ctor::ctor]), so it’s easy to miss with standard logging/instrumentation
Silent: no warning when variables are stripped
Only hits certain setups (non‑RPATH, custom enterprise libs, CUDA/MKL)
Users see LD_LIBRARY_PATH set in their shell, but inside codex exec it’s empty
The shell shows LD_LIBRARY_PATH is set, but inside Codex it's stripped

Supporting Evidence (Issue Reports / Timeline)

bolinfest collaborator · 6 months ago

@johnzfitch wow, this an amazing piece of investigative work: thank you!

The codex_process_hardening::pre_main_hardening() library function I added to Codex CLI in https://github.com/openai/codex/pull/4521 was originally designed with another executable in mind, our Responses API proxy:

https://github.com/openai/codex/tree/main/codex-rs/responses-api-proxy

As explained in:

https://github.com/openai/codex/tree/main/codex-rs/responses-api-proxy#hardening-details

Process hardening is "non-negotiable" for that binary because it was written primarily for https://github.com/openai/codex-action, which has the user's OPENAI_API_KEY in memory and we don't want a bad actor to be able to leak it.

Admittedly, Codex CLI likely also has your Codex auth token in memory (which seemed like a good reason to add hardening), though for end-users, it is likely already sitting in ~/.codex/auth.json, which is much easier for an attacker to define.

Based on all of the information you presented, it feels like we should remove codex_process_hardening::pre_main_hardening() ASAP and if we want to revisit hardening later, then we should consider your suggestions above (i.e., remove LD_LIBRARY_PATH from the Codex process, but allow processes spawned via the shell tool to inherit it).

bolinfest collaborator · 6 months ago

Just put up this PR to address this: https://github.com/openai/codex/pull/8951.

johnzfitch · 6 months ago

Verified: rust-v0.80.0-alpha.5 fixes the regression

Test Environment:

Platform: Arch Linux 6.17.9-arch1-1 x86_64
Binaries tested: codex-x86_64-unknown-linux-musl, codex-x86_64-unknown-linux-gnu

---

### Test 1: LD_LIBRARY_PATH Preservation

| Binary | LD_LIBRARY_PATH | Result |
|--------|-----------------|--------|
| MUSL | /opt/cuda/lib64:/opt/miniconda3/lib | PRESERVED |
| GNU | /opt/cuda/lib64:/opt/miniconda3/lib | PRESERVED |

---

### Test 2: CUDA Library Loading

| Library | Status |
|---------|--------|
| libcudart.so | LOADED |
| libcublas.so | LOADED |
| libcufft.so | LOADED |

---

### Test 3: MKL Performance Benchmark (10x 2000x2000 matmul)

| Configuration | Run 1 | Run 2 | Run 3 | Average | Speedup |
|---------------|-------|-------|-------|---------|---------|
| MKL via Codex (fixed) | 0.308s | 0.302s | 0.308s | 0.306s | 53x |
| System cblas (fallback) | 16.435s | 16.317s | 16.198s | 16.317s | 1x |

---

### Test 4: PyTorch CPU Benchmark

| Test | Result |
|------|--------|
| PyTorch version | 2.9.1 |
| CUDA libs loaded | Yes |
| CUDA device access | Blocked by sandbox (expected) |
| CPU matmul (10x 2000x2000) | 0.155s |

---

Summary:
Environment variables are correctly preserved and inherited by child processes.
MKL-accelerated workloads run 53x faster than fallback.
CUDA libraries load successfully via LD_LIBRARY_PATH.

The fix works.

<details>
<summary>Full Verification Logs</summary>

### Version
$ /tmp/alpha5/codex-x86_64-unknown-linux-musl --version
codex-cli 0.80.0-alpha.5

### Test 1: LD_LIBRARY_PATH Preservation (MUSL)
```bash
$ LD_LIBRARY_PATH=/opt/cuda/lib64:/opt/miniconda3/lib codex exec "echo \ $LD_LIBRARY_PATH"

/usr/bin/zsh -lc 'echo "$LD_LIBRARY_PATH"' succeeded in 2ms:
/opt/cuda/lib64:/opt/miniconda3/lib

Test 2: LD_LIBRARY_PATH Preservation (GNU)

$ LD_LIBRARY_PATH=/opt/cuda/lib64:/opt/miniconda3/lib /tmp/alpha5-gnu/codex-x86_64-unknown-linux-gnu exec "echo \$LD_LIBRARY_PATH"

/usr/bin/zsh -lc 'echo LD_LIBRARY_PATH=$LD_LIBRARY_PATH' succeeded in 3ms:
LD_LIBRARY_PATH=/opt/cuda/lib64:/opt/miniconda3/lib

Test 3: CUDA Library Loading

$ LD_LIBRARY_PATH=/opt/cuda/lib64 codex exec "python3 -c \"import ctypes; ctypes.CDLL('libcudart.so'); ctypes.CDLL('libcublas.so')\""

/usr/bin/zsh -lc "python3 -c \"import ctypes; print('libcudart:', ctypes.CDLL('libcudart.so')); print('libcublas:', ctypes.CDLL('libcublas.so'))\"" succeeded in 39ms:
libcudart: <CDLL 'libcudart.so', handle 55ccf5b0d1d0 at 0x7ffbf7937770>
libcublas: <CDLL 'libcublas.so', handle 55ccf5b47630 at 0x7ffbf7579950>

Test 4: CUDA Library Proof (Full)

$ LD_LIBRARY_PATH=/opt/cuda/lib64:/opt/miniconda3/lib codex exec "python3 cuda_lib_proof.py"

LD_LIBRARY_PATH is SET: /opt/cuda/lib64:/opt/miniconda3/lib

Paths included:
[EXISTS] /opt/cuda/lib64
[EXISTS] /opt/miniconda3/lib

CUDA LIBRARY LOADING TEST
[OK] libcudart.so: LOADED via search path
[OK] libcublas.so: LOADED via search path
[OK] libcufft.so: LOADED via search path

Test 5: MKL Performance (via Codex with LD_LIBRARY_PATH)

$ LD_LIBRARY_PATH=/opt/miniconda3/lib codex exec "/opt/miniconda3/bin/python matmul_benchmark.py"

NumPy: 2.3.5
BLAS: mkl-sdl (version: 2025)

Run 1: 0.308s
Run 2: 0.302s
Run 3: 0.308s
Average: 0.306s (10x 2000x2000 matmul)

Test 6: System cblas Performance (baseline - no MKL)

$ /usr/bin/python3 matmul_benchmark.py

NumPy: 2.3.5
BLAS: cblas

Run 1: 16.435s
Run 2: 16.317s
Run 3: 16.198s
Average: 16.317s (10x 2000x2000 matmul)

Test 7: PyTorch CPU Benchmark

$ LD_LIBRARY_PATH=/opt/cuda/lib64:/opt/miniconda3/lib codex exec "python3 pytorch_benchmark.py"

PyTorch: 2.9.1
CUDA available: False (sandbox blocks device access - expected)
CPU (10x 2000x2000 matmul): 0.155s

Summary

LD_LIBRARY_PATH (MUSL): ✓ PRESERVED
LD_LIBRARY_PATH (GNU): ✓ PRESERVED
CUDA libraries: ✓ LOADED (libcudart, libcublas, libcufft)
MKL performance: ✓ 53x faster than fallback

bolinfest collaborator · 6 months ago

@johnzfitch awesome: glad to hear it! I need to audit the other changes that have landed on main since the last release. Assuming there isn't anything that needs more bake time, we should be able to promote this to 0.80.0-alpha.5 as 0.80.0 tomorrow!

johnzfitch · 6 months ago

@bolinfest Thanks for prioritizing this - really appreciate the quick turnaround!

The 53x performance improvement for MKL workloads is significant. CUDA/ML users will definitely appreciate this in 0.80.0.