http-client: rustls root store + ClientConfig rebuilt from native certs on every websocket connect / client build (no memoization)

Summary

build_rustls_client_config() (codex-rs/http-client/src/custom_ca.rs) rebuilds the entire trust store from scratch on every call: it calls rustls_native_certs::load_native_certs(), allocates a fresh RootCertStore, parses every platform root cert into it, and builds a new rustls::ClientConfig. There is no process-level memoization, even though the result is identical for the lifetime of the process (it depends only on the platform trust store plus CODEX_CA_CERTIFICATE / SSL_CERT_FILE).

Every websocket connection pays this cost, because WebSocketConnector::new() → build_rustls_client_config_with_custom_ca() is called per connect, not per process:

• codex-rs/websocket-client/src/lib.rs (WebSocketConnector::new_with_tls_mode, WebSocketTlsMode::ExplicitCodexTls)
• codex-rs/codex-api/src/endpoint/responses_websocket.rs::connect_websocket — called for every Responses websocket connect (prewarm + every needs_new reconnect, via ModelClient::connect_websocket in codex-rs/core/src/client.rs)
• codex-rs/websocket-client/src/dialer.rs also calls it again when tunnelling through an HTTPS proxy and no config was supplied
• codex-rs/codex-api/src/endpoint/realtime_websocket/methods.rs

The equivalent cost exists on the reqwest path (build_reqwest_client_with_custom_ca), which re-reads/re-parses the configured CA PEM per client build. #29369 covers the reqwest client/connection-pool half of this; this issue is specifically about the unmemoized native-root RootCertStore + ClientConfig, which #29369 does not cover and which also affects the websocket path that #29369 explicitly says is shielded.

Measurements

Standalone measurement of exactly what build_rustls_client_config does (rustls_native_certs::load_native_certs() → RootCertStore::add_parsable_certificates → ClientConfig::builder().with_root_certificates(..).with_no_client_auth()), Ubuntu 24.04 x86-64, 146 certs in /etc/ssl/certs/ca-certificates.crt, warm page cache, release build:

per build:            ~1.5 ms CPU
retained per config:  ~102 KB RSS (200 live configs => +20.3 MB)

1.5 ms per connect sounds small, but it is fixed setup cost paid on the hot path, it is single-threaded work per connection, and it lands in bursts: when N threads/turns start concurrently, all N pay it at the same time, contending on the same crypto provider/global initialization. In a headless multi-tenant harness this showed up as visible CPU spikes at turn-start rather than as steady-state cost.

Production impact (downstream harness)

We run codex app-server headless on Linux x86-64 as an agent fleet (many short-lived threads/turns per process, 6 worker containers, ~50 concurrent turns per container). On rust-v0.146.0, profiling (perf + strace) attributed the largest per-turn CPU cost to this per-client trust-store rebuild — in that version it sat in codex-rs/http-client/src/client_builder.rs on the client-build path, so it was paid per turn, not just per websocket connect.

Caching the ClientConfig once per process in a LazyLock and cloning the Arc into each client, together with caching the user-agent OS probe (filed separately) and disabling unused features, took us from 0.36–0.39 CPU-seconds per turn at ~250 MB RSS to 0.15 CPU-s/turn at ~128 MB RSS on real authenticated turns (~60% less CPU, ~half the memory). In a model-free benchmark that isolates app-server overhead (mock Responses API, 64 turns at concurrency 4), the same patches cut CPU per turn from 0.355 to 0.026 CPU-s (−93%) and peak RSS from 209 MB to 144 MB. Those numbers combine several changes, so treat them as the ceiling for the whole set rather than for this one fix; the isolated cost of this specific code path is the 1.5 ms / 102 KB per build measured above.

Proposed fix

Memoize the config at process level and clone the Arc for each client/connector:

static SHARED_TLS_CONFIG: LazyLock<Option<Arc<rustls::ClientConfig>>> = LazyLock::new(|| {
    ensure_rustls_crypto_provider();
    build_rustls_client_config_with_custom_ca().ok()
});

Arc<ClientConfig> is already what callers hold, so this is a drop-in change: WebSocketConnector and the reqwest path clone the shared Arc instead of rebuilding. On the reqwest side the same shared config can be installed via ClientBuilder::use_preconfigured_tls, which also removes the per-build CA-file re-read.

Two details worth deciding explicitly:

• Keying: if a variant must stay dynamic (custom CA path/fingerprint, sandboxed vs not), a small OnceLock/map keyed by the CA bundle path + mtime keeps correctness while still eliminating the common case. Callers that need structured CA-failure handling can keep using the uncached build_rustls_client_config_with_custom_ca().
• Reloading: today a rotated system trust store is picked up on the next client build. Caching for the process lifetime changes that, which seems acceptable for a CLI/app-server but should be a conscious choice (an explicit invalidation hook would preserve it).

Happy to send a PR if you'd like this shape.

Environment

• Repo state inspected: main @ 0042b00986b9cc73c82c93f94e93d747818228be
• Production numbers measured against a patched fork of tag rust-v0.146.0
• Linux x86-64 (Ubuntu 24.04), headless codex app-server via JSON-RPC (no TUI)