app-server: app/installed rebuilds the codex_apps MCP session per call, pinning ~120% of a core while idle

Summary

On a Linux host running codex app-server --listen unix:// for a Codex Desktop SSH-remote session, the app-server sat at 120–127% of one CPU core while completely idle (no active turn, no user present). The cause is that app/installed / app/list requests rebuild the entire codex_apps MCP session from scratch on every call, including a full system-CA-bundle re-parse and a fresh TLS handshake, and then drop the RunningService without close().

Restarting the app-server does not help — the Desktop client reconnects and resumes the same poll loop. Dropping the client connection (killing the codex app-server proxy relay) stops it immediately.

This is adjacent to #30606 but a different mechanism: no ambient-suggestion turns are involved here, and the hot path is MCP session construction rather than SQLite log churn.

Environment

• Remote: Debian, kernel 6.12.95, x86_64
• codex-cli 0.144.0, vendored musl build (codex-linux-x64/vendor/x86_64-unknown-linux-musl/bin/codex)
• Listener: codex -c features.code_mode_host=true app-server --listen unix://
• Client: Codex Desktop over SSH remote, app_server.client_name="Codex Desktop"
• Enabled bundled plugins: sites@openai-bundled, visualize@openai-bundled (these back the codex_apps MCP server, 45 tools)

Observed behaviour

The client repeatedly issued only these requests — no others — from a single connection:

app/installed          0.40/s
app/list               0.10/s
mcpServerStatus/list   0.02/s

Each one drove a complete codex_apps rebuild cycle:

1. New HTTPS client built per call, re-reading and re-parsing the system CA bundle:

``
INFO codex_http_client::custom_ca transport_worker{name="StreamableHttpClientWorker"}:
using system root certificates because no CA override environment variable was selected
codex_ca_certificate_configured=false ssl_cert_file_configured=false
``
Measured at 4.7/s.

1. No connection reuse — the whole pool is torn down each cycle:

``
TRACE hyper_util::client::legacy::pool pool dropped, dropping pooled (("https", chatgpt.com))
TRACE hyper_util::client::legacy::pool pool closed, canceling idle interval
TRACE hyper_util::client::legacy::pool checkout waiting for idle connection: ("https", chatgpt.com)
TRACE hyper_util::client::legacy::connect::http Http::connect; scheme=Some("https"), host=Some("chatgpt.com")
``

1. Full initialize + ListResources + ListTools every time, with a ~29.5 KB ListResources payload serialized into a TRACE record:

``
TRACE rmcp::service new{server_name=codex_apps}:start_server_task{server_name=codex_apps}:initialize:serve_inner:
new event evt=PeerMessage(Response(JsonRpcResponse { ... ListResourcesResult { ... resources: [...] } }))
``

1. 664 KB tool-schema cache rewritten each cycle at ~/.codex/cache/codex_apps_tools/<hash>.json.

1. RunningService dropped without close(), cancelling the transport:

``
DEBUG rmcp::service RunningService dropped without explicit close(). The connection will be
closed asynchronously. For guaranteed cleanup, call close() or cancel() before dropping.
INFO rmcp::service ...:serve_inner: serve finished quit_reason=Cancelled
INFO rmcp::service ...:serve_inner: task cancelled
DEBUG rmcp::transport::worker worker quit with reason: Cancelled
DEBUG rmcp::transport::streamable_http_client transport_worker{name="StreamableHttpClientWorker"}: cancelled
``

1. The tool cache is never used. On every single pass:

``
TRACE codex_mcp::connection_manager list_all_tools{mcp_server_count=1}: waiting for MCP server tools
while building tool list server_name=codex_apps has_cached_tools=true startup_complete=false
TRACE codex_mcp::connection_manager list_all_tools{mcp_server_count=1}: listed MCP server tools
while building tool list server_name=codex_apps tool_count=45
`
startup_complete=false never latches true despite has_cached_tools=true`, so the fast path is dead code in this state.

Notably there were zero WARN and zero ERROR log entries throughout. The transport is not failing on the network — Codex cancels it itself, every cycle.

Measurements

Taken on the live process, with care not to attribute diagnostic overhead to the workload.

| Metric | Runaway | After dropping the client connection |
|---|---|---|
| app-server CPU | 120–127% of one core (sustained, 30 s windows) | 0.0–0.4% |
| Log rows written to logs_2.sqlite | 21.4 rows/s | 0.0 rows/s |
| CA-bundle reloads | 4.7/s | 0 |
| rchar | 3.0 MB/s (read_bytes = 0 — entirely page cache) | ~0 |
| write_bytes | 0.5 MB/s | ~0 |
| RSS | 278 ↔ 355 MB sawtooth, ~12 s period | flat |
| Voluntary ctx switches | ~1,600/s per worker, ~13k/s across 8 workers | idle |
| CPU package temp | 82–87 °C | 56–58 °C |

Load was spread evenly across all 8 tokio-rt-worker threads (work stealing), which is why no single thread looked hot in top -H and why this is easy to misread as a runtime-level spin rather than a repeated workload. The sqlx-sqlite-worker threads were cold (~70 jiffies total), ruling out query execution as the hot path.

A healthy reconnected client costs a single ~5 s burst peaking at 237% during handshake, then settles to 0%. So client presence is not the problem — only the stuck poll loop is.

Reproduction

1. Run codex app-server --listen unix:// on a Linux remote with the bundled sites / visualize plugins enabled.
2. Connect Codex Desktop to it over SSH remote.
3. Leave it idle. When the client enters the app/installed poll loop, the app-server pins ~120% of a core indefinitely.
4. kill -TERM the codex app-server proxy relay → CPU drops to 0% immediately. Restarting the app-server instead does not help, because the client reconnects into the same loop.

Suggested fixes

1. Reuse a pooled HTTPS client instead of constructing one per MCP session. Rebuilding the rustls root store from the system CA bundle 4.7 times a second is the single largest cost and is pure waste — the trust anchors do not change between calls.
2. Latch startup_complete so has_cached_tools=true actually short-circuits the rebuild in list_all_tools. As written, the cache is populated and then ignored.
3. Call close() on RunningService rather than dropping it — rmcp explicitly warns about this, and the cancel/respawn churn is what multiplies the cost across the Tokio workers.
4. Consider rate-limiting or coalescing app/installed / app/list, so a misbehaving client cannot amplify one poll into a full remote MCP handshake.

Side observation

~/.codex/logs_2.sqlite had grown to 1.76 GB with only 0.37 GB live data — 309,706 of 461,104 pages were freelist (~1.18 GB, 67%, reclaimable). Retention deletes rows but never VACUUMs, so the file only grows. Not the cause of the CPU issue, but it makes every log insert progressively more expensive and is worth a periodic VACUUM or auto_vacuum=INCREMENTAL.