macOS Tahoe: residual native-tls (Secure Transport) connections stall minutes at startup and on quit — coretls TLS-metric ASL logging blocks when diagnosticd is wedged

Open 💬 1 comment Opened Aug 15, 2026 by mapleberry

Summary

On macOS Tahoe 26.6, codex (CLI 0.146.0, npm install) intermittently takes 3–5 minutes to run a trivial codex exec "print ok" (~1s of CPU), and interactive sessions hang for minutes on quit after the answer has already been printed. Stack sampling shows the hang is inside Apple's legacy Secure Transport (coretls): every TLS handshake and every TLS context teardown synchronously logs a "TLS metric" through ASL/MessageTracer, and on this machine that logging call blocks on an XPC round-trip to a diagnostics endpoint that is not answering (diagnosticd shows state = not running, never spawned since boot). One blocked thread holds the process-wide ASL mutex, and every other thread that touches TLS queues behind it.

The model call itself succeeds (rustls path), which matches the known mixed TLS backend situation in codex-rs (#6874, #18772): most connections use rustls, but at least one auxiliary connection still goes through reqwest's default native-tls = Secure Transport on macOS. That residual connection is the one that stalls — at startup (handshake metric) and at shutdown (context-destroy metric, producing the quit hang).

Request: finish the rustls migration (default-features = false on reqwest workspace-wide, rustls-tls explicitly everywhere) so no codex connection depends on Secure Transport. That is the same fix already identified in #6874/#18772 for the TLS-1.3 failures; this issue adds a second, more severe reason for it.

Environment

  • macOS Tahoe 26.6 (Darwin 25.6.0), Apple Silicon
  • codex-cli 0.146.0 via npm (@openai/codex, vendored aarch64-apple-darwin binary)
  • Machine condition that triggers it: launchctl print system/com.apple.diagnosticdstate = not running, last exit code = (never exited); analyticsd stuck at spawn scheduled. (This machine-side wedge is an Apple bug — being reported to Apple separately — but codex is the only tool on the machine that hangs because of it, because almost nothing else still uses Secure Transport.)

Measurements (same command, codex exec -s read-only --skip-git-repo-check "print ok" < /dev/null)

| condition | wall time |
|---|---|
| healthy window (right after reboot) | 6–20 s |
| degraded (returns within ~1–2 h of any reboot) | 28 s → 57 s → 3:00–5:00, monotonically worsening |
| all hang runs | ~1s user CPU; answer often already printed; process alive minutes after |

codex mcp list is always instant (no TLS).

Stack evidence (from sample during hangs)

Startup-side stall — a tokio worker inside the handshake, holding the ASL lock, blocked on XPC:

SSLHandshake (Security)
  SSLHandshakeProceed
    tls_handshake_process (libcoretls)
      SSLProcessHandshakeRecordInner
        SSLAdvanceHandshake
          tls_metric_client_finished
            tls_metric_event_add_string
              asl_log (libsystem_asl)
                _asl_lib_vlog
                  _asl_send_message
                    _asl_mt_shim_send_message
                      _xpc_pipe_routine
                        _xpc_pipe_mach_msg
                          mach_msg   <-- blocked here for minutes

Shutdown-side stall (the quit hang) — TLS teardown takes the same path:

_CFRelease (CoreFoundation)
  SSLContextDestroy (Security)
    tls_handshake_destroy (libcoretls)
      tls_metric_event_done
        asl_log
          _asl_evaluate_send
            _pthread_mutex_firstfit_lock_wait   <-- queued behind the thread above

Additional threads (including one in syslog$DARWIN_EXTSN → _vsyslog → _asl_evaluate_send) pile up on the same mutex.

Ruled out by A/B testing (each falsified with interleaved timed runs + mid-hang samples)

  • mcp_servers.node_repl (the ChatGPT-desktop-injected server, 120s startup timeout): disabling it via -c 'mcp_servers.node_repl.enabled=false' — run still stalled 4:55; no node_repl child process is ever spawned during the hang.
  • features.apps=false (the codex_apps streamable-HTTP connector): stalls occur equally in both arms (16.8s / 3:02.9 disabled vs 5.8s / >3:20 control, same sampled stacks).
  • ASL_DISABLE=1: no effect (the MessageTracer shim path is not gated by it) — 4:31.
  • Machine-side remedies short of an OS fix: reboot (relief decays within ~1–2 h), safe-mode boot cycle, unified-log store reset (/var/db/diagnostics, /var/db/uuidtext) — behavior unchanged. No pending macOS update.

Likely related issues

  • #6874, #18772 — prior failures traced to the residual native-tls/Secure Transport backend on macOS; same proposed fix.
  • #14223 (slow exit), #13715 (local clients hang) — plausibly the same mechanism for the macOS reporters, though those issues lack stack samples.

Full sample output files available on request.

View original on GitHub ↗

1 Comment

mapleberry · 12 days ago

Full sample excerpt from a live hang (codex-cli 0.146.0, macOS 26.6, captured today — thread stacks only, binary-image list omitted):

Call graph:
    2521 Thread_28275   DispatchQueue_1: com.apple.main-thread  (serial)
    + 2521 start  (in dyld) + 6992  [0x183c4c4e4]
    +   2521 main  (in codex) + 52  [0x103baef94]
    +     2521 _RNvNtCsg55jX0GwzBC_3std2rt19lang_start_internal  (in codex) + 952  [0x10c30f2cc]
    +       2521 ???  (in codex)  load address 0x102f4c000 + 0x1e144c0  [0x104d604c0]
    +         2521 ???  (in codex)  load address 0x102f4c000 + 0x1e149b0  [0x104d609b0]
    +           2521 ???  (in codex)  load address 0x102f4c000 + 0x1d642d4  [0x104cb02d4]
    +             2521 _pthread_join  (in libsystem_pthread.dylib) + 616  [0x18401a114]
    +               2521 __ulock_wait  (in libsystem_kernel.dylib) + 8  [0x183fd5af8]
    2521 Thread_28276: codex-main
    + 2521 thread_start  (in libsystem_pthread.dylib) + 8  [0x184012c1c]
    +   2521 _pthread_start  (in libsystem_pthread.dylib) + 136  [0x184017c58]
    +     2521 ???  (in codex)  load address 0x102f4c000 + 0x93cd26c  [0x10c31926c]
    +       2521 ???  (in codex)  load address 0x102f4c000 + 0x1e279d4  [0x104d739d4]
    +         2521 ???  (in codex)  load address 0x102f4c000 + 0x1e14ab8  [0x104d60ab8]
    +           2521 ???  (in codex)  load address 0x102f4c000 + 0x1f11634  [0x104e5d634]
    +             2521 ???  (in codex)  load address 0x102f4c000 + 0x94c5334  [0x10c411334]
    +               2521 ???  (in codex)  load address 0x102f4c000 + 0x94eebdc  [0x10c43abdc]
    +                 2521 ???  (in codex)  load address 0x102f4c000 + 0x94db108  [0x10c427108]
    +                   2521 ???  (in codex)  load address 0x102f4c000 + 0x94db584  [0x10c427584]
    +                     2521 parking_lot::condvar::Condvar::wait_until_internal::hf152c3e27a86a315  (in codex) + 520  [0x10bbb58e0]
    +                       2521 _pthread_cond_wait  (in libsystem_pthread.dylib) + 980  [0x184018128]
    +                         2521 __psynch_cvwait  (in libsystem_kernel.dylib) + 8  [0x183fd750c]
    2521 Thread_28282: tokio-rt-worker
    + 2521 thread_start  (in libsystem_pthread.dylib) + 8  [0x184012c1c]
    +   2521 _pthread_start  (in libsystem_pthread.dylib) + 136  [0x184017c58]
    +     2521 ???  (in codex)  load address 0x102f4c000 + 0x93cd26c  [0x10c31926c]
    +       2521 ???  (in codex)  load address 0x102f4c000 + 0x94e51e4  [0x10c4311e4]
    +         2521 std::sys::backtrace::__rust_begin_short_backtrace::h43c2ee9d40ad8800  (in codex) + 224  [0x10c417df4]
    +           2521 ???  (in codex)  load address 0x102f4c000 + 0x94c6290  [0x10c412290]
    +             2521 ???  (in codex)  load address 0x102f4c000 + 0x94d0e90  [0x10c41ce90]
    +               2521 ???  (in codex)  load address 0x102f4c000 + 0x94ed038  [0x10c439038]
    +                 2521 tokio::runtime::scheduler::multi_thread::worker::run::hb0c4d53bf4dd0239  (in codex) + 2412  [0x10c42a6cc]
    +                   2521 ???  (in codex)  load address 0x102f4c000 + 0x94e13ac  [0x10c42d3ac]
    +                     2521 ???  (in codex)  load address 0x102f4c000 + 0x56a44d0  [0x1085f04d0]
    +                       2521 ???  (in codex)  load address 0x102f4c000 + 0x566b9a8  [0x1085b79a8]
    +                         2521 ???  (in codex)  load address 0x102f4c000 + 0x5666c54  [0x1085b2c54]
    +                           2521 ???  (in codex)  load address 0x102f4c000 + 0x56bb798  [0x108607798]
    +                             2521 ???  (in codex)  load address 0x102f4c000 + 0x566c8a4  [0x1085b88a4]
    +                               2521 ???  (in codex)  load address 0x102f4c000 + 0x5654ddc  [0x1085a0ddc]
    +                                 2521 ???  (in codex)  load address 0x102f4c000 + 0x5652dd0  [0x10859edd0]
    +                                   2521 ???  (in codex)  load address 0x102f4c000 + 0x5655c8c  [0x1085a1c8c]
    +                                     2521 _$LT$reqwest..async_impl..client..PendingRequest$u20$as$u20$core..future..future..Future$GT$::poll::h77fe8c5885c257a6  (in codex) + 116  [0x10bea00e4]
    +                                       2521 ???  (in codex)  load address 0x102f4c000 + 0x8f1da10  [0x10be69a10]
    +                                         2521 ???  (in codex)  load address 0x102f4c000 + 0x8f22054  [0x10be6e054]
    +                                           2521 ???  (in codex)  load address 0x102f4c000 + 0x8eef580  [0x10be3b580]
    +                                             2521 ???  (in codex)  load address 0x102f4c000 + 0x8ef4938  [0x10be40938]
    +                                               2521 ???  (in codex)  load address 0x102f4c000 + 0x8eb294c  [0x10bdfe94c]
    +                                                 2521 ???  (in codex)  load address 0x102f4c000 + 0x8eb2430  [0x10bdfe430]
    +                                                   2521 ???  (in codex)  load address 0x102f4c000 + 0x8ee7b1c  [0x10be33b1c]
    +                                                     2521 ???  (in codex)  load address 0x102f4c000 + 0x8ea39d8  [0x10bdef9d8]
    +                                                       2521 ???  (in codex)  load address 0x102f4c000 + 0x8ea568c  [0x10bdf168c]
    +                                                         2521 ???  (in codex)  load address 0x102f4c000 + 0x8e7f658  [0x10bdcb658]
    +                                                           2521 ???  (in codex)  load address 0x102f4c000 + 0x8e8080c  [0x10bdcc80c]
    +                                                             2521 ???  (in codex)  load address 0x102f4c000 + 0x8f267ec  [0x10be727ec]
    +                                                               2521 SSLHandshake  (in Security) + 228  [0x18795a750]
    +                                                                 2521 SSLHandshakeProceed  (in Security) + 168  [0x18795a870]
    +                                                                   2521 tls_handshake_process  (in libcoretls.dylib) + 112  [0x1930b233c]
    +                                                                     2521 SSLProcessHandshakeRecordInner  (in libcoretls.dylib) + 200  [0x1930b29a4]
    +                                                                       2521 SSLAdvanceHandshake  (in libcoretls.dylib) + 1388  [0x1930afabc]
    +                                                                         2521 tls_metric_client_finished  (in libcoretls.dylib) + 684  [0x1930b6064]
    +                                                                           2521 tls_metric_event_add_string  (in libcoretls.dylib) + 228  [0x1930b774c]
    +                                                                             2521 asl_log  (in libsystem_asl.dylib) + 216  [0x18a7d0698]
    +                                                                               2521 _asl_lib_vlog  (in libsystem_asl.dylib) + 464  [0x18a7d0890]
    +                                                                                 2521 _asl_send_message  (in libsystem_asl.dylib) + 944  [0x18a7d0c94]
    +                                                                                   2521 _asl_mt_shim_send_message  (in libsystem_asl.dylib) + 152  [0x18a7d1630]
    +                                                                                     2521 _xpc_pipe_routine  (in libxpc.dylib) + 384  [0x183d17bd8]
    +                                                                                       2521 _xpc_pipe_mach_msg  (in libxpc.dylib) + 56  [0x183d184ec]
    +                                                                                         2521 mach_msg  (in libsystem_kernel.dylib) + 24  [0x183fd3fc0]
    +                                                                                           2521 mach_msg_overwrite  (in libsystem_kernel.dylib) + 480  [0x183fdc9c0]
    +                                                                                             2521 mach_msg2_internal  (in libsystem_kernel.dylib) + 76  [0x183fe65a4]
    +                                                                                               2521 mach_msg2_trap  (in libsystem_kernel.dylib) + 8  [0x183fd3c34]
    2521 Thread_28284: tokio-rt-worker
    + 2521 thread_start  (in libsystem_pthread.dylib) + 8  [0x184012c1c]
    +   2521 _pthread_start  (in libsystem_pthread.dylib) + 136  [0x184017c58]
    +     2521 ???  (in codex)  load address 0x102f4c000 + 0x93cd26c  [0x10c31926c]
    +       2521 ???  (in codex)  load address 0x102f4c000 + 0x94e51e4  [0x10c4311e4]
    +         2521 std::sys::backtrace::__rust_begin_short_backtrace::h43c2ee9d40ad8800  (in codex) + 224  [0x10c417df4]
    +           2521 ???  (in codex)  load address 0x102f4c000 + 0x94c6290  [0x10c412290]
    +             2521 ???  (in codex)  load address 0x102f4c000 + 0x94d0e90  [0x10c41ce90]
    +               2521 ???  (in codex)  load address 0x102f4c000 + 0x94ed038  [0x10c439038]
    +                 2521 tokio::runtime::scheduler::multi_thread::worker::run::hb0c4d53bf4dd0239  (in codex) + 2412  [0x10c42a6cc]
    +                   2521 ???  (in codex)  load address 0x102f4c000 + 0x94e1470  [0x10c42d470]
    +                     2521 ???  (in codex)  load address 0x102f4c000 + 0x1feeec4  [0x104f3aec4]
    +                       2521 ???  (in codex)  load address 0x102f4c000 + 0x962250  [0x1038ae250]
    +                         2521 ???  (in codex)  load address 0x102f4c000 + 0xe0f554  [0x103d5b554]
    +                           2521 codex_app_server::thread_state::ThreadState::clear_listener::h9f6a587cd02d79ba  (in codex) + 160  [0x105b9caac]
    +                             2521 ???  (in codex)  load address 0x102f4c000 + 0x2d53884  [0x105c9f884]
    +                               2521 _$LT$codex_file_watcher..WatchRegistration$u20$as$u20$core..ops..drop..Drop$GT$::drop::h3d7f99c33e0bdce1  (in codex) + 1780  [0x108aef6b8]
    +                                 2521 ???  (in codex)  load address 0x102f4c000 + 0x5b9dc8c  [0x108ae9c8c]
    +                                   2521 ???  (in codex)  load address 0x102f4c000 + 0x8b59bfc  [0x10baa5bfc]
    +                                     2521 ???  (in codex)  load address 0x102f4c000 + 0x8b5ef1c  [0x10baaaf1c]
    +                                       2521 _pthread_join  (in libsystem_pthread.dylib) + 616  [0x18401a114]
    +                                         2521 __ulock_wait  (in libsystem_kernel.dylib) + 8  [0x183fd5af8]
    2521 Thread_28297: reqwest-internal-sync-runtime
    + 2521 thread_start  (in libsystem_pthread.dylib) + 8  [0x184012c1c]
    +   2521 _pthread_start  (in libsystem_pthread.dylib) + 136  [0x184017c58]
    +     2521 ???  (in codex)  load address 0x102f4c000 + 0x93cd26c  [0x10c31926c]
    +       2521 ???  (in codex)  load address 0x102f4c000 + 0x8f2a0c8  [0x10be760c8]
    +         2521 std::sys::backtrace::__rust_begin_short_backtrace::hb7e12e51ce7806ee  (in codex) + 1100  [0x10be73f14]
    +           2521 ???  (in codex)  load address 0x102f4c000 + 0x8ede028  [0x10be2a028]
    +             2521 ???  (in codex)  load address 0x102f4c000 + 0x8f28638  [0x10be74638]
    +               2521 ???  (in codex)  load address 0x102f4c000 + 0x8ede9c8  [0x10be2a9c8]
    +                 2521 ???  (in codex)  load address 0x102f4c000 + 0x8edc070  [0x10be28070]
    +                   2521 ???  (in codex)  load address 0x102f4c000 + 0x8ee1b3c  [0x10be2db3c]
    +                     2521 _$LT$reqwest..async_impl..client..PendingRequest$u20$as$u20$core..future..future..Future$GT$::poll::h77fe8c5885c257a6  (in codex) + 116  [0x10bea00e4]
    +                       2521 ???  (in codex)  load address 0x102f4c000 + 0x8f1da10  [0x10be69a10]
    +                         2521 ???  (in codex)  load address 0x102f4c000 + 0x8f22054  [0x10be6e054]
    +                           2521 ???  (in codex)  load address 0x102f4c000 + 0x8eef580  [0x10be3b580]
    +                             2521 ???  (in codex)  load address 0x102f4c000 + 0x8ef4938  [0x10be40938]
    +                               2521 ???  (in codex)  load address 0x102f4c000 + 0x8eb294c  [0x10bdfe94c]
    +                                 2521 ???  (in codex)  load address 0x102f4c000 + 0x8eb2430  [0x10bdfe430]
    +                                   2521 ???  (in codex)  load address 0x102f4c000 + 0x8ee7b1c  [0x10be33b1c]
    +                                     2521 ???  (in codex)  load address 0x102f4c000 + 0x8ea39d8  [0x10bdef9d8]
    +                                       2521 ???  (in codex)  load address 0x102f4c000 + 0x8ea568c  [0x10bdf168c]
    +                                         2521 ???  (in codex)  load address 0x102f4c000 + 0x8e7f658  [0x10bdcb658]
    +                                           2521 ???  (in codex)  load address 0x102f4c000 + 0x8e8080c  [0x10bdcc80c]
    +                                             2521 ???  (in codex)  load address 0x102f4c000 + 0x8f267ec  [0x10be727ec]
    +                                               2521 SSLHandshake  (in Security) + 228  [0x18795a750]