Codex Cloud can't fetch deps from hex.pm (Elixir package manager)

I'd love to try codex web but it's currently unusable for Elixir projects.

I'm seeing the same SSL issues with Pkl, which means I can't set up hk for pre-commit verification.

Every run, the agent complains "hk fix failed in this environment due to TLS certificate validation when Pkl attempted to fetch from GitHub".

/workspace/scrapier$ /bin/bash -lc 'mise x -- hk fix'
thread 'main' (28687) panicked at src/settings.rs:90:30:
Failed to load configuration: failed to analyze pkl

Caused by:
   0: failed to run pkl
   1: Failed to evaluate Pkl config at /workspace/scrapier/hk.pkl

      Exit code: 1

      Error output:
      –– Pkl Error ––
      Exception when making request `GET https://github.com/jdx/hk/releases/download/v1.32.0/hk@1.32.0`:
      Error during SSL handshake with host `github.com`:
      unable to find valid certification path to requested target

      1 | amends "package://github.com/jdx/hk/releases/download/v1.32.0/hk@1.32.0#/Config.pkl"
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      at pkl.analyze#importGraph (file:///workspace/scrapier/hk.pkl, line 1)

      6 | value = analyze.importGraph(importStrings)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      at text#output.value (repl:text)

      128 | renderer.renderDocument(value)
                                    ^^^^^
      at pkl.base#Module.output.text (https://github.com/apple/pkl/blob/0.30.2/stdlib/base.pkl#L128)


Location:
    src/config.rs:382:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Any update on fixing this?

Sad to see codex cloud so close to being really useful, but blocked by not being able to get packages.

+1

Adding more evidence from a Codex Cloud sandbox debugging session today, since this looks like a proxy ↔ Erlang :httpc interaction rather than a Hex/Mix bug. Other languages on the same image transit Envoy fine; only :httpc 503s.

Environment

Codex Cloud sandbox, mise-installed Elixir 1.18.3 / OTP 27.

HTTP_PROXY=http://proxy:8080
HTTPS_PROXY=http://proxy:8080
NO_PROXY=localhost,127.0.0.1,::1
SSL_CERT_FILE=/usr/local/share/ca-certificates/envoy-mitmproxy-ca-cert.crt
CODEX_PROXY_CERT=/usr/local/share/ca-certificates/envoy-mitmproxy-ca-cert.crt

The Envoy MITM CA is merged into /etc/ssl/certs/ca-certificates.crt.

curl through the same Envoy: all 200 OK

$ curl -sSI https://builds.hex.pm/ | head -3
HTTP/1.1 200 OK
server: envoy

$ curl -sSI https://repo.hex.pm/ | head -3
HTTP/1.1 200 OK
server: envoy

$ curl -sSI https://github.com/ | head -3
HTTP/1.1 200 OK
server: envoy

Every :httpc request: 503

$ mix local.hex --force
** (Mix) httpc request failed with: {:bad_status_code, 503}
Could not install Hex because Mix could not download metadata at https://builds.hex.pm/installs/hex-1.x.csv.

$ mix local.rebar --force
** (Mix) httpc request failed with: {:bad_status_code, 503}

$ HEX_MIRROR=https://hexpm.upyun.com mix deps.get
** (Mix) httpc request failed with: {:bad_status_code, 503}
Could not install Hex because Mix could not download metadata at https://hexpm.upyun.com/installs/hex-1.x.csv.

Things that did not help

| Knob | Result |
|---|---|
| HEX_UNSAFE_HTTPS=1 | 503 |
| HEX_CACERTS_PATH=/etc/ssl/certs/ca-certificates.crt | 503 |
| HEX_MIRROR=https://hexpm.upyun.com | 503 |
| HEX_HTTP_CONCURRENCY=1, HEX_HTTP_TIMEOUT=120 | 503 |
| Retries with backoff | 503 every attempt |

Switching upstream to UpYun produces the same Envoy 503, which suggests Envoy is bouncing the request before reaching any upstream — the failure is in the proxy's handling of :httpc's connection, not at hex.pm.

Why this looks proxy-side

Every other language runtime on the Codex Cloud image (Python requests/urllib3, Node npm, Ruby gem, cargo, go mod, apt, git/libcurl) transits the same Envoy fine. The only thing that 503s is Erlang's :httpc. :httpc is much thinner than libcurl/requests/reqwest and likely sends a CONNECT framing or TLS handshake shape (cipher list, ALPN, SNI) that the Envoy MITM rejects. A fix on the Envoy side would unblock every Elixir/Erlang user on Codex Cloud without requiring changes to Mix, Hex, or OTP — and would address the Pkl case @sargunv reported above for the same reason.

Working workaround

Until this is fixed I'm pre-vendoring deps/ into the repo and using a setup script that:

• installs rebar3 via curl (libcurl works),
• installs Hex via mix archive.install github hexpm/hex branch latest (uses git/libcurl, not :httpc),
• sets HEX_OFFLINE=1 and runs mix deps.compile against the vendored sources — never mix deps.get.

Works reliably but deps/ re-vendor on every dep bump is rough. Happy to provide raw :httpc traces or test proxy-config changes.

I got it working and put my solution into a small golang proxy you can plug into your Codex: https://github.com/dvcrn/hexpm-envoy-proxy

The issue is that httpc is sending empty headers like

te:

which is valid spec, but envoy is still imploding.

The golang proxy I wrote is stripping empty headers out before forwarding the requests to the envoy proxy that Codex is using. Set up the maintenance script like this:

mise use -g go:github.com/dvcrn/hexpm-envoy-proxy@latest
if ! pgrep -f hexpm-envoy-proxy > /dev/null 2>&1; then
  nohup mise x -- hexpm-envoy-proxy > /tmp/hexpm-envoy-proxy.log 2>&1 &
  disown
  sleep 2
fi
# Wait for proxy to be ready (up to 5s)
for i in $(seq 1 10); do
  if curl -s -o /dev/null -w '' http://127.0.0.1:8787/ 2>/dev/null; then
    echo "hexpm-envoy-proxy is ready"
    break
  fi
  sleep 0.5
done

cat /tmp/hexpm-envoy-proxy.log
mix deps.get

And set the following environment variables in your Codex env:

HEX_MIRROR = http://127.0.0.1:8787
HEX_BUILDS_URL = http://127.0.0.1:8787/builds

This will tell hex to use localhost:8787 as mirror, which just strips empty headers and forwards them to the local HTTP_PROXY (envoy)

I can now natively resolve hex packages in my Codex environments