›_ Codex Issues
Codex Issues / #39421

Marketplace upgrade staging leak: 559 GB / 4,972 orphaned dirs in 41 days; cleanup janitor exists for curated clones but not marketplaces

Open 💬 4 comments Opened Aug 19, 2026 by numberyy
💡 Likely answer: A maintainer (github-actions[bot], contributor) responded on this thread — see the highlighted reply below.

Summary

Marketplace auto-upgrade leaked 4,972 staging directories totaling 559 GB in ~/.codex/.tmp/marketplaces/.staging/ over 41 days on my machine. That is the largest instance reported so far (previous high: 277 GB in #29994), so I'm adding it as a data point together with a consolidated root-cause read, because the seven existing reports are all still open with no fix PR.

Environment

  • codex-cli 0.147.0 (musl standalone), Fedora 43
  • ChatGPT Desktop for Linux with automation extensions active (app-server plugin sync)
  • Two configured marketplaces (claude-plugins-official, cctools-codex-plugins), ~115 MB per full clone

What happened

  • First leaked dir 2026-07-09, newest 2026-08-19; 4,972 dirs, 559 GB, ~10 GB/day while the desktop app was open.
  • Completed leaked dirs are ~112–117 MB (full clones). The newest one was 18 MB, a partial clone — the process was killed mid-git clone. This matches the 30-second MARKETPLACE_UPGRADE_GIT_TIMEOUT trigger described in #38770: a ~115 MB marketplace on a residential connection can't finish in 30 s, so the upgrade dies every attempt, the revision metadata never updates, and the next sync retries — one orphaned clone per attempt, forever.

Root cause (from reading current main)

  1. codex-rs/core-plugins/src/marketplace_upgrade.rs stages each upgrade into tempfile::Builder::new().prefix("marketplace-upgrade-").tempdir_in(install_root.join(".staging")).
  2. TempDir cleanup only runs in its destructor. When the upgrade process is killed (clone timeout, OOM, SIGKILL, app-server teardown), the destructor never runs and the directory stays.
  3. Nothing ever sweeps .staging: the only code in the repo touching that path is the code that creates entries (marketplace_upgrade.rs, marketplace_add/install.rs). There is no startup janitor for it.

The kicker is that the janitor pattern already exists in this codebase for the sibling leak: curated plugin startup sync has remove_stale_curated_repo_temp_dirs(...) (see #16004). It was never applied to marketplace staging.

Prior reports, all open

#21005 (2026-05-04, 187 GB) · #29994 (277 GB) · #30620 · #30794 · #32058 (also covers marketplace-backup-*) · #38770 (timeout trigger) · #39332 (filed 2026-08-19). Amplifiers: #36093 (concurrent processes), #34128 (annotated-tag upgrade loop), #24815 (the 30 s timeout itself).

Suggested fix

  • On startup (or before each upgrade attempt), remove .staging/marketplace-upgrade-* entries older than some small age, mirroring remove_stale_curated_repo_temp_dirs.
  • Optionally make the clone timeout proportional to marketplace size or resumable; shallow clones (#29994 suggestion) shrink the per-leak cost but don't stop the leak.

Happy to open a PR porting the existing janitor pattern to installed_marketplaces.rs if maintainers confirm that's the preferred shape.

View original on GitHub ↗

4 Comments

github-actions[bot] contributor · 9 days ago

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

  • #39332
  • #38770

Powered by Codex Action

lwz20210407 · 8 days ago

Adding a Windows data point, since every detailed report on this leak so far (including this one, #39332, and #29994) is from Linux/macOS. The behaviour reproduces identically on Windows 11, and the leak rate matches yours almost exactly.

Environment

  • Windows 11 Pro 10.0.26200
  • codex-cli 0.145.0-alpha.4
  • Codex desktop app (MS Store OpenAI.Codex) also installed and running during the day
  • 2 configured marketplaces (sendbird, text-to-cad) + openai-bundled; ~160 MB per full text-to-cad clone

What I found

%USERPROFILE%\.codex\.tmp\marketplaces\.staging\ had accumulated 101 orphaned directories / 13.78 GB / 96,174 files. The .git directories alone accounted for 10.96 GB across 97 clones.

Directory naming was 101/101 marketplace-upgrade-<6 chars> — zero unexpected entries.

Remote census (all 101 enumerated, not sampled)

| count | remote.origin.url |
|---|---|
| 90 | https://github.com/earthtojake/text-to-cad.git |
| 5 | https://github.com/sendbird/codex-marketplace.git |
| 6 | no remote configured |

Those 6 clones with no remote at all are, I think, the most useful part of this report: they are clones killed before git remote add completed. That is direct corroboration of the timeout-kill chain you and #38770 describe — the process dies mid-clone, TempDir's destructor never runs, revision metadata never updates, next sync retries, one orphan per attempt.

git status --porcelain on a 5-directory spot check: all clean, no user data.

Leak rate

I sampled timestamps on 20 of the 101 directories. All 20 fell on a single day, spread across the working day:

07:15  07:37  07:53  08:03  08:41  08:57  09:01  09:15  09:59  10:37
10:39  10:57  12:11  12:25  13:37  13:53  14:16  15:25  18:20  20:13

That is roughly one orphan every 20–40 minutes, ≈ 13.8 GB/day at ~160 MB per clone — versus the 13.6 GB/day implied by your 559 GB / 41 days. Two very different platforms, two different marketplace sets, essentially the same rate. That's consistent with the leak being driven by the sync retry interval rather than by anything environment-specific.

Impact here: a 200 GB system drive went to 8.5 GB free.

Cleanup is safe (data point for anyone landing here from a search)

I deleted .staging/ outright and verified afterwards:

  • ~/.codex/sessions/ — 873 files / 9.78 GB, byte-identical to the pre-deletion baseline; oldest (Feb 2026) and newest sessions still parse
  • codex --version, codex plugin marketplace list, codex plugin list — all normal, 3 marketplaces detected, all plugins installed, enabled
  • live marketplaces/sendbird and marketplaces/text-to-cad clones intact with valid HEADs
  • logs_2.sqlite was regenerated automatically

Manual workaround on Windows until a janitor lands:

rd /s /q "%USERPROFILE%\.codex\.tmp\marketplaces\.staging"

Note on the duplicate pile

This is now at least the 9th open report of the same defect (#21005, #29994, #30620, #30794, #32058, #38770, #39332, #39421, plus amplifiers #36093 / #34128 / #24815), the earliest from 2026-05-05. I deliberately commented here instead of opening a tenth. +1 to your offer to port remove_stale_curated_repo_temp_dirs to the marketplace staging path — that seems like the smallest change that stops the bleeding, independent of whatever happens to the 30 s clone timeout.

Srcanesen · 2 days ago

Fresh datapoint confirming this is still unfixed in 0.147.0 (macOS, Apple Silicon)

Accumulation: 375 orphaned marketplace-upgrade-* dirs under ~/.codex/.tmp/marketplaces/.staging/ (~14 GB total, incl. 45 completely empty = aborted attempts) accumulated between Aug 21–25 while CodexBar ran recurring background codex exec sessions. ~41k files written in the last 24h alone.

Root cause analysis (codex-rs/core-plugins, 0.147.0):

  • marketplace_upgrade.rs:251-268 creates staging clones via TempDir::tempdir_in(...) with prefix marketplace-upgrade-; cleanup relies solely on TempDir::Drop, which never runs on SIGKILL or when MARKETPLACE_UPGRADE_GIT_TIMEOUT (30 s, marketplace_upgrade/git.rs) aborts the operation.
  • There is no sweeper for marketplace temp dirs — contrast curated clones, which do have one (startup_sync.rs:400 remove_stale_curated_repo_temp_dirs) — so every orphan persists forever.
  • manager.rs:362/2557 gates upgrades behind a process-local in_flight: bool; concurrent processes (menu-bar apps, multiple terminals) each stage their own full clone. This matches the app-server trigger reported here and explains why background session runners amplify the leak so dramatically.
  • Sibling leaks share the same pattern: activation.rs:76 marketplace-backup-*, marketplace_add.rs:148 .keep(), and 0.147.0 additionally leaks under plugins/.marketplace-plugin-source-staging/.

Suggested fix (mirroring the existing curated-repo janitor): sweep .staging entries matching marketplace-{upgrade,add,backup}-* older than ~10 minutes while holding an OS-level lock (mtime alone is unsafe mid-flight, cf. #32058), and/or adopt the 24 h age-gated prune approach from #30620.

Happy to provide additional local traces (lsof snapshots, per-dir timestamps) if useful.

lwz20210407 · 2 days ago

7-day follow-up from the Windows reporter above — regrowth measured, plus a natural experiment that supports the timeout hypothesis

I deleted .staging/ entirely on 2026-08-19 (the 101 dirs / 13.78 GB I reported here). Re-measured today, 2026-08-26, same machine, same codex-cli 0.145.0-alpha.4, no config changes.

It regrew, and nothing self-healed

17 new orphans / 2.67 GB between 2026-08-20 09:45 and 2026-08-26 10:17. Confirms there is no startup janitor and no lazy cleanup on the next successful upgrade — deleting the directory is purely symptomatic.

The rate is not constant — it tracks usage, not wall-clock

| window | orphans | size | rate |
|---|---|---|---|
| 2026-08-19 (~13 h) | 101 | 13.78 GB | ~13.8 GB/day |
| 2026-08-20 → 08-26 (6.0 d) | 17 | 2.67 GB | ~0.44 GB/day |

A 31× swing on an unchanged install. The first window was a heavy Codex day; the second was light. That's inconsistent with a fixed-interval poller and consistent with one orphan per upgrade attempt, i.e. the leak scales with how often the app triggers a sync.

The useful bit: only the large marketplace leaks

Remote census of all 17 (not sampled):

| count | remote |
|---|---|
| 17 | https://github.com/earthtojake/text-to-cad.git (~170 MB) |
| 0 | https://github.com/sendbird/codex-marketplace.git (~2 MB) |

Both marketplaces are configured and both get upgraded. In my 2026-08-19 census the split was 90 / 5 / 6-no-remote — so sendbird did leak occasionally before. Over these 6 days it produced zero orphans while text-to-cad produced 17.

That is close to a controlled experiment on one machine: same client, same scheduler, same network, two marketplaces differing by ~85× in clone size, and only the large one leaks. It's hard to explain that with anything except a size-dependent failure — i.e. the 30 s MARKETPLACE_UPGRADE_GIT_TIMEOUT from #38770 / #24815.

Per-directory sizes back that up

2026-08-20 09:45      3 MB   marketplace-upgrade-Ckdxl0   <-- partial, killed early
2026-08-20 10:23    169 MB   marketplace-upgrade-iZOngF
2026-08-20 10:58    169 MB   marketplace-upgrade-UuxdWd
2026-08-20 11:01    169 MB   marketplace-upgrade-G4VqBy
2026-08-20 11:05    169 MB   marketplace-upgrade-THJPZn
2026-08-20 11:07    169 MB   marketplace-upgrade-0VkEaP
2026-08-20 11:19    169 MB   marketplace-upgrade-x5VcTq
2026-08-20 15:38    169 MB   marketplace-upgrade-AomJoq
2026-08-22 12:54    171 MB   marketplace-upgrade-WxfIsl
2026-08-22 14:59    171 MB   marketplace-upgrade-dybpSl
2026-08-25 00:01    172 MB   marketplace-upgrade-MckTwN
2026-08-25 02:22    172 MB   marketplace-upgrade-BvNmQm
2026-08-25 04:12    172 MB   marketplace-upgrade-C7bOyF
2026-08-25 06:12    172 MB   marketplace-upgrade-6oH0SZ
2026-08-25 10:22    172 MB   marketplace-upgrade-MFZujp
2026-08-25 15:43    172 MB   marketplace-upgrade-wKBUF5
2026-08-26 10:17    173 MB   marketplace-upgrade-3pmqDo

Three things worth noting:

  1. Most orphans are complete clones (169–173 MB), not partials. So the clone frequently finishes and the directory is still abandoned. The timeout explains the partials, but a completed-then-orphaned clone points at the TempDir destructor also being skipped on the success path (or the process being torn down after clone but before promote). Worth checking both paths, not just the timeout one.
  2. Bursts, not a steady drip — 5 orphans inside 56 minutes on 08-20 (10:23→11:19), then 2 days of nothing. Retry storms while the app is open.
  3. Sizes creep 169 → 173 MB across the week, tracking upstream repo growth — so the per-leak cost grows over time on its own.

Also of note: 4 of the 17 landed between 00:01 and 06:12 on 08-25, when I was not using the machine interactively. So the sync fires unattended.

Restating the ask

Still no fix on any of the ten open reports. Porting remove_stale_curated_repo_temp_dirs to the marketplace staging path would cap the damage regardless of which of the two failure paths above is responsible. Happy to test a build on Windows if that helps.

Windows workaround, unchanged:

rd /s /q "%USERPROFILE%\.codex\.tmp\marketplaces\.staging"

Verified again today that this is safe: ~/.codex/sessions is untouched, and codex plugin marketplace list / codex plugin list come back clean afterwards.