/goal long-running loop produces 480 KB single log lines via chain-nested turn{} tracing spans (34 GB log in one day)
Codex version
codex-cli 0.130.0
Plan
ChatGPT Pro
Model
gpt-5.5
Platform
Darwin 25.1.0 arm64 arm (macOS, Apple Silicon)
Terminal
VS Code integrated terminal v1.118.1 (no multiplexer)
Codex doctor report
not available (codex 0.130.0 does not have doctor subcommand yet)
---
What issue are you seeing?
After running a /goal overnight (status remained active for ~24 hours), my ~/.codex/log/codex-tui.log grew from ~412 MB (8 months of normal usage) to 34 GB in a single day.
1. Single log lines reach 481,817 bytes (~480 KB)
Line length distribution from a 200 MB tail sample:
p50: 359 bytes
p90: 479,776 bytes
p99: 479,910 bytes
max: 481,817 bytes
avg: 56,276 bytes
2. Each monster line contains 691 chain-nested turn{} tracing spans
Truncated example (real content, only IDs anonymized):
2026-05-18T17:11:38Z WARN turn{otel.name="session_task.turn" thread.id=<TID> turn.id=AAAA model=gpt-5.5 codex.turn.reasoning_effort=xhigh codex.turn.token_usage.input_tokens=0 codex.turn.token_usage.input_tokens=0 codex.turn.token_usage.cached_input_tokens=0 codex.turn.token_usage.cached_input_tokens=0 ...}
:turn{otel.name="session_task.turn" thread.id=<TID> turn.id=BBBB ... input_tokens=0 input_tokens=0 ...}
:turn{otel.name="session_task.turn" thread.id=<TID> turn.id=CCCC ...}
... (688 more turn spans nested) ...
: codex_core_skills::loader: ignoring interface.icon_large: icon path must not contain '..'
Verification:
$ awk 'length($0) > 100000' ~/.codex/log/codex-tui.log \
| head -1 \
| grep -oE 'turn\.id=[a-f0-9-]+' | sort -u | wc -l
691
3. Token usage fields duplicated per nesting level
Note the pattern input_tokens=0 input_tokens=0, cached=N cached=N repeated in every span level. This strongly suggests child spans inherit and re-record parent fields without deduplication.
4. Side effect: threads.tokens_used shows impossible value
sqlite> SELECT tokens_used FROM threads WHERE id = '<TID>';
1248543402 -- 1.25 billion
sqlite> SELECT tokens_used FROM thread_goals WHERE thread_id = '<TID>';
138610 -- 138K (the truthful value)
The threads-level counter is inflated ~9000× by cumulative accumulation across nested spans. The goals-level counter (recorded at a different layer) is accurate.
5. Time distribution of the 34 GB log
| Window | Bytes written |
|---|---|
| 2025-09-26 → 2026-05-17 (8 months) | 412 MB |
| 2026-05-17 single day | 1.6 GB |
| 2026-05-18 single day (goal running) | 30 GB |
The 24h of goal execution wrote ~78× the prior 8-month total.
Related issues
- #11202 (file_watcher INFO spam) — similar symptom (rapid log growth), different root cause
- #4345, #4766 (token usage multiplied) — I believe my data provides the mechanism: nested span field duplication
- #21786 (
/goalself-loops with brainstorming) — adjacent area but different layer (product loop vs tracing instrumentation)
---
What steps can reproduce the bug?
- Codex CLI 0.130.0 with
[features] goals = truein~/.codex/config.toml(default) - Start
/goalwith an open-ended self-iterating objective such as "keep iterating, don't stop until you've polished every aspect, I'll check tomorrow morning" - Let it run unattended for several hundred turns (mine reached 691)
- Observe
~/.codex/log/codex-tui.loggrowing rapidly (~30 GB / 24h in my case)
Diagnostics:
# Confirm nested turn spans in single line
awk 'length($0) > 100000' ~/.codex/log/codex-tui.log \
| head -1 | grep -oE 'turn\.id=[a-f0-9-]+' | sort -u | wc -l
# Compare token counters between threads and thread_goals tables
sqlite3 ~/.codex/state_5.sqlite \
"SELECT t.id, t.tokens_used AS threads_count, g.tokens_used AS goal_count
FROM threads t JOIN thread_goals g ON g.thread_id = t.id
WHERE g.status = 'active'"
Thread id available on request (omitted from public issue for privacy).
---
What is the expected behavior?
Each turn span should be a sibling under the goal span, not a chained child of the prior turn. Specifically:
turnspans must be dropped at the end of each iteration so subsequent turns start at the same depth- Fields recorded on a child span should not duplicate into parent span output
- Long-running
/goalsessions should not accumulate unbounded tracing depth
Likely culprits in goal_loop instrumentation:
Span::enter()guard stored in a long-lived struct, never dropped between iterations#[instrument]on a function that recursively self-calls (forming parent→child chain instead of siblings)- Span context leak across
tokio::spawnboundaries withoutInstrument::in_current_spanorSpan::none()
Either fix would also resolve the inflated threads.tokens_used counter, which appears to be an artifact of the same nesting bug.
---
Additional information
- Active goal had
token_budget = 32,507,871and realtokens_used = 138,610(0.4% consumed). The 1.25B figure is purely a span-nesting artifact, not real consumption - 691 distinct
turn.idUUIDs confirmed via grep — these are real turn iterations, not display artifacts - Pre-2026-05 logs in the same file don't exhibit this issue because turns numbered in tens, not hundreds. Hundreds of turns appears to be the threshold where the bug becomes catastrophic (quadratic data amplification: N turns × N field re-records per log line)
- After
: > ~/.codex/log/codex-tui.log, Codex continued writing normally — no functional impact beyond disk space - I'm happy to provide a sanitized excerpt of the actual 480 KB log line if it would help; please ask
11 Comments
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
Thanks for the dedup signal! After reviewing #22008 and #23188, I believe all three are related but distinct surfaces of what may be a single underlying bug:
| # | Surface | Numeric symptom |
|---|---------|-----------------|
| #22008 | Hidden memory_consolidation subagent drained 5h quota after Stop hook failure |
input_tokens=668,017,079in a single turn || #23188 | Weekly quota dropped 70%→7% at reset boundary |
total_tokens=4,035,352per turn withcached=3,993,728|| #23340 (this) |
/goallong-running loop produces 480 KB log lines with 691 nestedturn{}spans |threads.tokens_used=1,248,543,402whilethread_goals.tokens_used=138,610|Common pattern across all three:
codex.turn.token_usage.*reporting nonsensically large numbers (hundreds of millions / billions) for individual turns or accumulators.In this issue I traced the mechanism: child spans inherit and re-record parent span fields without deduplication, so when a long-running goal accumulates N nested
turn{}spans, every token field is recorded N times.Fixing the span lifecycle in
goal_loop(so turns become siblings, not chained children) would likely resolve the inflated counters reported in #22008 and #23188 as well — though those issues also expose separate concerns (quota accounting visibility for hidden subagents, weekly-reset boundary reconciliation) that should remain tracked.Keeping #23340 open as it documents the log-disk-explosion symptom and the span-nesting root cause specifically.
The log-growth bug and the inflated hreads.tokens_used counter look like the same provenance defect showing up in two places: turn state is being accumulated as if each turn still owns all prior turn fields. I would treat the fix as a lifecycle boundary, not just a logging trim. Each turn needs a fresh receipt, and aggregate counters should roll up from bounded child receipts instead of inheriting the full parent span over and over. That is also the seam where long-running loops start lying to operators about spend in MartinLoop-style wrappers.
@Keesan12, you're generating a lot of noise in our issue tracker. Please refrain from posting large numbers of AI-generated responses.
+1, same bug still present in codex-cli 0.131.0 on macOS 26.2 — log file
~/.codex/log/codex-tui.loggrew to 67 GB overnight after/goalhit the usage limit.Confirms this is not fixed in the latest version, and the manifestation is getting worse (34 GB → 67 GB).
Truncating the log recovered the space immediately:
Disk filled to the point that Codex itself failed to save state:
(Filed #23722 before seeing this one — closing as dup.)
There are three issues here:
/goalfeature, the agent doesn't stop when it hits an impasse or a usage limit.Issue 1 is addressed in 0.132.0. The goal feature now allows the agent to mark the goal as "blocked" if it hits an impasse, and the harness marks the goal as "usage limited" if it hits a usage limit.
We're working on addressing 2 and 3.
I'm not sure what you mean by "bounded per-turn receipt instead of inheriting prior turn state"? Can you elaborate?
My assumption is that the log size problem is independent of goals. It's just exacerbated when you have very long-running tasks, which are made possible by the
/goalfeature.Me too, already created 100+GB logs, just one day.
What I meant by a bounded per-turn receipt is: each
/goalcycle should emit a fixed, inspectable record for that turn only, instead of letting counters or trace payloads implicitly accumulate across turns.So even if the root log bug is independent of
/goal, the lifecycle boundary is still useful for debugging it. A minimal receipt would be something like:If those stay bounded, then a 100GB log tells you "one turn emitted a pathological payload" instead of forcing you to guess whether the whole session kept inheriting prior state.
The receipt structure here gets a lot clearer if you separate log volume from decision volume. A tiny decision artifact that says why the attempt continued, stopped, or reopened is what operators actually need when /goal traces get noisy.
This is now fixed in https://github.com/openai/codex/pull/24809.