/goal long-running loop produces 480 KB single log lines via chain-nested turn{} tracing spans (34 GB log in one day)

Resolved 💬 11 comments Opened May 18, 2026 by 1yhy Closed May 30, 2026
💡 Likely answer: A maintainer (github-actions[bot], contributor) responded on this thread — see the highlighted reply below.

Codex version

codex-cli 0.130.0

Plan

ChatGPT Pro

Model

gpt-5.5

Platform

Darwin 25.1.0 arm64 arm (macOS, Apple Silicon)

Terminal

VS Code integrated terminal v1.118.1 (no multiplexer)

Codex doctor report

not available (codex 0.130.0 does not have doctor subcommand yet)

---

What issue are you seeing?

After running a /goal overnight (status remained active for ~24 hours), my ~/.codex/log/codex-tui.log grew from ~412 MB (8 months of normal usage) to 34 GB in a single day.

1. Single log lines reach 481,817 bytes (~480 KB)

Line length distribution from a 200 MB tail sample:

p50:     359 bytes
p90: 479,776 bytes
p99: 479,910 bytes
max: 481,817 bytes
avg:  56,276 bytes

2. Each monster line contains 691 chain-nested turn{} tracing spans

Truncated example (real content, only IDs anonymized):

2026-05-18T17:11:38Z  WARN turn{otel.name="session_task.turn" thread.id=<TID> turn.id=AAAA model=gpt-5.5 codex.turn.reasoning_effort=xhigh codex.turn.token_usage.input_tokens=0 codex.turn.token_usage.input_tokens=0 codex.turn.token_usage.cached_input_tokens=0 codex.turn.token_usage.cached_input_tokens=0 ...}
:turn{otel.name="session_task.turn" thread.id=<TID> turn.id=BBBB ... input_tokens=0 input_tokens=0 ...}
:turn{otel.name="session_task.turn" thread.id=<TID> turn.id=CCCC ...}
... (688 more turn spans nested) ...
: codex_core_skills::loader: ignoring interface.icon_large: icon path must not contain '..'

Verification:

$ awk 'length($0) > 100000' ~/.codex/log/codex-tui.log \
  | head -1 \
  | grep -oE 'turn\.id=[a-f0-9-]+' | sort -u | wc -l
691

3. Token usage fields duplicated per nesting level

Note the pattern input_tokens=0 input_tokens=0, cached=N cached=N repeated in every span level. This strongly suggests child spans inherit and re-record parent fields without deduplication.

4. Side effect: threads.tokens_used shows impossible value

sqlite> SELECT tokens_used FROM threads WHERE id = '<TID>';
1248543402   -- 1.25 billion

sqlite> SELECT tokens_used FROM thread_goals WHERE thread_id = '<TID>';
138610       -- 138K (the truthful value)

The threads-level counter is inflated ~9000× by cumulative accumulation across nested spans. The goals-level counter (recorded at a different layer) is accurate.

5. Time distribution of the 34 GB log

| Window | Bytes written |
|---|---|
| 2025-09-26 → 2026-05-17 (8 months) | 412 MB |
| 2026-05-17 single day | 1.6 GB |
| 2026-05-18 single day (goal running) | 30 GB |

The 24h of goal execution wrote ~78× the prior 8-month total.

Related issues

  • #11202 (file_watcher INFO spam) — similar symptom (rapid log growth), different root cause
  • #4345, #4766 (token usage multiplied) — I believe my data provides the mechanism: nested span field duplication
  • #21786 (/goal self-loops with brainstorming) — adjacent area but different layer (product loop vs tracing instrumentation)

---

What steps can reproduce the bug?

  1. Codex CLI 0.130.0 with [features] goals = true in ~/.codex/config.toml (default)
  2. Start /goal with an open-ended self-iterating objective such as "keep iterating, don't stop until you've polished every aspect, I'll check tomorrow morning"
  3. Let it run unattended for several hundred turns (mine reached 691)
  4. Observe ~/.codex/log/codex-tui.log growing rapidly (~30 GB / 24h in my case)

Diagnostics:

# Confirm nested turn spans in single line
awk 'length($0) > 100000' ~/.codex/log/codex-tui.log \
  | head -1 | grep -oE 'turn\.id=[a-f0-9-]+' | sort -u | wc -l

# Compare token counters between threads and thread_goals tables
sqlite3 ~/.codex/state_5.sqlite \
  "SELECT t.id, t.tokens_used AS threads_count, g.tokens_used AS goal_count
   FROM threads t JOIN thread_goals g ON g.thread_id = t.id
   WHERE g.status = 'active'"

Thread id available on request (omitted from public issue for privacy).

---

What is the expected behavior?

Each turn span should be a sibling under the goal span, not a chained child of the prior turn. Specifically:

  • turn spans must be dropped at the end of each iteration so subsequent turns start at the same depth
  • Fields recorded on a child span should not duplicate into parent span output
  • Long-running /goal sessions should not accumulate unbounded tracing depth

Likely culprits in goal_loop instrumentation:

  • Span::enter() guard stored in a long-lived struct, never dropped between iterations
  • #[instrument] on a function that recursively self-calls (forming parent→child chain instead of siblings)
  • Span context leak across tokio::spawn boundaries without Instrument::in_current_span or Span::none()

Either fix would also resolve the inflated threads.tokens_used counter, which appears to be an artifact of the same nesting bug.

---

Additional information

  • Active goal had token_budget = 32,507,871 and real tokens_used = 138,610 (0.4% consumed). The 1.25B figure is purely a span-nesting artifact, not real consumption
  • 691 distinct turn.id UUIDs confirmed via grep — these are real turn iterations, not display artifacts
  • Pre-2026-05 logs in the same file don't exhibit this issue because turns numbered in tens, not hundreds. Hundreds of turns appears to be the threshold where the bug becomes catastrophic (quadratic data amplification: N turns × N field re-records per log line)
  • After : > ~/.codex/log/codex-tui.log, Codex continued writing normally — no functional impact beyond disk space
  • I'm happy to provide a sanitized excerpt of the actual 480 KB log line if it would help; please ask

View original on GitHub ↗

11 Comments

github-actions[bot] contributor · 2 months ago

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

  • #22008
  • #23188

Powered by Codex Action

1yhy · 2 months ago

Thanks for the dedup signal! After reviewing #22008 and #23188, I believe all three are related but distinct surfaces of what may be a single underlying bug:

| # | Surface | Numeric symptom |
|---|---------|-----------------|
| #22008 | Hidden memory_consolidation subagent drained 5h quota after Stop hook failure | input_tokens=668,017,079 in a single turn |
| #23188 | Weekly quota dropped 70%→7% at reset boundary | total_tokens=4,035,352 per turn with cached=3,993,728 |
| #23340 (this) | /goal long-running loop produces 480 KB log lines with 691 nested turn{} spans | threads.tokens_used=1,248,543,402 while thread_goals.tokens_used=138,610 |

Common pattern across all three: codex.turn.token_usage.* reporting nonsensically large numbers (hundreds of millions / billions) for individual turns or accumulators.

In this issue I traced the mechanism: child spans inherit and re-record parent span fields without deduplication, so when a long-running goal accumulates N nested turn{} spans, every token field is recorded N times.

Fixing the span lifecycle in goal_loop (so turns become siblings, not chained children) would likely resolve the inflated counters reported in #22008 and #23188 as well — though those issues also expose separate concerns (quota accounting visibility for hidden subagents, weekly-reset boundary reconciliation) that should remain tracked.

Keeping #23340 open as it documents the log-disk-explosion symptom and the span-nesting root cause specifically.

Keesan12 · 2 months ago

The log-growth bug and the inflated hreads.tokens_used counter look like the same provenance defect showing up in two places: turn state is being accumulated as if each turn still owns all prior turn fields. I would treat the fix as a lifecycle boundary, not just a logging trim. Each turn needs a fresh receipt, and aggregate counters should roll up from bounded child receipts instead of inheriting the full parent span over and over. That is also the seam where long-running loops start lying to operators about spend in MartinLoop-style wrappers.

etraut-openai contributor · 2 months ago

@Keesan12, you're generating a lot of noise in our issue tracker. Please refrain from posting large numbers of AI-generated responses.

RochoMerino · 2 months ago

+1, same bug still present in codex-cli 0.131.0 on macOS 26.2 — log file ~/.codex/log/codex-tui.log grew to 67 GB overnight after /goal hit the usage limit.

Confirms this is not fixed in the latest version, and the manifestation is getting worse (34 GB → 67 GB).

Truncating the log recovered the space immediately:

: > ~/.codex/log/codex-tui.log

Disk filled to the point that Codex itself failed to save state:

⚠ Failed to save the conversation transcript; Codex will continue retrying.
  Error: thread-store internal error: No space left on device (os error 28)

(Filed #23722 before seeing this one — closing as dup.)

etraut-openai contributor · 2 months ago

There are three issues here:

  1. With the /goal feature, the agent doesn't stop when it hits an impasse or a usage limit.
  2. The harness is logging too much data in some logging points.
  3. TUI logs are not rotated.

Issue 1 is addressed in 0.132.0. The goal feature now allows the agent to mark the goal as "blocked" if it hits an impasse, and the harness marks the goal as "usage limited" if it hits a usage limit.

We're working on addressing 2 and 3.

etraut-openai contributor · 2 months ago

I'm not sure what you mean by "bounded per-turn receipt instead of inheriting prior turn state"? Can you elaborate?

My assumption is that the log size problem is independent of goals. It's just exacerbated when you have very long-running tasks, which are made possible by the /goal feature.

avatasia · 2 months ago

Me too, already created 100+GB logs, just one day.

Keesan12 · 2 months ago

What I meant by a bounded per-turn receipt is: each /goal cycle should emit a fixed, inspectable record for that turn only, instead of letting counters or trace payloads implicitly accumulate across turns.

So even if the root log bug is independent of /goal, the lifecycle boundary is still useful for debugging it. A minimal receipt would be something like:

  • attempt/turn id
  • bytes written this turn
  • trace/span count this turn
  • token/accounting delta this turn
  • stop / blocked / usage-limited state for this turn

If those stay bounded, then a 100GB log tells you "one turn emitted a pathological payload" instead of forcing you to guess whether the whole session kept inheriting prior state.

Keesan12 · 1 month ago

The receipt structure here gets a lot clearer if you separate log volume from decision volume. A tiny decision artifact that says why the attempt continued, stopped, or reopened is what operators actually need when /goal traces get noisy.

etraut-openai contributor · 1 month ago