Repeated regressions, false “done” signals, and time-draining syntax loops

Resolved 💬 2 comments Opened Aug 31, 2025 by carli2 Closed Nov 10, 2025
💡 Likely answer: A maintainer (etraut-openai, contributor) responded on this thread — see the highlighted reply below.

What version of Codex is running?

codex-cli 0.27.0

Which model were you using?

gpt5-high

What platform is your computer?

arm64 on raspberry pi

What steps can reproduce the bug?

  • I’m repeatedly seeing Codex spend hours chasing a simple syntax error, and in >50% of sessions it breaks unit tests while still claiming the task is complete.

Issues Observed

  • Endless syntax churn: gets stuck in cycles “fixing” parentheses/quoting, often introducing new syntax errors elsewhere.
  • Test regressions: modifies code in ways that break existing tests unrelated to the request.
  • False completion: reports “done” while the relevant unit test still fails.
  • Fragile planning: changes ripple without guardrails; no rollback when a patch increases failures.
  • Poor local validation: patches aren’t gated by a fast syntax check or a focused test run.
  • sometimes, codex just deletes the tests instead of fixing the code

Impact

  • Lost time (hours to a full day) rescuing builds, untangling trivial syntax/aliasing errors, and re-stabilizing tests.
  • Erosion of trust: I must re-verify every “done” because the stated outcome often doesn’t match reality.

Minimal Repro Pattern

  • I tested on https://github.com/launix-de/memcp/
  • in e18f434c653f36a796784b862515edf418837d75 I added a failing unit test
  • try to let codex fix it - it will mess up the whole code base or delete the tests

What Would Help

  • Mandatory local checks: run a fast syntax/parse step and the targeted test(s) before stating completion.
  • “No-regression” mode: forbid merging patches that increase failing tests relative to baseline.
  • Precise scope control: restrict edits to files directly implicated by the task unless explicitly widened.
  • Atomic patches: apply, test, and auto-rollback if failures increase; report diff and reason.
  • Honest status: if tests can’t run, say so; don’t declare “done” without verification.
  • Confidence + proof: include the exact test output (pass/fail counts) when claiming completion.

Context

  • This occurred in a real repo with a planner/SQL dialect; the pattern is systemic, not project-specific.

Please prioritize reliability over zeal: pass the test, or clearly state why not, with actionable next steps.

What is the expected behavior?

_No response_

What do you see instead?

_No response_

Additional information

_No response_

View original on GitHub ↗

2 Comments

carli2 · 11 months ago

I noticed it on 0.25.0 but I updated to 0.29 now.

Maybe it's just normal for an AI. My employee behave the same...

etraut-openai contributor · 9 months ago

Thanks for the bug report. We've shared the feedback with the team members who are responsible for training our codex models. We're tracking this feedback and similar reports so we can make the model more robust in future iterations.

If you see other model behavior that you'd like to report, please use the recently-added /feedback command in the CLI. That will provide us with additional details including logs and conversation text.