Codex can mislead users by proposing fixes before verifying disputed diagnoses

Request

Please improve Codex's debugging behavior so that:

• user counterevidence is treated as high-value input
• unresolved doubts block implementation more strongly
• deciding evidence is gathered before Codex argues for a theory
• minimal runtime reproduction is preferred earlier
• Codex more clearly separates guesses from confirmed facts
• explanation rigor is increased before implementation begins

Summary

Codex tended to move into implementation and workaround design before resolving core technical doubts raised by the user.

The problem was not only that some conclusions were wrong. The deeper problem was that Codex:

• did not sufficiently acknowledge user-provided counterevidence
• explained its own theory without first gathering the evidence needed to accept or reject that theory
• continued proposing fixes before the disputed behavior had been decisively verified

This creates a real risk in debugging and infrastructure work, because wrong assumptions can quickly turn into wrong code changes, wasted engineering time, and unnecessary complexity.

Environment

• Product: Codex coding agent
• Context: long multi-step debugging / refactoring session
• Domain: Docker, packaging, ONNX Runtime, GPU provider behavior, dependency resolution
• User context: experienced engineer providing contradictory runtime observations from real deployments

What happened

During the session, the user repeatedly raised technically meaningful objections such as:

• the explanation did not match real-world behavior in x86_64 deployments
• if the explanation were true, the upstream project should already have visible issues
• another explanation (runtime fallback) seemed more plausible

Instead of treating those objections as strong signals to stop and verify, Codex often continued in this pattern:

1. form a theory from partial evidence
2. explain that theory with too much confidence
3. start proposing implementation changes or workaround designs
4. only much later gather the evidence that should have been collected first

Eventually, after repeated pushback from the user and a direct runtime reproduction, the key disputed behavior was established:

• requesting CUDAExecutionProvider in ONNX Runtime did not immediately crash in the tested scenario
• instead, it emitted a warning and fell back to CPU

That fact should have been established before multiple rounds of explanation and workaround discussion.

Why this is a problem

The main issue is not simply "Codex made a factual mistake."

The main issue is that Codex showed a debugging posture that can be dangerous in real engineering work:

• it tried to solve before fully understanding
• it underweighted contradictory operational evidence from the user
• it did not clearly separate:
• confirmed facts
• unverified guesses
• implementation proposals
• it argued for its current theory before collecting the evidence needed to decide whether that theory should survive at all

In packaging/runtime debugging, this can mislead users into changing Dockerfiles, dependency flows, or runtime architecture based on assumptions that have not yet been proven.

Concrete behavior that should be improved

1. Stronger response to user counterevidence

If the user says:

• "that does not match behavior in production"
• "that should already be an upstream issue if true"
• "your explanation is less plausible than another one"

then Codex should reduce confidence aggressively and switch into evidence-gathering mode.

These are not minor objections. They are signals that the current explanation may be invalid.

2. Codex should gather deciding evidence before arguing for a theory

This is the core request.

When Codex has an explanation for a disputed behavior, it should not immediately:

• defend that explanation
• elaborate on it
• build workaround logic on top of it

Instead, it should first ask:

• what evidence would decide whether this explanation is valid?
• what is the smallest runtime or metadata check that would confirm or reject it?

Then it should gather that evidence first.

Only after that should it narrow the explanation space and discuss fixes.

3. Minimal runtime verification should happen much earlier

In this session, the decisive issue was whether ONNX Runtime would:

• throw an exception
• or emit a warning and fall back to CPU

That is a runtime fact. It should have been tested directly much earlier.

Codex should prefer direct runtime verification early whenever:

• behavior is disputed
• user observations contradict the current theory
• platform-specific packaging/runtime differences may exist

4. Codex should not present analysis-like explanations without actually doing the analysis

It is fine to propose a tentative idea.

It is not fine to speak as though the behavior is already understood when:

• the relevant runtime path has not been executed
• contradictory deployment evidence exists
• platform-specific package behavior has not been checked
• no minimal reproduction has been run

Codex should more clearly distinguish:

• "this is confirmed"
• "this is a guess"
• "this needs to be verified before we reason further"

5. Implementation momentum should be dampened during debugging

Codex appears optimized to keep moving toward edits. That is often useful.

But in this kind of session, that same trait became harmful. When the main problem is unresolved ambiguity, the agent should slow down and verify first, rather than continuing to accumulate workaround proposals.

What the better behavior would have been

A better sequence would have been:

1. user says the proposed explanation does not match deployment reality
2. Codex explicitly acknowledges that its current explanation is only tentative
3. Codex determines what evidence is required to accept or reject that explanation
4. Codex gathers that evidence before arguing further for the explanation
5. Codex then narrows the explanation space based on verified evidence
6. Codex only after that proposes implementation changes or workaround designs

This is not just "be more cautious."

The request is specifically: when the diagnosis itself is contested, Codex should gather deciding evidence before continuing to reason from that diagnosis.

Impact

This behavior caused:

• unnecessary back-and-forth
• repeated misleading explanations
• premature workaround design
• wasted user time and attention
• reduced confidence in the debugging process