Agent can bypass user-defined hard gates for local artifact delivery and visual QA

Summary

Codex Desktop can appear to acknowledge user-defined hard gates for local artifact delivery and technical visual QA, but still produce a final answer that bypasses those gates. In one local report-generation workflow, the agent repeatedly delivered stale or semantically outdated HTML/image artifacts and later marked a circuit-diagram audit as passing even though the rendered figure visibly violated the required terminal connectivity.

This is a trust and reliability issue for agent workflows that depend on local previews, generated images, SVG/PNG diagrams, or user-visible reports.

Environment

• Codex App: 26.623.81905
• CFBundleVersion: 4598
• Platform: macOS desktop app
• Workflow type: local HTML report with generated circuit diagrams and browser preview

Reproduction Pattern

1. Ask Codex to create a local HTML technical report with strict requirements, including:

• use a fresh immutable version for regenerated user-visible assets;
• verify the actual browser-rendered page, not only the filesystem;
• inspect every schematic device terminal connection before delivery;
• produce a required set of diagrams.

1. After the user reports stale content or visual/circuit errors, add stronger hard gates in local instructions/memory, such as:

• never reuse stale local artifact paths after a cache incident;
• serve the next preview from a no-cache HTTP server;
• verify the exact opened URL and page DOM;
• audit every TFT device for expected p/n type, D/S/G net connections, and visible gate/control electrode.

1. Ask Codex to revise and deliver again.
2. Observe that the agent may still:

• leave the user on an older file:// artifact or hand off an old semantic version;
• create a new audit table that checks its own written claims rather than the rendered image;
• mark the audit as passing while the rendered circuit has a visibly floating/incorrect gate connection;
• omit one of the explicitly required diagrams while still treating the report as deliverable.

Actual Behavior

• The agent acknowledged the user's hard gates in conversation and even wrote helper validation scripts, but the final delivery could still be stale or incomplete.
• A generated schematic image with a visibly incorrect TFT gate connection was treated as passing because the audit JSON was manually authored from intended topology rather than verified against the rendered figure.
• The agent's final response could imply completion after partial verification, even when requested deliverable count and rendered diagram semantics did not match the user request.

Expected Behavior

When a user defines explicit hard gates for local artifacts or visual QA, Codex should treat them as completion blockers, not advisory preferences.

In particular:

• After a stale-content report, the next delivery should be forced to use a new immutable artifact path, new asset paths, no-cache HTTP preview, exact URL verification, and stale-token scanning before any completion claim.
• For diagrams/schematics, any audit should be tied to the actual rendered artifact or to a single structured source-of-truth model used to generate both the diagram and the audit. The agent should not be able to pass an audit by writing a detached checklist that merely restates intended connections.
• If the user requested N diagrams or N deliverables, Codex should verify that count before final response.
• If a hard gate fails, the agent should stop and report the failure instead of handing over the artifact.

Why This Matters

This failure mode is especially damaging because it looks like the agent is becoming more rigorous while still allowing the same class of mistakes through. The user sees the agent acknowledge the rule, create a gate, claim verification, and still deliver an artifact that violates the rule. That breaks trust more severely than a simple first-pass mistake.

Possible Product Improvements

• Add a first-class "delivery gate" concept for local artifacts: exact preview URL, response headers, asset version tokens, opened-tab URL, DOM evidence, and requested deliverable count.
• Add a mechanism for user-defined hard gates to become explicit final-response blockers.
• For visual artifacts, require evidence binding between the rendered output and the audit result, especially when the audit concerns geometry, connectivity, or visual semantics.
• Surface a warning when the agent's validation script checks a manually authored assertion file rather than the actual rendered artifact.
• Make stale local file:// previews safer by nudging agents toward no-cache local HTTP previews after any cache/staleness complaint.