`codex exec review` accepts but ignores `--output-schema`

Open 💬 4 comments Opened Aug 14, 2026 by roonius

Component

Codex CLI bundled with the ChatGPT desktop app for macOS.

Version

  • Reproduced on codex-cli 0.148.0-alpha.9
  • Previously reproduced on 0.147.0-alpha.6.6
  • macOS 26.6.1 (arm64)

Summary

codex exec review accepts and advertises --output-schema, but the review command ignores the schema and writes its normal prose review to --output-last-message. The command exits 0, so automation cannot distinguish this from successful schema enforcement.

Ordinary codex exec honors the same schema.

Minimal reproduction

Create a repository containing a commit with an obvious defect:

# example.py
def divide(a, b):
    return a / 0

Use this schema:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "required": ["verdict"],
  "properties": {
    "verdict": {
      "type": "string",
      "enum": ["pass", "block"]
    }
  }
}

Run:

codex exec review \
  --ephemeral \
  --ignore-user-config \
  -m gpt-5.6-terra \
  -c 'model_reasoning_effort="low"' \
  --commit HEAD \
  --output-schema minimal-schema.json \
  -o review.txt

Actual result

The command exits 0 and review.txt contains prose:

The newly added function cannot successfully divide any inputs because it unconditionally divides by zero.

Review comment:

- [P1] Divide by the supplied denominator — .../example.py:2-2
  This always evaluates a / 0 ...

The response does not match the supplied schema.

Control

The same schema works with ordinary exec:

codex exec \
  --ephemeral \
  --ignore-user-config \
  -m gpt-5.6-terra \
  -c 'model_reasoning_effort="low"' \
  --output-schema minimal-schema.json \
  -o control.txt \
  'Return verdict pass. Do not inspect files or call tools.'

control.txt contains:

{"verdict":"pass"}

Expected result

Either:

  1. codex exec review forwards --output-schema to the review turn and guarantees that the final message matches it, or
  2. the review subcommand rejects --output-schema as unsupported instead of accepting it as a silent no-op.

Impact

This makes exec review unsafe for machine-readable CI gates: the documented flag appears to provide schema enforcement, but automation receives unstructured prose with a successful exit code.

View original on GitHub ↗

4 Comments

jacobbabula · 13 days ago

I traced this on current main (22bf16a37ed45006c0226541874abd7449c29911). The schema is being lost at a specific dispatch boundary rather than in the output writer.

  • codex-rs/exec/src/cli.rs defines Cli::output_schema as a global exec option, so Clap accepts it before the nested review subcommand.
  • run_main() extracts that value into output_schema_path and passes it into run_exec_session().
  • In run_exec_session(), the ExecCommand::Review match arm constructs InitialOperation::Review { review_request } immediately. The normal, resume, and prompted-fork arms call load_output_schema(...) and carry the value in InitialOperation::UserTurn; the review arm never reads it.
  • Dispatching InitialOperation::Review sends ReviewStartParams { thread_id, target, delivery }. There is currently no output_schema field at the review/start boundary.

So --output-schema is syntactically valid, survives top-level parsing, and is then silently discarded before the review turn is created. That also explains the successful exit with ordinary prose.

There are two coherent contracts:

  1. Support it: add an optional schema to the v2 review/start request and propagate it into the review turn, preserving the same final-output validation used by an ordinary user turn.
  2. Reject it: until that API is intentionally supported, fail before session/model work when ExecCommand::Review is combined with output_schema_path.is_some().

I would favor the second as the minimum safe correction unless structured review output is an intended API: it removes the false-success mode without prematurely extending the app-server review contract. A focused regression should invoke codex exec review --output-schema ... and assert a nonzero exit plus an actionable error before any model request. If propagation is preferred, the stronger regression belongs at the review/start boundary and should assert that the resulting Responses request contains the requested output format; the existing ordinary-exec output-schema tests do not exercise this branch.

I have not opened a PR because the repository requires an explicit maintainer invitation for external code contributions. If the early-rejection contract matches the intended behavior, I would be happy to implement the guard and focused tests.

weivwang · 13 days ago

Root cause and a ready fix for this.

Root cause. --output-schema is a global flag on the exec CLI (codex-rs/exec/src/cli.rs), so clap accepts it for every subcommand. But the review path never reads it: in run_exec_session (codex-rs/exec/src/lib.rs), the ExecCommand::Review arm calls build_review_request(review_cli) and issues review/start, while only the UserTurn/Resume/Fork arms call load_output_schema(...). ReviewStartParams (codex-rs/app-server-protocol/src/protocol/v2/review.rs) has no schema field, so the flag is dropped before any model request is built. That is why plain codex exec honors the same schema but codex exec review writes prose and still exits 0.

Fix (option 2 from the issue). Reject the flag instead of silently ignoring it. build_review_request is the single choke point for both codex exec review and top-level codex review, so checking the resolved --output-schema path there bails with:

--output-schema is not supported for `codex exec review`; review output is free-form prose

This fires before the app-server/API work starts, so it fails loudly with a non-zero exit instead of returning unstructured prose.

I implemented this with a unit test and verified it (cargo check -p codex-exec, just test -p codex-exec review_request, clippy + fmt clean). The change is on this branch: https://github.com/weivwang/codex/tree/fix/exec-review-reject-output-schema

Diff summary:

 fn build_review_request(args: &ReviewArgs) -> anyhow::Result<ReviewRequest> {
+fn build_review_request(
+    args: &ReviewArgs,
+    output_schema_path: Option<&Path>,
+) -> anyhow::Result<ReviewRequest> {
+    if output_schema_path.is_some() {
+        anyhow::bail!(
+            "--output-schema is not supported for `codex exec review`; review output is free-form prose"
+        );
+    }
     let target = if args.uncommitted {

Option 1 (actually forwarding the schema to the review turn so the final message matches it) is the better long-term fix, but it requires plumbing output_schema through ReviewStartParams → the app-server review/start handler → the core Op::Review loop, which is a larger cross-crate change. The reject-first approach makes the CLI honest now and can be followed by the forwarding work.

Happy to open the PR if a maintainer is able to invite/allow it.

jdcodes1 · 10 days ago

Confirmed on main @ 1f41cc5d92, and the mechanism is a one-glance read in the exec dispatch:

https://github.com/openai/codex/blob/1f41cc5d92722748e45cae9cecc6d883a4e7cbb1/codex-rs/exec/src/lib.rs#L706-L710

The ExecCommand::Review arm builds InitialOperation::Review { review_request } and never calls load_output_schema — while the plain-prompt, Fork, and Resume arms all do and thread the schema into their InitialOperation::UserTurn (#L734, #L756, and the root arm). --output-schema is a top-level codex exec flag, so clap happily accepts it for the review subcommand, and the operation it maps to simply has no schema slot. Exit 0 + prose output follows — exactly your automation hazard.

Three fix options, not mutually exclusive:

  1. Fail loudly today: reject --output-schema (and arguably --output-last-message semantics differences) for review until it's supported — the same accept-or-reject principle applies here as anywhere: silently ignoring an accepted flag is the worst of the three states. One bail! in the Review arm.
  2. Honor it: extend InitialOperation::Review / the review turn to carry output_schema like UserTurn does. Doable, but note the review flow drives a purpose-built prompt whose output the pipeline itself consumes — arbitrary user schemas may fight the internal contract.
  3. Probably the best end-state: the review pipeline already produces a native structured resultReviewOutputEvent { findings: Vec<ReviewFinding>, overall_correctness, overall_explanation, overall_confidence_score } (protocol/src/protocol.rs#L3195-L3225). Exposing that via codex exec review --json (or writing it to --output-last-message as JSON) gives automation a stable, richer contract than a user-supplied schema squeezed into a prose turn — your {"verdict": pass|block} example is a trivial projection of overall_correctness.

(1) is a one-liner that removes the silent failure immediately; (3) is the durable answer for CI-gating use cases like yours.

naipi11 · 9 days ago

codex exec review --output-schema is accepted by clap and then dropped. The review arm never loads the schema, so you get prose and exit 0.

Method:

  1. Do not use --output-schema with codex exec review or top-level codex review for automation.
  2. If you need a schema, run ordinary codex exec --output-schema schema.json -o out.json. That path does honor the file.
  3. If you still want the review command, treat -o as free text and validate it yourself. Exit 0 is not proof the schema was applied.

Evidence: on current main, ExecCommand::Review builds InitialOperation::Review and never calls load_output_schema. The plain-prompt / resume / fork arms do.

Independent community workaround; not an official OpenAI fix.