`codex exec review` accepts but ignores `--output-schema`
Component
Codex CLI bundled with the ChatGPT desktop app for macOS.
Version
- Reproduced on
codex-cli 0.148.0-alpha.9 - Previously reproduced on
0.147.0-alpha.6.6 - macOS 26.6.1 (arm64)
Summary
codex exec review accepts and advertises --output-schema, but the review command ignores the schema and writes its normal prose review to --output-last-message. The command exits 0, so automation cannot distinguish this from successful schema enforcement.
Ordinary codex exec honors the same schema.
Minimal reproduction
Create a repository containing a commit with an obvious defect:
# example.py
def divide(a, b):
return a / 0
Use this schema:
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"additionalProperties": false,
"required": ["verdict"],
"properties": {
"verdict": {
"type": "string",
"enum": ["pass", "block"]
}
}
}
Run:
codex exec review \
--ephemeral \
--ignore-user-config \
-m gpt-5.6-terra \
-c 'model_reasoning_effort="low"' \
--commit HEAD \
--output-schema minimal-schema.json \
-o review.txt
Actual result
The command exits 0 and review.txt contains prose:
The newly added function cannot successfully divide any inputs because it unconditionally divides by zero.
Review comment:
- [P1] Divide by the supplied denominator — .../example.py:2-2
This always evaluates a / 0 ...
The response does not match the supplied schema.
Control
The same schema works with ordinary exec:
codex exec \
--ephemeral \
--ignore-user-config \
-m gpt-5.6-terra \
-c 'model_reasoning_effort="low"' \
--output-schema minimal-schema.json \
-o control.txt \
'Return verdict pass. Do not inspect files or call tools.'
control.txt contains:
{"verdict":"pass"}
Expected result
Either:
codex exec reviewforwards--output-schemato the review turn and guarantees that the final message matches it, or- the review subcommand rejects
--output-schemaas unsupported instead of accepting it as a silent no-op.
Impact
This makes exec review unsafe for machine-readable CI gates: the documented flag appears to provide schema enforcement, but automation receives unstructured prose with a successful exit code.
4 Comments
I traced this on current
main(22bf16a37ed45006c0226541874abd7449c29911). The schema is being lost at a specific dispatch boundary rather than in the output writer.codex-rs/exec/src/cli.rsdefinesCli::output_schemaas a global exec option, so Clap accepts it before the nestedreviewsubcommand.run_main()extracts that value intooutput_schema_pathand passes it intorun_exec_session().run_exec_session(), theExecCommand::Reviewmatch arm constructsInitialOperation::Review { review_request }immediately. The normal, resume, and prompted-fork arms callload_output_schema(...)and carry the value inInitialOperation::UserTurn; the review arm never reads it.InitialOperation::ReviewsendsReviewStartParams { thread_id, target, delivery }. There is currently nooutput_schemafield at thereview/startboundary.So
--output-schemais syntactically valid, survives top-level parsing, and is then silently discarded before the review turn is created. That also explains the successful exit with ordinary prose.There are two coherent contracts:
review/startrequest and propagate it into the review turn, preserving the same final-output validation used by an ordinary user turn.ExecCommand::Reviewis combined withoutput_schema_path.is_some().I would favor the second as the minimum safe correction unless structured review output is an intended API: it removes the false-success mode without prematurely extending the app-server review contract. A focused regression should invoke
codex exec review --output-schema ...and assert a nonzero exit plus an actionable error before any model request. If propagation is preferred, the stronger regression belongs at thereview/startboundary and should assert that the resulting Responses request contains the requested output format; the existing ordinary-exec output-schema tests do not exercise this branch.I have not opened a PR because the repository requires an explicit maintainer invitation for external code contributions. If the early-rejection contract matches the intended behavior, I would be happy to implement the guard and focused tests.
Root cause and a ready fix for this.
Root cause.
--output-schemais aglobalflag on the exec CLI (codex-rs/exec/src/cli.rs), so clap accepts it for every subcommand. But the review path never reads it: inrun_exec_session(codex-rs/exec/src/lib.rs), theExecCommand::Reviewarm callsbuild_review_request(review_cli)and issuesreview/start, while only theUserTurn/Resume/Forkarms callload_output_schema(...).ReviewStartParams(codex-rs/app-server-protocol/src/protocol/v2/review.rs) has no schema field, so the flag is dropped before any model request is built. That is why plaincodex exechonors the same schema butcodex exec reviewwrites prose and still exits 0.Fix (option 2 from the issue). Reject the flag instead of silently ignoring it.
build_review_requestis the single choke point for bothcodex exec reviewand top-levelcodex review, so checking the resolved--output-schemapath there bails with:This fires before the app-server/API work starts, so it fails loudly with a non-zero exit instead of returning unstructured prose.
I implemented this with a unit test and verified it (
cargo check -p codex-exec,just test -p codex-exec review_request, clippy + fmt clean). The change is on this branch: https://github.com/weivwang/codex/tree/fix/exec-review-reject-output-schemaDiff summary:
Option 1 (actually forwarding the schema to the review turn so the final message matches it) is the better long-term fix, but it requires plumbing
output_schemathroughReviewStartParams→ the app-serverreview/starthandler → the coreOp::Reviewloop, which is a larger cross-crate change. The reject-first approach makes the CLI honest now and can be followed by the forwarding work.Happy to open the PR if a maintainer is able to invite/allow it.
Confirmed on
main@ 1f41cc5d92, and the mechanism is a one-glance read in the exec dispatch:https://github.com/openai/codex/blob/1f41cc5d92722748e45cae9cecc6d883a4e7cbb1/codex-rs/exec/src/lib.rs#L706-L710
The
ExecCommand::Reviewarm buildsInitialOperation::Review { review_request }and never callsload_output_schema— while the plain-prompt,Fork, andResumearms all do and thread the schema into theirInitialOperation::UserTurn(#L734, #L756, and the root arm).--output-schemais a top-levelcodex execflag, so clap happily accepts it for thereviewsubcommand, and the operation it maps to simply has no schema slot. Exit 0 + prose output follows — exactly your automation hazard.Three fix options, not mutually exclusive:
--output-schema(and arguably--output-last-messagesemantics differences) forreviewuntil it's supported — the same accept-or-reject principle applies here as anywhere: silently ignoring an accepted flag is the worst of the three states. Onebail!in the Review arm.InitialOperation::Review/ the review turn to carryoutput_schemalikeUserTurndoes. Doable, but note the review flow drives a purpose-built prompt whose output the pipeline itself consumes — arbitrary user schemas may fight the internal contract.ReviewOutputEvent { findings: Vec<ReviewFinding>, overall_correctness, overall_explanation, overall_confidence_score }(protocol/src/protocol.rs#L3195-L3225). Exposing that viacodex exec review --json(or writing it to--output-last-messageas JSON) gives automation a stable, richer contract than a user-supplied schema squeezed into a prose turn — your{"verdict": pass|block}example is a trivial projection ofoverall_correctness.(1) is a one-liner that removes the silent failure immediately; (3) is the durable answer for CI-gating use cases like yours.
codex exec review --output-schemais accepted by clap and then dropped. The review arm never loads the schema, so you get prose and exit 0.Method:
--output-schemawithcodex exec reviewor top-levelcodex reviewfor automation.codex exec --output-schema schema.json -o out.json. That path does honor the file.-oas free text and validate it yourself. Exit 0 is not proof the schema was applied.Evidence: on current main,
ExecCommand::ReviewbuildsInitialOperation::Reviewand never callsload_output_schema. The plain-prompt / resume / fork arms do.Independent community workaround; not an official OpenAI fix.