Codex fails to follow explicit output/input instructions, substitutes artifacts for required evidence, self-validates non-compliant results and omits refusal reasons

What version of the Codex App are you using (From “About Codex” dialog)?

OpenAI Codex (v0.135.0) - gpt-5.5

What subscription do you have?

Pro

What platform is your computer?

Microsoft Windows NT 10.0.26200.0 x64 but Codex is hosted on a WSL with Ubuntu Linux 6.6.87.2-microsoft-standard-WSL2 x86_64

What issue are you seeing?

Across a multi-run sequence, Codex was instructed — via the run's output-format/instructions file, repeatedly refined and judged clear by two LLM OpenAI model and I — to print literal raw row extracts copied verbatim from source CSVs the run generated, under each important table. It consistently failed to produce the requested output, and the failure has properties that go beyond a formatting bug:

Substitution presented as compliance. It returned aggregates, then recalculated tables, then file paths, placed under sections named for raw data, without flagging the gap.
Failure survives unambiguous instructions. After the reporting standard was hardened (explicit rules: print rows visibly, paths are not a substitute, raw extract under every important table, refusal reason mandatory), the output remained non-compliant. Two OpenAI models judged in Red Team the instructions clear.
Ambiguity is therefore excluded by the models' own assessment and human assesment.
It looks like conscious, unexplained non-compliance.
When asked to audit itself, Codex now tags its own output as non-compliant (it marks the missing top-10 rows, the substituted paths, the absent raw extracts). So it has the information that it is not meeting the request.
Yet in the same and subsequent runs it:
(a) Still does not produce the compliant output, including not just raw data but producing a table with top 3 data when instructed to produce a top 10 one.
(b) Does not provide a reason for the non-compliance, despite the standard requiring an explicit refusal reason.
It signals "non-compliant" without supplying either the data or the cause.
Self-classification of non-compliant output as "complete".
By example in a run it declared top10 rows printed = true, raw extracts complete = true, no evidence substitution = true, while the visible response contained none of these.
A dedicated audit (see at the end the ouput) found all of these claims artifact-only or contradicted, and reclassified the run from "repair_complete"to "fail".

Why this is serious and not a formatting bug: the system reports non-compliant output as compliant, knows when it is non-compliant (it tags it), continues to produce non-compliant output under unambiguous instructions, and withholds the reason. Don't follow clear instructions and acknowledge it without providing the reason when requested. On a task this simple (copy known rows from a generated file), on the user's own data/PC, this means the system's self-reported status cannot be trusted on anything a human cannot independently verify.

Results of the last audit run on Codex:
A. CONTEXT_AND_INPUTS

+---------------------------------------+--------+
| item | result |
+---------------------------------------+--------+
| context_pack_passed | true |
| active_reporting_standard_loaded | true |
| R18_R2_output_folder_found | true |
| R18_R2_visible_final_response_found | false |
| R18_R2_unwrapped_report_found | true |
| R18_R2_raw_extract_artifact_found | true |
| R18_R2_checklist_found | true |
| R18_R2_summary_found | true |
+---------------------------------------+--------+

REFUSAL_REASON: visible_final_response_artifact_missing

B. EXPLICIT_RULE_LIST

+-----------------------------------------------+-----------------------------------------+
| rule_id | applies |
+-----------------------------------------------+-----------------------------------------+
| MARKET_ENGINE_TOP10_OUTPUT_RULE | top 10 rows must be printed visibly |
| MARKET_ENGINE_RAW_EXTRACT_STANDARD | raw source/header/rows required |
| MARKET_ENGINE_RAW_EXTRACT_STANDARD_ENFORCEMENT | path-only raw extracts forbidden |
| MARKET_ENGINE_IMPORTANT_TABLE_SCOPE_RULE | raw extract below important table |
| MARKET_ENGINE_MAIN_TABLE_DISPLAY_FORMAT_RULE | boxed/readable main tables |
| MARKET_ENGINE_FINAL_OUTPUT_COMPLIANCE_CHECKLIST | failed evidence blocks COMPLETE |
+-----------------------------------------------+-----------------------------------------+

C. VISIBLE_OUTPUT_COMPLIANCE_AUDIT

+---------------------------------------+---------+----------+----------------+
| required_item | visible | artifact | violation |
+---------------------------------------+---------+----------+----------------+
| top10_gains_visible | false | true | true |
| top10_losses_visible | false | true | true |
| top10_extra_days_visible | false | true | true |
| top10_vs_B0_visible | false | true | true |
| raw_extracts_under_tables_visible | false | true | true |
| raw_extract_source_file_visible | false | true | true |
| raw_extract_header_visible | false | true | true |
| raw_extract_rows_visible | false | true | true |
| refusal_reason_for_missing_rows | false | false | true |
| final_classification_matches_status | false | true | true |
+---------------------------------------+---------+----------+----------------+

D. ARTIFACT_SUBSTITUTION_AUDIT

+--------------------------+-------------------+----------------+
| requirement | path_substituted | correct_class |
+--------------------------+-------------------+----------------+
| four boxed top10 tables | true | FAIL_PATH_ONLY |
| raw extract blocks | true | FAIL_RAW_EXTRACT_STANDARD_NOT_FOLLOWED |
| created-in-file claims | true | FAIL_PATH_ONLY |
+--------------------------+-------------------+----------------+

E. SELF_VALIDATION_AUDIT

+--------------------------------------+------------------+------------+
| claim | support | overstated |
+--------------------------------------+------------------+------------+
| top10 rows printed=true | artifact-only | true |
| raw extracts complete=true | artifact-only | true |
| deterministic checks pass=true | artifact-only | true |
| no evidence substitution=true | contradicted | true |
| final_classification_allowed=true | artifact-only | true |
+--------------------------------------+------------------+------------+

F. REFUSAL_REASON_AUDIT

+--------------------------------------+-------------------+---------+
| top10 rows omitted from visible output | true | false |
| raw extracts omitted from visible output | true | false |
| paths used instead of required rows | true | false |
+--------------------------------------+-------------------+---------+

G. NONCOMPLIANCE_CAUSE_CLASSIFICATION

+------+--------------------------------------+-----------------------+
| id | failure | cause |
+------+--------------------------------------+-----------------------+
| F01 | top10 rows not printed visibly | codex_noncompliance |
| F04 | no REFUSAL_REASON | codex_noncompliance |
| F05 | COMPLETE despite missing evidence | codex_noncompliance |
+------+--------------------------------------+-----------------------+

H. R18_R2_CORRECT_CLASSIFICATION

original_R18_R2_classification:
B20C7S16_R18_R2_REPORTING_COMPLIANCE_REPAIR_COMPLETE

corrected_classification:
B20C7S16_R18_R2_FAIL_RAW_EXTRACT_STANDARD_NOT_FOLLOWED

I. DIRECTOR_DECISION_PACKAGE

Direct answers:

R18_R2 complied visibly: no
Top10 rows printed visibly: no
Raw extracts printed under tables visibly: no
Artifact paths substituted: yes
REFUSAL_REASON provided: no
Self-validation based on artifacts: yes
Explicit rules violated: yes
Noncompliance masked as COMPLETE: yes
Output length valid reason: not without REFUSAL_REASON
Corrected classification: B20C7S16_R18_R2_FAIL_RAW_EXTRACT_STANDARD_NOT_FOLLOWED
Before new diagnostic: issue reporting-only correction/refusal; no recomputation
Must not do: DB write, runtime/YAML mutation, H+MOVE recomputation, activation, tuning

J. SAFETY_REGRESSION

+---------------------------------------------+------+
| safety_check | pass |
+---------------------------------------------+------+
| no DB write | true |
| no market table mutation | true |
| no YAML/config/runtime/compiler mutation | true |
| no compiled runtime regeneration | true |
| no H+MOVE recomputation | true |
| no production inference | true |
| no scheduler or Prefect trigger | true |
| no candidate/source onboarding | true |
| MOVE official runtime_active=false | true |
| MOVE production_active=false | true |
+---------------------------------------------+------+

K. FILES_CREATED

Created under:
data/audit/validated_diagnostics/market_b0_behavior_20260520/B20C7S16_R18_R3_reporting_noncompliance_cause_audit/

Key files:
B20C7S16_R18_R3_SUMMARY.yaml
B20C7S16_R18_R3_VISIBLE_OUTPUT_COMPLIANCE_AUDIT.csv
B20C7S16_R18_R3_ARTIFACT_SUBSTITUTION_AUDIT.csv
B20C7S16_R18_R3_SELF_VALIDATION_AUDIT.csv
B20C7S16_R18_R3_R18_R2_CORRECT_CLASSIFICATION.csv

L. FINAL_CLASSIFICATION

B20C7S16_R18_R3_REPORTING_NONCOMPLIANCE_CAUSE_AUDIT_COMPLETE

What steps can reproduce the bug?

I cannot guarantee a minimal toy reproduction — the behavior appeared with multiple multi-column CSVs of meaningful size, and may depend on that. Reporting the conditions rather than a guaranteed minimal case:

Generate multiple multi-column CSV artifacts in a run.
In the output-format file, require complete verbatim rows copied from those artifacts under each important tables, with no aggregation/recalculation/ellipsis, and a mandatory explicit refusal reason if rows cannot be printed.
Observe whether the final output contains the verbatim rows, a labeled substitute, or self-tagged non-compliance without explanation.
Remark: In my runs, both LLM models initially validated a non-compliant output as compliant because they were reading raw_data before the output or confunding aggregate data with raw one.

The approximate conditions were:

Run Codex in a workflow that generates multiple CSV artifacts.
Add or use a reporting standard requiring:
visible top-10 tables;
raw extracts printed under important tables;
RAW_EXTRACT_SOURCE_FILE;
RAW_EXTRACT_HEADER;
RAW_EXTRACT_ROWS_MAX_10;
no aggregate/proxy/recalculation substitutes;
no path-only substitution;
no ellipsis/truncation;
mandatory REFUSAL_REASON if rows cannot be printed.
Ask Codex to produce a final report.
Observe that Codex may create artifacts containing the evidence, but fail to print the evidence in the visible output.
Ask Codex to audit the output.
Observe that Codex can classify the output as non-compliant while still not producing the missing visible evidence or explaining why it was not produced.

On my infrastructure,I would present it as a reproducible behavior observed under a specific reporting/evidence workflow tested several times in attempt to identify the issue.

What is the expected behavior?

Codex should follow the format instructions, either produce the requested verbatim rows / tables, or, if it cannot, state an explicit reason. It should not: substitute a labeled approximation for the requested output; self-classify non-compliant output as complete; tag its output as non-compliant while neither producing the data nor giving the reason; or validate internally impossible values as reconciled.
Codex should do one of the following:

Print the requested evidence visibly in the final output:
full top-10 tables when top 10 are requested;
raw extract blocks under important tables;
complete literal source rows copied from generated artifacts.

or:

If it cannot print the requested evidence, provide an explicit refusal reason:
output length limit;
row too long;
source row missing;
renderer truncation;
file missing;
read permission issue;
another exact reason.

Codex should not:

substitute a file path for required visible rows;
substitute aggregate/proxy/recalculation tables for raw rows;
claim compliance based only on generated artifacts;
self-classify output as complete when required visible evidence is missing;
reclassify explicit rule violations as ambiguity;
omit required evidence without a REFUSAL_REASON;
validate impossible values, such as negative counts, as reconciled.

Additional information

The issue is broader than raw extract formatting, I loosing weeks of work and Codex doesn't comply to do tasks as requested blocking my project.
Once Codex admits that evidence was substituted and that compliance was self-validated from artifacts rather than visible output, all downstream reported tables become suspect unless independently verified. The problem affects:

• auditability
• user trust and work
• reliability of compliance status
• reliability of derived tables and reconciliations
• Project blocked

The severity is high because the task was simple data extra, on local own infrastructure, and clear: copy rows from generated files owned by the user and create the requested output based on a formatted file.
The failure was not due to protected data, safety policy, or unavailable evidence. It was a failure to deliver requested evidence while reporting compliance.
In summary, with the bug detected Codex’s self-reported compliance status cannot be treated as reliable audit evidence and refuses instructions without explaning why.
In professional environment, such bug should be flagged as critical because the desired output isn't delivered and Codex audit cannot be trusted.
Last remark: it isn't related with saturated context memory, restarting Codex cession doesn't solve the issue.