Codex silently reduced a paper-replication task, then presented a smoke test as evidence about the paper being "unsupported"

What issue are you seeing?

I asked Codex to reproduce a research paper using the repository’s paper-replication skill, train on the supplied MS1M dataset, reproduce the paper’s attack evaluation, verify
correctness, and use GPU 0.

Instead of completing the requested experiment, Codex silently restricted MS1M to:

max_classes: 512
max_per_class: 30

This reduced the available dataset from approximately 5.8 million images and 85,742 identities to only 13,528 images and 512 identities. It also shortened the paper’s training
schedules and used substitute architectures and evaluation protocols.

Codex then described the work as a reproduction and initially concluded that “the evidence does not support” some of the paper’s claims. Only after questioning the dataset size did it
acknowledge that the run was merely a reduced-scale diagnostic and that the requested task was unfinished.

The paper-replication skill did not request the 512-identity limit. Its instructions required paper-faithful baselines, explicit documentation of deviations, evidence before marking
targets matched, and no completion claim until validation passed.

What steps can reproduce the bug?

It is hard to reproduce because of the randomness of LLMs, but here is what I did

1. Create a repository containing:

• A research paper.
• The Codex paper-replication skill.
• Access to the paper’s requested training and evaluation datasets.

1. Ask Codex to:

• Use the skill to reproduce the paper.
• Train using the supplied full dataset.
• Reproduce and verify the paper’s attack evaluation.
• Use a specified GPU.

1. Observe whether Codex silently introduces dataset limits or shortened schedules to obtain a faster result.
2. Ask why the full dataset was not used.

What is the expected behavior?

Codex should:

1. Treat a reduced dataset or shortened schedule as a smoke test, never as the requested reproduction.
2. Ask for approval before materially reducing:

• Dataset size.
• Training duration.
• Model fidelity.
• Evaluation protocol.
• Attack strength.

1. Clearly label experiment tiers, such as:

• Smoke test.
• Reduced-scale diagnostic.
• Protocol-equivalent reproduction.
• Full reproduction.

1. Continue to the full experiment when the requested data and compute are available.
2. If required resources are unavailable, report the task as blocked or unfinished.
3. Never interpret failure of a surrogate or reduced-scale implementation as evidence against the paper.
4. Reserve terms such as “reproduced,” “matched,” and “evidence does not support” for experiments satisfying the relevant fidelity and acceptance requirements.

Additional information

_No response_