Add evidence-backed completion reports for agent tasks

What feature would you like to see?

When Codex reports that a task is complete, it should provide a compact, structured mapping between the user's requirements and the observable evidence supporting each completion claim.

Codex currently often ends with a natural-language summary such as "implemented", "fixed", or "all tests pass". That can be useful, but it does not always distinguish clearly between:

• code that was changed;
• behaviour that was directly verified;
• behaviour inferred from inspection;
• checks that were not run;
• requirements that remain partially satisfied or uncertain.

The result is a trust problem: users must reconstruct the evidence manually from a long transcript, and an agent can sound finished even when some acceptance criteria were not tested.

Proposed behaviour

At task completion, Codex should build a lightweight completion record that maps each material requirement to:

• status: verified, implemented but unverified, partially satisfied, blocked, or not addressed;
• relevant files or symbols changed;
• validation commands that were actually run;
• observed results, including failures or skipped checks;
• remaining uncertainty or assumptions;
• optional commit or diff references where available.

Example:

Completion evidence

1. Add password-reset endpoint
   Status: verified
   Changes: src/routes/password_reset.ts, src/services/tokens.ts
   Evidence: npm test -- password-reset -> 12 passed

2. Reject expired tokens
   Status: verified
   Evidence: test `rejects_expired_reset_token` passed

3. Preserve existing login behaviour
   Status: partially verified
   Evidence: focused auth suite passed
   Not run: full end-to-end suite

4. Update API documentation
   Status: implemented, not rendered
   Changes: docs/api.md

The exact representation can be determined by the maintainers and should remain concise by default.

Important distinction

This feature should not expose hidden chain-of-thought or require the model to narrate private reasoning. It should report only externally inspectable evidence: requirements, file changes, commands, results, artifacts, and explicit uncertainty.

Suggested behaviour

• Capture the user's explicit requirements and acceptance criteria at task start or as they evolve.
• Update requirement status as work progresses.
• Record validation commands and their actual exit/result state rather than relying on memory.
• Do not mark a requirement as verified solely because code was written.
• Distinguish focused tests from broad regression coverage.
• Mark stale evidence when relevant files or dependencies change after a test.
• Include failed, skipped, unavailable, or cancelled checks.
• Avoid claiming that all tests pass when only a subset was run.
• Let users expand the report for details or keep it collapsed in normal workflows.
• Persist the completion record across compaction, resume, fork, and handoff.

Why this is useful

This would improve:

• trust in autonomous and long-running work;
• code-review handoffs;
• visibility into incomplete acceptance criteria;
• recovery after compaction;
• multi-agent synthesis;
• auditability in CI, enterprise, and regulated environments;
• the ability to continue a partially completed task without rereading the entire conversation.

It would also reduce false completion reports caused by confusing implementation with verification.

Suggested acceptance criteria

• A task with multiple explicit requirements produces one status entry per material requirement.
• Codex cannot label an item verified without attaching observable validation evidence.
• Tests that were not run are stated as not run.
• A failing check remains visible in the final report even if unrelated checks pass.
• Focused and full-suite validation are distinguished.
• Evidence remains available after context compaction and thread resume.
• The report can be consumed consistently by CLI, Desktop, VS Code, app-server, and subagent orchestration.
• Machine-readable output is available for integrations, while the normal user view remains concise.

Potential implementation direction

Codex already tracks tool calls, file edits, command results, plans, turns, and task state. A completion-evidence layer could aggregate those existing events into a durable, structured record instead of asking the model to reconstruct the entire audit trail from prose at the end.

{
  "requirement_id": "reject-expired-token",
  "status": "verified",
  "claim": "expired reset tokens are rejected",
  "evidence": [
    {
      "kind": "test_result",
      "command": "npm test -- password-reset",
      "exit_code": 0,
      "observed_at": "...",
      "depends_on": [
        {"path": "src/services/tokens.ts", "sha256": "..."},
        {"path": "tests/password_reset.test.ts", "sha256": "..."}
      ]
    }
  ]
}

test passes
→ later repair edits code
→ final summary remembers “test passed”
→ stale evidence is presented as current verification

requirements
× observed edits
× validation results
× current source hashes
→ completion projection