Codex should require explicit approval before stateful infrastructure mutations
Summary
Codex Desktop agent autonomy allowed an assistant to perform a high-blast-radius stateful infrastructure operation while trying to complete an otherwise routine deployment-monitoring task. The agent did not recognize that it had crossed from application/service deployment into environment-level infrastructure mutation, and it did not require explicit user approval before running an environment deploy that replaced and deleted an Aurora/RDS database cluster.
This resulted in catastrophic data-loss risk for a customer: a staging database cluster was replaced by CloudFormation, the old cluster was deleted with final snapshot skipped, and normal manual recovery paths were not available afterward.
Why this is a Codex safety/product issue
The user had a long-running productive workflow with Codex, and Codex had been intentionally helpful and autonomous for normal engineering tasks: code edits, tests, commits, pushes, GitHub Actions monitoring, staging deploy verification, and operational troubleshooting.
In this incident, that same autonomy model failed to distinguish:
- a routine app/service deploy from an environment/infrastructure deploy
- stateless resources from stateful persistent resources
- recoverable deployment work from potentially irreversible data-plane mutation
- "staging" as lower customer-facing risk from "staging database" as potentially critical business/IP storage
The agent optimized for "keep going until deployment succeeds" and used a broader deploy command to apply an environment addon. The command mutated CloudFormation-managed environment resources and triggered RDS/Aurora replacement.
Concrete failure mode
The agent ran a Copilot environment deployment to apply an environment-level WAF addon after a service deploy had completed. CloudFormation determined that the database cluster resource required replacement. The old database cluster was deleted by CloudFormation. CloudTrail showed the delete operation skipped final snapshot creation. Afterward, retained automated backups / automated snapshots for the deleted cluster were not visible through normal RDS/AWS Backup APIs.
The agent did not:
- stop before running an environment deploy against shared infrastructure
- classify the operation as high risk because it could touch stateful resources
- inspect the generated CloudFormation changes for RDS replacement
- verify or create a fresh recoverable backup/snapshot first
- ask for explicit approval before a command capable of replacing/deleting persistent state
- preserve before repair once database replacement symptoms appeared
Expected behavior
Codex should have a strong built-in guardrail for stateful infrastructure mutations. It should require explicit user confirmation before running commands or applying IaC plans that may affect persistent state, including but not limited to:
copilot env deploy- CloudFormation stack updates involving
AWS::RDS::*,AWS::ElastiCache::*,AWS::OpenSearchService::*,AWS::S3::Bucket,AWS::KMS::*,AWS::SecretsManager::*, queues/topics with retained messages, or backup resources - Terraform/Pulumi/CDK/apply operations involving persistent data stores
- RDS/Aurora create/replace/delete/restore/modify operations
- backup retention, deletion protection, snapshot, or PITR policy changes
- DNS/security/certificate changes that may affect production auth or data access
Before proceeding, Codex should present a concise risk summary and ask for explicit approval. For database-impacting work, it should require or strongly encourage:
- confirmed current restore path or fresh manual snapshot
- understood
DeletionPolicy/UpdateReplacePolicy/ final snapshot behavior - rollback or recovery plan
- identification of affected environment and resources
Suggested product/model guardrail
Introduce a "stateful infrastructure stop rule" in Codex:
- Detect commands and IaC diffs likely to mutate persistent infrastructure.
- Downgrade autonomy: do not proceed on implicit task intent.
- Ask explicit approval with resource class, environment, and failure mode.
- For RDS/Aurora and similar resources, require a backup/restore verification step before mutation.
- If an infra operation behaves unexpectedly, stop and diagnose read-only rather than continuing with broader fix-forward actions.
This should apply even for staging environments, because staging databases can contain critical product data, customer data, or intellectual property.
Impact
Potential loss of critical business data/IP. This was not merely a failed deploy. The agent took an action that could destroy state in a way the customer could not manually recover.
Requested action
Please treat this as a high-priority Codex autonomy/safety bug. Codex needs stronger stateful-infrastructure guardrails so it can remain highly autonomous for low-risk engineering work while forcing explicit human approval for hard-to-recover infrastructure mutations.
1 Comment
Worth separating two mechanisms here. Asking the model to recognise "stateful infrastructure mutation" and pause is behaviour, and behaviour under a keep-going objective is exactly what failed. A deterministic pre-execution hook is the stronger control. Every shell command passes through a small script before it runs; the script matches against a blocklist (destructive verbs, protected resource names, force flags) and blocks the call before the tool executes, regardless of what the model believes about its task. The model never gets to argue with it.
Two design points from months of running this. First, tier by mechanical impact, not by how the action reads.
aws rds delete-db-clustersits in a different tier from a read no matter how confident the plan sounds. Second, the guard fails open. A bug in the hook must never wedge the session, so its failure mode is allow-plus-log, and the blocklist stays small and boring.A ten-line check catches most accidental damage for almost nothing.
I maintain a reference architecture documenting this exact pattern: https://github.com/jimy-r/agent-workspace-architecture/blob/main/PATTERNS.md#7-a-cheap-hook-beats-a-careful-agent