Native GUI / Computer Use Tools for Desktop Automation

What variant of Codex are you using?

CLI, App

What feature would you like to see?

Codex currently operates exclusively through CLI tools — it cannot see, click, or interact with graphical applications. Many real-world development tasks require GUI interaction: testing desktop apps, navigating browser-based tools, or automating repetitive UI workflows.

Claude Code shipped native computer use capabilities in its March 2026 research preview. Codex should have this too.

Working implementation

I've built a working implementation: codex-cua — a fork of Codex with native GUI / computer use tools integrated into the Rust codebase. It is fully functional on macOS today. See PR #3 for the full implementation.

Demo:

!GUI Demo

Tool set:

| Tool | Purpose |
|------|---------|
| gui_observe | Capture a screenshot and optionally ground a semantic target |
| gui_click | Click a UI element by semantic description or coordinates |
| gui_type | Type text into a focused or targeted input field |
| gui_key | Press keyboard shortcuts and special keys |
| gui_scroll | Scroll within a targeted area or the current surface |
| gui_drag | Drag from one element/location to another |
| gui_move | Move the cursor to an absolute position |
| gui_wait | Wait for a visual condition to appear on screen |

Implementation details

• Semantic grounding — Rather than relying solely on the model to output raw pixel coordinates, the agent describes what it wants to interact with (e.g., target: "the Save button in the toolbar") and a grounding engine resolves the description to screen coordinates. This decouples targeting accuracy from the primary model's vision capabilities and is more robust across resolutions and layouts.
• First-class observability — 8 GUI tools integrated as BuiltinToolCall turn items (like exec commands and MCP tool calls) with full lifecycle tracking (inProgress → completed/failed), visible in TUI/CLI and available in JSONL output.
• Safety-first — Experimental feature flag (disabled by default), emergency stop via physical key (Escape), physical resource locking to prevent concurrent sessions, host window exclusion to prevent the agent from interacting with its own terminal, and automatic hiding of non-target apps during GUI actions.
• Platform abstraction — A trait-based platform layer allows incremental platform support. macOS is implemented with a Windows stub ready.
• Grounding engine — Multi-round validation with refinement cropping, HiDPI-aware capture across Retina displays, and dual grounding modes (simple prediction or complex multi-round validation with simulation overlays).

Background

The design of codex-cua's GUI tools was inspired by Understudy, my open-source local AI agent project (TypeScript). Understudy operates your entire computer — GUI, browser, shell, messaging — from a single instruction. The semantic grounding approach, observe-act-verify loop, safety layers, and platform abstraction pattern in codex-cua all originate from Understudy's architecture.

Additional information

Would love to hear from the community:

• Is native GUI / computer use a direction the Codex team is considering?
• What safety and security concerns should be addressed first?
• Would a PR toward upstream integration be welcome?