[Windows][Computer Use] Add target-window-only capture, background input, and no-overlay mode for desktop EXE automation

Open 💬 0 comments Opened Aug 28, 2026 by lanfengc

What variant of Codex are you using?

app

What feature would you like to see?

Summary

I use Codex on Windows for software development and frequently need Computer Use to test and operate external desktop EXE applications.

The current Computer Use implementation works functionally, but on Windows it can cause significant desktop/UI lag and shows a blue visual overlay during interaction.

For desktop application development, especially when operating a specific known target HWND, capturing or compositing more of the desktop than necessary is inefficient.

I would like to propose a dedicated window-scoped/background automation mode for the Computer Use plugin.

The desired behavior is:

Target HWND
    ↓
Capture only target window
    ↓
Model analyzes screenshot
    ↓
Return client-relative coordinates
    ↓
Send background mouse/keyboard input to target HWND
    ↓
Capture target window again

without continuously affecting the entire Windows desktop.

---

Use case

My primary workflow is:

Codex modifies application code
        ↓
Builds the Windows EXE
        ↓
Launches the EXE
        ↓
Uses Computer Use to test the UI
        ↓
Clicks buttons / enters values / verifies state
        ↓
Finds bugs
        ↓
Modifies code and repeats

Many of the external applications I work with do not expose a useful UI Automation/accessibility tree.

Examples include:

custom-drawn Win32 applications
Qt applications
Delphi applications
CEF/Electron applications
GPU-rendered/custom UI
legacy industrial/business software

For those applications, screenshot-based visual grounding is necessary.

The ideal workflow is therefore:

screenshot(target HWND)
        ↓
vision model
        ↓
(x, y) in client coordinates
        ↓
background click(target HWND, x, y)

This is similar to how traditional Windows automation tools operate.

---

Current problems

1. Computer Use can cause system-wide lag on Windows

When Computer Use is active, mouse movement and desktop interaction can become noticeably less responsive.

The problem is not necessarily high CPU usage; it appears related to the Windows capture/compositor/overlay path.

Disabling Computer Use significantly improves responsiveness.

---

2. Blue overlay / visual layer adds unnecessary desktop rendering work

Computer Use shows a blue visual overlay during interaction.

For automated EXE testing this visual overlay is usually unnecessary.

I would prefer an option such as:

computer_use.visual_overlay = false

or:

headless_visual_mode = true

The agent should still receive screenshots, but the Windows desktop should not need to render an agent cursor, highlight, border, or overlay.

---

3. Target-window capture should be explicitly supported

If the agent has already selected a specific target window/HWND, Computer Use should capture only that window.

Conceptually:

get_window_state({
    window: targetWindow,
    include_screenshot: true
})

should guarantee that the screenshot corresponds only to that requested window.

It should not depend on monitor-wide capture unless a fallback is explicitly required.

This would also reduce the risk of accidentally capturing unrelated windows.

---

4. Add an explicit capture backend / capture policy

A possible API/configuration:

[computer_use.windows]

capture_scope = "window"
capture_backend = "auto"

show_overlay = false
show_agent_cursor = false

allow_monitor_fallback = false

Possible capture pipeline:

Target HWND

1. PrintWindow / equivalent window capture
        ↓
2. Windows.Graphics.Capture CreateForWindow(HWND)
        ↓
3. optional visible screen-region fallback

Monitor-level screenshot should be the final fallback, not the default behavior when a valid target HWND exists.

---

5. Add background mouse input

For many traditional Windows applications, Computer Use does not need to move the physical user cursor.

It could provide:

background_click(window, x, y)
background_double_click(window, x, y)
background_right_click(window, x, y)

background_drag(
    window,
    start_x,
    start_y,
    end_x,
    end_y
)

background_scroll(window, x, y, delta)

Possible Windows implementation:

WM_MOUSEMOVE
WM_LBUTTONDOWN
WM_LBUTTONUP
WM_LBUTTONDBLCLK
WM_RBUTTONDOWN
WM_RBUTTONUP
WM_MOUSEWHEEL

When appropriate, the implementation could resolve the deepest child HWND using the target client coordinates before sending the message.

Coordinates should be client-relative, not desktop absolute coordinates.

---

6. Add background keyboard input

Similarly, support a background keyboard strategy:

WM_SETTEXT
WM_CHAR
WM_KEYDOWN / WM_KEYUP

with foreground SendInput only as an explicit fallback.

If background input is unsupported by the target application, Computer Use should return something like:

{
    "status": "background_input_unavailable",
    "foreground_fallback_available": true
}

It should not silently take control of the user's physical keyboard/mouse.

---

7. Separate vision from physical desktop control

For applications without accessibility trees, the model still needs visual screenshots.

But visual grounding does not inherently require foreground desktop control.

A more efficient architecture would be:

                  Codex
                    │
                    ▼
              Computer Use
                    │
        ┌───────────┴───────────┐
        │                       │
 Window Capture            Background Input
        │                       │
 target HWND                 target HWND
        │                       │
 screenshot                  PostMessage
        │                       │
        └─────────┬─────────────┘
                  │
                  ▼
              Target.exe

The user's IDE and mouse could remain usable while Codex tests the application in the background.

---

8. Proposed operating modes

Computer Use could expose three Windows modes.

Mode A — Background / Window Scoped

Preferred for development/testing:

Capture: target HWND only
Input: background
Overlay: off
Physical cursor: untouched
Foreground switching: disabled

Mode B — Hybrid

Capture: target HWND
Input: background first
Fallback: foreground input with confirmation
Overlay: optional

Mode C — Current interactive desktop mode

For cases where true physical desktop interaction is required.

Capture: desktop/window as needed
Input: foreground
Overlay: enabled

---

Why this matters for Codex specifically

Computer Use inside a coding agent has different requirements from a general-purpose GUI assistant.

A coding-agent workflow repeatedly performs:

edit
→ build
→ launch
→ inspect
→ click
→ verify
→ edit

The target application is often already known.

Therefore Computer Use does not need to behave like an agent taking over the whole physical desktop.

For development workflows, target-window automation should be a first-class mode.

---

Security / privacy benefit

Window-scoped capture would also improve privacy.

When the target is:

MyApplication.exe

the model should ideally receive only:

MyApplication.exe pixels

rather than potentially capturing:

email
messaging applications
browser windows
IDE content
unrelated desktop notifications

A strict target-window mode provides a much cleaner security boundary.

---

Suggested acceptance criteria

A future Windows Computer Use version would ideally support:

[ ] enumerate and select target HWND
[ ] capture only selected HWND
[ ] capture an occluded target window where technically possible
[ ] no blue overlay when configured
[ ] no agent cursor animation when configured
[ ] return screenshot directly to the model
[ ] client-relative coordinates
[ ] background click
[ ] background double-click
[ ] background right-click
[ ] background drag
[ ] background mouse wheel
[ ] background text/key input
[ ] explicit foreground fallback
[ ] no silent takeover of physical mouse/keyboard
[ ] no monitor capture when strict window mode is enabled
[ ] screenshot identity is validated against the requested target HWND

---

Expected result

For a target application running behind another window:

Visual Studio / IDE remains foreground
User continues using mouse normally

Target.exe remains background
        ↓
Computer Use captures Target.exe
        ↓
model sees target screenshot
        ↓
model determines (x, y)
        ↓
Computer Use sends background click
        ↓
captures Target.exe again
        ↓
model verifies result

There should be:

No full-desktop visual overlay
No physical mouse movement
No foreground-window stealing
No significant Windows/DWM stutter

---

Additional note

There are already third-party/open-source Windows automation implementations that demonstrate that a useful subset of this architecture is practical using combinations of:

HWND
PrintWindow
Windows Graphics Capture
PostMessage
WM_CHAR / WM_SETTEXT
client-relative coordinates

It would be very useful if the official Codex Computer Use plugin exposed this as a supported Windows mode instead of requiring external MCP tools for desktop application testing.

Additional information

#36091 — System-wide lag disappears after stopping codex-computer-use.exe, suggesting that the Computer Use process or its capture/compositing path can significantly affect Windows desktop responsiveness.
#36677 — Codex Windows app becomes noticeably laggy or unresponsive when Computer Use is enabled, while disabling Computer Use restores normal responsiveness.
#38710 — Active Codex sessions can cause mouse lag, DWM stuttering, and degraded responsiveness across the Windows desktop.
#36603 — The screenshot returned by Computer Use may not always correspond correctly to the requested target window. This also highlights the need for a strict target-HWND capture mode for both correctness and privacy.

View original on GitHub ↗