Connect to Codex CLI from the ChatGPT desktop app

yes plz

Related #2614

While the Desktop App can connected to local instances in theory, often instances are running on remote servers and would require an outbound connection from codex CLI instead, but both should be supported an the UI should be similar in the ChatGPT Mobile App, web app and desktop app, just how it connects can differ.

You are suggesting to show it in the chat view, while I suggested to instead show in the Codex view. Right now the Codex is not available native in the desktop app, only the web app and mobile app.

One big disadvantage of showing it in the regular chat view is that it would make the chat list incredibly cluttered unless it switched the chat list betwen the codex agent chats and regular chats when switching models.

It might be better to show in the codex view rather than the chat view, otherwise it will just be cluttered, but that requires the codex view to be implemented in the desktop app too. In any case the UI of either the chat view or the codex view would need to be reworked a bit to get a good UX.

Below is a concrete, path to make Codex CLI feel first‑class across Desktop, Web, and Mobile—without cluttering the chat experience or compromising security.

---

TL;DR – Recommended direction

• Architecture: Support two connection modes behind one UX.

1. Local Bridge (Desktop ↔ 127.0.0.1) for the fastest, private loopback.
2. Cloud Relay (CLI ↔ outbound WebSocket) so Web/Mobile/Desktop can reach remote or headless CLI instances with no inbound ports.

• UX: A Codex workspace in Desktop (plus Web/Mobile), with an Agent List and a Session view. Keep regular chats uncluttered. Offer lightweight chat-side attach for “send this message to an agent” without moving users out of chat.
• Security & privacy: Explicit scopes (FS, shell, network, git), per‑folder allowlists, capability negotiation, and opt‑in history/indexing.
• MVP scope: Local discovery + selection; single-agent sessions; history persisted and labeled; basic error states; “Stop/Interrupt”; minimal settings for scopes.

---

1) Critique of the two proposals (and high‑impact fixes)

A. “Show Codex CLI as a selectable model in Chat”

Drawbacks

• Chat list clutter: dozens of agent sessions pollute the single chat rail.
• Mental model collision: regular model chats vs. stateful/long‑running agents with tools and FS access.
• Missing agent constructs: diffs, runs, approvals, and logs don’t map cleanly into a plain chat transcript.

Fixes

• Scoped list switching: when a Codex agent is the current “model,” the left rail switches to Agent Sessions only. A clear breadcrumb (“Chat ▸ Codex ▸ AgentName”) prevents confusion.
• Inline agent sidecar: show Runs, Diffs, and Tools in a right‑hand side panel; chat stays center.
• Smart routing: if a user tries to attach files or request FS actions, guide them to the Codex workspace (one click), not the regular chat view.

B. “Show Codex in a separate Codex view (Agents list)”

Drawbacks

• Discoverability: developers living in chat may never open Codex view.
• Context switching: jumping between chat and Codex can feel heavy.

Fixes

• Hybrid navigation:

• A Codex workspace (Agents list + Sessions) for deep work.
• A chat-side attach action: from any chat message, “Send to Agent…” (choose agent → opens a lightweight session drawer or deep‑links into Codex workspace).
• A global switcher (⌘K / Ctrl‑K) that lists Chats, Agents, Sessions in one searchable surface.

Bottom line: Use a Codex workspace as the primary surface and chat-side attach as the bridge. Avoid polluting the main chat list.

---

2) Architecture options

| Option | How it connects | Where it works | Pros | Cons | Recommendation |
| --------------- | -------------------------------------------------------- | ------------------ | -------------------------------- | ------------------------------------ | -------------- |
| Local Bridge | Desktop app probes 127.0.0.1:PORT (WebSocket) | Desktop only | Zero cloud, low latency | No Web/Mobile reach; WSL/path quirks | MVP |
| Cloud Relay | CLI keeps outbound TLS WS to a relay; clients fan‑in | Desktop/Web/Mobile | NAT‑proof, remote servers, teams | Requires auth & relay infra | Phase 2 |
| P2P (STUN/TURN) | NAT traversal between client↔CLI | Desktop/Web/Mobile | Potentially lower relay cost | Complex, brittle | Later/Optional |

Handshake (both modes)

• Capability advert: CLI sends version, tools, FS scopes, project root(s), sandbox flags.
• Auth: Device‑link or OAuth device code (“Run codex link → enter code in app”), short‑lived tokens, mTLS between relay and CLI where applicable.
• Health: heartbeats, backoff, retry, status lights.

---

3) UX system

A. Agents list (Codex workspace left rail)

• Sections: Online (green), Idle (yellow = waiting for input), Offline (gray).
• Grouping: by Project (root path) and Host. Deduplicate by (host, projectRoot); allow aliases when two CLIs attach to the same dir.
• Actions on hover: Start/stop, Pin, Share (team), Settings (scopes, approvals).

B. Session view (center)

• Top bar: Agent name, project root, status pill, “Stop/Interrupt,” “Open project folder.”
• Main stream: Interleaved Messages, Run steps, Diffs, Command output.
• Right side panel (toggle):

• Files (allow‑listed tree; quick open),
• Tools (Drive/Git/Shell toggles),
• Runs (history),
• Approvals (pending prompts).
• Input affordances: Dictation, drag‑and‑drop files, “Run in terminal” vs “Let agent apply.”

C. Chat-side attach (bridge from normal chat)

• Right‑click / ellipsis on a message → “Send to Agent…” → choose agent → confirm context items (message, files).
• Creates/links a session and shows a mini session drawer; deep‑link available.

D. Mobile/Web parity

• Cloud Relay agents appear the same as Desktop agents.
• Push notifications when an agent requests input or completes a run.

---

4) Permissions, safety, and privacy

• Scopes at connect time: FS(Read/Write), Shell, Network, Git, Package Manager, Custom Tools.

• Each scope can be Allow, Ask, or Deny (policy per agent).
• Folder allowlists: macOS security‑scoped bookmarks / Windows folder picker / Linux directory picker. Persisted per agent.
• Command preview: For dangerous actions (e.g., rm -rf), require inline approval; show exact command and impact.
• History & indexing:

• Default: store locally in Desktop app; cloud indexing off.
• Per‑agent toggle to sync transcripts and diffs for cross‑device search; redact secrets by default.
• Audit & export: Per session event log (who, what, when), one‑click export of diffs/messages.

---

5) Handling multi‑instance, WSL, and remote hosts

• Instance identity: (agentId, hostName, projectRoot, capabilitiesHash).
• WSL & path mapping: Detect WSL; normalize paths (C:\… ↔ /mnt/c/...); show active environment (WSL1/2, distro).
• Remote naming: Show hostName and an optional label (e.g., gpu-rig).

---

6) Feature mapping (MVP → Phases)

MVP (Desktop + Local Bridge)

• Discovery on 127.0.0.1:PORT (configurable; single port or mDNS).
• Agents list + Session view in Desktop.
• History persistence (tagged as Codex); offline states; Stop/Interrupt.
• Basic error handling: agent unavailable on phone/web; show last known state.

Phase 2 (Cloud Relay + Parity)

• CLI outbound registration (codex link), team sharing, push notifications, Web/Mobile use.
• File/image forwarding, secure upload/download channel.
• Drive/Git/Shell tools proxied by CLI; cloud tools (e.g., Drive search) proxied by client—see Tool Broker below.

Phase 3 (DX polish)

• One‑click install & run from Desktop; auto‑update CLI (opt‑in).
• Templates (Node, Python, monorepo) with pre‑scoped folders.
• Diff visualizations (three‑pane, inline comments), code map, and run timelines.

---

7) Tool Broker: where calls should execute

• Local/host‑sensitive tools (FS, Shell, Git): CLI executes; app shows previews and asks for approvals.
• Cloud tools (Drive search, issue trackers): Client/App executes via user’s cloud auth; results forwarded to agent.
• Conflict behavior: If both can run a tool, prefer the actor with richer auth/context; expose a per‑agent override.

---

8) Interrupts, runs, and commands

• Stop button maps to CLI interrupt (SIGINT or cooperative cancel).
• Command mode: Prefix (!git status) or button converts the next instruction to a terminal command preview; user approves and the CLI runs it.
• Run queue: One active run per agent by default; queued runs display ETAs; users can reorder/cancel.

---

9) Answers to the Open Questions

• Should local Codex chats sync to cloud search?

Default off. Offer per‑agent opt‑in with redaction. Show a shield icon when a session is local‑only.

• Should tool‑calls be proxied by the app or Codex?

Split: host‑sensitive via CLI; cloud/SAAS via client. Use a Tool Broker to route and show provenance.

• How would it work with projects?

Agents attach to Project Roots; Sessions inherit project context (env, scopes). Agents can be grouped under Projects in the rail.

• Different system prompt?

Yes—use a Codex‑operator system prompt that emphasizes reproducibility, conservative edits, and tool etiquette. Keep it visible and overridable per agent.

• Multiple Codex instances for the same directory?

Allowed; show as siblings under the same project with different host labels. Offer a “prefer fastest/closest” default.

• Sandbox and approval options in UI?

Yes—per‑agent Settings: scopes, approvals, environment, redaction, indexing. Quick toggles in the right panel.

• Manual run vs background process?

MVP: manual (codex start). Phase 2: optional background helper managed by Desktop (with clear menu bar tray and Quit).

• Interrupts and CLI‑only features (!git status)?

Supported as command mode with preview + approval. Stop maps to interrupt.

• Desktop‑only or also web?

Desktop: Local Bridge + Cloud Relay. Web/Mobile: Cloud Relay only. The UI looks the same; the transport differs.

• Will developers prefer a richer UI to a TUI?

Expect both camps. Provide workspace UX for rich interactions, terminal‑first workflow for purists, and chat‑side attach for casual use.

---

10) Key UI flows (concrete)

Connect a local agent (MVP):

1. User runs codex start (defaults to port 8719).
2. Desktop app detects agent → shows toast “Found Codex @ ~/repo” → Connect.
3. User selects allowed folders and scopes → agent appears Online in Codex workspace.

Link a remote agent (Phase 2):

1. On the remote machine: codex link → device code displayed.
2. In Desktop/Web/Mobile: Add Agent → Enter code → agent shows under “Remote”.
3. Push notifications when input is requested.

Send a chat message to an agent:

• From any chat message → “Send to Agent…” → pick agent → a mini session drawer opens with the forwarded content and a link to the full session.

---

11) Reliability & diagnostics

• Status footer in Session: latency, last heartbeat, queue length.
• Diagnostics panel: recent errors, last 50 logs, “copy bundle” for bug reports.
• Graceful degradation: when agent goes offline, keep read‑only transcript with a Reconnect CTA.

---

12) Security checklist

• Outbound‑only networking for relay; no inbound ports required for Web/Mobile.
• Short‑lived tokens; per‑agent revocation; optional mTLS between relay and CLI.
• Content redaction for secrets; opt‑in cloud indexing; explicit scope gates.
• Clear UX that distinguishes local‑only vs synced sessions.

---

13) What to build first (bite‑sized backlog)

1. Local Bridge detection + connect flow with folder picker and scopes.
2. Codex workspace (Agents list + Session view) in Desktop.
3. Session transcript with Runs, Diffs, Approvals, Stop.
4. Chat-side attach action.
5. Robust offline/unknown state handling.
6. Path normalization for WSL/macOS/Windows.
7. Export session (JSON + patch bundle).

Phase 2 adds Cloud Relay, mobile/Web parity, push notifications, team sharing, and file/image forwarding.

---

14) “Where this could go” (optional polish)

• Code map + blame overlays in the right panel.
• Run recipes (e.g., “Bump dependency safely”) as one‑click macros.
• Session templates per framework (React, Django, Go).
• Latency‑aware routing (prefer closest agent).

In the Codex web you also launch an agent, and it doesn’t clutter the chat list. After all, you’re running the main session, while the internally launched ones are stored within it.

it also need to come with tabs, like how terminal / browser we can keep open fresh chat while not closing it. then well be really great.

ram leak n performance issue still there. just a plain bring app in front cost 54% on m4 cpu.
less gui more hotkey make ux better, eg raycast chat.

that say chatgpt.com seems also can intergrat this.

The local bridge idea seems right, but I would separate two product goals:

1. Use the ChatGPT app as a richer UI for Codex CLI.
2. Preserve the local CLI session as the execution authority, so the user is still controlling the same process/workdir/approval state rather than a second conversation that happens to share history.

The second goal is especially important once phone access enters the picture. A mobile/desktop app can be nicer than a TUI, but if it does not act on the same live local session, users still get stale context, duplicated threads, or unclear approval state.

I built Faryo as a narrower open-source experiment around that second goal: Codex/Claude/shell stay running inside tmux on the owner machine, and phone/desktop browser acts as a lightweight workbench for compact output, short input, approve/interrupt, and handoff. It is not a ChatGPT app integration, but it may be a useful comparison for the "local Codex remains the live source of truth" design: https://github.com/Snailflyer/faryo

Would your desired MVP require ChatGPT's full history/search UI, or would a small same-session workbench be enough if it controls the existing Codex CLI process reliably?