Dropped app-server events corrupt resumed Codex sessions

Open 💬 4 comments Opened Aug 12, 2026 by diegocabello
💡 Likely answer: A maintainer (github-actions[bot], contributor) responded on this thread — see the highlighted reply below.

What version of Codex CLI is running?

codex-cli 0.147.0

What subscription do you have?

$20/mo

Which model were you using?

gpt-5.6-sol

What platform is your computer?

Darwin 25.0.0 arm64 arm

What terminal emulator and version are you using (if applicable)?

Ghostty 1.2.0; no multiplexer

Codex doctor report

{
  "schemaVersion": 1,
  "generatedAt": "1786558899s since unix epoch",
  "overallStatus": "ok",
  "codexVersion": "0.147.0",
  "checks": {
    "app_server.status": {
      "id": "app_server.status",
      "category": "app-server",
      "status": "ok",
      "summary": "background server is not running",
      "details": {
        "control socket": "/Users/REDACTED/.codex/app-server-control/app-server-control.sock",
        "daemon state dir": "/Users/REDACTED/.codex/app-server-daemon",
        "mode": "ephemeral",
        "pid file": "/Users/REDACTED/.codex/app-server-daemon/app-server.pid (missing)",
        "settings": "/Users/REDACTED/.codex/app-server-daemon/settings.json (missing)",
        "status": "not running",
        "update-loop pid file": "/Users/REDACTED/.codex/app-server-daemon/app-server-updater.pid (missing)"
      },
      "remediation": null,
      "durationMs": 6
    },
    "auth.credentials": {
      "id": "auth.credentials",
      "category": "auth",
      "status": "ok",
      "summary": "auth is configured",
      "details": {
        "auth file": "/Users/REDACTED/.codex/auth.json",
        "auth storage mode": "File",
        "stored API key": "false",
        "stored ChatGPT tokens": "true",
        "stored agent identity": "false",
        "stored auth mode": "chatgpt"
      },
      "remediation": null,
      "durationMs": 0
    },
    "config.load": {
      "id": "config.load",
      "category": "config",
      "status": "ok",
      "summary": "config loaded",
      "details": {
        "CODEX_HOME": "/Users/REDACTED/.codex",
        "config.toml": "/Users/REDACTED/.codex/config.toml",
        "config.toml parse": "ok",
        "cwd": "/Users/REDACTED/mystuff/Coding/cascade-browser",
        "enabled feature flags": "shell_tool, view_image, unified_exec, shell_snapshot, code_mode_host, terminal_resize_reflow, sqlite, hooks, enable_request_compression, multi_agent, apps, tool_search_always_defer_mcp_tools, tool_suggest, plugins, in_app_browser, in_app_updates, browser_use, browser_use_full_cdp_access, browser_use_external, computer_use, remote_plugin, plugin_sharing, image_generation, resize_all_images, item_ids, skill_mcp_dependency_install, skill_search, mentions_v2, steer, guardian_approval, goals, collaboration_modes, tool_call_mcp_elicitation, auth_elicitation, personality, fast_mode, tui_app_server, remote_compaction_v2, workspace_dependencies",
        "feature flag overrides": "none",
        "feature flags enabled": "39",
        "log dir": "/Users/REDACTED/.codex/log",
        "mcp servers": "2",
        "model": "gpt-5.6-luna",
        "model provider": "openai",
        "sqlite home": "/Users/REDACTED/.codex"
      },
      "remediation": null,
      "durationMs": 1
    },
    "git.environment": {
      "id": "git.environment",
      "category": "git",
      "status": "ok",
      "summary": "git version 2.50.1 (Apple Git-155)",
      "details": {
        ".git entry": "directory",
        "PATH git #1": "/usr/bin/git",
        "PATH git entries": "1",
        "git branch": "master",
        "git build options": "git version 2.50.1 (Apple Git-155); cpu: arm64; no commit associated with this build; sizeof-long: 8; sizeof-size_t: 8; shell-path: /bin/sh; feature: fsmonitor--daemon; libcurl: 8.7.1; zlib: 1.2.12; SHA-1: SHA1_DC; SHA-256: SHA256_BLK",
        "git exec path": "/Applications/Xcode.app/Contents/Developer/usr/libexec/git-core",
        "git version": "git version 2.50.1 (Apple Git-155)",
        "repo detected": "true",
        "repo root": "/Users/REDACTED/mystuff/Coding/cascade-browser",
        "selected git": "/usr/bin/git"
      },
      "remediation": null,
      "durationMs": 1515
    },
    "installation": {
      "id": "installation",
      "category": "install",
      "status": "ok",
      "summary": "installation looks consistent",
      "details": {
        "PATH codex #1": "/opt/homebrew/bin/codex",
        "current executable": "/opt/homebrew/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/bin/codex",
        "install context": "npm (package /opt/homebrew/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin, bin /opt/homebrew/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/bin, resources /opt/homebrew/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/codex-resources, path /opt/homebrew/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/codex-path)",
        "managed by bun": "false",
        "managed by npm": "true",
        "managed by pnpm": "false",
        "managed package root": "/opt/homebrew/lib/node_modules/@openai/codex",
        "npm update target": "/opt/homebrew/lib/node_modules/@openai/codex"
      },
      "remediation": null,
      "durationMs": 1738
    },
    "mcp.config": {
      "id": "mcp.config",
      "category": "mcp",
      "status": "ok",
      "summary": "MCP configuration is locally consistent",
      "details": {
        "configured servers": "2",
        "disabled servers": "1",
        "stdio servers": "2"
      },
      "remediation": null,
      "durationMs": 0
    },
    "network.env": {
      "id": "network.env",
      "category": "network",
      "status": "ok",
      "summary": "network-related environment looks readable",
      "details": {
        "proxy env vars": "none"
      },
      "remediation": null,
      "durationMs": 0
    },
    "network.provider_reachability": {
      "id": "network.provider_reachability",
      "category": "reachability",
      "status": "ok",
      "summary": "active provider endpoints are reachable over HTTP",
      "details": {
        "ChatGPT base URL": "https://chatgpt.com/backend-api/ reachable (HTTP 403)",
        "reachability mode": "ChatGPT auth"
      },
      "remediation": null,
      "durationMs": 1733
    },
    "network.websocket_reachability": {
      "id": "network.websocket_reachability",
      "category": "websocket",
      "status": "ok",
      "summary": "Responses WebSocket handshake succeeded",
      "details": {
        "DNS": "2 IPv4, 0 IPv6, first IPv4",
        "auth mode": "chatgpt",
        "connect timeout": "15000 ms",
        "endpoint": "wss://chatgpt.com/backend-api/<redacted>",
        "handshake result": "HTTP 101 Switching Protocols",
        "model provider": "openai",
        "models etag present": "true",
        "provider name": "OpenAI",
        "proxy env vars": "none",
        "reasoning header": "false",
        "server model present": "false",
        "supports websockets": "true",
        "wire API": "responses"
      },
      "remediation": null,
      "durationMs": 2412
    },
    "runtime.provenance": {
      "id": "runtime.provenance",
      "category": "runtime",
      "status": "ok",
      "summary": "running npm on macos-aarch64",
      "details": {
        "commit": "unknown",
        "current executable": "/opt/homebrew/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/bin/codex",
        "install method": "npm (package /opt/homebrew/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin, bin /opt/homebrew/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/bin, resources /opt/homebrew/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/codex-resources, path /opt/homebrew/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/codex-path)",
        "platform": "macos-aarch64",
        "version": "0.147.0"
      },
      "remediation": null,
      "durationMs": 0
    },
    "runtime.search": {
      "id": "runtime.search",
      "category": "search",
      "status": "ok",
      "summary": "search is OK (bundled)",
      "details": {
        "search command": "/opt/homebrew/lib/node_modules/@openai/codex/node_modules/@openai/codex-darwin-arm64/vendor/aarch64-apple-darwin/codex-path/rg",
        "search command readiness": "file exists",
        "search provider": "bundled"
      },
      "remediation": null,
      "durationMs": 17
    },
    "sandbox.helpers": {
      "id": "sandbox.helpers",
      "category": "sandbox",
      "status": "ok",
      "summary": "sandbox configuration is readable",
      "details": {
        "approval policy": "OnRequest",
        "codex-linux-sandbox helper": "none",
        "execve wrapper helper": "/Users/REDACTED/.codex/tmp/arg0/codex-arg0msOPmE/codex-execve-wrapper",
        "filesystem sandbox": "restricted",
        "network sandbox": "restricted"
      },
      "remediation": null,
      "durationMs": 0
    },
    "state.paths": {
      "id": "state.paths",
      "category": "state",
      "status": "ok",
      "summary": "state paths and databases are inspectable",
      "details": {
        "CODEX_HOME": "/Users/REDACTED/.codex (dir)",
        "active rollout files": "128 files, 211775217 total bytes, 1654493 average bytes",
        "archived rollout files": "0 files, 0 total bytes, 0 average bytes",
        "goals DB": "/Users/REDACTED/.codex/goals_1.sqlite (file)",
        "goals DB integrity": "ok",
        "log DB": "/Users/REDACTED/.codex/logs_2.sqlite (file)",
        "log DB integrity": "ok",
        "log dir": "/Users/REDACTED/.codex/log (dir)",
        "memories DB": "/Users/REDACTED/.codex/memories_1.sqlite (file)",
        "memories DB integrity": "ok",
        "queue DB": "/Users/REDACTED/.codex/queue_1.sqlite (file)",
        "queue DB integrity": "ok",
        "sqlite home": "/Users/REDACTED/.codex (dir)",
        "state DB": "/Users/REDACTED/.codex/state_5.sqlite (file)",
        "state DB integrity": "ok",
        "thread history DB": "/Users/REDACTED/.codex/thread_history_1.sqlite (file)",
        "thread history DB integrity": "ok"
      },
      "remediation": null,
      "durationMs": 3540
    },
    "state.rollout_db_parity": {
      "id": "state.rollout_db_parity",
      "category": "threads",
      "status": "ok",
      "summary": "rollout files and state DB thread inventory agree",
      "details": {
        "default model provider": "openai",
        "rollout DB active files": "128",
        "rollout DB active rows": "128",
        "rollout DB archive mismatches": "0",
        "rollout DB archived files": "0",
        "rollout DB archived rows": "0",
        "rollout DB duplicate DB paths": "0",
        "rollout DB duplicate rollout thread ids": "0",
        "rollout DB malformed file names": "0",
        "rollout DB missing active rows": "0",
        "rollout DB missing archived rows": "0",
        "rollout DB model providers": "openai=128",
        "rollout DB rows": "128",
        "rollout DB scan cap reached": "false",
        "rollout DB scan errors": "0",
        "rollout DB sources": "exec=47, subagent:thread_spawn=39, cli=18, vscode=17, subagent:other=7",
        "rollout DB stale rows": "0"
      },
      "remediation": null,
      "durationMs": 2253
    },
    "system.environment": {
      "id": "system.environment",
      "category": "system",
      "status": "ok",
      "summary": "OS language en-US",
      "details": {
        "EDITOR": "set",
        "GH_PAGER": "set",
        "GIT_PAGER": "set",
        "LANG": "C.UTF-8",
        "LC_ALL": "C.UTF-8",
        "LC_CTYPE": "C.UTF-8",
        "PAGER": "set",
        "VISUAL": "not set",
        "os": "Mac OS 26.0.1 [64-bit]",
        "os language": "en-US",
        "os type": "Mac OS",
        "os version": "26.0.1"
      },
      "remediation": null,
      "durationMs": 127
    },
    "terminal.env": {
      "id": "terminal.env",
      "category": "terminal",
      "status": "ok",
      "summary": "terminal metadata was detected",
      "details": {
        "COLORTERM": "present",
        "NO_COLOR": "1",
        "TERMINFO": "/Applications/Ghostty.app/Contents/Resources/terminfo (dir)",
        "TERM_PROGRAM": "ghostty",
        "color output": "disabled (NO_COLOR)",
        "effective locale": "C.UTF-8",
        "stderr is terminal": "false",
        "stdin is terminal": "false",
        "stdout is terminal": "false",
        "terminal": "Ghostty",
        "terminal size": "80x24",
        "terminal version": "1.2.0"
      },
      "remediation": null,
      "durationMs": 70
    },
    "terminal.title": {
      "id": "terminal.title",
      "category": "title",
      "status": "ok",
      "summary": "terminal title default",
      "details": {
        "terminal title activity": "true",
        "terminal title items": "activity, project-name",
        "terminal title project source": "git repo root",
        "terminal title project value": "cascade-browser",
        "terminal title source": "default"
      },
      "remediation": null,
      "durationMs": 1
    },
    "updates.status": {
      "id": "updates.status",
      "category": "updates",
      "status": "ok",
      "summary": "update configuration is locally consistent",
      "details": {
        "cached latest version": "0.147.0",
        "check for update on startup": "true",
        "last checked at": "2026-08-12T00:12:12.876114Z",
        "latest version": "0.147.0",
        "latest version status": "current version is not older",
        "npm update target": "/opt/homebrew/lib/node_modules/@openai/codex",
        "update action": "npm install -g @openai/codex",
        "version cache": "/Users/REDACTED/.codex/version.json"
      },
      "remediation": null,
      "durationMs": 1731
    }
  }
}

What issue are you seeing?

A long-running codex exec resume --json session emitted repeated app-server lag errors and dropped hundreds of events. Later attempts to resume the same session could not continue because a custom tool-call result was missing. The process stayed alive and repeatedly printed the same error without producing a response.

What steps can reproduce the bug?

  1. Start or resume a Codex session through codex exec resume --json.
  2. Let the session perform a long-running workflow with repeated tool calls and polling.
  3. Allow the in-process app-server event consumer to fall behind.
  4. Observe event-drop errors.
  5. Resume the same Codex session with another prompt.

What is the expected behavior?

Expected behavior

Event-stream backpressure should not permanently corrupt session state. If events must be dropped, Codex should preserve required tool-call/result pairs, recover the missing state, or terminate the turn with a clear retryable error. A resumed session should not wait indefinitely for an output that can never arrive.

Additional information

Actual behavior

The original run emitted many errors in this form:

in-process app-server event stream lagged; dropped 43 events
in-process app-server event stream lagged; dropped 171 events
in-process app-server event stream lagged; dropped 854 events

Afterward, every resumed run repeatedly emitted:

ERROR codex_core::util: Custom tool call output is missing for call id: call_lgAAyVMavptKZY5NFEwPIkAo

The resumed process produced no assistant response and remained stuck until canceled. Starting additional turns on that same session reproduced the missing-output error.

Impact

  • The session becomes unusable across subsequent resumes.
  • The harness sees a live process but receives no meaningful assistant output.
  • User prompts queue behind or repeatedly re-enter the broken session.
  • Recovery currently requires abandoning or rotating the session rather than simply retrying the turn.

Relevant timing

  • Event-drop burst began: 2026-08-12 17:58:29 UTC
  • First missing tool-output error after resume: 2026-08-12 17:59:53 UTC
  • Error continued across another resumed run through at least 2026-08-12 18:05:56 UTC

View original on GitHub ↗

4 Comments

github-actions[bot] contributor · 15 days ago

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

  • #38169
  • #37577
  • #38006

Powered by Codex Action

shleder · 14 days ago

Hey, your missing tool output / dropped events case looks very relevant to something I’m testing.

I built Codex Rescue to inspect broken Codex sessions without modifying them. If you still have that session, would you be open to testing it and sharing the sanitized result?

https://github.com/shleder/codex-rescue

jdcodes1 · 10 days ago

Code shape on main @ 1f41cc5d92 that explains why lag becomes permanent corruption: the in-process app-server event stream is a tokio::sync::broadcast channel — lossy by design — and the consumer's only lag handling is a warning:

  • server side: RecvError::Lagged → "receiver lagged; skipping resync" (app-server/src/in_process.rs#L540-L542)
  • exec side: InProcessServerEvent::Lagged { skipped }process_warning and carry on (exec/src/lib.rs#L1120-L1124)

The unrecoverable case is that the dropped events aren't all fire-and-forget: they can include requests that require a client response (custom/client-executed tool calls, approvals). If the request event is lost to lag, the client never runs the tool, the turn waits for an output that can never arrive, and the rollout permanently records a call without its result — which then poisons every later resume, exactly your report.

Fixes: (1) split request-bearing messages onto a lossless backpressured channel (mpsc), leaving broadcast for pure notifications; (2) on Lagged, resync — re-fetch pending requests from thread state instead of warn-and-continue; (3) resume-side repair: detect call-without-output in history and synthesize an error output so existing corrupted threads become continuable (same detect-but-never-repair family as #38792).

shleder · 4 days ago

For dropped app-server events, please capture a redacted diagnosis and source hash before retrying; the adapter is evidence-only and does not repair the event stream.

Thanks for the detailed report. This is a good candidate for a bounded recovery check. Vetto 0.2.0-alpha.2 is available from npm and adds a read-only, copy-only Codex rescue adapter. It does not resume Codex, edit rollout files, or write vendor SQLite. On a disposable copy, try: npm install --global @shleddy/vetto@next; vetto rescue --adapter codex --root <CODEX_HOME> --json scan; then use the exact returned key with diagnose and snapshot ... --output ./vetto-recovery/session.jsonl. Please report OS, Codex/Vetto versions, sanitized JSON, and source SHA-256 before/after; never upload raw transcripts, auth/config, prompts, or tokens. An explicit unavailable/unsupported result is useful too.