Automatic updater replaces app-server while a parent response is still in progress

Open 💬 0 comments Opened Jul 10, 2026 by midhunmonachan

What version of Codex CLI is running?

codex-cli 0.144.0. The automatic updater installed 0.144.1 during the affected turn.

What subscription do you have?

Not relevant to this local daemon lifecycle failure.

What platform is your computer?

Linux 6.12.0-203.76.7.1.el10uek.aarch64 aarch64 unknown

What issue are you seeing?

I was using a standalone Codex installation through codex remote-control start when the automatic updater initiated a managed app-server restart.

The app-server entered graceful restart draining while one assistant turn was running. A subagent completed during that drain, and the parent received its result. The old app-server then logged response.created followed by response.in_progress for the parent's continuation.

Those were the old process's final records. The daemon PID record reports that the replacement app-server started at 23:14:08, immediately after the old process logged the parent response as still in_progress at 23:14:07.964448940Z.

The parent rollout contains no task_complete between the affected task_started record and the later turn I started manually. The conversation stopped without a terminal event and continued only after manual input.

What steps can reproduce the bug?

  1. Install Codex using the standalone installer.
  2. Start the managed remote-control daemon with codex remote-control start.
  3. Start a parent turn that launches and waits for a subagent.
  4. Allow the automatic updater to install a newer release while the parent turn is active.
  5. Have the subagent complete while graceful restart draining is active.
  6. Observe that the replacement app-server can start while the parent's continuation response is still in_progress, leaving the parent turn without a terminal event.

I observed this during an actual version transition and do not currently have a release-independent reproducer.

What is the expected behavior?

An automatic update should not replace the app-server until every active turn has emitted a terminal completion, failure, cancellation, or interruption event.

A parent turn must remain protected by graceful draining while it processes a subagent result and starts or streams its continuation.

Additional information

Codex 0.144.0 gives the managed app-server a 60-second grace period before force termination. In this incident, the replacement process started about 8-9 seconds after the app-server logged that it was waiting for the active turn:

The evidence establishes that graceful draining failed while the parent response remained active. It does not establish whether the running-turn count became zero prematurely or whether the old app-server exited through another path.

The existing restart tests cover a delayed single-response turn and repeated signals, but not a parent continuation after a subagent completes:

Related: #23954 reports managed updater WebSocket disconnections. This report is narrower: the old app-server disappears while a parent response is still explicitly in_progress and before the parent turn records a terminal event.

I can contribute a focused integration regression test and a minimal fix if a maintainer confirms the intended drain semantics and invites a pull request.

<details>
<summary>Sanitized logs and rollout records (UTC)</summary>

{
  "timestamp": "2026-07-09T23:04:12.223Z",
  "type": "event_msg",
  "payload": {
    "type": "task_started",
    "turn_id": "<redacted: interrupted parent turn id>",
    "started_at": 1783638252,
    "model_context_window": 353400,
    "collaboration_mode_kind": "plan"
  }
}

{
  "timestamp": "2026-07-09T23:13:59.848Z",
  "type": "event_msg",
  "payload": {
    "type": "task_complete",
    "turn_id": "<redacted: subagent turn id>",
    "last_agent_message": "<redacted: subagent result content>",
    "completed_at": 1783638839,
    "duration_ms": 376732,
    "time_to_first_token_ms": 4807
  }
}

2026-07-09T23:13:59.848629933Z	INFO	codex_otel.agent_communication	<redacted: subagent thread id>	pid:2382582:f629d049-9d6c-4c16-b716-e3d05a2c4907
session_loop{thread_id=<redacted: subagent thread id>}:submission_dispatch{otel.name="op.dispatch.inter_agent_communication" submission.id="<redacted: subagent turn id>" codex.op="inter_agent_communication"}:turn{otel.name="session_task.turn" thread.id=<redacted: subagent thread id> turn.id=<redacted: subagent turn id> model=gpt-5.6-sol codex.turn.reasoning_effort=ultra codex.turn.token_usage.input_tokens=2794924 codex.turn.token_usage.cached_input_tokens=2340352 codex.turn.token_usage.non_cached_input_tokens=454572 codex.turn.token_usage.output_tokens=17591 codex.turn.token_usage.reasoning_output_tokens=12484 codex.turn.token_usage.total_tokens=2812515}: agent communication event.name="codex.agent_communication" communication_id="<redacted: communication id>" kind="result" state="send" sender_thread_id=<redacted: subagent thread id> receiver_thread_id=<redacted: parent thread id> content="<redacted: subagent result content>"

2026-07-09T23:13:59.848639213Z	INFO	codex_app_server		pid:2382582:f629d049-9d6c-4c16-b716-e3d05a2c4907
shutdown signal restart: waiting for 1 running assistant turn(s) to finish

{
  "timestamp": "2026-07-09T23:14:00.733Z",
  "type": "response_item",
  "payload": {
    "type": "agent_message",
    "author": "<redacted: subagent name>",
    "recipient": "<redacted: parent agent name>",
    "content": [
      {
        "type": "input_text",
        "text": "<redacted: subagent result content>"
      }
    ],
    "internal_chat_message_metadata_passthrough": {
      "turn_id": "<redacted: subagent turn id>"
    }
  }
}

2026-07-09T23:14:07.953742020Z	TRACE	codex_api::sse::responses		pid:2382582:f629d049-9d6c-4c16-b716-e3d05a2c4907
SSE event:
{
  "type": "response.created",
  "response": {
    "id": "<redacted: response id>",
    "object": "response",
    "created_at": 1783638845,
    "status": "in_progress",
    "background": false,
    "completed_at": null,
    "error": null,
    "frequency_penalty": 0,
    "incomplete_details": null,
    "instructions": null,
    "max_output_tokens": null,
    "max_tool_calls": null,
    "model": "gpt-5.6-sol",
    "moderation": null,
    "output": [],
    "parallel_tool_calls": false,
    "presence_penalty": 0,
    "previous_response_id": null,
    "prompt_cache_key": "<redacted: parent thread id>",
    "prompt_cache_retention": "24h",
    "reasoning": {
      "context": "all_turns",
      "effort": "max",
      "mode": "standard",
      "summary": null
    },
    "safety_identifier": "<redacted: user safety identifier>",
    "service_tier": "auto",
    "store": false,
    "temperature": 1,
    "text": {
      "format": {
        "type": "text"
      },
      "verbosity": "low"
    },
    "tool_choice": "auto",
    "tool_usage": {
      "image_gen": {
        "input_tokens": 0,
        "input_tokens_details": {
          "image_tokens": 0,
          "text_tokens": 0
        },
        "output_tokens": 0,
        "output_tokens_details": {
          "image_tokens": 0,
          "text_tokens": 0
        },
        "total_tokens": 0
      },
      "web_search": {
        "num_requests": 0
      }
    },
    "tools": "<redacted: tool schemas and internal tool instructions>",
    "top_logprobs": 0,
    "top_p": 0.98,
    "truncation": "disabled",
    "usage": null,
    "user": null,
    "metadata": {}
  },
  "sequence_number": 0
}

2026-07-09T23:14:07.964288860Z	TRACE	codex_api::sse::responses		pid:2382582:f629d049-9d6c-4c16-b716-e3d05a2c4907
SSE event:
{
  "type": "response.in_progress",
  "response": {
    "id": "<redacted: response id>",
    "object": "response",
    "created_at": 1783638845,
    "status": "in_progress",
    "background": false,
    "completed_at": null,
    "error": null,
    "frequency_penalty": 0,
    "incomplete_details": null,
    "instructions": null,
    "max_output_tokens": null,
    "max_tool_calls": null,
    "model": "gpt-5.6-sol",
    "moderation": null,
    "output": [],
    "parallel_tool_calls": false,
    "presence_penalty": 0,
    "previous_response_id": null,
    "prompt_cache_key": "<redacted: parent thread id>",
    "prompt_cache_retention": "24h",
    "reasoning": {
      "context": "all_turns",
      "effort": "max",
      "mode": "standard",
      "summary": null
    },
    "safety_identifier": "<redacted: user safety identifier>",
    "service_tier": "auto",
    "store": false,
    "temperature": 1,
    "text": {
      "format": {
        "type": "text"
      },
      "verbosity": "low"
    },
    "tool_choice": "auto",
    "tool_usage": {
      "image_gen": {
        "input_tokens": 0,
        "input_tokens_details": {
          "image_tokens": 0,
          "text_tokens": 0
        },
        "output_tokens": 0,
        "output_tokens_details": {
          "image_tokens": 0,
          "text_tokens": 0
        },
        "total_tokens": 0
      },
      "web_search": {
        "num_requests": 0
      }
    },
    "tools": "<redacted: tool schemas and internal tool instructions>",
    "top_logprobs": 0,
    "top_p": 0.98,
    "truncation": "disabled",
    "usage": null,
    "user": null,
    "metadata": {}
  },
  "sequence_number": 1
}

2026-07-09T23:14:07.964448940Z	TRACE	codex_api::sse::responses		pid:2382582:f629d049-9d6c-4c16-b716-e3d05a2c4907
unhandled responses event: response.in_progress

{
  "pid": 2400015,
  "processStartTime": "Thu Jul  9 23:14:08 2026"
}

{
  "timestamp": "2026-07-09T23:43:12.166Z",
  "type": "event_msg",
  "payload": {
    "type": "task_started",
    "turn_id": "<redacted: manual continuation turn id>",
    "started_at": 1783640592,
    "model_context_window": 353400,
    "collaboration_mode_kind": "default"
  }
}

</details>

View original on GitHub ↗