app-server daemon: auto-update force-kills active turns after a 60s drain budget, and cannot be disabled

Verified your source refs against current main (37a5149822, 2026-08-27) — they all hold. Updated line numbers (your rust-v0.150.0 refs → main):

| Your ref (v0.150.0) | Current main | Symbol |
|---|---|---|
| pid.rs:23-24 | pid.rs:23-24 | STOP_GRACE_PERIOD = 60s, STOP_TIMEOUT = 70s |
| pid.rs:520 | pid.rs:240 | PidBackend::stop() — SIGTERM via terminate_process (pid.rs:517) |
| pid.rs:253-277 | pid.rs:255, 274 | grace deadline + force_terminate_process → SIGKILL (pid.rs:532) |
| lib.rs:804-818 | lib.rs:804 | restart_decision — version-only comparison, no turn awareness |
| lib.rs:363-372 | lib.rs:363 | try_restart_if_running — Busy = operation lock only |
| lib.rs:608-612, 619 | lib.rs:587, 619 | bootstrap always starts the update loop; auto_update_enabled: true literal (unchanged) |
| app-server/lib.rs:250-262 | app-server/lib.rs:235, 251, 256, 276 | ShutdownState::on_signal / update() drain |

Two additions to the mechanism picture, both relevant to your request 3 (defer rather than kill):

1. The daemon currently has no running-turn visibility. ProbeInfo (client.rs:38-41) carries only app_server_version — the restart decision literally cannot see turns. "Turns running → defer" therefore needs a small additive protocol change: expose the running-turn count over the control socket. The good news is the app-server already maintains exactly this number for the drain — ThreadWatchManager::running_turn_count_tx (thread_status.rs:22, 235) → subscribe_running_assistant_turn_count (thread_processor.rs:3514, message_processor.rs:819) → consumed at app-server/src/lib.rs:918. Extending the probe (or adding a status request) to surface it is straightforward.
2. The retry cadence in the update loop can't support a minutes-long defer. update_once (update_loop.rs:109-135) installs the new standalone before the restart decision and only re-loops with RESTART_RETRY_INTERVAL = 50ms on Busy. A turn-drain defer needs its own longer interval (e.g. 30–60s) so the daemon doesn't spin at 20 Hz against the control socket for the whole drain.

Design notes:

• Unbounded drain is safe only because of the second-signal force path. ShutdownState::on_signal sets forced on a second SIGTERM and update() returns ShutdownAction::Finish immediately (app-server/src/lib.rs:235-245, 261-271). So "no automatic SIGKILL" should still keep an explicit force — a second SIGTERM (not SIGKILL) for the operator, with the daemon's STOP_TIMEOUT as a final fallback. This matters because of the #40306 hang class (a turn awaiting exec with a dead client never completes): an unbounded drain on such a turn would otherwise wedge the daemon forever.
• A configurable grace affects user-initiated stop too — Daemon::stop and the update path share PidBackend::stop(). Keeping the default at 60/70s leaves behavior unchanged unless the operator opts in.

Implementation gotcha for your request 2: DaemonSettings (settings.rs:9-12) derives Deserialize without #[serde(default)], so adding a non-Option field would break loading of existing settings.json files (missing-field error). New fields (auto_update_enabled, grace/timeout) need #[serde(default)] or Option for backward compatibility — and the same applies to the stop_grace_period/stop_timeout fields in request 1.

Happy to implement this. The smallest high-value slice (mirroring requests 1+2, both low-risk and fully testable in the daemon crate):

• configurable stop_grace_period_secs / stop_timeout_secs in DaemonSettings (null = never auto-SIGKILL), plumbed into PidBackend, default 60/70 unchanged;
• auto_update_enabled setting + enable-auto-update / disable-auto-update mirroring the existing remote-control pair, gating the update loop at bootstrap and replacing the hardcoded autoUpdateEnabled status literal.

Defer-on-by-default (your request 3) is the nicer long-term behavior but needs the probe extension above; happy to take that on as a follow-up once maintainers confirm the intended default semantics.

I've prepared an implementation for requests 1 + 2 (configurable/unbounded drain budget + auto-update toggle). Design summary:

Settings (settings.json, daemon-managed)

| field | meaning |
|---|---|
| stopGracePeriodSecs | seconds after SIGTERM before force-terminating (SIGKILL). Absent → 60 (unchanged); 0 → unbounded (never auto-SIGKILL while draining); n → n seconds |
| stopTimeoutSecs | total stop budget before giving up. Absent → 70 (unchanged); 0 → wait indefinitely |
| autoUpdateEnabled | false → update loop not started at bootstrap; running loop stopped by the toggle |

0 = "no limit" follows the systemd TimeoutStopSec=0 convention. New fields are optional with skip_serializing_if — existing settings.json files keep working unchanged, and the default behavior (60/70, auto-update on) is bit-for-bit today's.

Design decisions worth confirming

1. Unbounded drain stays interruptible: the app-server's existing second-SIGTERM force path (ShutdownState::forced → immediate ShutdownAction::Finish) is preserved, so an operator can still force-exit an unbounded drain explicitly. This matters because of the #40306 hang class — a turn awaiting exec with a dead client never completes on its own, so a truly unkillable drain would wedge the daemon.
2. disable-auto-update stops the running update loop immediately (mirrors your workaround of killing the pid-update-loop process, but as a supported command pair enable-auto-update / disable-auto-update).
3. No re-bootstrap resurrection: is_bootstrapped is extended to also treat a running app-server backend as bootstrapped, so codex app-server daemon start after disabling auto-update won't re-bootstrap and silently restart the update loop.
4. Configurable grace applies to user-initiated stop too (both paths share PidBackend::stop()), default unchanged.

Scope

Touch: DaemonSettings (settings.rs), PidBackend::stop() policy (backend/pid.rs), BackendPaths plumbing (backend/mod.rs), bootstrap gating + set_auto_update (lib.rs), CLI subcommand pair (cli/main.rs). Request 3 (defer rather than kill by default) is intentionally out of this slice — it needs the probe protocol change from my earlier comment and is a natural follow-up.

I can push this as a PR — I have the patch ready. (Note: my fork push has been intermittently blocked by a local sandbox restriction; will land as soon as it clears.)

Implementation update — requests 1 + 2 are now built and tested on a fork branch (no PR yet; still waiting on the invitation gate for PR creation).

Branch: argszero/codex:fix/app-server-daemon-stop-policy-and-auto-update-toggle @ 293a744e9a

What's in it:

Configurable stop policy (settings.json, absent = current behavior, 0 = unbounded):

{
  "remoteControlEnabled": false,
  "stopGracePeriodSecs": 0,
  "stopTimeoutSecs": 0,
  "autoUpdateEnabled": false
}

• stopGracePeriodSecs — grace after SIGTERM before SIGKILL; default 60, 0 = never auto-SIGKILL while draining. Second-signal force is preserved, so an operator can still interrupt an unbounded drain.
• stopTimeoutSecs — total stop budget; default 70, 0 = wait indefinitely.
• autoUpdateEnabled — default true. Disables the update loop at bootstrap; codex app-server daemon disable-auto-update / enable-auto-update stop/start a running loop and persist the setting.

Key semantics

• bootstrap_locked loads existing settings instead of replacing them → a disabled auto-update survives re-bootstrap (this was the trap in the current updater.start() unconditional path).
• is_bootstrapped treats a running app-server backend as bootstrapped → ensure_remote_control_started won't re-bootstrap and resurrect the update loop.
• The update-loop process always uses the default stop policy (no turns to drain) so an unbounded drain setting can never wedge disable-auto-update.
• PidBackend::new/new_update_loop signatures unchanged (zero test churn); only with_stop_policy is added.

Tests: cargo test -p codex-app-server-daemon 36 passed; cargo test -p codex-cli --bin codex 279 passed (new parse tests included); clippy + fmt clean.

Request 3 ("defer rather than kill") still needs the probe-protocol change (expose running-turn count in the daemon's probe) so the daemon can choose to defer restart — separate follow-up.

Design for request 3 — "defer rather than kill" on auto-update (follow-up to my P1 implementation comment)

Key facts that shape the design

1. The app-server's graceful drain has no internal timeout: on first SIGTERM it enters graceful-restart drain and waits indefinitely for running turns (ShutdownState::update → ShutdownAction::Noop until running_turn_count == 0; only a second signal forces). So with P1's stopGracePeriodSecs: 0, auto-update already stops killing — the app-server drains on the old binary, exits on its own when idle, and the daemon then restarts with the new binary.
2. What request 3 still needs on top of P1:

• Visibility: the daemon silently waits while turns drain — no signal that an update is pending.
• Bounded defer: a stuck turn (the #40306 hang class — turn awaiting exec with a dead client never completes) would wedge the update forever with unbounded drain. Operators need "wait for idle, but not forever", with a fallback to the configured stop policy.

Proposed design

1. Probe protocol — expose the running-turn count

• Add an optional field to InitializeResponse:

#[serde(default, skip_serializing_if = "Option::is_none")] running_turn_count: Option<usize> (camelCase: runningTurnCount).

• Plumbing is cheap: the app-server already publishes this via message_processor::subscribe_running_assistant_turn_count() → watch::Receiver<usize>. Pass a cloned receiver into InitializeRequestProcessor (it derives Clone, and watch::Receiver::borrow() gives the latest value per request).
• Backward compatible: older app-servers omit the field → the daemon treats it as "unknown".

2. Daemon side — bounded defer on restart

• ProbeInfo gains running_turn_count: Option<usize>; probe() parses it.
• New optional setting deferRestartTimeoutSecs (absent → 0 → no defer, current P1 behavior):
• > 0 and auto-update restart is pending with runningTurnCount > 0 → poll the probe every ~2–5 s until the count reaches 0 or the deadline; log a warn! per poll ("auto-update deferred: waiting for N running assistant turn(s)") so the pending update is visible.
• On deadline: warn!("deferred restart timed out with N running assistant turn(s); proceeding with configured stop policy") and fall through to the normal graceful stop (SIGTERM → drain → stopGracePeriodSecs → SIGKILL).

3. Escape hatches compose, not conflict

• deferRestartTimeoutSecs only delays the start of the graceful stop; it is orthogonal to stopGracePeriodSecs / stopTimeoutSecs.
• Stuck-turn recovery stays as-is: second SIGTERM forces the app-server to exit; stopGracePeriodSecs: 0 still means "never auto-SIGKILL".

Why this shape

• One optional protocol field, no new JSON-RPC method (the probe already does initialize).
• Old binaries + default settings → behavior identical to today (and to P1).
• The defer window is bounded by default opt-in, avoiding the unbounded-wedge regression while still giving long-turn operators the "finish on the old binary" behavior they asked for.

I can implement this on the same branch pattern as P1 (fork branch + tests) once it looks reasonable — happy to adjust the knob shape or naming if you'd prefer waitForIdleTimeoutSecs or similar.