multi_agent_v1.close_agent can hang for hours when closing an unresponsive subagent

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #23700
• #23971

Powered by Codex Action

Thanks. Ill review #23700 and #23971 and close this if its a duplicate.

Fix: multi_agent_v1.close_agent Hangs for Hours When Closing an Unresponsive Subagent

Issue: openai/codex #24389
Severity: High — blocks parent agent indefinitely
Component: codex-rs/core/src/agent/control.rs — AgentControl::shutdown_live_agent
Labels: multi-agent, hang, timeout, shutdown

---

1. Root Cause Analysis

1.1 The Problem

When close_agent is called on an unresponsive subagent, the call can hang for hours (or indefinitely). The root cause is an unbounded await in the shutdown path:

// codex-rs/core/src/agent/control.rs — shutdown_live_agent()
pub(crate) async fn shutdown_live_agent(&self, agent_id: ThreadId) -> CodexResult<String> {
    let state = self.upgrade()?;
    let result = if let Ok(thread) = state.get_thread(agent_id).await {
        thread.codex.session.ensure_rollout_materialized().await;
        thread.codex.session.flush_rollout().await?;
        let result = if matches!(thread.agent_status().await, AgentStatus::Shutdown) {
            Ok(String::new())
        } else {
            state.send_op(agent_id, Op::Shutdown {}).await
        };
        thread.wait_until_terminated().await;  // <-- NO TIMEOUT: hangs indefinitely
        result
    } else {
        state.send_op(agent_id, Op::Shutdown {}).await
    };
    let _ = state.remove_thread(&agent_id).await;
    self.state.release_spawned_thread(agent_id);
    result
}

1.2 Why It Hangs

The call chain is:

1. close_agent(agent_id) → shutdown_agent_tree(agent_id) → shutdown_live_agent(agent_id)
2. shutdown_live_agent sends Op::Shutdown to the subagent's submission channel
3. Then calls thread.wait_until_terminated().await — which awaits session_loop_termination
4. session_loop_termination is a Shared<BoxFuture<'static, ()>> wrapping a JoinHandle<()>
5. The JoinHandle only completes when the submission_loop task exits

The submission_loop (in codex-rs/core/src/session/handlers.rs) processes operations sequentially:

pub(super) async fn submission_loop(
    sess: Arc<Session>,
    config: Arc<Config>,
    rx_sub: Receiver<Submission>,
) {
    while let Ok(sub) = rx_sub.recv().await {
        // ... processes each op sequentially ...
        Op::Shutdown => shutdown(&sess, sub.id.clone()).await,  // only reached after current op completes
    }
}

If the session loop is currently:

• Waiting for a model API response that never returns (network hang)
• Executing a long-running tool call
• Blocked on a permission prompt awaiting user input
• Deadlocked on a mutex or channel

Then Op::Shutdown sits in the channel queue behind the current operation, and wait_until_terminated() blocks forever.

1.3 Impact Scope

| Component | Affected | Description |
|-----------|----------|-------------|
| close_agent tool (v1) | Yes | Calls shutdown_live_agent |
| close_agent tool (v2) | Yes | Same code path |
| shutdown_agent_tree | Yes | Cascades to all descendants |
| shutdown_live_agent | Yes | Direct culprit |
| CodexThread::shutdown_and_wait | Yes | Same unbounded await pattern |
| Session::shutdown_and_wait | Yes | Same unbounded await pattern |

1.4 Why Tests Do Not Catch This

The test infrastructure uses ThreadManager::with_models_provider_for_tests() which operates in a test mode where:

• The session loop is mocked/simulated
• captured_ops() records ops without actually running a real session loop
• Shutdown completes immediately because there is no real blocking I/O

// multi_agents_tests.rs — uses mock infrastructure
fn thread_manager() -> ThreadManager {
    ThreadManager::with_models_provider_for_tests(
        CodexAuth::from_api_key("dummy"),
        built_in_model_providers(...)["openai"].clone(),
    )
}

---

2. Step-by-Step Fix Approach

2.1 Design Goals

1. Add a configurable timeout to shutdown_live_agent so it never blocks indefinitely
2. Force-terminate the session loop task if the graceful shutdown times out
3. Log warnings when timeouts occur so operators can investigate the root cause
4. Maintain backward compatibility — graceful shutdown should still be the default path
5. Add a config option so users can tune the timeout

2.2 Implementation Steps

Step 1: Add shutdown_timeout_ms to MultiAgentV2Config

File: codex-rs/core/src/config/mod.rs

// In MultiAgentV2Config struct (around line 995):
pub struct MultiAgentV2Config {
    pub max_concurrent_threads_per_session: usize,
    pub min_wait_timeout_ms: i64,
    pub max_wait_timeout_ms: i64,
    pub default_wait_timeout_ms: i64,
    pub shutdown_timeout_ms: i64,  // NEW: timeout for graceful agent shutdown
    pub usage_hint_enabled: bool,
    pub usage_hint_text: Option<String>,
    pub root_agent_usage_hint_text: Option<String>,
    pub subagent_usage_hint_text: Option<String>,
    pub tool_namespace: Option<String>,
    pub hide_spawn_agent_metadata: bool,
    pub non_code_mode_only: bool,
}

// Default (around line 1010):
impl Default for MultiAgentV2Config {
    fn default() -> Self {
        Self {
            max_concurrent_threads_per_session: DEFAULT_MULTI_AGENT_V2_MAX_CONCURRENT_THREADS_PER_SESSION,
            min_wait_timeout_ms: DEFAULT_MULTI_AGENT_V2_MIN_WAIT_TIMEOUT_MS,
            max_wait_timeout_ms: DEFAULT_MULTI_AGENT_V2_MAX_WAIT_TIMEOUT_MS,
            default_wait_timeout_ms: DEFAULT_MULTI_AGENT_V2_DEFAULT_WAIT_TIMEOUT_MS,
            shutdown_timeout_ms: DEFAULT_SHUTDOWN_TIMEOUT_MS,  // NEW
            usage_hint_enabled: true,
            usage_hint_text: None,
            root_agent_usage_hint_text: None,
            subagent_usage_hint_text: None,
            tool_namespace: None,
            hide_spawn_agent_metadata: false,
            non_code_mode_only: false,
        }
    }
}

// Add constant near other defaults (around line 100):
const DEFAULT_SHUTDOWN_TIMEOUT_MS: i64 = 30_000; // 30 seconds

Step 2: Store the JoinHandle in Codex for forceful termination

File: codex-rs/core/src/session/mod.rs

// Change SessionLoopTermination to also store the JoinHandle:
pub(crate) type SessionLoopTermination = Shared<BoxFuture<'static, ()>>;

pub struct Codex {
    pub(crate) tx_sub: Sender<Submission>,
    pub(crate) rx_event: Receiver<Event>,
    pub(crate) agent_status: watch::Receiver<AgentStatus>,
    pub(crate) session: Arc<Session>,
    pub(crate) session_loop_termination: SessionLoopTermination,
    // NEW: Store the JoinHandle for forceful abort capability
    pub(crate) session_loop_handle: Option<tokio::task::JoinHandle<()>>,
}

Update Codex::spawn to store the handle:

// Around line 662 in session/mod.rs:
let session_loop_handle = tokio::spawn(async move {
    submission_loop(sess.clone(), config, rx_sub).await;
});

// Store the handle alongside the termination future:
session_loop_termination: session_loop_termination_from_handle(session_loop_handle.clone()),
session_loop_handle: Some(session_loop_handle),

Step 3: Add wait_until_terminated_with_timeout to CodexThread

File: codex-rs/core/src/codex_thread.rs

use std::time::Duration;
use tokio::time::timeout;

impl CodexThread {
    /// Wait until the session loop terminates, with a timeout.
    /// Returns Ok(()) if terminated normally, Err(()) if timed out.
    pub async fn wait_until_terminated_with_timeout(&self, timeout_duration: Duration) -> Result<(), ()> {
        let termination = self.codex.session_loop_termination.clone();
        match timeout(timeout_duration, termination).await {
            Ok(()) => Ok(()),
            Err(_) => {
                tracing::warn!(
                    thread_id = %self.codex.session.conversation_id,
                    timeout_ms = timeout_duration.as_millis(),
                    "session loop did not terminate within timeout"
                );
                Err(())
            }
        }
    }

    /// Forcefully abort the session loop task. This is a last-resort measure
    /// when graceful shutdown has timed out.
    pub fn abort_session_loop(&self) {
        if let Some(ref handle) = self.codex.session_loop_handle {
            if !handle.is_finished() {
                tracing::warn!(
                    thread_id = %self.codex.session.conversation_id,
                    "forcefully aborting unresponsive session loop"
                );
                handle.abort();
            }
        }
    }
}

Step 4: Update shutdown_live_agent with timeout and fallback

File: codex-rs/core/src/agent/control.rs

use std::time::Duration;
use tokio::time::timeout;

impl AgentControl {
    /// Submit a shutdown request for a live agent without marking it explicitly closed in
    /// persisted spawn-edge state.
    ///
    /// If the agent does not shut down within the configured timeout, the session loop
    /// is forcefully aborted and cleanup proceeds anyway.
    pub(crate) async fn shutdown_live_agent(&self, agent_id: ThreadId) -> CodexResult<String> {
        let state = self.upgrade()?;
        let result = if let Ok(thread) = state.get_thread(agent_id).await {
            thread.codex.session.ensure_rollout_materialized().await;
            thread.codex.session.flush_rollout().await?;

            let result = if matches!(thread.agent_status().await, AgentStatus::Shutdown) {
                Ok(String::new())
            } else {
                state.send_op(agent_id, Op::Shutdown {}).await
            };

            // NEW: Wait with timeout, then force abort if needed
            let shutdown_timeout = self.get_shutdown_timeout(&thread).await;
            match thread
                .wait_until_terminated_with_timeout(shutdown_timeout)
                .await
            {
                Ok(()) => {
                    tracing::debug!(
                        thread_id = %agent_id,
                        "agent shut down gracefully"
                    );
                }
                Err(()) => {
                    // Graceful shutdown timed out — force abort
                    tracing::warn!(
                        thread_id = %agent_id,
                        timeout_ms = shutdown_timeout.as_millis(),
                        "agent shutdown timed out, force-aborting session loop"
                    );
                    thread.abort_session_loop();
                    // Wait briefly for the abort to take effect
                    let _ = thread
                        .wait_until_terminated_with_timeout(Duration::from_secs(5))
                        .await;
                }
            }

            result
        } else {
            state.send_op(agent_id, Op::Shutdown {}).await
        };

        let _ = state.remove_thread(&agent_id).await;
        self.state.release_spawned_thread(agent_id);
        result
    }

    /// Get the configured shutdown timeout for a thread.
    async fn get_shutdown_timeout(&self, thread: &crate::CodexThread) -> Duration {
        let config = thread.codex.session.get_config().await;
        let timeout_ms = config.multi_agent_v2.shutdown_timeout_ms;
        Duration::from_millis(timeout_ms.max(1000) as u64) // minimum 1 second
    }
}

Step 5: Add config parsing for shutdown_timeout_ms

File: codex-rs/core/src/config/mod.rs

Add to MultiAgentV2ConfigToml and the parsing function:

// In MultiAgentV2ConfigToml struct:
pub struct MultiAgentV2ConfigToml {
    // ... existing fields ...
    pub shutdown_timeout_ms: Option<i64>,
}

// In resolve_multi_agent_v2_config (around line 2197):
fn resolve_multi_agent_v2_config(config_toml: &ConfigToml) -> MultiAgentV2Config {
    let base = multi_agent_v2_toml_config(config_toml.features.as_ref());
    // ... existing resolution ...
    MultiAgentV2Config {
        // ... existing fields ...
        shutdown_timeout_ms: base
            .and_then(|b| b.shutdown_timeout_ms)
            .unwrap_or(DEFAULT_SHUTDOWN_TIMEOUT_MS),
        // ...
    }
}

Add validation:

// In validate_config (around line 2999):
validate_multi_agent_v2_wait_timeout(
    "features.multi_agent_v2.shutdown_timeout_ms",
    multi_agent_v2.shutdown_timeout_ms,
)?;

2.3 Alternative Approach: Abort Without Config

If adding a config option is too invasive, the simpler approach uses a hardcoded default:

// Minimal fix — just add timeout with a hardcoded default
const SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(30);

pub(crate) async fn shutdown_live_agent(&self, agent_id: ThreadId) -> CodexResult<String> {
    let state = self.upgrade()?;
    let result = if let Ok(thread) = state.get_thread(agent_id).await {
        thread.codex.session.ensure_rollout_materialized().await;
        thread.codex.session.flush_rollout().await?;
        let result = if matches!(thread.agent_status().await, AgentStatus::Shutdown) {
            Ok(String::new())
        } else {
            state.send_op(agent_id, Op::Shutdown {}).await
        };
        // NEW: Timeout after 30s, then force abort
        if timeout(SHUTDOWN_TIMEOUT, thread.wait_until_terminated()).await.is_err() {
            tracing::warn!(thread_id = %agent_id, "shutdown timed out, force-aborting");
            thread.abort_session_loop();
        }
        result
    } else {
        state.send_op(agent_id, Op::Shutdown {}).await
    };
    let _ = state.remove_thread(&agent_id).await;
    self.state.release_spawned_thread(agent_id);
    result
}

---

3. Code Snippets

3.1 Full shutdown_live_agent with Fix

use std::time::Duration;
use tokio::time::timeout;

/// Submit a shutdown request for a live agent without marking it explicitly closed in
/// persisted spawn-edge state.
///
/// If the agent does not respond to the shutdown signal within the configured timeout,
/// the session loop task is forcefully aborted. This prevents the parent agent from
/// hanging indefinitely when a subagent is unresponsive (e.g., stuck on a network call,
/// infinite loop, or deadlocked).
pub(crate) async fn shutdown_live_agent(&self, agent_id: ThreadId) -> CodexResult<String> {
    let state = self.upgrade()?;
    let result = if let Ok(thread) = state.get_thread(agent_id).await {
        // Ensure rollout is persisted before shutdown
        thread.codex.session.ensure_rollout_materialized().await;
        thread.codex.session.flush_rollout().await?;

        let result = if matches!(thread.agent_status().await, AgentStatus::Shutdown) {
            Ok(String::new())
        } else {
            state.send_op(agent_id, Op::Shutdown {}).await
        };

        // Wait for graceful shutdown with timeout
        let shutdown_timeout = self.get_shutdown_timeout(&thread).await;
        let terminated_gracefully = timeout(
            shutdown_timeout,
            thread.wait_until_terminated(),
        )
        .await
        .is_ok();

        if !terminated_gracefully {
            // Graceful shutdown timed out — force abort the session loop
            tracing::warn!(
                thread_id = %agent_id,
                timeout_ms = shutdown_timeout.as_millis(),
                "agent shutdown timed out; forcefully aborting session loop"
            );
            thread.abort_session_loop();

            // Give the abort a short window to take effect
            let _ = timeout(
                Duration::from_secs(5),
                thread.wait_until_terminated(),
            )
            .await;
        }

        result
    } else {
        state.send_op(agent_id, Op::Shutdown {}).await
    };

    let _ = state.remove_thread(&agent_id).await;
    self.state.release_spawned_thread(agent_id);
    result
}

3.2 CodexThread Extensions

use std::time::Duration;
use tokio::time::timeout;

impl CodexThread {
    /// Wait until the underlying session loop has terminated, with a timeout.
    ///
    /// Returns `Ok(())` if the session loop terminated normally within the timeout,
    /// or `Err(())` if the timeout elapsed.
    pub async fn wait_until_terminated_with_timeout(
        &self,
        timeout_duration: Duration,
    ) -> Result<(), ()> {
        let termination = self.codex.session_loop_termination.clone();
        match timeout(timeout_duration, termination).await {
            Ok(()) => Ok(()),
            Err(_) => Err(()),
        }
    }

    /// Forcefully abort the session loop task.
    ///
    /// This is a last-resort measure when graceful shutdown has timed out.
    /// The abort will cancel any in-flight futures in the session loop,
    /// but may leave resources in an inconsistent state.
    pub fn abort_session_loop(&self) {
        if let Some(ref handle) = self.codex.session_loop_handle {
            if !handle.is_finished() {
                handle.abort();
            }
        }
    }
}

3.3 Config Example

# codex.toml
[features.multi_agent_v2]
# Timeout for graceful shutdown of subagents (milliseconds).
# If a subagent does not shut down within this time, its session loop
# is forcefully aborted. Default: 30000 (30 seconds).
shutdown_timeout_ms = 30000

# Minimum timeout for wait_agent tool calls
min_wait_timeout_ms = 2500

# Maximum timeout for wait_agent tool calls
max_wait_timeout_ms = 120000

# Default timeout for wait_agent tool calls
default_wait_timeout_ms = 30000

---

4. Testing Strategy

4.1 Unit Tests

Test 1: Graceful Shutdown Within Timeout

#[tokio::test]
async fn shutdown_live_agent_completes_gracefully_when_responsive() {
    let (mut session, turn) = make_session_and_context().await;
    let manager = thread_manager();
    let config = turn.config.as_ref().clone();
    let thread = manager.start_thread(config).await.expect("start thread");
    let agent_id = thread.thread_id;

    // Submit shutdown op to make the agent responsive
    thread.submit(Op::Shutdown {}).await.expect("submit shutdown");

    let result = manager
        .agent_control()
        .shutdown_live_agent(agent_id)
        .await;

    assert!(result.is_ok(), "shutdown should succeed for responsive agent");
}

Test 2: Timeout and Force Abort

#[tokio::test]
async fn shutdown_live_agent_times_out_and_force_aborts_unresponsive_agent() {
    let (mut session, turn) = make_session_and_context().await;
    let manager = thread_manager_for_unresponsive_test();
    let config = turn.config.as_ref().clone();

    // Create a thread that will never respond to shutdown
    let thread = manager
        .start_thread_with_blocking_session_loop(config)
        .await
        .expect("start thread");
    let agent_id = thread.thread_id;

    let start = Instant::now();
    let result = manager
        .agent_control()
        .shutdown_live_agent(agent_id)
        .await;

    let elapsed = start.elapsed();

    // Should complete within ~35s (30s timeout + 5s abort wait)
    assert!(
        elapsed < Duration::from_secs(60),
        "shutdown should complete within timeout, took {:?}",
        elapsed
    );
    assert!(result.is_ok(), "shutdown should succeed even after force abort");

    // Verify the thread was cleaned up
    assert_eq!(
        manager.agent_control().get_status(agent_id).await,
        AgentStatus::NotFound,
        "agent should be removed after force abort"
    );
}

Test 3: Cascade Shutdown with Mixed Responsiveness

#[tokio::test]
async fn shutdown_agent_tree_handles_mixed_responsiveness() {
    // Spawn parent -> child1 (responsive) -> child2 (unresponsive)
    // Closing parent should eventually complete despite child2 being unresponsive
}

Test 4: Config Timeout Respected

#[tokio::test]
async fn shutdown_uses_configured_timeout() {
    let mut config = test_config();
    config.multi_agent_v2.shutdown_timeout_ms = 5000; // 5 seconds

    // Verify the 5s timeout is used, not the default 30s
}

4.2 Integration Tests

Test 5: Real Unresponsive Agent (API Hang Simulation)

#[tokio::test]
async fn close_agent_handles_api_hang_scenario() {
    // Simulate a subagent that is stuck waiting for an API response
    // by blocking the session loop model client
    // Verify close_agent completes within a reasonable time
}

Test 6: Multiple Concurrent Close Operations

#[tokio::test]
async fn close_multiple_agents_concurrently() {
    // Spawn 5 agents, make 2 unresponsive
    // Close all 5 concurrently
    // Verify all complete within timeout
}

4.3 Regression Tests

Test 7: Existing Tests Still Pass

All existing tests in multi_agents_tests.rs and control_tests.rs should continue to pass:

just test -p codex-core -- multi_agent
just test -p codex-core -- agent::control::tests

4.4 Performance Tests

Test 8: Timeout Does Not Affect Normal Shutdown

#[tokio::test]
async fn graceful_shutdown_latency_unchanged() {
    // Measure time for responsive agent shutdown
    // Should be < 100ms (no timeout overhead)
}

---

5. Impact Assessment

5.1 Positive Impacts

| Area | Impact |
|------|--------|
| Reliability | Parent agents no longer hang indefinitely when closing unresponsive subagents |
| Observability | Warning logs provide visibility into shutdown issues |
| Resource Management | Resources are eventually cleaned up even for stuck agents |
| User Experience | Users see timely feedback instead of silent hangs |

5.2 Potential Risks

| Risk | Mitigation |
|------|------------|
| Force abort may leave resources inconsistent | The abort only cancels the session loop task; cleanup (thread removal, spawn slot release) still proceeds. MCP connections are handled by shutdown_session_runtime which runs before the timeout. |
| Rollout data loss | ensure_rollout_materialized() and flush_rollout() are called before the timeout, so rollout data is persisted before any force abort |
| Timeout too short for slow agents | Configurable timeout allows users to adjust. Default of 30s is generous for normal operations. |
| Breaking change for long-running tools | Agents with intentionally long-running operations (>30s) may be force-aborted. Users can increase the timeout. |

5.3 Backward Compatibility

• Config: New shutdown_timeout_ms field is optional with a sensible default (30s)
• API: No changes to public interfaces; close_agent still returns CloseAgentResult
• Behavior: Graceful shutdown path is unchanged; timeout only activates when shutdown is slow
• Tests: Existing tests continue to pass (test mode completes immediately)

5.4 Files Modified

| File | Changes |
|------|---------|
| codex-rs/core/src/agent/control.rs | Add timeout to shutdown_live_agent, add get_shutdown_timeout helper |
| codex-rs/core/src/codex_thread.rs | Add wait_until_terminated_with_timeout and abort_session_loop |
| codex-rs/core/src/session/mod.rs | Store JoinHandle in Codex struct |
| codex-rs/core/src/config/mod.rs | Add shutdown_timeout_ms to MultiAgentV2Config |
| codex-rs/core/src/agent/control_tests.rs | Add tests for timeout and force abort |
| codex-rs/core/src/tools/handlers/multi_agents_tests.rs | Add integration tests for unresponsive agents |

5.5 Performance Impact

• Normal case (responsive agent): Negligible — timeout is only checked after graceful shutdown completes
• Timeout case (unresponsive agent): Adds ~30s worst-case delay (vs. indefinite hang before)
• Memory: Minimal — one additional Option<JoinHandle> per Codex instance

5.6 Security Considerations

• Force abort does not introduce new security vulnerabilities
• The abort mechanism is the same as tokio::task::JoinHandle::abort() which is a standard Rust pattern
• No user input is involved in the timeout decision

---

6. Rollout Plan

Phase 1: Core Fix (PR #1)

1. Add shutdown_timeout_ms to config with default of 30s
2. Add timeout to shutdown_live_agent
3. Add abort_session_loop capability
4. Add unit tests

Phase 2: Observability (PR #2)

1. Add metrics for shutdown timeouts
2. Add structured logging for force aborts
3. Add telemetry for shutdown latency distribution

Phase 3: Tuning (PR #3)

1. Monitor production timeout rates
2. Adjust default if needed based on real-world data
3. Add documentation for the new config option

---

7. References

• Issue: openai/codex #24389
• Related files:
• codex-rs/core/src/agent/control.rs — AgentControl::shutdown_live_agent
• codex-rs/core/src/codex_thread.rs — CodexThread::wait_until_terminated
• codex-rs/core/src/session/mod.rs — Codex struct, SessionLoopTermination
• codex-rs/core/src/session/handlers.rs — submission_loop
• codex-rs/core/src/config/mod.rs — MultiAgentV2Config
• Similar patterns in codebase:
• app-server/src/in_process.rs — SHUTDOWN_TIMEOUT = Duration::from_secs(5)
• app-server-client/src/remote.rs — SHUTDOWN_TIMEOUT for remote connections

I hit a related long-running subagent lifecycle failure and inspected the current public source. The close_agent hang described here looks consistent with an unbounded shutdown await in the shared close path.

Both multi_agents/close_agent.rs and multi_agents_v2/close_agent.rs call AgentControl::close_agent(...), which reaches shutdown_agent_tree(...) and then shutdown_live_agent(...).

In codex-rs/core/src/agent/control.rs, shutdown_live_agent sends Op::Shutdown and then awaits:

thread.wait_until_terminated().await;

without an outer timeout or fallback. If the child session loop is stuck, blocked in a tool call, transport-stalled, or otherwise unable to process/complete shutdown, the parent close_agent tool can remain blocked indefinitely.

I hit this issue in Codex Desktop as well and prepared a minimal branch with a bounded shutdown wait:

https://github.com/tianpeng-dev/codex/tree/fix/bound-agent-shutdown-wait

Commit: ae4888b fix(core): bound live agent shutdown wait

Local evidence points to the same lifecycle path described here: shutdown_live_agent submits Op::Shutdown and then waits on thread.wait_until_terminated() without a timeout. If the child session loop is stalled, close_agent can remain blocked and the UI stays in Closing ....

The branch keeps the close_agent result schema unchanged and adds a bounded wait around live agent termination. If the shutdown request was submitted but the agent does not terminate, it returns CodexErr::RequestTimeout and continues the existing cleanup path.

Validated locally with:

• cargo fmt --check -p codex-core
• cargo test -p codex-core agent::control::legacy::tests::wait_for_live_agent_termination_times_out_when_agent_never_exits
• cargo test -p codex-core close_agent_submits_shutdown_and_returns_previous_status
• RUST_MIN_STACK=8388608 cargo test -p codex-core shutdown_agent_tree_closes_live_descendants

Note: shutdown_agent_tree_closes_live_descendants overflowed the default test stack locally, but passed with RUST_MIN_STACK=8388608.

Would the Codex team be open to this minimal approach, and if so, could you invite a PR for this branch?

Fresh reproduction from June 17, 2026, on a local Codex Desktop session. This is the same failure mode: multi_agent_v1.close_agent waited until the user aborted the parent turn, and subsequent subagent work hit the thread cap.

The affected target in both long closes was 019ed2f7-e5e2-7320-94cb-c90c6e56c428.

2026-06-17T00:36:02.855Z  function_call         multi_agent_v1 close_agent call_xRw562qjZJKLy8HUPPcn9DAb {"target":"019ed2f7-e5e2-7320-94cb-c90c6e56c428"}
2026-06-17T02:18:00.063Z  function_call_output                         call_xRw562qjZJKLy8HUPPcn9DAb aborted by user after 6116.8s
2026-06-17T20:58:22.407Z  function_call_output                         call_dhhgM6qfUlsSTdHpHBoH4VxC collab spawn failed: agent thread limit reached
2026-06-17T20:58:28.346Z  function_call         multi_agent_v1 close_agent call_Kujn6yqMZf6RASeBXYGYeasT {"target":"019ed2f7-e5e2-7320-94cb-c90c6e56c428"}
2026-06-17T21:26:03.701Z  function_call_output                         call_Kujn6yqMZf6RASeBXYGYeasT aborted by user after 1655.3s
2026-06-17T21:34:02.091Z  function_call_output                         call_j53aGpcWIHbyIdSlj1GvSKil collab spawn failed: agent thread limit reached
2026-06-17T21:46:08.557Z  function_call_output                         call_K08XWC50awPzYwUGPS88eJm7 collab spawn failed: agent thread limit reached
2026-06-17T21:47:29.771Z  function_call_output                         call_ii0uQP0YZySgYSkipxp0nWp6 collab spawn failed: agent thread limit reached

I traced the current main path to codex-rs/core/src/agent/control/legacy.rs: shutdown_live_agent submits Op::Shutdown and then awaits thread.wait_until_terminated() without bounding the wait. A focused patch is forthcoming that interrupts an active agent task before queuing shutdown and wraps the termination wait in a timeout so the parent close tool cannot hang indefinitely.

A focused patch is available here:

• Fork draft PR: https://github.com/bobashopcashier/codex/pull/1
• Branch: https://github.com/bobashopcashier/codex/tree/fix/multi-agent-close-hang
• Commit: 45b61dc0a5 fix: bound multi-agent close shutdown

I attempted to open the PR directly against openai/codex, but this account is blocked from creating upstream pull requests:

pull request create failed: GraphQL: bobashopcashier does not have the correct permissions to execute `CreatePullRequest` (createPullRequest)

The REST endpoint was also unavailable from this token:

POST /repos/openai/codex/pulls -> HTTP 404 Not Found

Net change in the branch:

• Interrupt an active agent task before submitting Op::Shutdown.
• Bound the live-agent termination wait with a 10 second timeout.
• Add a regression test for shutdown_live_agent while the agent has an active turn task.

Local verification on the branch:

git diff --check
cargo fmt --check
CARGO_NET_GIT_FETCH_WITH_CLI=true cargo test -p codex-core shutdown_live_agent_closes_agent_with_active_turn
CARGO_NET_GIT_FETCH_WITH_CLI=true RUST_MIN_STACK=8388608 cargo test -p codex-core shutdown_agent_tree
CARGO_NET_GIT_FETCH_WITH_CLI=true RUST_MIN_STACK=8388608 cargo test -p codex-core agent::control::tests::

Fresh Codex Desktop datapoint from 2026-06-20 that looks like the same close-hang family, with an extra UI/registry mismatch signal.

Environment:

• Codex Desktop app: 26.616.32156 (4157)
• Bundled app CLI: codex-cli 0.142.0-alpha.1
• Local CLI: codex-cli 0.141.0
• Platform: macOS 15.7.7 / Darwin 24.6.0 arm64
• Surface: Codex Desktop app, local/worktree thread with multiple subagents

Observed sequence:

1. A long multi-subagent worktree task auto-compacted the parent context.
2. The parent continued after compaction and began closing a subagent named Zeno.
3. The parent transcript stayed stuck at a progress state equivalent to "closing 1 agent" / step 4 of 5.
4. At the same time, the Desktop right rail Subagents list no longer contained Zeno; it only showed other subagents such as Hooke, Halley, Kepler, and Turing.
5. No visible work was running in Zeno, and there was no visible subagent card/row to inspect for it.
6. The parent turn remained blocked waiting for the close operation.

Why this seems useful for this issue:

• This is not the stale-card variant where the Desktop UI still shows a closed subagent.
• It is the opposite mismatch: the UI/listing no longer shows the target subagent, but the parent orchestrator still appears to hold a close waiter for it.
• The likely trigger involved parent context compaction/resume, so this may intersect with #26728, but the user-visible stuck point is still the close path.

Expected behavior:

• Closing a missing, already-detached, already-shutdown, or no-longer-listed subagent should be idempotent and complete promptly.
• After compaction/resume, the parent's active close/wait set should reconcile with the live subagent registry/UI state.
• A missing target should be reported as already_closed, already_shutdown, or not_found_after_spawn, not block the parent turn indefinitely.

I am not attaching the raw screenshot/logs here because they include workspace/task details, but the key visible contradiction was:

Parent transcript: closing subagent "Zeno" / closing 1 agent
Desktop right rail: no "Zeno" entry in Subagents list
Parent state: still blocked on close

This supports the same fix direction discussed above: make close_agent bounded and idempotent, and make the close path treat missing/terminal/already-detached child state as a terminal cleanup result rather than waiting forever on a termination future or stale handle.

I can reproduce a closely matching case on Codex Desktop where a normal delegated message to an old parent thread completes quickly, but asking that same parent thread to close an already-unresponsive subagent leaves the parent turn in progress.

Environment / context:

• Codex Desktop on macOS (local host)
• Parent thread id: 019ee0a0-c3b5-7843-951c-0b6d93258937
• Subagent id: 019ef395-2511-7d03-9cef-4aef2fa407fd
• Subagent role/nickname from session meta: explorer / Mendel
• Parent cwd: /Users/fujiwarakasei/Documents/Project/LAUNOVA
• Subagent session log: ~/.codex/sessions/2026/06/23/rollout-2026-06-23T17-24-58-019ef395-2511-7d03-9cef-4aef2fa407fd.jsonl

Observed sequence:

1. The subagent session log only contains session metadata, task start, and turn abort records. It produced no assistant output and no tool output.
2. In the parent thread, earlier wait_agent calls for that subagent timed out with empty status, then send_input(... interrupt=true ...) also failed to produce a usable result.
3. The parent attempted multi_agent_v1.close_agent for that subagent, and the user interrupted the turn while it was still stuck around the close operation.
4. As a control, I sent a minimal delegated probe to the same parent thread: “do not use tools, just reply received”. It completed successfully in about 9.7 seconds.
5. I then sent another delegated diagnostic prompt asking the same parent thread to close subagent 019ef395-2511-7d03-9cef-4aef2fa407fd. The parent emitted “开始关闭 sub-agent。” and then stayed inProgress; after more than two minutes there was still no close_agent return or final response.

This makes the failure mode look specific to multi_agent_v1.close_agent on an aborted/unresponsive subagent, rather than the parent thread being generally unable to receive or answer new messages.

The local evidence strongly matches this issue’s report: wait_agent gives no useful status for the subagent, and close_agent can block the parent turn instead of returning promptly or reporting a bounded failure.

This issue maps closely to LS work on bounded lifecycle transitions, execution leases, and durable termination receipts.

The central distinction is:

termination requested
≠ termination completed
≠ cleanup completed
≠ parent must remain blocked

A control-plane operation such as "close_agent" should never depend indefinitely on the responsiveness of the child it is trying to terminate.

A possible termination request:

{
"schema_version": "agent-termination-request/v0.1",
"request_id": "terminate-184",
"dispatch_id": "dispatch-review-05",
"parent_agent_id": "agent-root",
"child_agent_id": "agent-reviewer",
"runtime_instance_id": "runtime-7712",
"mode": "GRACEFUL",
"requested_at": "2026-06-24T20:10:00Z",
"grace_timeout_ms": 10000,
"hard_timeout_ms": 30000
}

A bounded shutdown path could be:

request graceful cancellation
→ wait within grace timeout
→ escalate to forced termination
→ persist terminal or unresolved state
→ return control to parent
→ continue asynchronous cleanup if necessary

The key invariant would be:

«Failure to confirm child termination must not block the parent agent indefinitely.»

A second invariant:

«"close_agent" should be idempotent and return the current lifecycle state when the child is already aborted, terminated, missing, or under cleanup.»

Possible results:

TERMINATED
ALREADY_TERMINATED
TERMINATION_PENDING
FORCE_TERMINATED
NOT_FOUND
TERMINATION_FAILED
CLEANUP_INCOMPLETE

A structured result could look like:

{
"request_id": "terminate-184",
"child_agent_id": "agent-reviewer",
"status": "CLEANUP_INCOMPLETE",
"runtime_terminated": true,
"remaining_resources": [
"mcp-process:xcodebuildmcp",
"node-repl:process-889"
],
"parent_blocked": false,
"follow_up": "BACKGROUND_RECONCILIATION"
}

This separates three concerns that should not be collapsed:

child task status
child runtime status
descendant-resource cleanup status

For example, the child turn may already be "ABORTED", while an MCP descendant remains alive. That should not require keeping the parent turn blocked for hours.

The parent should receive a bounded receipt:

{
"event_type": "AGENT_TERMINATION_RECEIPT",
"child_agent_id": "agent-reviewer",
"terminal_status": "ABORTED",
"termination_confirmed": true,
"cleanup_confirmed": false,
"elapsed_ms": 30000,
"decision": "RETURN_TO_PARENT"
}

A reconciliation worker could then asynchronously inspect:

agent registry
↔ operating-system process tree
↔ MCP child-process registry
↔ launch-slot accounting

and emit a later cleanup receipt.

A deterministic conformance fixture could test:

1. responsive child closes gracefully;
2. unresponsive child reaches grace timeout;
3. forced termination returns within the hard timeout;
4. already-aborted child returns "ALREADY_TERMINATED";
5. repeated close requests are idempotent;
6. missing child does not block;
7. leaked MCP process produces "CLEANUP_INCOMPLETE";
8. parent regains control even when cleanup is incomplete;
9. launch slot is released only when the relevant runtime state is reconciled;
10. termination events remain replayable after session restart.

Related LS work:

• causal audit:

https://github.com/safal207/LS/issues/594

• execution and evidence gates:

https://github.com/safal207/LS/issues/595

• deterministic replay:

https://github.com/safal207/LS/issues/597

• recovered continuation state:

https://github.com/safal207/LS/pull/651

Would a small vendor-neutral "AgentTerminationRequest", "TerminationReceipt", and asynchronous cleanup fixture help make shutdown latency, escalation, idempotency, and parent non-blocking behavior machine-testable?

Adding a fresh matching datapoint from 2026-06-24; I posted the detailed timeline on #25870 because that issue focuses on stale/unresponsive child cleanup, but this is the same user-visible failure class.

Short version: a native explorer subagent timed out under wait_agent three times, then multi_agent_v1.close_agent was called at 2026-06-24T04:51:56Z and did not return until the user aborted at 2026-06-24T11:56:27Z. The outputs were aborted by user after 24930.6s / 24930.5s. The child transcript had only session metadata, task_started, and turn_aborted; no assistant/tool output.

This reinforces that close_agent needs a bounded shutdown/cleanup path and should not keep the parent turn blocked while waiting on an unresponsive child runtime.

Follow-up: I pushed a minimal candidate fix from this reproduction to a public fork branch:

• branch: https://github.com/alygg77/codex/tree/codex/bound-v1-close-agent-shutdown
• commit: ca2890159e3e264e2f52599ac5d122ee4f66717f
• compare: https://github.com/openai/codex/compare/main...alygg77:codex/bound-v1-close-agent-shutdown?expand=1

The patch bounds the v1 live-agent termination wait and returns the existing request-timeout error if the child accepts shutdown but does not terminate, while leaving the still-live thread tracked instead of removing/releasing it. I attempted to open a draft PR, but GitHub rejected both CreatePullRequest and REST pull creation for this token/repo permission set.

Adding a fresh sanitized datapoint from Codex Desktop on 2026-06-27. This looks like the same multi-agent lifecycle failure family described here, with one nuance: the user-visible stuck point was a close_agent attempt on an old timed-out review subagent, while the persisted parent transcript later shows the parent turn still in progress around multi-agent wait/cleanup orchestration.

Environment:

• Codex Desktop on macOS 15.3.1 arm64 / Darwin 24.3.0
• Bundled/local CLI: codex-cli 0.142.3
• Parent session originator: Codex Desktop
• Parent session source: vscode
• Parent session id: 019f083a-d4d6-7b12-a4be-254b1393943f
• Local project path and task details redacted

Observed sequence, sanitized:

2026-06-27T15:58:13Z spawn_agent -> review child 019f09cd-8838-76c0-bc13-874ef554a598
2026-06-27T15:58:20Z wait_agent timeout_ms=180000
2026-06-27T16:01:20Z wait_agent -> {"status":{},"timed_out":true}
2026-06-27T16:01:29Z send_input interrupt=true to the review child
2026-06-27T16:01:38Z wait_agent timeout_ms=60000
2026-06-27T16:02:17Z wait_agent returned a completed review result
2026-06-27T16:02:27Z close_agent review child
2026-06-27T16:02:27Z close_agent returned previous_status promptly

2026-06-27T16:06:13Z spawn_agent -> narrow follow-up review child 019f09d4-ded7-77f0-a22c-78447b67eea9
2026-06-27T16:06:22Z wait_agent timeout_ms=120000
2026-06-27T16:07:31Z wait_agent returned completed PASS
2026-06-27T16:07:52Z close_agent follow-up review child
2026-06-27T16:07:53Z close_agent returned previous_status promptly

2026-06-27T16:08:35Z spawn_agent -> worker child 019f09d7-08ee-7bb3-b742-684405012763
2026-06-27T16:08:44Z parent wait_agent timeout_ms=300000

At this point the parent transcript stopped with the wait_agent call and no corresponding output in the local JSONL transcript. The Codex app still reports that parent turn as inProgress with completedAt: null.

The child transcript for 019f09d7-08ee-7bb3-b742-684405012763 shows it continued running and applying changes after the parent wait began. It progressed at least until about 2026-06-27T16:13:07Z, including successful local command outputs, but the child transcript also has no final completed/subagent notification record. So the user-visible effect was that the parent orchestration was stuck even though child work had continued.

Additional UI-side observation from the same parent thread summary:

• The user asked whether it had stuck again.
• The assistant answered that it was not project code stuck; it was an attempt to close a previous timed-out review agent where close_agent itself got stuck, and it would stop touching that old agent.
• Later in the same thread, after avoiding that old agent, the parent was still left inProgress around subsequent subagent lifecycle management.

Local evidence checked:

• Parent transcript: ~/.codex/sessions/2026/06/27/rollout-2026-06-27T16-38-21-019f083a-d4d6-7b12-a4be-254b1393943f.jsonl
• Child transcripts under ~/.codex/sessions/2026/06/28/ for the child ids above
• read_thread for the parent still reports status inProgress, completedAt: null

I am not attaching raw logs because they include workspace and task details. The sanitized takeaway is:

1. Timed-out review agents make the parent orchestration fragile.
2. close_agent can be the user-visible stuck point for an old timed-out review child.
3. Even when later close_agent calls return promptly, the parent/child lifecycle can still fail to reconcile: a child continues doing work, but the parent wait_agent never receives a terminal result and the parent turn remains inProgress.

Expected behavior remains the same as discussed above: child close/wait lifecycle operations should be bounded and idempotent, and parent control should return with a structured timeout/partial-cleanup result instead of leaving the parent turn open indefinitely.

Adding another fresh Codex Desktop datapoint. This looks like the same close-agent lifecycle bug, but with one detail that may be useful: after a rollback/resume, the parent tried to close the same stale child again and hit the same hang a second time.

Environment:

• Codex Desktop app: 26.623.42026 (4514)
• Bundled/local session CLI in the current rollout: 0.142.3
• Parent session originator: Codex Desktop
• Parent source: vscode
• Platform: Darwin 25.4.0 arm64 arm
• Parent session id: 019f0a4c-dddb-7561-951f-31015132d59f
• Target subagent id: 019f09ce-7776-75f0-9dca-491a96971b27
• Target subagent role/nickname: explorer / Fermat

Observed sequence:

target subagent rollout:
2026-06-27T16:13:57Z turn_aborted reason=interrupted duration_ms=883237

parent rollout:
line 12805 close_agent target=019f09ce-7776-75f0-9dca-491a96971b27
line 12807 aborted by user after 4978.7s

later, after continuing/rollback:
line 13116 close_agent target=019f09ce-7776-75f0-9dca-491a96971b27
line 13117 aborted by user after 1889.1s
line 13141 turn_aborted
line 13143 thread_rolled_back num_turns=3

So this was not just a slow child task. The child had already reached an interrupted/aborted state, but close_agent still put the parent turn into a blocking close path. After the user interrupted and the parent thread rolled back, the same cleanup action was attempted again and wedged the parent again.

Why this matters:

• Cleanup became the thing that blocked useful work.
• Interrupting the stuck close did not leave the conversation in a clean state; the parent recorded thread_rolled_back.
• Retrying the cleanup did not behave idempotently. It re-entered the same long wait.
• From the user side, this feels like "closing one subagent blocks every later conversation."

This matches the current source shape discussed above. In current main, the v1 close handler still awaits session.services.agent_control.close_agent(agent_id), and the legacy shutdown path sends Op::Shutdown and then awaits thread.wait_until_terminated() without a bounded close-specific timeout. Interrupted is also not treated as final by the public status helper, which seems relevant for stale/interrupted children that are terminal enough for cleanup but not final enough for the lifecycle machinery.

Expected behavior:

• Closing an already interrupted, aborted, completed, missing, or stale child should return promptly.
• close_agent should be idempotent across rollback/resume/retry.
• If runtime termination cannot be confirmed, the tool should return a bounded result such as shutdown_timed_out, already_interrupted, cleanup_pending, or not_found, rather than holding the parent turn open.
• A failed close should not require the user to interrupt and risk rolling back the parent conversation.

This feels adjacent to #25870 and #25426, but the extra signal here is the repeated close after rollback/resume: the same stale interrupted child can keep re-poisoning the parent turn until the user stops touching it.