TUI freeze during agent fan-out

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #16384

Powered by Codex Action

Do you have reason to believe this is a recent regression? If you downgrade to 0.116.0, for example, are you able to repro the problem still? I ask because this affects prioritization.

Do you have reason to believe this is a recent regression? If you downgrade to 0.116.0, for example, are you able to repro the problem still? I ask because this affects prioritization.

Happens in that version as well, not sure if it ever worked corretly. If it never did, Let me know if I can open a PR with the fix that works for me locally so maybe we can discuss there what's the best solution.

You mentioned that you've "already fixed the issue locally". I presume that you've analyzed the problem and understand the root cause. Can you explain what you've found?

Starting 100 subagents is outside the bounds of the current design. By default, we limit the max number to 6. You've presumably overridden that default value of agents.max_threads in your config file. If you start significantly more than 6 subagents, you'll likely hit 429's on the responses API because you'll trigger DDoS detection, so even if the client didn't hang here, you probably wouldn't be able to get this to work. If there's a straightforward fix to prevent hangs, we will entertain it, but I think we'd consider this a low priority.

You mentioned that you've "already fixed the issue locally". I presume that you've analyzed the problem and understand the root cause. Can you explain what you've found? Starting 100 subagents is outside the bounds of the current design. By default, we limit the max number to 6. You've presumably overridden that default value of agents.max_threads in your config file. If you start significantly more than 6 subagents, you'll likely hit 429's on the responses API because you'll trigger DDoS detection, so even if the client didn't hang here, you probably wouldn't be able to get this to work. If there's a straightforward fix to prevent hangs, we will entertain it, but I think we'd consider this a low priority.

Actually the API doesn't throw a 429, it could be because my test agents don't do that much work.
Also keep in mind i could reproduce the issue even with 10 agents (I'm running an i7 9700K and 16GBs of DDR4 3200 RAM which seems reasonable for such a workload).
As for the issues I found:

• Blocking thread_read RPC in the TUI event loop

hydrate_collab_agent_metadata_for_notification was making a synchronous thread_read RPC to the embedded app-server for every agent notification. When the app-server was busy handling 100 concurrent spawn requests, that RPC queued behind all the spawn work and blocked the TUI event loop. While this was happening, crossterm kept queueing tick events, so after the block finally resolved, each queued tick triggered an expensive render (4+ seconds each), compounding the freeze.

• SQLite connection pool contention

Each agent was calling StateRuntime::init(), which opens 2 new SQLite connection pools (state + logs, max_connections(5) each), all pointing at the same DB files, runs migrations (write lock) and attempts VACUUM (exclusive lock).
With 100 agents: 200 pools, up to 1,000 connection attempts, all fighting for SQLite's file-level locks. Process inspection showed 483 threads, mostly blocked on SQLite.

• node --version subprocess per agent

Not that important, but i still fixed it: Codex::spawn_internal calls resolve_compatible_node() up to twice per agent (once for JsRepl, once for CodeMode) and each call spawns node.exe as a subprocess to run node --version.

• spawn_agent serialized through a write lock

spawn_agent had supports_parallel_tool_calls = false, which in ToolCallRuntime::parallel forced every spawn call to acquire an exclusive RwLock write guard. With 100 spawn tool calls from a single turn, they executed one at a time, each waiting for the previous to complete session init. This doesn't seem to cause any regressions from the automated tests or my own testing, but maybe there was a reason why supports_parallel_tool_calls was set to false.

I’ve been consistently hitting this in long-lived sessions, and it seems to have existed since Codex moved to the app-server + TUI frontend split.

When a session has accumulated many historical sub-agents, typing /agent and pressing Tab can freeze the entire TUI. Normally that opens the picker and shows all sub-agents, including running, stopped, and completed ones. But once the history gets large enough, the UI can
become completely unresponsive at the moment Tab is pressed, even though the underlying task is still running.

I’ve waited 20+ minutes in this state, then killed Codex and resumed the same session with codex resume, and found that the work had already finished long before. So this looks like a TUI-side freeze rather than the session itself being stuck.

My current workaround is to quit Codex and immediately codex resume once /agent has accumulated too many historical sub-agents. After resuming, those historical sub-agents no longer appear in /agent, and the picker becomes usable again.

I’ve been consistently hitting this in long-lived sessions, and it seems to have existed since Codex moved to the app-server + TUI frontend split. When a session has accumulated many historical sub-agents, typing /agent and pressing Tab can freeze the entire TUI. Normally that opens the picker and shows all sub-agents, including running, stopped, and completed ones. But once the history gets large enough, the UI can become completely unresponsive at the moment Tab is pressed, even though the underlying task is still running. I’ve waited 20+ minutes in this state, then killed Codex and resumed the same session with codex resume, and found that the work had already finished long before. So this looks like a TUI-side freeze rather than the session itself being stuck. My current workaround is to quit Codex and immediately codex resume once /agent has accumulated too many historical sub-agents. After resuming, those historical sub-agents no longer appear in /agent, and the picker becomes usable again.

If you want when im home I can push my branch of codex to git so you can test it and lmk if it fixes your issue

I'm not able to repro the problem as stated. I've set max_threads to 100, then asked the main agent to spawn 100 subagents, and then used /agents to view the list and switch to various subagents. It takes about a second to display the full list, but at no point does the TUI appears to be "unresponsive".

When you say that you type /agents and then press Tab, are you pressing Return before Tab (to display the agent list)? Or are you pressing tab immediately after /agents? In neither case do I see any freeze.

If you think that you have a "fix" for this issue, please describe the root cause. I'm not looking for a list of speculations but the verified root cause.

I can reproduce this on 0.121 on Linux.
In a clean session, I opened sub-agents one by one without forking the main-session context, closed each immediately, and paused 5 seconds between iterations. Around 20 agents, /agent + Tab already took ~5s to show the menu; around 50 agents, ~10s.
In a real session, once it goes past ~20, even waiting 10 minutes does not recover it.

I can reproduce this in my local machine. Codex stuck forever when spawning even like 4+ agents and i have to manually kill the whole window (Ctrl+C, Ctrl+Z do not work either). Btw I'm on a NFS machine (high latency on read/writes) . I'm not sure if it's related.

--
update: after mentioning these, i tried moving the whole ~/.codex to a local ssd mount point. it works way much better. Spawning is fast but navigating through /agent still sometimes stucks.

I can reproduce this in my local machine. Codex stuck forever when spawning even like 4+ agents and i have to manually kill the whole window (Ctrl+C, Ctrl+Z do not work either). Btw I'm on a NFS machine (high latency on read/writes) . I'm not sure if it's related. -- update: after mentioning these, i tried moving the whole ~/.codex to a local ssd mount point. it works way much better. Spawning is fast but navigating through /agent still sometimes stucks.

Hi,

since i bought exact for one month Pro, i got this updates for you:

For the 128 update:

• Spawning more agents as 9 gets instant TUI crash, wait for 15-30 mins. The reason is, its only the TUI the work will be done.
• Using NFS: I did not have any issues regarding this, due i use SSD as a backend, so only the Gbit is the "slowness" for startup.
• Setting logs_2.sqlite to read only, fixed a lot of problems with TUI crashes, i dont know what exactly it prevents, but it stops growing infinite and the tui lasted for me longer (as said, i do not recommended it, but i also using the "YOLO" mode on Codex....)

-r--r--r--  1 codex codex        0 30. Apr 23:22 logs_2.sqlite
````

So it seems still, there is a problem, the only thing i did not tried to say codex, to wait x seconds after each spawn, if it will help or crash directly


For the dev:
Uploaded thread: 019d4a69-9705-76f3-9d72-0d417bb7f17d

I'm not able to repro the problem as stated. I've set max_threads to 100, then asked the main agent to spawn 100 subagents, and then used /agents to view the list and switch to various subagents. It takes about a second to display the full list, but at no point does the TUI appears to be "unresponsive". When you say that you type /agents and then press Tab, are you pressing Return before Tab (to display the agent list)? Or are you pressing tab immediately after /agents? In neither case do I see any freeze. If you think that you have a "fix" for this issue, please describe the root cause. I'm not looking for a list of speculations but the verified root cause.

are you running on windows or linux? It could be a windows only issue.

also I don't think I ever said I was sending a /agents command, all I did was run a skill that spawned the agents and the TUI would freeze.

i've listed the issues i've found in my second message, if you are asking which one I think is reponsible for the TUI freeze than it's the first one I've listed, the others are performance issues, but they are not strictly required to fix the TUI freezing.

Here is my fork, in which the TUI doesn't freeze: https://github.com/Auties00/codex

hey I told my codex to take a sleep 0.5 between the spawns and to let me work with 12 agents,maybe a try worth

hey I told my codex to take a sleep 0.5 between the spawns and to let me work with 12 agents,maybe a try worth

Do you mind compiling my branch and trying it out? Just to see if it fixes your issue as well (if you ask codex it should be able to compile it for you quiite easily)

I found the regression. PR #21870 stopped doing synchronous thread/read metadata hydration on the TUI render path, which was the right performance direction because large fan-outs could block the UI. The side effect is that collab rows can now render before the TUI has
learned each child agent’s nickname/role, so wait/follow-up rows fall back to raw thread UUIDs even though /agent later has the proper names.