performance: token delay in codex-cli/src/utils/agent/agent-loop.ts

Resolved 💬 2 comments Opened Apr 24, 2025 by tomascupr Closed Apr 25, 2025
💡 Likely answer: A maintainer (tomascupr, contributor) responded on this thread — see the highlighted reply below.

Is the 10ms token delay on purpose to simulate streaming? Codex sometimes feels slow, and 10ms doesn't feel necessary for all output.

I would propose (and submit PR) for:

  1. Adaptive Delay rather than no delay:
  • Use a smaller delay (perhaps 2-3ms) that still preserves readability
  • Adjust delay based on token volume (faster for code, slower for explanations)
  1. Batch Processing:
  • Process tokens immediately internally, but render in small batches
  • This gives UI benefits of streaming without the processing overhead
  1. Token Throttling:
  • Process all tokens immediately but throttle output rendering
  • Implement a "max tokens per second" display rate (e.g., 30-50 tokens/sec)

Simulating token streaming performance for 500 tokens:

Running balanced benchmark (optimized with a small delay)...

Balanced approach results (new implementation):

  • Total token processing time: 5.78ms
  • Average token delay: 0.01ms
  • P95 token delay: 0.01ms
  • Tokens processed per second: 86509.55
  • Total wall-clock time: 10.17ms

Running setTimeout benchmark (original implementation)...
setTimeout results (original implementation):

  • Total token processing time: 5.95ms
  • Average token delay: 0.01ms
  • P95 token delay: 0.01ms
  • Tokens processed per second: 84054.80
  • Total wall-clock time: 17.10ms

====== PERFORMANCE COMPARISON ======
Balanced approach vs Original (setTimeout):

  • Wall-clock time reduction: 40.54%
  • Average token delay reduction: 2.84%

Real-world impact using balanced approach:

  • Small response (100 tokens): ~0.7s faster
  • Medium response (500 tokens): ~3.5s faster
  • Large response (2000 tokens): ~14.0s faster

The balanced implementation provides:

  1. Significantly faster response time than the original implementation
  2. Still maintains a natural streaming effect for better readability
  3. Reduces wait time by ~70% while preserving UX quality

View original on GitHub ↗

2 Comments

tomascupr contributor · 1 year ago
tibo-openai collaborator · 1 year ago

PR was merged, closing