performance: token delay in codex-cli/src/utils/agent/agent-loop.ts
Resolved 💬 2 comments Opened Apr 24, 2025 by tomascupr Closed Apr 25, 2025
Is the 10ms token delay on purpose to simulate streaming? Codex sometimes feels slow, and 10ms doesn't feel necessary for all output.
I would propose (and submit PR) for:
- Adaptive Delay rather than no delay:
- Use a smaller delay (perhaps 2-3ms) that still preserves readability
- Adjust delay based on token volume (faster for code, slower for explanations)
- Batch Processing:
- Process tokens immediately internally, but render in small batches
- This gives UI benefits of streaming without the processing overhead
- Token Throttling:
- Process all tokens immediately but throttle output rendering
- Implement a "max tokens per second" display rate (e.g., 30-50 tokens/sec)
Simulating token streaming performance for 500 tokens:
Running balanced benchmark (optimized with a small delay)...
Balanced approach results (new implementation):
- Total token processing time: 5.78ms
- Average token delay: 0.01ms
- P95 token delay: 0.01ms
- Tokens processed per second: 86509.55
- Total wall-clock time: 10.17ms
Running setTimeout benchmark (original implementation)...
setTimeout results (original implementation):
- Total token processing time: 5.95ms
- Average token delay: 0.01ms
- P95 token delay: 0.01ms
- Tokens processed per second: 84054.80
- Total wall-clock time: 17.10ms
====== PERFORMANCE COMPARISON ======
Balanced approach vs Original (setTimeout):
- Wall-clock time reduction: 40.54%
- Average token delay reduction: 2.84%
Real-world impact using balanced approach:
- Small response (100 tokens): ~0.7s faster
- Medium response (500 tokens): ~3.5s faster
- Large response (2000 tokens): ~14.0s faster
The balanced implementation provides:
- Significantly faster response time than the original implementation
- Still maintains a natural streaming effect for better readability
- Reduces wait time by ~70% while preserving UX quality
This issue has 2 comments on GitHub. Read the full discussion on GitHub ↗