State startup query scans the full threads table
What issue are you seeing?
Codex state runtime initialization executes this query when it seeds its thread timestamp cache:
SELECT MAX(threads.updated_at_ms), MAX(threads.recency_at_ms)
FROM threads
SQLite performs a full table scan even though idx_threads_updated_at_ms and idx_threads_recency_at_ms already exist. On a local state database with 23,739 threads and a file size of about 944 MB, the query took approximately 1.87 seconds and repeatedly triggered the SQLx slow-statement warning during worker startup. I found 376 instances of this warning across local worker transcripts.
The same query remains present on current upstream main at commit a7b8c074b577f897111c14de3a5e127b91e2a479.
What steps can reproduce the bug?
Run the current aggregate query against a sufficiently populated Codex state database:
sqlite3 -readonly ~/.codex/state_5.sqlite \
'EXPLAIN QUERY PLAN SELECT MAX(threads.updated_at_ms), MAX(threads.recency_at_ms) FROM threads;'
Observed plan:
SCAN threads
On the database described above:
real 1.87
Then inspect the equivalent query with independent scalar aggregates:
SELECT
(SELECT MAX(updated_at_ms) FROM threads),
(SELECT MAX(recency_at_ms) FROM threads)
Its plan searches the two existing covering indexes:
SEARCH threads USING COVERING INDEX idx_threads_updated_at_ms
SEARCH threads USING COVERING INDEX idx_threads_recency_at_ms
On the same database it completed below the resolution of /usr/bin/time:
real 0.00
What is the expected behavior?
State initialization should use the existing timestamp indexes and avoid a full scan of the threads table, so startup does not become slower as local thread history grows and does not emit a slow-query warning under normal use.
Additional information
The root cause is the combined pair of MAX aggregates over one table source. SQLite cannot use both single-column indexes for that form, while independent scalar subqueries allow each aggregate to use its matching covering index.
I have a small fix that:
- rewrites the initialization query as two scalar subqueries;
- adds a regression test using
EXPLAIN QUERY PLANthat requires both covering indexes; - passes
just fmt,just fix -p codex-state, all 171codex-stateunit tests, and its doc test.
The patch is ready here:
- Branch: https://github.com/assimelha/codex/tree/codex/optimize-thread-recency-max
- Commit: https://github.com/assimelha/codex/commit/429644613
Per the contribution policy, I am filing this issue before opening a PR and requesting an invitation to submit the prepared patch.
2 Comments
Could you assign this to me
Status update: this appears fixed on current
main@ 1f41cc5d92 — the startup seed query is now the scalar-subquery form, which SQLite optimizes into two index seeks instead of a scan:https://github.com/openai/codex/blob/1f41cc5d92722748e45cae9cecc6d883a4e7cbb1/codex-rs/state/src/runtime.rs#L221
SELECT (SELECT MAX(updated_at_ms) FROM threads), (SELECT MAX(recency_at_ms) FROM threads)— each inner query qualifies for SQLite's single-MAX index optimization againstidx_threads_updated_at_ms/idx_threads_recency_at_ms. You can confirm on your 944 MB store withEXPLAIN QUERY PLANon that exact statement (expect twoSEARCH ... USING COVERING INDEXrows, noSCAN). Worth a re-test once a release containing it reaches you, then this can close.