Semantic codebase indexing and search

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• #3504

Powered by Codex Action

Is this something you're actively working on? If so and/or the maintainers indicate approval, I'd be interested to work on it.

The biggest obstacle I see is that the model already seems to rely too heavily on the semantics of function and variable names, and not enough on analyzing actual code flow. This is particularly troublesome in projects (ahem) where meaning of identifiers is... let's say local to the person who created them. I wonder if it wouldn't actually help to run a light obfuscation before asking the model to generate identifiers on its own, which could then be mapped back to the original identifiers for the index. The model could also highlight large discrepancies, like "this function is called X but it really does Y".

Thanks for raising that, I’m not actively working on it yet, but this is exactly the kind of discussion I was hoping for before starting the implementation.

You make a great point about identifier semantics. I’ve noticed the same issue: Codex (and even embeddings-based search) can over-fit to surface tokens rather than real data or control flow. That’s actually one of the motivations for the indexing approach, it gives us an opportunity to add an optional preprocessing layer before embedding.

Your idea of lightweight obfuscation is really interesting. It could help normalize identifiers so the embeddings capture actual logic and relationships instead of names. Conceptually, that could mean:
• Mapping identifiers to generic placeholders (func_1, var_2, etc.) before computing embeddings
• Keeping a mapping file for reverse-lookup when displaying search results
• Comparing embedding similarity between obfuscated and original versions to flag “semantic drift”, like your example of a function named X that actually does Y

If maintainers are open to it, this could even become a flag (e.g. --normalize-identifiers) during indexing, so users can toggle it depending on their repo’s naming quality.

I’ll hold off on coding until there’s confirmation from maintainers, but I really like this idea, it would make the index much more robust across different naming conventions.

I wonder if the model would get confused by too many similar placeholders all at once. The indexer might walk (a cache of) the project both from the entry point as well as the folder structure, applying generated names as it goes and falling back on the original identifiers for hints when needed (for data structures, interfaces, etc.). There might be something in the Reverse Engineering space that is doing something similar already, since it's essentially the same problem.

Here, it wouldn't be about trying to untangle potentially malicious code per se, but nudging the model toward analyzing the control flow more than the identifiers. Probably not so important as a first step, but it seems like a good long-term goal (IMO). It seems like a semantic index could also cut down massively on agentic workloads.

I've only been watching this repo for a short time, but I don't see a lot of activity from maintainers, especially in Issues, but it does appear that there are already some semantic indexing projects designed as MCP servers...

https://github.com/elastic/semantic-code-search-mcp-server
https://github.com/yichuan-w/LEANN

That’s a really solid observation, I share your concern about placeholder overload. Too many generic identifiers (like hundreds of var_n or func_n) could definitely collapse embedding diversity and confuse the vector space. I like your idea of walking the project graph instead of doing a flat text replacement, building a semantic skeleton rather than a literal rename pass.

A hybrid approach could work well here:
• Selective obfuscation: Replace identifiers only where their naming adds little value (local variables, temporary helpers), while preserving or lightly normalizing domain-heavy names (e.g., CustomerAccount → EntityType1_CustomerAccount instead of Class_23).
• Context-weighted normalization: Use folder hierarchy and imports to group and disambiguate placeholders, so embeddings retain some structural signal even after normalization.
• Dual embedding: We could even maintain two indices here, one for original source, one for normalized and compare them to highlight where identifier semantics diverge from actual behavior (your “X does Y” scenario). That could be incredibly insightful for large, aging codebases.

And agreed the reverse-engineering analogy is spot on. Tools like IDA Pro or Ghidra essentially reconstruct call graphs and reassign symbolic names to drive deeper analysis. The same principles apply here, just in a benign, developer-oriented context.

As for existing MCP servers like the ones you linked (Elastic’s and LEANN), I think those strengthen the case rather than replace it, they show that semantic indexing is valuable, but those projects live as external MCP integrations. What I’m proposing here is to bring that capability natively into Codex CLI, so that both local users and the Codex agent itself can leverage it out of the box without separate setup.

Long-term, that could indeed cut down on agentic workloads massively. Instead of repeatedly scanning or prompting over raw files, the agent could query the precomputed semantic index and operate on compressed representations, effectively turning Codex into a self-optimizing retrieval layer for code.

Definitely appreciate your insight, I’d be interested in collaborating if this moves forward or if maintainers give the green light.

Apologies if this is obvious — just hoping to contribute something useful.

I think the concern about Codex over-indexing function names is valid — but it might be a case of premature optimization.

A good next step could be to prototype a minimal eval setup to observe the behavior of a toy code-indexing system. You could even run this directly on the Codex CLI codebase or another well-known open-source project. That would make the evals easy to generate and the results self-evident.

In general, when you notice misbehavior, the fastest way to improve the model is to capture it in a small, reproducible eval and then iterate until those cases disappear.

That’s a great point,I completely agree that before adding abstraction or normalization layers, it makes sense to observe the current model’s behavior in a controlled setup.

A minimal eval on a known repo (like Codex CLI itself or a smaller open-source project) would definitely clarify how much the model currently relies on naming semantics versus actual code flow. It would also help quantify the practical gain of adding a semantic index at all.

I was planning to start with a prototype indexer that only chunks and embeds files, then run a few evaluation passes:

1. Baseline: direct keyword search or name-based retrieval
2. Semantic-only: query using embeddings without any identifier normalization
3. Hybrid: embedding after simple normalization (e.g., variable → generic token)

That should produce some measurable signal about whether semantic indexing actually improves retrieval quality or reduces context misses.

If you or others are already running lightweight evals internally, I’d be happy to align with whatever metrics or harness you use, the goal would be to get a reproducible dataset of “retrieval errors” we can iterate on.

Thanks for surfacing this, I think you’re right that it’s the pragmatic next step to turn the idea into something testable.

Hi everyone — I’m currently working on this project and running some experiments related to the topics being discussed here.

The repo is available at: https://github.com/allenanswerzq/llmcc

My goal is to set up a lightweight indexing system, integrate it with Codex, and observe how it behaves — essentially what @CaliLuke mentioned earlier.

It can already parse the Codex repo — you can try it out if interested with:

cargo run --release -- --dir ./codex

If anyone is exploring similar ideas, feel free to take a look or collaborate. Contributions and feedback are very welcome!

On the other hand, been trying the new Claude Haiku 4.5 model in github copilot, feels its already very very good in terms of speed and context understanding... maybe an indexing isn't need after all in the future, with model grow stronger and stronger, who would know ;-).

@allenanswerzq I am also working on it and I have a full coded solution that I never got the chance to test :D

I'm looking forward to see yours succeed!

I think there are many approaches to "correct indexing" as it is not trivial, so any approach is welcome.
My repo:
https://github.com/teabranch/agentic-code-indexer

@OriNachum thanks for the nice comments, I feel right now the direction is more "agentic" (like using tools rg, grep etcc) of doing everything, so a very simple indexing tool might be useful in this scenarios.

Did anything ever come of this?

this is a very useful feature. is it being worked on?

For anyone still looking for this: I built something that does exactly what this issue describes. It's a semantic codebase indexing and search tool that works as a Claude Code plugin, but the core approach applies to any AI coding assistant.

How it works: it chunks your codebase, generates embeddings locally via Ollama, and stores them in SQLite with sqlite-vec for vector search and FTS5 for keyword matching. Queries run a hybrid search that combines semantic similarity with BM25 keyword scoring and identifier boosting. So searching "database connection pooling" finds create_pool, get_connection, release_conn even when there's zero string overlap.

The indexing hooks into file change events so it stays in sync automatically. No manual reindexing needed.

Fully local by default, no API keys required. Optional cloud embeddings if you prefer that route. Hitting about 98% retrieval accuracy in my benchmarks across mixed language codebases.

Would love feedback from anyone who tries it: https://github.com/sagarmk/beacon-plugin

Implemented this on a fresh branch and pushed it to my fork for review.

Branch: issue-5181-semantic-search
Commit: f7cbded0bb7dc6bce6b0e7cc6a6334707d3b785c
Compare: https://github.com/openai/codex/compare/main...dineshkumarkummara:issue-5181-semantic-search?expand=1

What is included:

• codex index to build a local semantic index for the workspace
• codex search for natural-language retrieval over indexed chunks
• optional --normalize-identifiers and --dual-embeddings modes to reduce over-reliance on local naming
• local .codex_index/index.json storage with chunk metadata and embeddings
• mocked end-to-end CLI tests for indexing and search

Validation run locally:

• just fix -p codex-cli
• cargo test -p codex-cli

I also tried to open the upstream PR directly from the branch, but GitHub blocked CreatePullRequest for my account, so I’m sharing the compare link here in the meantime.

Following the contributor policy in docs/contributing.md, I am not opening an unsolicited PR.

The implementation is ready on my fork if the team wants to invite a PR or cherry-pick it:

• branch: dineshkumarkummara:issue-5181-semantic-search
• commit: f7cbded0bb7dc6bce6b0e7cc6a6334707d3b785c
• compare: https://github.com/openai/codex/compare/main...dineshkumarkummara:issue-5181-semantic-search?expand=1

It includes:

• codex index / codex search
• local semantic index storage
• optional identifier normalization and dual embeddings
• mocked end-to-end CLI coverage

Validation run locally:

• just fix -p codex-cli
• cargo test -p codex-cli

If this aligns with the intended direction, could one of the maintainers advise whether you’d want this submitted as an invited PR or picked up internally?

cc @pakrym-oai @aibrahim-oai @charley-oai

I built a working prototype for a narrower version of this problem that may be useful as an intermediate step before full semantic/vector search.

Instead of adding a brand new search surface, the approach improves the existing grep_files path by routing between:

• rg
• a local shared trigram index

based on:

• repo size
• query shape
• index availability / freshness

The goal is not to replace ripgrep everywhere. The goal is to speed up the search-heavy cases where Codex currently relies on repeated repo scans, while preserving safe fallback
behavior.

Behavior:

• small repos: keep using rg
• medium/large repos with no ready index: use rg immediately and build the index in the background
• medium/large repos with a warm index and compatible query shape: use the local index
• unsupported / stale / poor-fit cases: fall back to rg

Why I think this is relevant to this issue:

• it directly addresses codebase search performance on larger repos
• it does not require Codex to jump straight to semantic search as the first step
• it keeps the current grep_files tool contract intact
• it may also help reduce some of the repeated broad-rg pain seen in issues like:
• #7572
• #13717
• #15159
• #16016

I tested this approach in a Codex integration prototype and recorded a live side-by-side on torvalds/linux where Codex is visibly using grep_files to solve the same task on both
sides.

Observed result on that workload:

• baseline Codex: 60.3s mean
• Codex + adaptive trigram index: 25.8s mean
• same final answer across all benchmark runs
• mean speedup: 2.33x

Important limitations:

• this is not a blanket claim that indexing beats rg everywhere
• small repos still favor rg
• query shape matters
• the value is in routing intelligently, not replacing ripgrep wholesale

This is a different approach from the broader semantic/vector direction discussed here, but it may still fit under the same umbrella as a lower-risk or earlier-step improvement.

I am not opening an unsolicited PR. If this direction is interesting, I can share:

• a short design note focused only on grep_files backend routing
• benchmark summary
• the live side-by-side demo

Adding a Mac desktop app UX perspective here after reading through the thread.

Most of the discussion above seems focused on the core semantic indexing/search capability, CLI commands such as codex index / codex search, embedding strategy, identifier normalization, and external plugin/MCP-style implementations. I think those are all valuable, but I want to add a related angle that I do not see fully covered yet: the Codex Mac app product experience.

In Cursor, the workspace-level Codebase Indexing feature is visible and persistent. In practice, this makes file/code-area retrieval feel much more efficient, because the tool can rely on an existing workspace index instead of spending a long time repeatedly exploring and reading files before each project-level task.

For the Codex Mac desktop app, I would love to see this exposed as a first-class, user-controllable project/worktree indexing experience, not only as a terminal command:

• visible indexing/syncing status for each selected project or worktree
• pause, resume, rebuild, and delete index controls
• incremental updates when files change
• .codexignore support in addition to .gitignore
• easy exclusion of build outputs, generated files, logs, vendor folders, secrets, large binaries, and other irrelevant files
• clear privacy documentation about what is stored locally and whether embeddings/metadata ever leave the machine
• use of the index to reduce repeated long Exploring phases in the Mac app before implementation/debugging work begins
• final verification by reading current source files before editing, so the index helps with retrieval and prioritization but is not blindly trusted

The main user-facing pain point is that the Mac app can feel like it repeatedly starts repository discovery from scratch, even when working on the same project across related tasks. A persistent project index UI would make the desktop workflow feel much closer to how users expect a project-aware coding app to behave, while still preserving Codex's safety model of reading the real current files before making edits.

Pinging this issue again because this is still one of the biggest practical gaps between Codex and tools like Cursor.

Cursor exposes codebase indexing as a visible feature: it indexes the repository, supports semantic search over the codebase, and gives users a clear mental model of how the assistant finds relevant code.

With Codex, it is much less clear what happens. From the user side, it often looks like Codex relies mostly on explicit context, open files, shell searches, and manual exploration. That can work, but for larger repositories it creates uncertainty around token efficiency, relevance, and whether Codex is repeatedly rediscovering the same project structure.

Could the team clarify a few things?

• Does Codex currently build any persistent local/project-level index of the repository?
• Is there any semantic search layer over code chunks, symbols, or documentation?
• If not, is this planned as a first-class feature?
• Should users who need this today build their own MCP-based indexing/search server, or would that duplicate work that Codex is expected to support natively?
• Are there recommended best practices for making Codex more token-efficient on large repositories?

Even a short official answer would help. If native codebase indexing is not on the roadmap, it would be useful to know that too, so users can design their own workflows around MCP, ripgrep, AST indexes, or external retrieval tools.

Thanks for reviving this. I agree the biggest missing piece right now is an official answer from the Codex team.

I cannot answer for maintainers on roadmap/current internals, but from the user side I think the thread has converged on a few separable pieces:

1. Lower-risk retrieval performance: @Sagart-cactus's adaptive grep_files / trigram-index direction looks like a practical Phase 0 because it preserves the existing tool contract, keeps rg fallback behavior, and avoids introducing embeddings/privacy concerns immediately.
2. Semantic retrieval: the prototype branch I shared earlier explores codex index / codex search, optional identifier normalization, and dual embeddings. I still think this solves the deeper large-repo discoverability problem, but it would need maintainer alignment before becoming a real contribution.
3. Product UX: @GGBondBlueWhale's Mac app points are important. If indexing exists, users should have visibility and control: status, pause/resume/rebuild/delete, incremental updates, ignore rules, clear privacy docs, and final verification against current source before edits.

So +1 to @igor-markin's questions. Official clarification would help a lot on:

• whether Codex currently has any persistent local/project-level index
• whether native semantic search over code chunks/symbols/docs is planned
• whether users should build MCP/indexing integrations today or wait for native work
• what best practices the team recommends for large repos and token efficiency

Until then, I think the safest community direction is to keep sharing prototypes, benchmarks, and design notes in this issue rather than opening unsolicited PRs, consistent with docs/contributing.md.