Codex Code Search with BM25: Reduce Exploration Tokens

One driver of the cost and latency of coding agents is inefficient discovery of candidates: using glob/grep plus sequential file reads. As a repository grows, agents open unrelated files, invent the next search term, and read again.

📑Table of Contents

1. What the paper showed — BM25 overtakes File-System Agents at scale
2. Mapping to Codex — a secondary BM25-fronted experiment
3. Beyond Codex — hybrid BM25 and when grep still wins
4. Adoption checklist and next actions
5. FAQ
6. Takeaways

Wang et al., BM25 Wins at Scale (arXiv:2607.26497), show a controlled crossover where BM25 overtakes a File-System Agent on accuracy-cost as corpora scale.

Engineering design questions

On the engineering side, this is the design question:

• whether to put BM25-style ranking in front
• whether to keep the agent for later reasoning

This article cross-checks the following claims against independent sources:

1. the paper’s crossover and Agent+BM25 control numbers
2. a secondary Codex experiment that fronts BM25 candidate search
3. independent hybrid/lexical analysis for coding agents
4. an adoption checklist

---

What you can decide after reading

After reading it, you should be able to decide:

• whether a BM25 front-end is worth a controlled A/B on your repo
• what to measure
• which claims not to treat as guarantees

---

Audience and grounding

The primary audience is:

• Codex / Claude Code users
• RAG and internal-search designers
• SWE platform owners

The discovery write-up is a Zenn secondary report by Knowledgesense. Factual grounding stays with arXiv and independent sources.

---

Learning outcomes

---

What the paper showed — BM25 overtakes File-System Agents at scale

As corpora grow, ranking candidates globally becomes both cheaper and more reliable than walking through raw files.

Experiment design

Wang et al. (arXiv:2607.26497v3, 2026-07-31) compare retrieval paradigms on EnterpriseRAG-Bench with this controlled setup:

• 28 strictly nested corpus tiers (~1,144→511,959 documents, ~450×)
• Fixed questions, gold/adversarial bedrock, and reader/judge
• Measured accuracy, build/query tokens, and latency
• Paradigms: lexical (BM25), dense, graph, and agentic (File-System Agent)

---

Headline results

These are the headline results (point estimates from the paper and independent summaries):

• On the smallest shared tier, the File-System Agent can lead on accuracy, but bedrock query tokens are about 39× BM25
• Near ~10M corpus tokens, BM25 crosses over and leads on later shared tiers
• At full scale, the BM25 lead approaches roughly 20 points (third-party summary example: BM25 50.5 vs Agent 30.7)
• oio-blog’s memo also records small-scale integrated scores around FS-Agent 77.4 / BM25 74.7 and small-scale query tokens BM25 5.8K vs Agent 226K

---

Agent+BM25 control

The Agent+BM25 control (same 150 questions) makes the role split concrete:

• raw-file agent: score 36.9 / 895K tokens per query
• Agent+BM25: score 69.4 / 101K tokens per query; document recall 36.8→72.4
• BM25 alone (memo figure): 54.8 / 5.8K

---

Practical implication

The practical implication is:

• treat lexical retrieval as the scalable default
• put agentic reasoning after ranked discovery
• failures often start in candidate discovery, not only in late-stage synthesis

---

Comparison table by setting

Setting / metric	File-System Agent (raw)	BM25	Agent+BM25 control
Small-corpus tendency	May lead on point estimate	Close / trailing	—
Bedrock query tokens	~39× BM25	Low-cost end (e.g. 5.8K)	Mid (e.g. 101K)
After ~10M corpus tokens	Relative drop as search space grows	Leads shared tiers	—
Full-scale accuracy gap	~20pt behind in summaries	Lead	36.9→69.4 (150-q control)
Control tokens/query	895K	~5.8K alone	101K

---

Sources (as of August 2026)

• arXiv:2607.26497
• AI Weekly alert
• oio-blog paper memo
• AI Native Foundation digest (2026-08-07)
• recsys.substack note
• Zhihu / Tommy Tang write-up

---

How to read the limits

The limits come first:

• single benchmark family
• single reader/judge stack
• limited coverage of hybrid BM25+rerank or agent-over-top-k
• read the claim as “cheap global ranking scales better than raw-file agency,” not “BM25 always wins.”

---

Mapping to Codex — a secondary BM25-fronted experiment

The paper studies controlled RAG for enterprise documents. Coding agents need secondary evidence.

Secondary-experiment framing

The Zenn report (Knowledgesense / Hideto Sudo, around 2026-08-11) compares Codex on an internal codebase with vs without a local BM25 front-end. It is a secondary experiment, not a paper reproduction.

---

Reported setup

The reported setup is:

• Baseline: Glob / Grep / file reads
• BM25 path: local BM25 (MCP) plus targeted follow-up reads
• Repo: ~7,536 files / 68.8MB; ~15,000 fragments; SQLite index 164MB built in ~27s
• Codex CLI 0.146.0-alpha.3.1, model gpt-5.6-sol high
• 4 easy / 4 medium / 4 hard questions; one run per condition; fresh sessions

---

Reported outcomes

The reported outcomes (all questions answered correctly in their setup) are:

• Session tokens: 1,812,066 → 1,283,250 (−29.2%)
• Per-question median reduction 28.5% (10 of 12 questions improved)
• Median wall time 50.8s → 30.0s (reference; median −34.2%)
• Median token cut by difficulty: easy 39.8% / medium 31.9% / hard 28.5% (one medium and one hard case worsened under BM25)

---

Implementation notes

These are the implementation notes from the report:

• camel/snake identifier splitting
• Japanese bigrams
• path-token boosts
• returns top paths, line ranges, and short snippets
• deep investigation stays with normal Codex reads

The hypothesized savings come from:

• fewer irrelevant reads
• fewer “invent next search term” loops

---

Secondary-result snapshot

Metric	Baseline search	BM25-fronted	Delta
Session token total	1,812,066	1,283,250	−29.2%
Per-question median cut	—	28.5%	10/12 improved
Median wall time (ref.)	50.8s	30.0s	−20.8s
Index build	—	~27s / 164MB SQLite	fixed adoption cost

Source: Zenn / Knowledgesense secondary report. Primary paper: arXiv:2607.26497.

---

How to read the ~30% figure

Reader handling for the ~30% figure:

• treat ~30% as conditioned on that repo, question set, and single-run protocol
• it is not the same scale as the paper’s 895K→101K control
• do not copy it as a guarantee
• use it as a hypothesis for a fixed-question A/B on your tree

For adjacent Codex cost layers beyond search, pair this with notes on GPT-5.6 × Codex serving/harness efficiency and Codex CLI release changes so you do not mix exploration cost with runtime/serving cost.

---

Beyond Codex — hybrid BM25 and when grep still wins

The paper’s split—lexical discovery first, then the agent—matches the engineering complaint that grep-then-read in a monorepo burns context.

Monorepo search cost

Particula Tech (Sebastian Mondragon, 2026-07-05) independently analyzes coding-agent search cost:

• hybrid stacks (AST chunks + dense + BM25) or MCP tools that return relevant functions
• roughly 40% token cuts at equal retrieval quality in vendor self-evals
• not third-party benches

---

Practical roles

The practical roles are:

• BM25: identifiers, rare tokens, path language
• Dense: semantic questions (“where do we authenticate?”)
• Agent: cross-file synthesis, edits, judgment after candidates exist
• Index ops: Merkle-tree style re-embed of changed files so maintenance tracks churn

---

When grep still wins

The same analysis lists many MCP clients (including Codex CLI) and also states the boundary:

• small repos, exact-match lookups, and one-off queries still favor grep
• fashionable hybrid defaults lose when index fixed cost never pays back

---

Situation guide

Situation	Prefer	Why / caveat
Mid/large monorepo, repeated exploration	BM25 front-end or hybrid	Candidate discovery dominates
Unique identifier, one-shot locate	grep / exact match	Index overhead rarely pays
Semantic “where is auth?” queries	hybrid (BM25+dense)	Dense covers lexical vocabulary gaps
High-churn monorepo	incremental re-index	Full rebuild ops fail in practice
Security boundary	local index / tight MCP scope	Shipping full trees externally is a different risk

---

Sources and security boundary

Sources (July–August 2026):

• Particula Tech analysis
• arXiv:2607.26497

⚠️ Even a local BM25/MCP approach still needs policy on index scope, tool permissions, and whether any payload leaves the machine. Cutting tokens by uploading a full private tree is a security trade, not only a cost win.

---

Adoption checklist and next actions

Paper figures (39× / ~20pt) and the secondary 29.2% cut are conditioned measurements. Start from your own sessions.

Now (30–60 minutes)

• ✓ Review last week of Codex/agent sessions: tool-call counts and rough tokens spent on search/read
• ✓ Split failures into “never found candidates” vs “failed after candidates”
• ✓ Note repo size (files/MB) and identifier-search vs semantic-search mix

---

Minimum experiment (half day to one day)

• ✓ Build a fixed 8–12 question set (easy/medium/hard); compare baseline vs BM25 front-end in fresh sessions on the same model
• ✓ Track session tokens, per-question median, answer/coverage quality, and wall time as a reference
• ✓ Record index build time, size, and update mode (full vs incremental)

---

Proceed to adopt when (examples)

• ✓ Tokens or time improve in a reproducible band without unacceptable quality loss
• ✓ Medium/hard regressions are identified and have fallbacks (grep / explicit paths)
• ✓ MCP/local-index permissions match org policy for secrets

---

Skip or hold when

• Small repo where grep already one-shots the answer
• Truly one-off questions that never amortize index cost
• You need the paper’s 39× / 20pt or the secondary 29.2% as a contractual guarantee for your environment

---

Next sequence

A durable sequence is the following:

1. measure
2. fixed-question A/B
3. design fallbacks for regressions
4. incremental index ops

Reading Codex Security CLI/SDK and serving and harness topics in parallel helps keep these separate:

• search cost
• execution, permission, and delivery cost

---

FAQ

Q1. Does “BM25 wins” transfer directly to coding agents?

The paper is controlled enterprise-document RAG. Coding transfer needs secondary experiments and implementation analysis; it is not an unconditional win claim.

---

Q2. Is “~30% fewer tokens” universally reproducible?

No. It is Knowledgesense’s secondary measurement on ~7.5k files, 12 questions, and a specific Codex/model setup (overall −29.2%). Re-measure with a fixed set on your tree.

---

Q3. If BM25 is added, is the agent obsolete?

In the paper control, Agent+BM25 beat the raw-file agent on score and tokens. Let lexical search own candidate discovery; keep the agent for synthesis and cross-cutting investigation.

---

Q4. Should dense / vector DBs replace BM25?

The paper shows dense can be efficient yet lose accuracy to BM25 in the studied regime. Code identifiers favor BM25; semantic queries often need hybrid—consistent with Particula’s framing.

---

Q5. How should indexes be refreshed?

The secondary report’s snapshot build is ~27s. High-churn trees need incremental re-embed (e.g. Merkle-style). Stale indexes silently degrade recall.

---

Q6. Does the same apply to Claude Code and other agents?

The candidate-discovery cost principle may transfer, but tools, MCP support, and measurement differ. Do not paste Codex numbers onto other products without retesting.

---

Takeaways

Core findings

• Wang et al. show BM25 overtaking File-System Agents on accuracy-cost as corpora grow, with agentic work best after ranked discovery (~10M crossover, ~39× query tokens, Agent+BM25 36.9→69.4 / 895K→101K)
• The secondary Codex experiment reports ~29% session-token savings with a BM25 front-end; treat it as a conditioned implementation hint, not the paper control
• Independent analysis strengthens the monorepo grep-then-read problem and the “grep still wins on small/exact/one-off” boundary
• Next action: measure your sessions, A/B on a fixed question set, define regression fallbacks and index ops, then decide adoption

---

Grounding note

Base facts on arXiv, independent secondary sources, and your own measurements—not popularity signals alone.

Related articles:

• OpenAI Codex Security CLI/SDK
• 11959