Hybrid search lost to BM25, and two other things I was wrong about
I had 434 documents and about 836,000 words that I could not search. Not "hard to search." Could not. I knew a particular number was in there somewhere, and the only way to find it was opening files one at a time.
So I built a retrieval pipeline over them. Chunking, embedding, retrieval, evaluation. No API calls, no network, fully reproducible on a laptop.
The second reason was less practical. I kept reading that hybrid retrieval beats plain lexical search, and I wanted to measure that claim on a corpus I knew well enough to grade the answers myself.
It did not beat lexical search. Not here. That was the first of three things I got wrong, and those three are the only interesting part of this post.
What the thing actually is
| Stage | Approach |
|---|---|
| Chunking | Markdown-structure-first. Splits on heading boundaries and keeps the heading path as metadata, so every result cites its section. Long sections window at 900 characters with 180 overlap, breaking on paragraph or sentence ends. 10,356 chunks, mean 93 tokens. |
| Embedding | Hashing trick plus signed random projection over word unigrams, bigrams and character trigrams, into 256 dimensions, L2-normalised. Deterministic, so an index rebuilds byte for byte. |
| Retrieval | BM25 (k1=1.4, b=0.75) and cosine similarity, fused with Reciprocal Rank Fusion (k=60). |
| Evaluation | 20 hand-verified cases, split into verbatim and paraphrase families. Recall@k, MRR and precision reported per mode. |
One line is worth pulling out of that table. Reciprocal Rank Fusion combines the two retrievers on rank, not on score. That matters because BM25 scores and cosine similarities live on different scales with no principled conversion between them. A naive weighted sum has to invent one. RRF never needs it.
The evaluation split matters too. Verbatim queries share vocabulary with their target document. Paraphrase queries do not. Every metric is reported per mode, which means fusion has to actually beat each single retriever. Assuming a hybrid is better is easy. Checking is not much work.
Finding one: hybrid lost
| mode | recall@8 | MRR | precision | MRR verbatim | MRR paraphrase |
|---|---|---|---|---|---|
| hybrid | 70.0% | 0.511 | 15.6% | 0.758 | 0.141 |
| lexical | 75.0% | 0.575 | 22.5% | 0.889 | 0.105 |
| dense | 50.0% | 0.383 | 8.8% | 0.611 | 0.042 |
BM25 alone beat the hybrid on recall, on MRR, and on precision. Not by a rounding error. Precision fell from 22.5% to 15.6% when I added the dense retriever to it.
The reason is not mysterious. My dense half is a hashing trick with a signed random projection, into 256 dimensions. It is deterministic and it costs nothing, which is exactly why I chose it. It is also not a learned embedding. It captures vocabulary shape. It does not capture meaning. Fusing it into BM25 adds noise more often than signal.
I kept fusion anyway, for one reason: the paraphrase column. Hybrid gets 0.141 MRR there against lexical's 0.105. That is the one place a fuzzy retriever is supposed to earn its keep, and it does, slightly.
"Hybrid is better" is a claim about a learned embedder. The word hybrid is doing none of the work. If your dense half is not learned, you are fusing noise into a retriever that was already working.
Finding two: the paraphrase number is bad, and I printed it
0.141 MRR on paraphrase queries is not good. I published it anyway, and I want to be clear about why.
That number is the honest ceiling of a non-semantic embedder. Take a query like "the trucking fuel card company" against a document that only ever says "OTR payments." There is no vocabulary overlap at all. No amount of parameter tuning bridges that, because the gap is semantic and my embedder has no semantics. Tuning k1 and b will not do it. More dimensions will not do it.
So I treat that number as a bar. If I swap in a real embedding API, 0.141 is what it has to beat to have justified the cost and the network dependency. That is why embed.mjs ships with a provider interface and not a hardcoded implementation. The upgrade is a decision I have already priced, not one I will discover later.
Most retrieval write-ups report the configuration that won. Publishing the ceiling of the configuration I could afford seemed more useful.
Finding three: the technique that made it worse
This is my favourite, because it is the one I was most confident about going in.
Pseudo-relevance feedback is well established. You run the query, take the top results, assume they are relevant, pull terms out of them, and re-run an expanded query. RM3 is the standard formulation. It works in the literature.
It made mine worse. Hybrid went from 70.0% recall and 0.511 MRR down to 50.0% and 0.400. It degraded the verbatim cases too, where I had least reason to expect trouble: MRR dropped from 0.758 to 0.583.
That is textbook query drift. Expansion assumes the first pass is good enough that its top chunks are genuinely relevant. Mine is not. So expansion pulls the query away from what was actually asked, and the second pass lands further from the answer than the first.
The code is still in the repository. It ships defaulted off, with those numbers written in the comment above it, and the failure reproduces with { prf: true }. I kept it because the negative result is worth more than deleting the evidence. If I come back with a learned embedder and a stronger first pass, this is worth re-testing, and I will want to know exactly how badly it failed the first time.
Two things I deliberately did not build
No generation. This is retrieval, not RAG end to end. Nothing synthesises an answer, so nothing can fabricate one. Every result is a verbatim span with a citation back to its heading path. That matters more than usual here, because the corpus is my own material. A system that confidently invents a detail about my own history is worse than no system at all.
No re-ranker, and no approximate nearest neighbour index. A cross-encoder would probably lift precision above 22.5%, and it would need a model. 10,356 chunks scan in well under a second. HNSW would be premature.
Naming what you did not build, and why, is worth as much as the feature list. Most write-ups only give you the second half.
The guard I care about most
eval.mjs exits 0 with a SKIP line when no index exists. A missing index can never produce a green build. An index that exists but is empty exits 1, because that is a real failure. The regression floor sits at 0.65 recall and 0.45 MRR, a notch under the measured baselines, so that growing the corpus does not turn the suite red while a genuine regression still does.
I use the same guard on an eval suite for a different project, and it comes from one rule I try not to break.
A green build on a test that never ran is worse than no test. A silent skip is not neutral. It is a false negative wearing a checkmark, and it will let a regression through on the exact day you stop watching.
Three findings, all of them negative. I would rather publish those than a tuned configuration and a confident conclusion.
← Back to writing