Vector search alone is not a retrieval system

Every RAG prototype demos well. You embed a few hundred documents, ask a question the documents obviously answer, and get a good response. Then it goes in front of actual users, someone searches for INV-2024-8871, and the system confidently returns four paragraphs about invoicing policy.

That failure isn’t a model problem. The generation model never had a chance — it was handed the wrong context. Almost everything that separates a prototype from a system people trust lives in the retrieval layer, before the LLM is involved at all.

Why pure vector search fails

Embeddings capture meaning. That’s the entire point, and it’s also the problem: they’re lossy about exact tokens by design.

INV-2024-8871 and INV-2024-8872 embed to nearly the same vector. So do two contract clauses that differ only in a threshold number. Semantic similarity has no way to express “this exact string, and not the one that differs by a digit.”

The failures cluster in predictable places:

• Identifiers — invoice numbers, case numbers, part numbers, patient IDs
• Rare proper nouns — a client name that appeared twice in the whole corpus
• Jargon and acronyms the embedding model never saw in training
• Negation and numeric thresholds — “not covered”, “under 30 days”

These are exactly the queries people type when they’re trying to get real work done. Nobody asks a document system open-ended thematic questions; they ask where a specific thing is.

Why keyword search alone also fails

BM25 handles every case above perfectly. It also completely misses “how do I expense a client dinner?” when the policy says “reimbursement for business entertainment,” because they share no terms.

Both methods fail. They fail on different queries. That’s the useful part.

Hybrid retrieval, then reranking

The architecture that has held up for me:

1. Run both retrievers. Dense vector search and BM25, independently, over the same chunks. Take the top ~50 from each.

2. Fuse with Reciprocal Rank Fusion. RRF scores each document by Σ 1/(k + rank) across both result lists. Its virtue is that it uses only ranks, not scores — so you never have to normalize a cosine similarity against a BM25 score, which is a comparison with no principled answer. A document both retrievers rank highly rises to the top; a document either one ranks first still surfaces.

3. Rerank with a cross-encoder. The first two stages optimize for recall — get the right chunk somewhere in the top 50. A cross-encoder reads the query and each candidate together rather than comparing precomputed vectors, which makes it far more accurate and far too slow to run over the whole corpus. Over 50 candidates it’s fast enough, and it’s what lets you pass only 5–8 chunks to the model instead of 20.

4. Then generate. With fewer, better chunks, the generation step gets easier and the answers get shorter and more grounded.

Stage 3 is the one people skip, and it’s the one with the largest quality-per-unit-effort payoff in the entire pipeline.

Chunking matters more than the embedding model

Teams spend weeks comparing embedding models and ten minutes on chunking. The ratio should be inverted.

A chunk is what gets retrieved and what the model sees. Cut it wrong and no retriever can save you: split a table from its header and the numbers become meaningless; split a clause from its defined terms and the model has to guess. Respect document structure — sections, headers, table boundaries — and keep enough overlap that a concept spanning a boundary survives in at least one chunk.

Attach the document title and section heading to each chunk’s text. It’s a one-line change that improves both retrievers, because it gives BM25 keywords and the embedding model context that the chunk body alone doesn’t carry.

Build the eval set first

You cannot tune any of this by feel. Twenty to fifty real questions with known-correct source documents is enough to start, and collecting them takes an afternoon.

Measure retrieval separately from generation — recall@k for the retriever, groundedness for the answer. When quality drops you need to know which half moved. Without that split you end up changing prompts to fix what is actually a chunking bug.

The eval set is also the only honest way to answer “did that change help?” Every improvement above sounds obviously correct. Some of them won’t help on your corpus.

The short version

Run dense and sparse retrieval together, fuse by rank, rerank the survivors with a cross-encoder, and spend your tuning time on chunk boundaries and an eval set rather than on the embedding model leaderboard.

None of this is exotic. It’s just the part that doesn’t fit in a demo.