Advanced RAG retrieval: parent-child, hybrid fusion, and two-stage reranking
Basic RAG retrieves a chunk and hopes. Production RAG over long, structured documents needs real retrieval architecture — small-to-big chunks, BM25+dense fusion, and a two-stage rerank. Here's the toolkit.
The RAG-fundamentals and RAG-improvement posts get you a working system; document-heavy domains (legal, technical, scientific) need more. When retrieval quality is the whole ballgame and the documents are long and structured, a few retrieval-architecture moves separate a demo from something you'd stake a decision on.
Chunk small, return big
There's a tension in chunking: small chunks match a query precisely but lack surrounding context; large chunks carry context but match fuzzily. The fix is to decouple what you match from what you return.
- Parent-child (small-to-big) — embed and match on small chunks, but return the parent section they belong to, so the model gets precision in retrieval and context in the prompt.
- Sentence-window — embed individual sentences, then return a window of the surrounding sentences around the best match.
- Section-aware chunking — for structured documents, split on the document's real structure (abstract, spec, claims, drawings) rather than fixed character counts, so a chunk is a coherent unit.
Hybrid search: BM25 + dense, fused
Dense embeddings capture meaning but miss exact tokens — part numbers, statute codes, precise terms — while BM25 nails lexical matches and misses paraphrase. Run both and fuse the results. Reciprocal Rank Fusion (RRF) combines two ranked lists by summing 1/(k + rank) for each document, which needs no score normalisation between the two very different scorers, and you can weight one retriever over the other when one is more trustworthy for your corpus.
Two-stage: retrieve wide, rerank narrow
An ANN/bi-encoder search is fast but approximate — it embeds query and documents separately. A cross-encoder reranker reads the query and each candidate together and scores relevance far more accurately, but it's expensive: one model pass per candidate. So you use both. Retrieve a wide shortlist cheaply (say top 100), then rerank it down to the best few (top 5) with a cross-encoder or a hosted reranker like Cohere Rerank. The economics work because reranking cost is bounded by the shortlist size — keep it capped.
High recall vs high precision
The retrieval goal isn't universal — it depends on the task. A chat assistant wants high precision: the top few results must be right, because that's all the model sees. A prior-art or discovery search wants high recall: missing the one relevant document is the failure, and some noise is acceptable. Tune k, fusion weights, and reranking for the goal you actually have, not the default one.
The failure modes to design against
- Lost-in-the-middle — a relevant chunk buried in the middle of a long context gets ignored; put the best chunks first (which is exactly what reranking buys you).
- Chunk-boundary loss — the answer is split across two chunks and neither is retrieved whole; overlap and parent-child retrieval mitigate it.
- Stale index — the source changed but the embeddings didn't; re-index on change, and treat the index freshness as a monitored property.
Basic RAG asks 'what's similar?'. Production RAG asks 'what's similar, phrased how the corpus phrases it, ranked by a model that actually read the query' — and only then hands the model an answer.