Building RAG that doesn't hallucinate in front of customers

The gap between a RAG demo and a RAG product is enormous. The demo answers the three questions you tested it on. The product faces thousands of real queries, half of them phrased in ways you never anticipated. Here's what separates the two.

Retrieval quality is the whole game

The model can only be as grounded as the chunks you hand it. No matter how capable the LLM, it cannot answer accurately from context that's irrelevant, incomplete, or wrong — it's garbage in, garbage out. And this reframes where hallucinations come from: most of them aren't the model failing to reason, they're the retriever failing to find the right source material. That's why roughly 80% of RAG engineering effort belongs in retrieval, not in prompting the generator.

• Hybrid (BM25 + dense) beats either approach alone on real query distributions.
• Reranking the top candidates matters more than fetching more of them.
• Chunking strategy is a product decision, not a default — respect document structure.

Hybrid search: lexical precision and semantic recall

Dense embeddings and keyword search fail in opposite ways, which is exactly why production retrieval runs both. Lexical search (BM25) matches exact terms — it's what reliably finds a SKU like PROD-8821, an error code like ERR_404_NET, a proper name, or a niche acronym — but it has no notion of meaning, so a search for 'automobile' won't surface a document that only ever says 'car'. Dense vectors are the mirror image: they capture meaning and know 'car' and 'automobile' are the same idea, but they blur exact identifiers, happily confusing Part-101 with Part-102 because the surrounding text looks identical. Run both and you get lexical precision and semantic recall together; drop either and you open a blind spot.

Hybrid wins because real queries are a mix

This isn't a benchmark curiosity — it's about how real users actually type. Some fire terse keyword fragments ('error 502 setup'); others ask full conversational questions ('how do I fix my connection when it keeps dropping?'). A real query distribution is a mixture of the two, so keyword-only search misses the conversational half and vector-only search misses the identifier half. Hybrid covers the whole spectrum, which is why it beats either approach alone on live traffic — not on a curated test set, but on the messy queries people really send.

Reranking: high recall without the noise

There's a tension between recall and precision. Fetch the top 50 chunks instead of the top 5 and the right one is far more likely to be in the set — but you've also flooded the model with dozens of irrelevant chunks that dilute its attention and blow up latency and cost. Reranking resolves it. A dedicated cross-encoder (the bi-encoders-vs-cross-encoders post explains why it scores relevance so much better) re-scores those 50 candidates against the query and reorders them, and you pass only the best 3–5 to the model. You get the high recall of fetching many with the clean context of feeding few — the reranker is the fine-grained filter that sits between the two.

Chunking is a product decision, not a default

The lazy default — split every 500 characters with a 50-character overlap — ignores how human information is actually structured, and it does real damage: it slices a table in half, severs an equation from its definition, or splits a heading from the paragraph it introduces. Good chunking respects the document's structure so each chunk is a complete, self-contained thought, and what that means depends on the content: chunk a codebase by function or class, a legal document by clause or section, an article or Markdown file by headings and paragraphs. The right boundary is a design decision about the content — not a number you copied from a tutorial.

Ground every answer with citations

An answer without a source is a claim you can't verify. We require the model to cite the retrieved passages it used, which does two things: it gives users a way to check, and it makes hallucinations visible in evals instead of invisible in production.

Evaluate retrieval and generation separately

When an answer is wrong, you need to know whether retrieval failed to find the right context or generation failed to use it. Measuring them separately turns 'the bot is wrong sometimes' into a fixable engineering problem.

Grounding isn't a prompt trick. It's an architecture — retrieval, citations, and evals working together.