Building RAG that doesn't hallucinate in front of customers
A demo that answers your three test questions is easy. A retrieval system that stays grounded across thousands of real queries is a different discipline. Here's what actually matters.
The gap between a RAG demo and a RAG product is enormous. The demo answers the three questions you tested it on. The product faces thousands of real queries, half of them phrased in ways you never anticipated. Here's what separates the two.
Retrieval quality is the whole game
The model can only be as grounded as the chunks you hand it. Dense embeddings alone miss exact-match terms — product codes, names, acronyms — so we run hybrid retrieval: BM25 for lexical precision plus dense vectors for semantic recall, then rerank the merged set.
- Hybrid (BM25 + dense) beats either approach alone on real query distributions.
- Reranking the top candidates matters more than fetching more of them.
- Chunking strategy is a product decision, not a default — respect document structure.
Ground every answer with citations
An answer without a source is a claim you can't verify. We require the model to cite the retrieved passages it used, which does two things: it gives users a way to check, and it makes hallucinations visible in evals instead of invisible in production.
Evaluate retrieval and generation separately
When an answer is wrong, you need to know whether retrieval failed to find the right context or generation failed to use it. Measuring them separately turns 'the bot is wrong sometimes' into a fixable engineering problem.
Grounding isn't a prompt trick. It's an architecture — retrieval, citations, and evals working together.