Most RAG failures are retrieval, not generation — evaluate them separately

When a RAG answer is wrong, there are two very different causes: the right context was never retrieved, or it was retrieved and the model ignored it. Treating 'the AI is wrong' as one bug hides which of these it is — and the fixes are completely different. So measure the two halves separately.

Measure retrieval on its own

Build a set of questions labelled with the chunks that should answer them, then score retrieval directly:

• Recall@k — did the relevant chunk make it into the top k results at all? If not, generation never had a chance.
• MRR (mean reciprocal rank) — how high did the right chunk rank? Position matters, because models attend most to the top.
• NDCG — a rank-aware score that rewards putting the most relevant chunks highest, not just somewhere in the top k.
• Precision (and context precision) — how much of what you retrieved was actually relevant, versus noise crowding the context.

Diagnosing a low score

A low recall@k or MRR isn't a dead end — it's a three-stage diagnosis, worked in order. First, retrieval: if the right chunk isn't being fetched at all, add hybrid search (BM25 + dense) so exact terms and paraphrases both land. Second, reranking: if the right chunk is retrieved but ranked low, add or tune a cross-encoder reranker to push it toward the top — which is precisely what MRR rewards, since it scores the rank of the first relevant hit. Third, data preparation: if neither moves the number, the problem is upstream — a chunk size that splits the answer across two chunks, or missing metadata that leaves chunks context-less. Most low scores are fixed at stage one or stage three; reranking is the middle adjustment.

Measure generation on its own

Now hand the model the known-correct context and ask a different question: given the right information, does it produce a faithful, relevant answer? Faithfulness (or groundedness) scoring — often via LLM-as-judge — catches the model contradicting, ignoring, or embellishing its sources.

Why the split pays off

In practice, most RAG failures are retrieval failures. If your recall@k is low, no amount of prompt tuning will help — go fix chunking, hybrid search, or reranking (see the RAG-improvement post). If retrieval is solid but answers are still wrong, the problem is generation — prompt or model. Splitting the metric tells you which knob to turn.

'The AI answered wrong' isn't a diagnosis. Separate retrieval from generation and it becomes one: either you didn't find the answer, or you didn't use it.