Evaluate retrieval separately from generation

When a RAG answer is wrong, there are two very different causes: the right context was never retrieved, or it was retrieved and the model ignored it. Treating 'the AI is wrong' as one bug hides which of these it is — and the fixes are completely different. So measure the two halves separately.

Measure retrieval on its own

Build a set of questions labelled with the chunks that should answer them, then score retrieval directly:

• Recall@k — did the relevant chunk make it into the top k results at all? If not, generation never had a chance.
• MRR (mean reciprocal rank) — how high did the right chunk rank? Position matters, because models attend most to the top.
• Precision — how much of what you retrieved was actually relevant, versus noise crowding the context.

Measure generation on its own

Now hand the model the known-correct context and ask a different question: given the right information, does it produce a faithful, relevant answer? Faithfulness (or groundedness) scoring — often via LLM-as-judge — catches the model contradicting, ignoring, or embellishing its sources.

Why the split pays off

In practice, most RAG failures are retrieval failures. If your recall@k is low, no amount of prompt tuning will help — go fix chunking, hybrid search, or reranking (see the RAG-improvement post). If retrieval is solid but answers are still wrong, the problem is generation — prompt or model. Splitting the metric tells you which knob to turn.

'The AI answered wrong' isn't a diagnosis. Separate retrieval from generation and it becomes one: either you didn't find the answer, or you didn't use it.