Score retrieved context before you trust the answer
RAG failures often look like model failures because the model speaks with the confidence of the context it was given. The first move is to separate retrieval quality from generation quality.
Give each chunk three labels: directly answers, adjacent background, or contradiction. The answer should cite directly answering chunks first and name contradictions instead of smoothing them away.
A good retrieval review is boring and mechanical. It asks whether the evidence was even in the room before debating whether the model reasoned over it correctly.
Retrieved is not the same thing as relevant.
Aha moment
Retrieval only proves that documents were fetched. It does not prove the fetched text answers the question.
Try this
Label each retrieved chunk as direct support, background, contradiction, or irrelevant before reading the final answer.
Watch for
- Answers citing adjacent background as if it were direct support
- Contradictory chunks hidden by smooth synthesis
- High source count used as a substitute for relevance