GuidesRAG

Evaluating a RAG system: measure retrieval and answers

By the Banzai Labs teamUpdated on 2 min read

Test setRetrievalright passage found?Answerfaithful to sources?Score for each versionMeasure retrieval and answer separately
A test question set measures passage retrieval and answer faithfulness separately, for each version.

Two measures, two problems

Where RAG can fail
Retrieval
  • Is the right passage in the top k?
  • Fixed through chunking, embeddings, reranking
  • Simple measure, no judge model
Answer
  • Faithful to the passages, nothing made up?
  • Does it really answer the question?
  • Fixed through the prompt and the model

The test set

[
  { "question": "What is the return period?", "sources": ["terms.md#returns"], "expected": "30 days" },
  { "question": "Do you ship to Belgium?", "sources": ["shipping.md#countries"], "expected": "yes" }
]

The metrics

Metric Question asked
Recall at k Is the expected source among the k retrieved passages?
Faithfulness Is every claim backed by a passage?
Relevance Does the answer address the question?
Correctness Does it match the expected answer?

The routine

  1. 1A changeChunking, model, prompt, number of passages.
  2. 2The full setReplayed automatically.
  3. 3ComparisonKeep the change if it improves without breaking anything.

The basics: RAG on your documents and chunking your documents. For retrieval: hybrid search and reranking.

Frequently asked questions

How many test questions?
Fifty to a hundred to start with, taken from real requests, each with the document or documents that contain the answer.
Who grades the answers?
A judge model with a precise rubric grades quickly. Check it against a hand-graded sample before trusting it.
Are there tools for this?
Yes, open source libraries such as Ragas provide these metrics. A homemade script is often enough at first.

Read next

All guides