GuidesRAG
Evaluating a RAG system: measure retrieval and answers
Two measures, two problems
Retrieval
Answer
The test set
[
{ "question": "What is the return period?", "sources": ["terms.md#returns"], "expected": "30 days" },
{ "question": "Do you ship to Belgium?", "sources": ["shipping.md#countries"], "expected": "yes" }
]
The metrics
| Metric | Question asked |
|---|---|
| Recall at k | Is the expected source among the k retrieved passages? |
| Faithfulness | Is every claim backed by a passage? |
| Relevance | Does the answer address the question? |
| Correctness | Does it match the expected answer? |
The routine
- 1A changeChunking, model, prompt, number of passages.
- 2The full setReplayed automatically.
- 3ComparisonKeep the change if it improves without breaking anything.
The basics: RAG on your documents and chunking your documents. For retrieval: hybrid search and reranking.
Frequently asked questions
- How many test questions?
- Fifty to a hundred to start with, taken from real requests, each with the document or documents that contain the answer.
- Who grades the answers?
- A judge model with a precise rubric grades quickly. Check it against a hand-graded sample before trusting it.
- Are there tools for this?
- Yes, open source libraries such as Ragas provide these metrics. A homemade script is often enough at first.