All 29 lessons
Foundations: what RAG is and a working pipeline
Better retrieval: chunking and search
Answers you can trust, measure and debug
Production in a company
Agentic RAG
Frameworks: LangChain, LangGraph and LangSmith
- 24. The RAG toolbox: what each library does and why you import it
- 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
- 26. Lab: Ask Brightline in LangChain
- 27. Lab: agentic RAG as a LangGraph graph
- 28. Lab: tracing, evaluating and monitoring with LangSmith
- 29. Wrap-up: your production agentic RAG checklist
18 min
Measuring a RAG system with an evaluation set
Build a small test set and score retrieval and answers so every change is measured, not guessed.
By the end of this lesson you can
- Build a small evaluation set of real questions and expected sources
- Compute recall@k and mean reciprocal rank for the retriever
- Judge answer faithfulness with a model, and check the judge
From earlier lessons · 1 of 2
Ask Brightline's reply cites passage [4], but only three were retrieved. What should your code do?
Pick one answer.
From earlier lessons · 2 of 2
Why ask the model for JSON with an answer and a citation list?
Pick one answer.
In lesson 10, the reranker fixed the hotel approval question. We measured it on eight questions: top-1 accuracy rose from 75% to 88%. In the same run, "Can I stay home on Fridays?" fell out of the top three. Nobody would have noticed without the test.
Where we stand: changes by feel
Nine lessons of improvements, each checked on one or two questions. That's how most RAG projects drift. A fix for today's complaint quietly breaks last month's question, and nobody re-runs it.
An evaluation set fixes that: a list of real questions, each with the source it should retrieve and, where useful, a reference answer. You run the whole set after every change and compare numbers.
EVAL = [{"q": "How much can I spend on a hotel in Chicago?", "expect": "travel-hotel"},{"q": "Who has to approve a pricier hotel?", "expect": "travel-hotel"},{"q": "What does VPN-0x80 mean?", "expect": "it-vpn"},{"q": "Can I stay home on Fridays?", "expect": "hr-remote"},{"q": "Is wine with dinner covered?", "expect": "travel-meals"},{"q": "Where do I report a stolen laptop?", "expect": "sec-device"},{"q": "Do points from hotel stays belong to me?", "expect": "travel-loyalty"},{"q": "the dongle thing for remote access is dead", "expect": "it-vpn"},]
Two retrieval metrics
Recall@k is the share of questions whose expected chunk appears in the top k results. It answers one question: did the facts reach the prompt?
Mean reciprocal rank (MRR) rewards ranking the right chunk high. A question scores 1 if the chunk is first, 1/2 if second, 1/3 if third, and the scores are averaged. It tells you whether the best chunk sits where the model looks first.
def score(search, k=3):at_1 = at_k = rr = 0for case in EVAL:ids = search(case["q"])at_1 += ids[:1] == [case["expect"]]at_k += case["expect"] in ids[:k]rr += 1 / (ids.index(case["expect"]) + 1) if case["expect"] in ids else 0n = len(EVAL)return {"recall@1": round(at_1 / n, 2), f"recall@{k}": round(at_k / n, 2), "mrr": round(rr / n, 2)}
Score both versions
vector_only = lambda q: [d["id"] for d, _ in retrieve(q, k=10)]reranked = lambda q: [d["id"] for d, _ in rerank(q, retrieve(q, k=10), keep=10)]print("vector ", score(vector_only))print("reranked", score(reranked))
vector {'recall@1': 0.75, 'recall@3': 1.0, 'mrr': 0.85}reranked {'recall@1': 0.88, 'recall@3': 0.88, 'mrr': 0.89}
Read the numbers, then the cases. Reranking put more right answers first: recall@1 rose from 0.75 to 0.88. But recall@3 fell from 1.0 to 0.88, which is one question out of eight. Printing the per-question results shows which: "Can I stay home on Fridays?" The reranker preferred two travel chunks over the remote-work rule.
Neither version wins outright. Now you can make the trade-off on purpose. You might keep 5 chunks instead of 3, try another reranker, or fuse the reranker's order with the first stage. Then you re-run the set and check.
Scoring the answers, not just retrieval
Retrieval metrics can't tell you whether the answer is right. For that, three answer checks matter most:
- Faithfulness. Is every claim supported by the retrieved context? This catches made-up rules.
- Answer relevance. Does it answer the question that was asked?
- Correctness. Does it match a reference answer? This needs a reference answer written by a person.
The Ragas paper (Es et al., 2023) frames RAG evaluation the same way. It scores whether retrieval found relevant, focused context and whether the model used it faithfully. It does this without needing human-written answers for every question.
Faithfulness is usually checked by an LLM-as-judge: a second model call with a strict rubric. It costs one extra call per question. Before you trust the judge, label 20 answers yourself and count how often it agrees with you. A judge that agrees less than about 9 times in 10 needs a clearer rubric.
def judge_prompt(context, answer):return (f"Context:\n{context}\n\nAnswer:\n{answer}\n\n""Is every claim in the answer supported by the context? Reply with JSON only: "'{"supported": true or false, "unsupported": ["claims not in the context"]}')def faithful(answer, hits):context = "\n".join(d["text"] for d, _ in hits)return parse_reply(llm(judge_prompt(context, answer)))print(faithful("Hotels are $250 a night in Boston, and you get free parking.", retrieve("Boston hotel", k=2)))
Put in order
Put the evaluation loop in order
Drag, or focus an item and press Space, move with the arrow keys, then Space to drop.
- Label each with the chunk it should retrieve
- Run the set and record recall@k, MRR and faithfulness
- Collect real questions from users or tickets
- Re-run the whole set and compare per question
- Make one change to the pipeline
- Add every newly reported failure to the set
Worth keeping
- Re-run the whole evaluation set after every change, not just today's question.
- Recall@k says whether the facts arrived; MRR says how high they ranked.
- Judge faithfulness with a model, and check the judge against your own labels.
- Every reported failure becomes a new question in the set.
You computed recall, MRR and a faithfulness judge by hand. Teams rarely do: Ragas and DeepEval package these metrics and more, and DeepEval can fail a build on a bad score. Next lab runs both on Ask Brightline.
Leaderboard
Loading…
50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.
Sources for this lesson (1)
Each fact in the lesson is marked with the number of the page it comes from.