Black Friday50% off your first month on any monthly plan with EARLY50 See plans
All 29 lessons
  1. Foundations: what RAG is and a working pipeline

    1. 1. Getting started: what RAG is and when to use it
    2. 2. Types of RAG: a map from naive to agentic
    3. 3. Lab: build the smallest RAG pipeline end to end
    4. 4. How embeddings and vector search find meaning
    5. 5. Document types: what each one breaks, and how to prepare it
  2. Better retrieval: chunking and search

    1. 6. Chunking documents so answers survive
    2. 7. Chunking strategies compared: which to use when
    3. 8. Keyword search vs vector search, and fusing both
    4. 9. Retrieval algorithms compared: which to use when
    5. 10. Reranking the top results before the model reads them
    6. 11. Rewriting the question before you search
  3. Answers you can trust, measure and debug

    1. 12. Lab: grounded answers with citations and refusals
    2. 13. Measuring a RAG system with an evaluation set
    3. 14. Lab: evaluation methods and frameworks: Ragas and DeepEval
    4. 15. Troubleshooting a RAG system, symptom by symptom
  4. Production in a company

    1. 16. Access control: only retrieve what the user may see
    2. 17. Defending against prompt injection in retrieved text
    3. 18. Running RAG in production: freshness, cost and monitoring
    4. 19. Real-world RAG: what worked, what failed, and why
  5. Agentic RAG

    1. 20. Lab: from a fixed pipeline to an agent with a search tool
    2. 21. Self-correcting retrieval: grade, retry, fall back
    3. 22. GraphRAG and routing across many sources
    4. 23. Lab: numbers, tables and financial documents
  6. Frameworks: LangChain, LangGraph and LangSmith

    1. 24. The RAG toolbox: what each library does and why you import it
    2. 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
    3. 26. Lab: Ask Brightline in LangChain
    4. 27. Lab: agentic RAG as a LangGraph graph
    5. 28. Lab: tracing, evaluating and monitoring with LangSmith
    6. 29. Wrap-up: your production agentic RAG checklist
See the leaderboard

18 min

Measuring a RAG system with an evaluation set

Build a small test set and score retrieval and answers so every change is measured, not guessed.

By the end of this lesson you can

  • Build a small evaluation set of real questions and expected sources
  • Compute recall@k and mean reciprocal rank for the retriever
  • Judge answer faithfulness with a model, and check the judge

From earlier lessons · 1 of 2

Ask Brightline's reply cites passage [4], but only three were retrieved. What should your code do?

Pick one answer.

From earlier lessons · 2 of 2

Why ask the model for JSON with an answer and a citation list?

Pick one answer.

In lesson 10, the reranker fixed the hotel approval question. We measured it on eight questions: top-1 accuracy rose from 75% to 88%. In the same run, "Can I stay home on Fridays?" fell out of the top three. Nobody would have noticed without the test.

Where we stand: changes by feel

Nine lessons of improvements, each checked on one or two questions. That's how most RAG projects drift. A fix for today's complaint quietly breaks last month's question, and nobody re-runs it.

An evaluation set fixes that: a list of real questions, each with the source it should retrieve and, where useful, a reference answer. You run the whole set after every change and compare numbers.

python
EVAL = [
{"q": "How much can I spend on a hotel in Chicago?", "expect": "travel-hotel"},
{"q": "Who has to approve a pricier hotel?", "expect": "travel-hotel"},
{"q": "What does VPN-0x80 mean?", "expect": "it-vpn"},
{"q": "Can I stay home on Fridays?", "expect": "hr-remote"},
{"q": "Is wine with dinner covered?", "expect": "travel-meals"},
{"q": "Where do I report a stolen laptop?", "expect": "sec-device"},
{"q": "Do points from hotel stays belong to me?", "expect": "travel-loyalty"},
{"q": "the dongle thing for remote access is dead", "expect": "it-vpn"},
]
Eight questions, each with the chunk it should retrieve. Real sets have 30-100.

Two retrieval metrics

Recall@k is the share of questions whose expected chunk appears in the top k results. It answers one question: did the facts reach the prompt?

Mean reciprocal rank (MRR) rewards ranking the right chunk high. A question scores 1 if the chunk is first, 1/2 if second, 1/3 if third, and the scores are averaged. It tells you whether the best chunk sits where the model looks first.

python
def score(search, k=3):
at_1 = at_k = rr = 0
for case in EVAL:
ids = search(case["q"])
at_1 += ids[:1] == [case["expect"]]
at_k += case["expect"] in ids[:k]
rr += 1 / (ids.index(case["expect"]) + 1) if case["expect"] in ids else 0
n = len(EVAL)
return {"recall@1": round(at_1 / n, 2), f"recall@{k}": round(at_k / n, 2), "mrr": round(rr / n, 2)}

Score both versions

python
vector_only = lambda q: [d["id"] for d, _ in retrieve(q, k=10)]
reranked = lambda q: [d["id"] for d, _ in rerank(q, retrieve(q, k=10), keep=10)]
print("vector ", score(vector_only))
print("reranked", score(reranked))
retrieve() is lab 1's function over the 14 snippets; rerank() is from lesson 10.
text
vector {'recall@1': 0.75, 'recall@3': 1.0, 'mrr': 0.85}
reranked {'recall@1': 0.88, 'recall@3': 0.88, 'mrr': 0.89}
Real output with the models from lessons 3 and 10. Small library differences can shift a decimal.

Read the numbers, then the cases. Reranking put more right answers first: recall@1 rose from 0.75 to 0.88. But recall@3 fell from 1.0 to 0.88, which is one question out of eight. Printing the per-question results shows which: "Can I stay home on Fridays?" The reranker preferred two travel chunks over the remote-work rule.

Neither version wins outright. Now you can make the trade-off on purpose. You might keep 5 chunks instead of 3, try another reranker, or fuse the reranker's order with the first stage. Then you re-run the set and check.

Scoring the answers, not just retrieval

Retrieval metrics can't tell you whether the answer is right. For that, three answer checks matter most:

  • Faithfulness. Is every claim supported by the retrieved context? This catches made-up rules.
  • Answer relevance. Does it answer the question that was asked?
  • Correctness. Does it match a reference answer? This needs a reference answer written by a person.

The Ragas paper (Es et al., 2023) frames RAG evaluation the same way. It scores whether retrieval found relevant, focused context and whether the model used it faithfully. It does this without needing human-written answers for every question.

[8]

Faithfulness is usually checked by an LLM-as-judge: a second model call with a strict rubric. It costs one extra call per question. Before you trust the judge, label 20 answers yourself and count how often it agrees with you. A judge that agrees less than about 9 times in 10 needs a clearer rubric.

python
def judge_prompt(context, answer):
return (f"Context:\n{context}\n\nAnswer:\n{answer}\n\n"
"Is every claim in the answer supported by the context? Reply with JSON only: "
'{"supported": true or false, "unsupported": ["claims not in the context"]}')
def faithful(answer, hits):
context = "\n".join(d["text"] for d, _ in hits)
return parse_reply(llm(judge_prompt(context, answer)))
print(faithful("Hotels are $250 a night in Boston, and you get free parking.", retrieve("Boston hotel", k=2)))
Real output from our run: supported False, with two unsupported claims: the free parking, and "a flat $250" when the policy only sets a limit of up to $250. The judge caught a subtle misreading too. parse_reply is from lesson 12.

Put in order

Put the evaluation loop in order

Drag, or focus an item and press Space, move with the arrow keys, then Space to drop.

  1. Label each with the chunk it should retrieve
  2. Run the set and record recall@k, MRR and faithfulness
  3. Collect real questions from users or tickets
  4. Re-run the whole set and compare per question
  5. Make one change to the pipeline
  6. Add every newly reported failure to the set

Worth keeping

  1. Re-run the whole evaluation set after every change, not just today's question.
  2. Recall@k says whether the facts arrived; MRR says how high they ranked.
  3. Judge faithfulness with a model, and check the judge against your own labels.
  4. Every reported failure becomes a new question in the set.

You computed recall, MRR and a faithfulness judge by hand. Teams rarely do: Ragas and DeepEval package these metrics and more, and DeepEval can fail a build on a bad score. Next lab runs both on Ask Brightline.

Leaderboard

Loading…

50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.

Sources for this lesson (1)

Each fact in the lesson is marked with the number of the page it comes from.

  1. [8]Es et al. (2023), Ragas: Automated Evaluation of RAG