Black Friday50% off your first month on any monthly plan with EARLY50 See plans
All 29 lessons
  1. Foundations: what RAG is and a working pipeline

    1. 1. Getting started: what RAG is and when to use it
    2. 2. Types of RAG: a map from naive to agentic
    3. 3. Lab: build the smallest RAG pipeline end to end
    4. 4. How embeddings and vector search find meaning
    5. 5. Document types: what each one breaks, and how to prepare it
  2. Better retrieval: chunking and search

    1. 6. Chunking documents so answers survive
    2. 7. Chunking strategies compared: which to use when
    3. 8. Keyword search vs vector search, and fusing both
    4. 9. Retrieval algorithms compared: which to use when
    5. 10. Reranking the top results before the model reads them
    6. 11. Rewriting the question before you search
  3. Answers you can trust, measure and debug

    1. 12. Lab: grounded answers with citations and refusals
    2. 13. Measuring a RAG system with an evaluation set
    3. 14. Lab: evaluation methods and frameworks: Ragas and DeepEval
    4. 15. Troubleshooting a RAG system, symptom by symptom
  4. Production in a company

    1. 16. Access control: only retrieve what the user may see
    2. 17. Defending against prompt injection in retrieved text
    3. 18. Running RAG in production: freshness, cost and monitoring
    4. 19. Real-world RAG: what worked, what failed, and why
  5. Agentic RAG

    1. 20. Lab: from a fixed pipeline to an agent with a search tool
    2. 21. Self-correcting retrieval: grade, retry, fall back
    3. 22. GraphRAG and routing across many sources
    4. 23. Lab: numbers, tables and financial documents
  6. Frameworks: LangChain, LangGraph and LangSmith

    1. 24. The RAG toolbox: what each library does and why you import it
    2. 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
    3. 26. Lab: Ask Brightline in LangChain
    4. 27. Lab: agentic RAG as a LangGraph graph
    5. 28. Lab: tracing, evaluating and monitoring with LangSmith
    6. 29. Wrap-up: your production agentic RAG checklist
See the leaderboard

22 min

Lab: evaluation methods and frameworks: Ragas and DeepEval

Score Ask Brightline with the popular retrieval and answer metrics, using Ragas and DeepEval, and read what each score tells you.

By the end of this lesson you can

  • Compute hit rate, precision, recall, MRR and nDCG, and say what each low score means
  • Score faithfulness, answer relevancy and context precision and recall with Ragas
  • Run RAG checks as DeepEval tests that fail a build

From earlier lessons · 1 of 2

What does recall@k tell you?

Pick one answer.

From earlier lessons · 2 of 2

Before trusting an LLM judge, what should you do?

Pick one answer.

Ask Brightline answers "$250 a night" for a Denver hotel. Denver's limit is $175. Your recall@3 says 1.0, because the right chunk was retrieved. The answer is still wrong, and the metric you built in lesson 13 can't see it.

Lab: score Ask Brightline the standard ways

Lesson 13 hand-built two retrieval metrics and a judge. Real teams use shared metric definitions, so scores mean the same thing across projects. We'll add three more retrieval metrics, then run Ragas and DeepEval, the two most widely used open-source RAG evaluation frameworks. Versions were checked in November 2026: ragas 0.4.4 and deepeval 4.3.4. Judged scores vary a little between runs and judge models, so compare runs that use the same judge.

bash
pip install "ragas==0.4.4" "deepeval==4.3.4" # pin: unpinned installs can pull an old deepeval
export RAGAS_DO_NOT_TRACK=true DEEPEVAL_TELEMETRY_OPT_OUT=1 # both send usage data by default
export OPENAI_API_KEY=... # the default judge model in both frameworks' examples
Terminal. Steps 1, 2 and 5 run with no key; steps 3, 4 and 6 call a judge model.

Step 1: five retrieval metrics

Three questions, each with the top 3 retrieved chunk ids and graded relevance: 2 means the chunk answers the question, 1 means it helps. The third question's chunk was never retrieved.

python
import math
def hit_rate(ret, rel, k): return float(any(d in rel for d in ret[:k]))
def precision(ret, rel, k): return sum(d in rel for d in ret[:k]) / k
def recall(ret, rel, k): return sum(d in rel for d in ret[:k]) / len(rel)
def rr(ret, rel): return next((1 / r for r, d in enumerate(ret, 1) if d in rel), 0.0)
def ndcg(ret, grades, k):
dcg = sum(grades.get(d, 0) / math.log2(i + 1) for i, d in enumerate(ret[:k], 1))
ideal = sorted(grades.values(), reverse=True)[:k]
return dcg / sum(g / math.log2(i + 1) for i, g in enumerate(ideal, 1))
RUNS = [(["travel-meals", "travel-hotels", "travel-tiers"], {"travel-hotels": 2, "travel-tiers": 1}),
(["travel-flights", "travel-meals", "hr-pto"], {"travel-flights": 1}),
(["travel-hotels", "it-vpn", "hr-pto"], {"travel-meals": 1})]
text
hit@3 [1.0, 1.0, 0.0] mean=0.67
precision@3 [0.67, 0.33, 0.0] mean=0.33
recall@3 [1.0, 1.0, 0.0] mean=0.67
mrr [0.5, 1.0, 0.0] mean=0.50
ndcg@3 [0.67, 1.0, 0.0] mean=0.56
Real output, printing each metric over RUNS. The print loop is left out for brevity.

Each metric asks a different question. Hit rate asks whether anything relevant appeared. Precision@k asks how much of the top k is relevant, which measures noise. Recall@k asks how much of what's relevant appeared. MRR asks how high the first relevant chunk sits.

nDCG@k (normalised discounted cumulative gain) uses the grades. It rewards the best chunks for being first, and discounts each position by log2(rank + 1). Question 1 scores 0.67, not 1.0, because the best chunk sits second behind a useless one. Some sources use 2^grade − 1 as the gain, so say which version you report.

python
import asyncio
from ragas import SingleTurnSample
from ragas.metrics import IDBasedContextPrecision, IDBasedContextRecall
s = SingleTurnSample(retrieved_context_ids=["travel-meals", "travel-hotels", "travel-tiers"],
reference_context_ids=["travel-hotels", "travel-tiers"])
print(asyncio.run(IDBasedContextPrecision().single_turn_ascore(s)),
asyncio.run(IDBasedContextRecall().single_turn_ascore(s)))
Step 2, no model needed. Real output: 0.6666666666666666 1.0. These ID-based metrics live in Ragas's older API and warn that they'll be removed in v1.0.

Step 3: the four core Ragas metrics

Ragas 0.4 moved its metrics to ragas.metrics.collections. They take keyword arguments and return a result with .value and .reason. Many tutorials still show the older evaluate() API, which now warns that it's deprecated.

[32]

python
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.embeddings import embedding_factory
from ragas.metrics.collections import Faithfulness, AnswerRelevancy, ContextPrecision, ContextRecall
Q, ANS = "How much can I spend on a hotel in Denver?", "Up to $250 per night."
CTX = ["Hotels: up to $250 per night in Tier 1 cities (New York, San Francisco, Chicago, Boston) and $175 elsewhere."]
REF = "$175 per night, since Denver is not a Tier 1 city."
client = AsyncOpenAI()
llm, emb = llm_factory("gpt-4o-mini", client=client), embedding_factory("openai", model="text-embedding-3-small", client=client)
print(Faithfulness(llm=llm).score(user_input=Q, response=ANS, retrieved_contexts=CTX).value,
AnswerRelevancy(llm=llm, embeddings=emb).score(user_input=Q, response=ANS).value,
ContextPrecision(llm=llm).score(user_input=Q, reference=REF, retrieved_contexts=CTX).value,
ContextRecall(llm=llm).score(user_input=Q, retrieved_contexts=CTX, reference=REF).value)
Expected: faithfulness low (the context says $175 for non-Tier-1 cities), relevancy high (it answers the question asked), and both context scores high (the right chunk was there). Exact values vary by judge.

That pattern is the hook's bug, now visible. Retrieval was fine; generation misread the chunk. Here's what each core metric measures:

  • Faithfulness checks whether every claim in the answer is supported by the retrieved context. Low means the model invented or misread facts.
  • Answer relevancy checks whether the answer addresses the question. Low means it drifted or answered something else.
  • Context precision checks whether the relevant chunks rank above the irrelevant ones. Low means the right facts are buried in noise.
  • Context recall checks whether the context holds everything the reference answer needs. Low means retrieval missed facts.

Precision and recall need a reference answer written by a person. Faithfulness and answer relevancy don't, so they can also run on live traffic.

[33][34][35]

Step 4: a synthetic test set, and its limits

python
from langchain_core.documents import Document
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from ragas.testset import TestsetGenerator
docs = [Document(page_content=open("travel_policy.md").read(), metadata={"source": "travel_policy.md"})]
gen = TestsetGenerator.from_langchain(ChatOpenAI(model="gpt-4o-mini"), OpenAIEmbeddings())
testset = gen.generate_with_langchain_docs(docs, testset_size=10)
print(testset.to_pandas().columns.tolist())
Expected output: ['user_input', 'reference_contexts', 'reference', 'synthesizer_name']. It rejects documents of 100 tokens or fewer, so use a full policy, not a snippet.

A synthetic test set is a model writing questions and reference answers from your documents. It's a fast way to reach 50 questions before launch. But generated questions sound like the documents, not like your employees. Read them, delete the odd ones, and replace them with real questions as traffic arrives.

[36]

Step 5: tests that fail the build, with DeepEval

DeepEval wraps metrics in pytest-style tests. A failing score fails the test, so a bad change can't merge. Start with a cheap check that needs no model: does the answer contain the right amount?

python
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import PatternMatchMetric
cases = [LLMTestCase(input="Hotel limit in Chicago?", actual_output="Up to $250 per night.",
expected_output="$250 per night, since Chicago is a Tier 1 city.", retrieval_context=CTX),
LLMTestCase(input="Hotel limit in Denver?", actual_output="Up to $250 per night.",
expected_output="$175 per night, since Denver is not a Tier 1 city.", retrieval_context=CTX)]
@pytest.mark.parametrize("test_case", cases)
def test_amount(test_case):
amount = test_case.expected_output.split()[0].replace("$", r"\$")
assert_test(test_case, [PatternMatchMetric(pattern=rf".*{amount}.*")])
Save as test_policy.py. CTX is from step 3.
text
$ deepeval test run test_policy.py
Pass Rate: 50.0% | Passed: 1 | Failed: 1
FAILED test_amount[test_case1] - Pattern Match (score: 0.0, threshold: 1.0 ...)
Real output, with no key and no login. The Denver bug fails the build.

Step 6: DeepEval's RAG metrics

python
from deepeval.metrics import (FaithfulnessMetric, AnswerRelevancyMetric, ContextualPrecisionMetric,
ContextualRecallMetric, ContextualRelevancyMetric)
rag_metrics = [FaithfulnessMetric(threshold=0.7), AnswerRelevancyMetric(threshold=0.7),
ContextualPrecisionMetric(), ContextualRecallMetric(), ContextualRelevancyMetric()]
@pytest.mark.parametrize("test_case", cases)
def test_rag(test_case):
assert_test(test_case, rag_metrics)
Needs a judge key: without one, the metrics fail as soon as they're created. The default threshold is 0.5.

DeepEval's metrics mirror Ragas's, with one extra: contextual relevancy, the share of retrieved text that's relevant at all. Its docs recommend deepeval test run in CI over plain pytest, and offer flaky=True for judged tests that occasionally wobble. Each judged metric is a model call per test case. Run the cheap pattern checks on every commit, and the judged suite nightly or before a release.

[37][38]

Match

Match each low score to what it tells you

Pick an item on the left, then its match on the right.

Worth keeping

  1. Retrieval metrics are free and exact; run them on every change.
  2. Faithfulness and answer relevancy need no reference, so they also run on live traffic.
  3. Context precision and recall need reference answers, and split ranking noise from missing facts.
  4. Cheap checks on every commit, judged suites nightly; pin framework versions.

Metrics tell you that something is wrong. They don't tell you which stage broke, or why. Next lesson is a debugging playbook that turns one bad answer into a located, fixed stage.

Leaderboard

Loading…

50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.

Sources for this lesson (7)

Each fact in the lesson is marked with the number of the page it comes from.

  1. [32]Ragas docs: Migrating from v0.3 to v0.4
  2. [33]Ragas docs: Faithfulness
  3. [34]Ragas docs: Context Precision
  4. [35]Ragas docs: Context Recall
  5. [36]Ragas docs: Testset generation for RAG
  6. [37]DeepEval docs: Faithfulness metric
  7. [38]DeepEval docs: Unit testing in CI/CD