All 29 lessons
Foundations: what RAG is and a working pipeline
Better retrieval: chunking and search
Answers you can trust, measure and debug
Production in a company
Agentic RAG
Frameworks: LangChain, LangGraph and LangSmith
- 24. The RAG toolbox: what each library does and why you import it
- 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
- 26. Lab: Ask Brightline in LangChain
- 27. Lab: agentic RAG as a LangGraph graph
- 28. Lab: tracing, evaluating and monitoring with LangSmith
- 29. Wrap-up: your production agentic RAG checklist
22 min
Lab: evaluation methods and frameworks: Ragas and DeepEval
Score Ask Brightline with the popular retrieval and answer metrics, using Ragas and DeepEval, and read what each score tells you.
By the end of this lesson you can
- Compute hit rate, precision, recall, MRR and nDCG, and say what each low score means
- Score faithfulness, answer relevancy and context precision and recall with Ragas
- Run RAG checks as DeepEval tests that fail a build
From earlier lessons · 1 of 2
What does recall@k tell you?
Pick one answer.
From earlier lessons · 2 of 2
Before trusting an LLM judge, what should you do?
Pick one answer.
Ask Brightline answers "$250 a night" for a Denver hotel. Denver's limit is $175. Your recall@3 says 1.0, because the right chunk was retrieved. The answer is still wrong, and the metric you built in lesson 13 can't see it.
Lab: score Ask Brightline the standard ways
Lesson 13 hand-built two retrieval metrics and a judge. Real teams use shared metric definitions, so scores mean the same thing across projects. We'll add three more retrieval metrics, then run Ragas and DeepEval, the two most widely used open-source RAG evaluation frameworks. Versions were checked in November 2026: ragas 0.4.4 and deepeval 4.3.4. Judged scores vary a little between runs and judge models, so compare runs that use the same judge.
pip install "ragas==0.4.4" "deepeval==4.3.4" # pin: unpinned installs can pull an old deepevalexport RAGAS_DO_NOT_TRACK=true DEEPEVAL_TELEMETRY_OPT_OUT=1 # both send usage data by defaultexport OPENAI_API_KEY=... # the default judge model in both frameworks' examples
Step 1: five retrieval metrics
Three questions, each with the top 3 retrieved chunk ids and graded relevance: 2 means the chunk answers the question, 1 means it helps. The third question's chunk was never retrieved.
import mathdef hit_rate(ret, rel, k): return float(any(d in rel for d in ret[:k]))def precision(ret, rel, k): return sum(d in rel for d in ret[:k]) / kdef recall(ret, rel, k): return sum(d in rel for d in ret[:k]) / len(rel)def rr(ret, rel): return next((1 / r for r, d in enumerate(ret, 1) if d in rel), 0.0)def ndcg(ret, grades, k):dcg = sum(grades.get(d, 0) / math.log2(i + 1) for i, d in enumerate(ret[:k], 1))ideal = sorted(grades.values(), reverse=True)[:k]return dcg / sum(g / math.log2(i + 1) for i, g in enumerate(ideal, 1))RUNS = [(["travel-meals", "travel-hotels", "travel-tiers"], {"travel-hotels": 2, "travel-tiers": 1}),(["travel-flights", "travel-meals", "hr-pto"], {"travel-flights": 1}),(["travel-hotels", "it-vpn", "hr-pto"], {"travel-meals": 1})]
hit@3 [1.0, 1.0, 0.0] mean=0.67precision@3 [0.67, 0.33, 0.0] mean=0.33recall@3 [1.0, 1.0, 0.0] mean=0.67mrr [0.5, 1.0, 0.0] mean=0.50ndcg@3 [0.67, 1.0, 0.0] mean=0.56
Each metric asks a different question. Hit rate asks whether anything relevant appeared. Precision@k asks how much of the top k is relevant, which measures noise. Recall@k asks how much of what's relevant appeared. MRR asks how high the first relevant chunk sits.
nDCG@k (normalised discounted cumulative gain) uses the grades. It rewards the best chunks for being first, and discounts each position by log2(rank + 1). Question 1 scores 0.67, not 1.0, because the best chunk sits second behind a useless one. Some sources use 2^grade − 1 as the gain, so say which version you report.
import asynciofrom ragas import SingleTurnSamplefrom ragas.metrics import IDBasedContextPrecision, IDBasedContextRecalls = SingleTurnSample(retrieved_context_ids=["travel-meals", "travel-hotels", "travel-tiers"],reference_context_ids=["travel-hotels", "travel-tiers"])print(asyncio.run(IDBasedContextPrecision().single_turn_ascore(s)),asyncio.run(IDBasedContextRecall().single_turn_ascore(s)))
Step 3: the four core Ragas metrics
Ragas 0.4 moved its metrics to ragas.metrics.collections. They take keyword arguments and return a result with .value and .reason. Many tutorials still show the older evaluate() API, which now warns that it's deprecated.
from openai import AsyncOpenAIfrom ragas.llms import llm_factoryfrom ragas.embeddings import embedding_factoryfrom ragas.metrics.collections import Faithfulness, AnswerRelevancy, ContextPrecision, ContextRecallQ, ANS = "How much can I spend on a hotel in Denver?", "Up to $250 per night."CTX = ["Hotels: up to $250 per night in Tier 1 cities (New York, San Francisco, Chicago, Boston) and $175 elsewhere."]REF = "$175 per night, since Denver is not a Tier 1 city."client = AsyncOpenAI()llm, emb = llm_factory("gpt-4o-mini", client=client), embedding_factory("openai", model="text-embedding-3-small", client=client)print(Faithfulness(llm=llm).score(user_input=Q, response=ANS, retrieved_contexts=CTX).value,AnswerRelevancy(llm=llm, embeddings=emb).score(user_input=Q, response=ANS).value,ContextPrecision(llm=llm).score(user_input=Q, reference=REF, retrieved_contexts=CTX).value,ContextRecall(llm=llm).score(user_input=Q, retrieved_contexts=CTX, reference=REF).value)
That pattern is the hook's bug, now visible. Retrieval was fine; generation misread the chunk. Here's what each core metric measures:
- Faithfulness checks whether every claim in the answer is supported by the retrieved context. Low means the model invented or misread facts.
- Answer relevancy checks whether the answer addresses the question. Low means it drifted or answered something else.
- Context precision checks whether the relevant chunks rank above the irrelevant ones. Low means the right facts are buried in noise.
- Context recall checks whether the context holds everything the reference answer needs. Low means retrieval missed facts.
Precision and recall need a reference answer written by a person. Faithfulness and answer relevancy don't, so they can also run on live traffic.
Step 4: a synthetic test set, and its limits
from langchain_core.documents import Documentfrom langchain_openai import ChatOpenAI, OpenAIEmbeddingsfrom ragas.testset import TestsetGeneratordocs = [Document(page_content=open("travel_policy.md").read(), metadata={"source": "travel_policy.md"})]gen = TestsetGenerator.from_langchain(ChatOpenAI(model="gpt-4o-mini"), OpenAIEmbeddings())testset = gen.generate_with_langchain_docs(docs, testset_size=10)print(testset.to_pandas().columns.tolist())
A synthetic test set is a model writing questions and reference answers from your documents. It's a fast way to reach 50 questions before launch. But generated questions sound like the documents, not like your employees. Read them, delete the odd ones, and replace them with real questions as traffic arrives.
Step 5: tests that fail the build, with DeepEval
DeepEval wraps metrics in pytest-style tests. A failing score fails the test, so a bad change can't merge. Start with a cheap check that needs no model: does the answer contain the right amount?
import pytestfrom deepeval import assert_testfrom deepeval.test_case import LLMTestCasefrom deepeval.metrics import PatternMatchMetriccases = [LLMTestCase(input="Hotel limit in Chicago?", actual_output="Up to $250 per night.",expected_output="$250 per night, since Chicago is a Tier 1 city.", retrieval_context=CTX),LLMTestCase(input="Hotel limit in Denver?", actual_output="Up to $250 per night.",expected_output="$175 per night, since Denver is not a Tier 1 city.", retrieval_context=CTX)]@pytest.mark.parametrize("test_case", cases)def test_amount(test_case):amount = test_case.expected_output.split()[0].replace("$", r"\$")assert_test(test_case, [PatternMatchMetric(pattern=rf".*{amount}.*")])
$ deepeval test run test_policy.pyPass Rate: 50.0% | Passed: 1 | Failed: 1FAILED test_amount[test_case1] - Pattern Match (score: 0.0, threshold: 1.0 ...)
Step 6: DeepEval's RAG metrics
from deepeval.metrics import (FaithfulnessMetric, AnswerRelevancyMetric, ContextualPrecisionMetric,ContextualRecallMetric, ContextualRelevancyMetric)rag_metrics = [FaithfulnessMetric(threshold=0.7), AnswerRelevancyMetric(threshold=0.7),ContextualPrecisionMetric(), ContextualRecallMetric(), ContextualRelevancyMetric()]@pytest.mark.parametrize("test_case", cases)def test_rag(test_case):assert_test(test_case, rag_metrics)
DeepEval's metrics mirror Ragas's, with one extra: contextual relevancy, the share of retrieved text that's relevant at all. Its docs recommend deepeval test run in CI over plain pytest, and offer flaky=True for judged tests that occasionally wobble. Each judged metric is a model call per test case. Run the cheap pattern checks on every commit, and the judged suite nightly or before a release.
Match
Match each low score to what it tells you
Pick an item on the left, then its match on the right.
Worth keeping
- Retrieval metrics are free and exact; run them on every change.
- Faithfulness and answer relevancy need no reference, so they also run on live traffic.
- Context precision and recall need reference answers, and split ranking noise from missing facts.
- Cheap checks on every commit, judged suites nightly; pin framework versions.
Metrics tell you that something is wrong. They don't tell you which stage broke, or why. Next lesson is a debugging playbook that turns one bad answer into a located, fixed stage.
Leaderboard
Loading…
50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.
Sources for this lesson (7)
Each fact in the lesson is marked with the number of the page it comes from.