All 29 lessons
Foundations: what RAG is and a working pipeline
Better retrieval: chunking and search
Answers you can trust, measure and debug
Production in a company
Agentic RAG
Frameworks: LangChain, LangGraph and LangSmith
- 24. The RAG toolbox: what each library does and why you import it
- 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
- 26. Lab: Ask Brightline in LangChain
- 27. Lab: agentic RAG as a LangGraph graph
- 28. Lab: tracing, evaluating and monitoring with LangSmith
- 29. Wrap-up: your production agentic RAG checklist
18 min
Self-correcting retrieval: grade, retry, fall back
Add a grader that checks retrieved chunks and the draft answer, then rewrites, retries or escalates.
By the end of this lesson you can
- Grade retrieved chunks before the model answers from them
- Retry with a rewritten query, then fall back or escalate
- Check the final answer against the passages it used
From earlier lessons · 1 of 2
What ends the agent loop when the model keeps searching?
Pick one answer.
From earlier lessons · 2 of 2
When is the fixed pipeline better than the agent?
Pick one answer.
An employee asks whether they can bring their dog to the Denver office. Search returns the Denver office page: opening hours and visitor sign-in. The agent answers "Yes, sign your dog in at reception (office-denver)." Cited, confident and invented.
Where we stand: trusting every result
The lesson 20 agent treats whatever search returns as good evidence. Search always returns something: the three closest chunks, however far they are. On a question the documents don't answer, the closest chunks are near-misses, and the model stitches them into an answer.
We need a step between retrieval and answering that asks: do these passages actually answer the question? The Corrective RAG paper (Yan et al., 2024) adds exactly that. A lightweight evaluator rates retrieval quality, and its confidence decides what happens next: use the results, correct them, or search elsewhere.
Layer 1: grade what was retrieved
A retrieval grader is a cheap model call that labels each passage as relevant, partial or irrelevant to the question. Only relevant passages go forward. parse_reply is from lesson 12 and llm from rag.py.
def grade(question, hits):listing = "\n".join(f"{d['id']}: {d['text']}" for d, _ in hits)prompt = (f"Question: {question}\n\nPassages:\n{listing}\n\n""For each passage, does it help answer the question? Reply with JSON only, "'mapping id to "relevant", "partial" or "irrelevant".')return parse_reply(llm(prompt))q = "Can I bring my dog to the Denver office?"print(grade(q, rerank(q, retrieve(q, k=10), keep=3)))
Layer 2: retry, then give up honestly
If nothing is relevant, rewrite the query once with HyDE from lesson 11 and search again. If that fails too, return a clear marker so the agent says it doesn't know.
SEEN = [] # every passage the agent was allowed to use this turndef corrective_search(query, tries=2):for _ in range(tries):hits = rerank(query, retrieve(query, k=10), keep=3)labels = grade(query, hits)good = [(d, s) for d, s in hits if labels.get(d["id"]) == "relevant"]if good:SEEN.extend(good)return "\n".join(f"({d['id']}) {d['text']}" for d, _ in good)query = hyde(query)return "NO_RELEVANT_PASSAGES: tell the user this isn't covered."search_policies = corrective_search # lesson 20's run_tools() now calls this
The CRAG paper falls back to web search when the corpus fails. For a company assistant, that's usually wrong. A Brightline policy question must be answered from Brightline's documents, not from another company's rules on the internet. The right fallback is honesty and a person.
Layer 3: check the answer, then escalate
Grading the passages doesn't guarantee the answer uses them faithfully. The Self-RAG paper (Asai et al., 2023) trains one model to decide when to retrieve and to critique its own output with special reflection tokens. We get much of the benefit with a prompted check. Before showing an answer, run the faithfulness judge from lesson 13 on it.
def escalate(question):print("[stub] HR help desk ticket:", question) # a real system asks the user firstreturn "I couldn't confirm this in Brightline's policies, so I've sent it to the HR help desk."def answer_or_escalate(question):SEEN.clear()answer = agent(question)if SEEN and faithful(answer, SEEN)["supported"]:return answerreturn escalate(question)print(answer_or_escalate("Can I bring my dog to the Denver office?"))
Re-run the earlier cases
print(answer_or_escalate("Who has to approve a pricier hotel?"))
Each layer costs calls. A grade per search, a HyDE rewrite on a miss and a final judge can turn a 4-call agent answer into 6-8 calls. Use a small, fast model for grading and judging, and measure what the layers buy with your evaluation set. Keep them where wrong answers are expensive, such as HR, finance and security.
Match
Match each situation to the system's move
Pick an item on the left, then its match on the right.
Worth keeping
- Search always returns something, so grade results before answering from them.
- On a miss, rewrite once, then admit it and offer a person.
- For company policy, never fall back to the open web.
- Check the final answer against the passages it used before showing it.
Some questions can't be answered from any three passages, however well graded. "What are the most common reasons employees contact IT?" needs the whole ticket history. Next lesson: when retrieval isn't the right tool at all.
Leaderboard
Loading…
50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.
Sources for this lesson (2)
Each fact in the lesson is marked with the number of the page it comes from.