Black Friday50% off your first month on any monthly plan with EARLY50 See plans
All 29 lessons
  1. Foundations: what RAG is and a working pipeline

    1. 1. Getting started: what RAG is and when to use it
    2. 2. Types of RAG: a map from naive to agentic
    3. 3. Lab: build the smallest RAG pipeline end to end
    4. 4. How embeddings and vector search find meaning
    5. 5. Document types: what each one breaks, and how to prepare it
  2. Better retrieval: chunking and search

    1. 6. Chunking documents so answers survive
    2. 7. Chunking strategies compared: which to use when
    3. 8. Keyword search vs vector search, and fusing both
    4. 9. Retrieval algorithms compared: which to use when
    5. 10. Reranking the top results before the model reads them
    6. 11. Rewriting the question before you search
  3. Answers you can trust, measure and debug

    1. 12. Lab: grounded answers with citations and refusals
    2. 13. Measuring a RAG system with an evaluation set
    3. 14. Lab: evaluation methods and frameworks: Ragas and DeepEval
    4. 15. Troubleshooting a RAG system, symptom by symptom
  4. Production in a company

    1. 16. Access control: only retrieve what the user may see
    2. 17. Defending against prompt injection in retrieved text
    3. 18. Running RAG in production: freshness, cost and monitoring
    4. 19. Real-world RAG: what worked, what failed, and why
  5. Agentic RAG

    1. 20. Lab: from a fixed pipeline to an agent with a search tool
    2. 21. Self-correcting retrieval: grade, retry, fall back
    3. 22. GraphRAG and routing across many sources
    4. 23. Lab: numbers, tables and financial documents
  6. Frameworks: LangChain, LangGraph and LangSmith

    1. 24. The RAG toolbox: what each library does and why you import it
    2. 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
    3. 26. Lab: Ask Brightline in LangChain
    4. 27. Lab: agentic RAG as a LangGraph graph
    5. 28. Lab: tracing, evaluating and monitoring with LangSmith
    6. 29. Wrap-up: your production agentic RAG checklist
See the leaderboard

18 min

Self-correcting retrieval: grade, retry, fall back

Add a grader that checks retrieved chunks and the draft answer, then rewrites, retries or escalates.

By the end of this lesson you can

  • Grade retrieved chunks before the model answers from them
  • Retry with a rewritten query, then fall back or escalate
  • Check the final answer against the passages it used

From earlier lessons · 1 of 2

What ends the agent loop when the model keeps searching?

Pick one answer.

From earlier lessons · 2 of 2

When is the fixed pipeline better than the agent?

Pick one answer.

An employee asks whether they can bring their dog to the Denver office. Search returns the Denver office page: opening hours and visitor sign-in. The agent answers "Yes, sign your dog in at reception (office-denver)." Cited, confident and invented.

Where we stand: trusting every result

The lesson 20 agent treats whatever search returns as good evidence. Search always returns something: the three closest chunks, however far they are. On a question the documents don't answer, the closest chunks are near-misses, and the model stitches them into an answer.

We need a step between retrieval and answering that asks: do these passages actually answer the question? The Corrective RAG paper (Yan et al., 2024) adds exactly that. A lightweight evaluator rates retrieval quality, and its confidence decides what happens next: use the results, correct them, or search elsewhere.

[7]

Rendering diagram…
Two checks: one on what was retrieved, one on what was written. Both can send the question to a person.

Layer 1: grade what was retrieved

A retrieval grader is a cheap model call that labels each passage as relevant, partial or irrelevant to the question. Only relevant passages go forward. parse_reply is from lesson 12 and llm from rag.py.

python
def grade(question, hits):
listing = "\n".join(f"{d['id']}: {d['text']}" for d, _ in hits)
prompt = (f"Question: {question}\n\nPassages:\n{listing}\n\n"
"For each passage, does it help answer the question? Reply with JSON only, "
'mapping id to "relevant", "partial" or "irrelevant".')
return parse_reply(llm(prompt))
q = "Can I bring my dog to the Denver office?"
print(grade(q, rerank(q, retrieve(q, k=10), keep=3)))
Expected output: {'office-denver': 'partial', 'office-boston': 'irrelevant', ...}. The office page is about Denver, not dogs. Your labels may differ.

Layer 2: retry, then give up honestly

If nothing is relevant, rewrite the query once with HyDE from lesson 11 and search again. If that fails too, return a clear marker so the agent says it doesn't know.

python
SEEN = [] # every passage the agent was allowed to use this turn
def corrective_search(query, tries=2):
for _ in range(tries):
hits = rerank(query, retrieve(query, k=10), keep=3)
labels = grade(query, hits)
good = [(d, s) for d, s in hits if labels.get(d["id"]) == "relevant"]
if good:
SEEN.extend(good)
return "\n".join(f"({d['id']}) {d['text']}" for d, _ in good)
query = hyde(query)
return "NO_RELEVANT_PASSAGES: tell the user this isn't covered."
search_policies = corrective_search # lesson 20's run_tools() now calls this
hyde() is from lesson 11. Swapping the function keeps lesson 20's tool definition and loop unchanged.

The CRAG paper falls back to web search when the corpus fails. For a company assistant, that's usually wrong. A Brightline policy question must be answered from Brightline's documents, not from another company's rules on the internet. The right fallback is honesty and a person.

Layer 3: check the answer, then escalate

Grading the passages doesn't guarantee the answer uses them faithfully. The Self-RAG paper (Asai et al., 2023) trains one model to decide when to retrieve and to critique its own output with special reflection tokens. We get much of the benefit with a prompted check. Before showing an answer, run the faithfulness judge from lesson 13 on it.

[6]

python
def escalate(question):
print("[stub] HR help desk ticket:", question) # a real system asks the user first
return "I couldn't confirm this in Brightline's policies, so I've sent it to the HR help desk."
def answer_or_escalate(question):
SEEN.clear()
answer = agent(question)
if SEEN and faithful(answer, SEEN)["supported"]:
return answer
return escalate(question)
print(answer_or_escalate("Can I bring my dog to the Denver office?"))
Expected output: the stub ticket line, then the escalation message. faithful() is from lesson 13, agent() from lesson 20.

Re-run the earlier cases

python
print(answer_or_escalate("Who has to approve a pricier hotel?"))
Expected output: a director's written approval is needed above the nightly limit (travel-hotel). Answered questions still pass both checks.

Each layer costs calls. A grade per search, a HyDE rewrite on a miss and a final judge can turn a 4-call agent answer into 6-8 calls. Use a small, fast model for grading and judging, and measure what the layers buy with your evaluation set. Keep them where wrong answers are expensive, such as HR, finance and security.

Match

Match each situation to the system's move

Pick an item on the left, then its match on the right.

Worth keeping

  1. Search always returns something, so grade results before answering from them.
  2. On a miss, rewrite once, then admit it and offer a person.
  3. For company policy, never fall back to the open web.
  4. Check the final answer against the passages it used before showing it.

Some questions can't be answered from any three passages, however well graded. "What are the most common reasons employees contact IT?" needs the whole ticket history. Next lesson: when retrieval isn't the right tool at all.

Leaderboard

Loading…

50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.

Sources for this lesson (2)

Each fact in the lesson is marked with the number of the page it comes from.

  1. [6]Asai et al. (2023), Self-RAG
  2. [7]Yan et al. (2024), Corrective Retrieval Augmented Generation