Black Friday50% off your first month on any monthly plan with EARLY50 See plans
All 29 lessons
  1. Foundations: what RAG is and a working pipeline

    1. 1. Getting started: what RAG is and when to use it
    2. 2. Types of RAG: a map from naive to agentic
    3. 3. Lab: build the smallest RAG pipeline end to end
    4. 4. How embeddings and vector search find meaning
    5. 5. Document types: what each one breaks, and how to prepare it
  2. Better retrieval: chunking and search

    1. 6. Chunking documents so answers survive
    2. 7. Chunking strategies compared: which to use when
    3. 8. Keyword search vs vector search, and fusing both
    4. 9. Retrieval algorithms compared: which to use when
    5. 10. Reranking the top results before the model reads them
    6. 11. Rewriting the question before you search
  3. Answers you can trust, measure and debug

    1. 12. Lab: grounded answers with citations and refusals
    2. 13. Measuring a RAG system with an evaluation set
    3. 14. Lab: evaluation methods and frameworks: Ragas and DeepEval
    4. 15. Troubleshooting a RAG system, symptom by symptom
  4. Production in a company

    1. 16. Access control: only retrieve what the user may see
    2. 17. Defending against prompt injection in retrieved text
    3. 18. Running RAG in production: freshness, cost and monitoring
    4. 19. Real-world RAG: what worked, what failed, and why
  5. Agentic RAG

    1. 20. Lab: from a fixed pipeline to an agent with a search tool
    2. 21. Self-correcting retrieval: grade, retry, fall back
    3. 22. GraphRAG and routing across many sources
    4. 23. Lab: numbers, tables and financial documents
  6. Frameworks: LangChain, LangGraph and LangSmith

    1. 24. The RAG toolbox: what each library does and why you import it
    2. 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
    3. 26. Lab: Ask Brightline in LangChain
    4. 27. Lab: agentic RAG as a LangGraph graph
    5. 28. Lab: tracing, evaluating and monitoring with LangSmith
    6. 29. Wrap-up: your production agentic RAG checklist
See the leaderboard

18 min

Troubleshooting a RAG system, symptom by symptom

Trace a bad answer to the stage that caused it and apply the fix, using a repeatable debugging playbook.

By the end of this lesson you can

  • Follow a five-step playbook from a bad answer to a fix
  • Locate the failing stage with one diagnostic function
  • Match common production symptoms to their likely causes

From earlier lessons · 1 of 2

Context recall is high but context precision is low. What does that mean?

Pick one answer.

From earlier lessons · 2 of 2

Why check an LLM judge against your own labels?

Pick one answer.

An employee reports: "Ask Brightline told me I can't work from home on Fridays." The policy says up to three days a week. You have the question, the bad answer, and a trace. Where do you start?

The playbook

Most RAG debugging goes wrong in one way: people change the prompt first, because it's easiest to edit. Instead, use the same five steps every time:

  1. Reproduce the exact question, with the same user permissions.
  2. Read the trace: rewritten query, retrieved ids and scores, reranked ids, prompt and answer.
  3. Locate the stage. A document can be missing from the index (ingestion), indexed but not found (retrieval), found but cut (ranking), or present but misused (generation).
  4. Fix one thing at that stage.
  5. Re-run the evaluation set, then add this question to it.

Step 3 is the one people skip. This function does it for you, with lab 1's retrieve() and lesson 10's rerank().

python
def diagnose(question, expected_id, keep=3):
if not any(d["id"] == expected_id for d in DOCS):
return "ingestion: the document isn't in the index"
ranked = [d["id"] for d, _ in rerank(question, retrieve(question, k=20), keep=20)]
if expected_id not in ranked:
return "retrieval: indexed, but not in the top 20"
rank = ranked.index(expected_id) + 1
if rank > keep:
return f"ranking: found at rank {rank}, cut by keep={keep}"
return "generation: the chunk reached the prompt; check prompt, order and answer"

Round 1: the Friday question

python
print(diagnose("Can I stay home on Fridays?", "hr-remote"))
Real output: ranking: found at rank 8, cut by keep=3. After reranking, the remote-work rule sits eighth, as lesson 13's evaluation hinted.

Issue: the reranker cut the right chunk

The chunk was retrieved. Then the reranker preferred two travel chunks, and keep=3 dropped the remote-work rule. With no rule in front of it, the model answered from general knowledge. That's why the answer was confidently wrong rather than a refusal.

The fix belongs in ranking, not in the prompt. Raise keep to 5, or rewrite the query as lesson 11 did ("work from home on Fridays"). Then re-run the evaluation set to check the hotel approval question didn't break.

Round 2: the parking question

python
print(diagnose("Is there free parking at the Denver office?", "facilities-parking"))
Expected output: ingestion: the document isn't in the index.

Issue: the document never arrived

Facilities says a parking policy exists. It's a scanned PDF, and the parser extracted no text, so it was never chunked. No retrieval setting can find text that isn't there. Check ingestion logs for documents that produced zero chunks, and run text extraction or OCR on scans. Until then, the correct answer is the refusal from lesson 12.

Round 3: right chunk, wrong answer

python
print(diagnose("Who has to approve a pricier hotel?", "travel-hotel"))
Expected output: generation: the chunk reached the prompt. Reranked, travel-hotel is first.

Issue: the model misread good context

Suppose the answer said "Finance approves it." The hotel rule says a director approves, but the fin-approval chunk mentioning finance was also in the prompt. The model blended two passages.

Try generation fixes in order of cost. Put the best chunk first and keep fewer chunks. Tighten the rules ("answer from the most specific passage"). Run lesson 13's faithfulness judge to catch what slips through. Swap the model only after those fail.

Symptoms beyond one question

Some failures only show across many requests. Each points to a stage:

  • Old rules quoted after an update. Ingestion isn't re-indexing changes or deletes (lesson 18).
  • Users see documents they shouldn't. Permissions are missing from chunks or stale (lesson 16).
  • Refusal rate jumps overnight. A source connector stopped syncing, or a parser broke.
  • Outside links or odd instructions appear. Injected text in an indexed page (lesson 17).
  • Latency and cost spike. An agent loop, rewrite or judge now runs on every question (lesson 20).
  • Good evaluation scores, bad user reports. The evaluation set no longer looks like real questions, so refresh it from recent traffic.

Match

Match each symptom to the first thing to check

Pick an item on the left, then its match on the right.

Worth keeping

  1. Locate the failing stage before changing anything.
  2. Missing, not found, cut and misused each have different fixes.
  3. A confident wrong answer usually means the right chunk never reached the prompt.
  4. Every debugged question becomes a new evaluation case.

One question no evaluation set contains: an intern asking for the salary bands of their own team. Ask Brightline retrieves the HR compensation file and answers perfectly. Every metric says it did well. Next lesson explains why that's a disaster.

Leaderboard

Loading…

50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.