All 29 lessons
Foundations: what RAG is and a working pipeline
Better retrieval: chunking and search
Answers you can trust, measure and debug
Production in a company
Agentic RAG
Frameworks: LangChain, LangGraph and LangSmith
- 24. The RAG toolbox: what each library does and why you import it
- 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
- 26. Lab: Ask Brightline in LangChain
- 27. Lab: agentic RAG as a LangGraph graph
- 28. Lab: tracing, evaluating and monitoring with LangSmith
- 29. Wrap-up: your production agentic RAG checklist
18 min
Troubleshooting a RAG system, symptom by symptom
Trace a bad answer to the stage that caused it and apply the fix, using a repeatable debugging playbook.
By the end of this lesson you can
- Follow a five-step playbook from a bad answer to a fix
- Locate the failing stage with one diagnostic function
- Match common production symptoms to their likely causes
From earlier lessons · 1 of 2
Context recall is high but context precision is low. What does that mean?
Pick one answer.
From earlier lessons · 2 of 2
Why check an LLM judge against your own labels?
Pick one answer.
An employee reports: "Ask Brightline told me I can't work from home on Fridays." The policy says up to three days a week. You have the question, the bad answer, and a trace. Where do you start?
The playbook
Most RAG debugging goes wrong in one way: people change the prompt first, because it's easiest to edit. Instead, use the same five steps every time:
- Reproduce the exact question, with the same user permissions.
- Read the trace: rewritten query, retrieved ids and scores, reranked ids, prompt and answer.
- Locate the stage. A document can be missing from the index (ingestion), indexed but not found (retrieval), found but cut (ranking), or present but misused (generation).
- Fix one thing at that stage.
- Re-run the evaluation set, then add this question to it.
Step 3 is the one people skip. This function does it for you, with lab 1's retrieve() and lesson 10's rerank().
def diagnose(question, expected_id, keep=3):if not any(d["id"] == expected_id for d in DOCS):return "ingestion: the document isn't in the index"ranked = [d["id"] for d, _ in rerank(question, retrieve(question, k=20), keep=20)]if expected_id not in ranked:return "retrieval: indexed, but not in the top 20"rank = ranked.index(expected_id) + 1if rank > keep:return f"ranking: found at rank {rank}, cut by keep={keep}"return "generation: the chunk reached the prompt; check prompt, order and answer"
Round 1: the Friday question
print(diagnose("Can I stay home on Fridays?", "hr-remote"))
Issue: the reranker cut the right chunk
The chunk was retrieved. Then the reranker preferred two travel chunks, and keep=3 dropped the remote-work rule. With no rule in front of it, the model answered from general knowledge. That's why the answer was confidently wrong rather than a refusal.
The fix belongs in ranking, not in the prompt. Raise keep to 5, or rewrite the query as lesson 11 did ("work from home on Fridays"). Then re-run the evaluation set to check the hotel approval question didn't break.
Round 2: the parking question
print(diagnose("Is there free parking at the Denver office?", "facilities-parking"))
Issue: the document never arrived
Facilities says a parking policy exists. It's a scanned PDF, and the parser extracted no text, so it was never chunked. No retrieval setting can find text that isn't there. Check ingestion logs for documents that produced zero chunks, and run text extraction or OCR on scans. Until then, the correct answer is the refusal from lesson 12.
Round 3: right chunk, wrong answer
print(diagnose("Who has to approve a pricier hotel?", "travel-hotel"))
Issue: the model misread good context
Suppose the answer said "Finance approves it." The hotel rule says a director approves, but the fin-approval chunk mentioning finance was also in the prompt. The model blended two passages.
Try generation fixes in order of cost. Put the best chunk first and keep fewer chunks. Tighten the rules ("answer from the most specific passage"). Run lesson 13's faithfulness judge to catch what slips through. Swap the model only after those fail.
Symptoms beyond one question
Some failures only show across many requests. Each points to a stage:
- Old rules quoted after an update. Ingestion isn't re-indexing changes or deletes (lesson 18).
- Users see documents they shouldn't. Permissions are missing from chunks or stale (lesson 16).
- Refusal rate jumps overnight. A source connector stopped syncing, or a parser broke.
- Outside links or odd instructions appear. Injected text in an indexed page (lesson 17).
- Latency and cost spike. An agent loop, rewrite or judge now runs on every question (lesson 20).
- Good evaluation scores, bad user reports. The evaluation set no longer looks like real questions, so refresh it from recent traffic.
Match
Match each symptom to the first thing to check
Pick an item on the left, then its match on the right.
Worth keeping
- Locate the failing stage before changing anything.
- Missing, not found, cut and misused each have different fixes.
- A confident wrong answer usually means the right chunk never reached the prompt.
- Every debugged question becomes a new evaluation case.
One question no evaluation set contains: an intern asking for the salary bands of their own team. Ask Brightline retrieves the HR compensation file and answers perfectly. Every metric says it did well. Next lesson explains why that's a disaster.
Leaderboard
Loading…
50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.