All 29 lessons
Foundations: what RAG is and a working pipeline
Better retrieval: chunking and search
Answers you can trust, measure and debug
Production in a company
Agentic RAG
Frameworks: LangChain, LangGraph and LangSmith
- 24. The RAG toolbox: what each library does and why you import it
- 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
- 26. Lab: Ask Brightline in LangChain
- 27. Lab: agentic RAG as a LangGraph graph
- 28. Lab: tracing, evaluating and monitoring with LangSmith
- 29. Wrap-up: your production agentic RAG checklist
18 min
Lab: grounded answers with citations and refusals
Write the generation step so answers cite their sources and say 'I don't know' when the context is silent.
By the end of this lesson you can
- Write a generation prompt that asks for cited answers and allows refusals
- Parse the model's structured reply safely
- Check citations in code instead of trusting the model
From earlier lessons · 1 of 2
When should Ask Brightline rewrite a question before searching?
Pick one answer.
From earlier lessons · 2 of 2
Which words go into the final answer prompt after a rewrite?
Pick one answer.
Ask Brightline is asked whether the Denver office has free parking. No policy mentions parking. It answers anyway: "Yes, free parking is available in Lot B [2]." Passage 2 is the laptop security rule.
Lab: make the answer prove itself
You'll rebuild the generation step so every answer either cites the passages it used or says it can't answer. Then you'll check those citations in code. retrieve() is lab 1's function; llm() comes from rag.py.
import json, refrom rag import llmSYSTEM = "You answer Brightline employees' policy questions. You are careful and brief."
Step 1: a structured prompt
The prompt has four parts: who the model is (the system prompt), the numbered context, the question, then the rules. In a long prompt, put the rules and question after the context. Many providers' prompting guides recommend this, because the end of a long prompt gets the most attention, as lesson 10 showed.
Asking for JSON gives your code something to check.
RULES = """Rules:- Answer only from the numbered context, never from outside knowledge.- After each claim, cite its passage like [2].- If the context does not answer the question, set "answer" to null.Reply with JSON only: {"answer": string or null, "citations": [numbers]}"""def grounded_prompt(question, hits):context = "\n".join(f"[{n}] ({d['id']}) {d['text']}" for n, (d, _) in enumerate(hits, 1))return f"<context>\n{context}\n</context>\n\nQuestion: {question}\n\n{RULES}"
Step 2: parse the reply and ask
Models sometimes wrap JSON in a sentence or a code fence. Find the outermost braces before parsing, instead of failing on the first stray word.
def parse_reply(text):start, end = text.find("{"), text.rfind("}")return json.loads(text[start:end + 1])def ask(question, k=3):hits = retrieve(question, k)reply = parse_reply(llm(grounded_prompt(question, hits), system=SYSTEM))return reply, hitsreply, hits = ask("How many unused PTO days carry over?")print(reply)
Step 3: an honest refusal
reply, hits = ask("Is there free parking at the Denver office?")print(reply)if reply["answer"] is None:print("I couldn't find this in Brightline's policies. Try the HR portal.")
A refusal is a correct answer when the documents are silent. Your code, not the model, decides what the employee sees then: a fixed message and a place to ask. That keeps refusals consistent and easy to count, which lesson 13 needs.
Issue: a citation that points nowhere
Strict rules make made-up answers rarer, but not impossible. Here's the kind of reply they still allow, on a question about London meal allowances. We wrote it by hand so you can test the check.
bad = {"answer": "Meals are $75 per day [1], and London has a higher $110 allowance [4].","citations": [1]}
The $110 figure came from the model's memory, not from Brightline. The answer looks grounded because it has brackets. The fix is to check citations in code: they must exist, match the list, and every answer must have at least one.
def citation_problems(reply, hits):in_text = {int(n) for n in re.findall(r"\[(\d+)\]", reply["answer"] or "")}problems = []if not in_text <= set(range(1, len(hits) + 1)):problems.append("cites a passage that wasn't retrieved")if reply["answer"] and not in_text:problems.append("answer has no citations")if in_text != set(reply["citations"]):problems.append("citation list doesn't match the text")return problemsprint(citation_problems(bad, hits))
When the check fails, don't show the answer. Retry once with the problems added to the prompt, or fall back to the refusal message. This check catches citations that are missing or invented. It can't tell whether passage 1 really supports the claim; lesson 21 adds a grader for that.
This lab makes about five model calls.
Put in order
Put the generation step in order
Drag, or focus an item and press Space, move with the arrow keys, then Space to drop.
- Build the prompt: context, question, rules
- Retrieve and number the passages
- Check citations in code
- Call the model and parse its JSON
- Show the cited answer or the fallback message
Worth keeping
- Ask for structured output so your code can check the answer.
- A refusal is a correct answer when the documents are silent.
- Verify citations in code: they must exist and match the text.
- Your code decides what the user sees, never the raw model reply.
You've changed the chunker, the search, the reranker, the query and the prompt. Is Ask Brightline better than it was in lesson 3, or does it just feel better? Next lesson turns that feeling into a number.
Leaderboard
Loading…
50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.