Black Friday50% off your first month on any monthly plan with EARLY50 See plans
All 29 lessons
  1. Foundations: what RAG is and a working pipeline

    1. 1. Getting started: what RAG is and when to use it
    2. 2. Types of RAG: a map from naive to agentic
    3. 3. Lab: build the smallest RAG pipeline end to end
    4. 4. How embeddings and vector search find meaning
    5. 5. Document types: what each one breaks, and how to prepare it
  2. Better retrieval: chunking and search

    1. 6. Chunking documents so answers survive
    2. 7. Chunking strategies compared: which to use when
    3. 8. Keyword search vs vector search, and fusing both
    4. 9. Retrieval algorithms compared: which to use when
    5. 10. Reranking the top results before the model reads them
    6. 11. Rewriting the question before you search
  3. Answers you can trust, measure and debug

    1. 12. Lab: grounded answers with citations and refusals
    2. 13. Measuring a RAG system with an evaluation set
    3. 14. Lab: evaluation methods and frameworks: Ragas and DeepEval
    4. 15. Troubleshooting a RAG system, symptom by symptom
  4. Production in a company

    1. 16. Access control: only retrieve what the user may see
    2. 17. Defending against prompt injection in retrieved text
    3. 18. Running RAG in production: freshness, cost and monitoring
    4. 19. Real-world RAG: what worked, what failed, and why
  5. Agentic RAG

    1. 20. Lab: from a fixed pipeline to an agent with a search tool
    2. 21. Self-correcting retrieval: grade, retry, fall back
    3. 22. GraphRAG and routing across many sources
    4. 23. Lab: numbers, tables and financial documents
  6. Frameworks: LangChain, LangGraph and LangSmith

    1. 24. The RAG toolbox: what each library does and why you import it
    2. 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
    3. 26. Lab: Ask Brightline in LangChain
    4. 27. Lab: agentic RAG as a LangGraph graph
    5. 28. Lab: tracing, evaluating and monitoring with LangSmith
    6. 29. Wrap-up: your production agentic RAG checklist
See the leaderboard

18 min

Lab: grounded answers with citations and refusals

Write the generation step so answers cite their sources and say 'I don't know' when the context is silent.

By the end of this lesson you can

  • Write a generation prompt that asks for cited answers and allows refusals
  • Parse the model's structured reply safely
  • Check citations in code instead of trusting the model

From earlier lessons · 1 of 2

When should Ask Brightline rewrite a question before searching?

Pick one answer.

From earlier lessons · 2 of 2

Which words go into the final answer prompt after a rewrite?

Pick one answer.

Ask Brightline is asked whether the Denver office has free parking. No policy mentions parking. It answers anyway: "Yes, free parking is available in Lot B [2]." Passage 2 is the laptop security rule.

Lab: make the answer prove itself

You'll rebuild the generation step so every answer either cites the passages it used or says it can't answer. Then you'll check those citations in code. retrieve() is lab 1's function; llm() comes from rag.py.

python
import json, re
from rag import llm
SYSTEM = "You answer Brightline employees' policy questions. You are careful and brief."
Setup. Outputs vary from run to run, so your wording will differ from the examples.

Step 1: a structured prompt

The prompt has four parts: who the model is (the system prompt), the numbered context, the question, then the rules. In a long prompt, put the rules and question after the context. Many providers' prompting guides recommend this, because the end of a long prompt gets the most attention, as lesson 10 showed.

Asking for JSON gives your code something to check.

python
RULES = """Rules:
- Answer only from the numbered context, never from outside knowledge.
- After each claim, cite its passage like [2].
- If the context does not answer the question, set "answer" to null.
Reply with JSON only: {"answer": string or null, "citations": [numbers]}"""
def grounded_prompt(question, hits):
context = "\n".join(f"[{n}] ({d['id']}) {d['text']}" for n, (d, _) in enumerate(hits, 1))
return f"<context>\n{context}\n</context>\n\nQuestion: {question}\n\n{RULES}"

Step 2: parse the reply and ask

Models sometimes wrap JSON in a sentence or a code fence. Find the outermost braces before parsing, instead of failing on the first stray word.

python
def parse_reply(text):
start, end = text.find("{"), text.rfind("}")
return json.loads(text[start:end + 1])
def ask(question, k=3):
hits = retrieve(question, k)
reply = parse_reply(llm(grounded_prompt(question, hits), system=SYSTEM))
return reply, hits
reply, hits = ask("How many unused PTO days carry over?")
print(reply)
Expected output: {'answer': 'Up to 10 unused days carry over to the next year [1].', 'citations': [1]}

Step 3: an honest refusal

python
reply, hits = ask("Is there free parking at the Denver office?")
print(reply)
if reply["answer"] is None:
print("I couldn't find this in Brightline's policies. Try the HR portal.")
Expected output: {'answer': None, 'citations': []}, then the fallback message.

A refusal is a correct answer when the documents are silent. Your code, not the model, decides what the employee sees then: a fixed message and a place to ask. That keeps refusals consistent and easy to count, which lesson 13 needs.

Issue: a citation that points nowhere

Strict rules make made-up answers rarer, but not impossible. Here's the kind of reply they still allow, on a question about London meal allowances. We wrote it by hand so you can test the check.

python
bad = {"answer": "Meals are $75 per day [1], and London has a higher $110 allowance [4].",
"citations": [1]}
A reply that cites passage 4 when only 3 were retrieved, and lists one citation while using two.

The $110 figure came from the model's memory, not from Brightline. The answer looks grounded because it has brackets. The fix is to check citations in code: they must exist, match the list, and every answer must have at least one.

python
def citation_problems(reply, hits):
in_text = {int(n) for n in re.findall(r"\[(\d+)\]", reply["answer"] or "")}
problems = []
if not in_text <= set(range(1, len(hits) + 1)):
problems.append("cites a passage that wasn't retrieved")
if reply["answer"] and not in_text:
problems.append("answer has no citations")
if in_text != set(reply["citations"]):
problems.append("citation list doesn't match the text")
return problems
print(citation_problems(bad, hits))
Expected output: ["cites a passage that wasn't retrieved", "citation list doesn't match the text"]

When the check fails, don't show the answer. Retry once with the problems added to the prompt, or fall back to the refusal message. This check catches citations that are missing or invented. It can't tell whether passage 1 really supports the claim; lesson 21 adds a grader for that.

This lab makes about five model calls.

Put in order

Put the generation step in order

Drag, or focus an item and press Space, move with the arrow keys, then Space to drop.

  1. Build the prompt: context, question, rules
  2. Retrieve and number the passages
  3. Check citations in code
  4. Call the model and parse its JSON
  5. Show the cited answer or the fallback message

Worth keeping

  1. Ask for structured output so your code can check the answer.
  2. A refusal is a correct answer when the documents are silent.
  3. Verify citations in code: they must exist and match the text.
  4. Your code decides what the user sees, never the raw model reply.

You've changed the chunker, the search, the reranker, the query and the prompt. Is Ask Brightline better than it was in lesson 3, or does it just feel better? Next lesson turns that feeling into a number.

Leaderboard

Loading…

50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.