All 29 lessons
Foundations: what RAG is and a working pipeline
Better retrieval: chunking and search
Answers you can trust, measure and debug
Production in a company
Agentic RAG
Frameworks: LangChain, LangGraph and LangSmith
- 24. The RAG toolbox: what each library does and why you import it
- 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
- 26. Lab: Ask Brightline in LangChain
- 27. Lab: agentic RAG as a LangGraph graph
- 28. Lab: tracing, evaluating and monitoring with LangSmith
- 29. Wrap-up: your production agentic RAG checklist
18 min
Chunking documents so answers survive
Split long policy documents into chunks that keep their meaning and context, and measure the difference.
By the end of this lesson you can
- Spot a chunk boundary that separates a rule from its exception
- Chunk a document by its headings, with fixed-size splits and overlap as a fallback
- Add a context line to each chunk before embedding it
From earlier lessons · 1 of 2
Why must questions and documents use the same embedding model?
Pick one answer.
From earlier lessons · 2 of 2
Where is vector search weakest?
Pick one answer.
Ask Brightline now reads the full travel policy. An employee asks whether a $300 Boston hotel is allowed with a director's approval. It answers "No, the limit is $250". The approval rule is in the index. It just never reached the prompt.
Where we are and what breaks
Lab 1 used six one-paragraph snippets. Real policies are long documents with sections. Embedding a whole 40-page policy as one vector blurs every topic into one position, so nothing matches well. We have to cut documents into chunks: pieces small enough to match one question, but complete enough to answer it.
Here is a short version of the travel policy. We'll chunk it three ways and see which survives.
TRAVEL = """# Travel policy## HotelsBook through the travel portal. Limits per night: $250 in Tier 1 cities (New York,San Francisco, Chicago, Boston) and $175 elsewhere. Exceptions: stays above thelimit need a director's written approval before booking.## MealsUp to $75 per day. Alcohol is not reimbursable.## FlightsEconomy for flights under 6 hours. Premium economy is allowed above 6 hours."""
Naive version: fixed-size chunks
The simplest chunker counts words and cuts. Production chunkers count tokens, but words are close enough to see the problem.
def chunk_fixed(text, size=20, overlap=0):words = text.split()step = size - overlapreturn [" ".join(words[i:i + size]) for i in range(0, len(words), step)]for c in chunk_fixed(TRAVEL):print("-", c)
The second chunk holds the approval rule, but it says "the limit" and never mentions hotels. Its vector lands nowhere near a question about Boston hotels. This is context loss: a chunk that only makes sense next to its neighbours.
Layer 1: overlap
Overlap repeats the last few words of each chunk at the start of the next, so a sentence cut at a boundary appears whole in at least one chunk.
for c in chunk_fixed(TRAVEL, size=20, overlap=6):print("-", c)
Overlap rescues sentences cut at a boundary, but it can't tie this rule to the word hotel. Microsoft's Azure AI Search guidance suggests starting at about 512 tokens (roughly 2,000 characters) with 25% overlap, then testing on your own documents. Our 20-word chunks are tiny only so the cuts are visible.
Layer 2: chunk by structure
Policies already tell you where ideas begin and end: their headings. Structure-aware chunking splits on headings, and only falls back to fixed-size splits when a section is too long.
def chunk_by_heading(text, doc_title):chunks = []for block in text.split("\n## ")[1:]: # skip the title linesection, _, body = block.partition("\n")chunks.append({"doc": doc_title, "section": section, "text": body.strip()})return chunksCHUNKS = chunk_by_heading(TRAVEL, "Travel policy")print([c["section"] for c in CHUNKS])
Layer 3: add context before embedding
A chunk from page 30 may still say "these limits" with no subject. The fix is to embed each chunk with a short context line: the document title and section. Anthropic's contextual retrieval goes further and has a model write 50-100 tokens of context per chunk.
In their tests, that alone cut failed retrievals by 35%. With prompt caching, they put the one-time cost at about $1.02 per million document tokens. Those figures come from Anthropic's September 2024 post; measure on your own data.
import numpy as npfrom rag import embeddef embed_text(c):return f"{c['doc']} > {c['section']}: {c['text']}"vecs = embed([embed_text(c) for c in CHUNKS])def search(question, k=2):scores = vecs @ embed([question])[0]return [CHUNKS[i]["section"] for i in np.argsort(-scores)[:k]]print(search("Can I book a $300 hotel in Boston if my director approves?"))
We embed the text with its context line, but we still show the model the plain chunk with its source. Keep the two apart: one is for matching, the other is for reading.
Re-run the earlier cases
for q in ["How much can I spend on a hotel in Chicago?","Can I expense wine with dinner?","Do I fly economy to London?"]:print(q, "->", search(q, k=1))
Store metadata with every chunk: source document, section, last-updated date and the groups allowed to read it. You'll filter on dates in lesson 18 and on groups in lesson 16.
Treat chunking as a semi-permanent decision. Changing it means re-chunking and re-embedding everything, and re-running your evaluation set.
Match
Match each document to a good first chunking strategy
Pick an item on the left, then its match on the right.
Worth keeping
- Split by the document's own structure first, and by size only as a fallback.
- Overlap protects sentences cut at a boundary; start near 512 tokens with 25% overlap.
- Embed each chunk with its title and section, and show the model the plain chunk.
- Changing your chunking means re-embedding everything, so test before you commit.
Heading chunks fixed the travel policy. But Brightline also indexes chat transcripts with no headings, and a table of expense limits. Next lesson compares seven chunking strategies and when each one wins.
Leaderboard
Loading…
50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.
Sources for this lesson (2)
Each fact in the lesson is marked with the number of the page it comes from.