All 29 lessons
Foundations: what RAG is and a working pipeline
Better retrieval: chunking and search
Answers you can trust, measure and debug
Production in a company
Agentic RAG
Frameworks: LangChain, LangGraph and LangSmith
- 24. The RAG toolbox: what each library does and why you import it
- 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
- 26. Lab: Ask Brightline in LangChain
- 27. Lab: agentic RAG as a LangGraph graph
- 28. Lab: tracing, evaluating and monitoring with LangSmith
- 29. Wrap-up: your production agentic RAG checklist
15 min
Running RAG in production: freshness, cost and monitoring
Keep the index fresh, control cost and latency, and watch the system so failures surface before users report them.
By the end of this lesson you can
- Design an ingestion pipeline that keeps the index fresh, including deletes
- Estimate cost and latency per question, stage by stage
- Choose what to trace and which numbers to watch on a dashboard
From earlier lessons · 1 of 2
A wiki page tells the model to send users to an outside email address. Which defence catches it after the model obeys?
Pick one answer.
From earlier lessons · 2 of 2
Why should actions with side effects need human approval?
Pick one answer.
HR updates the PTO policy on Monday: carry-over drops from 10 days to 5. On Tuesday, Ask Brightline still tells 200 employees they can carry over 10. Nothing crashed. Every metric looks normal.
Keep the index fresh
A RAG system is only as current as its index. The demo indexed documents once. Production needs an ingestion pipeline that runs continuously:
- Detect changes with modified timestamps, change feeds or webhooks from the source system.
- Re-index incrementally. Re-chunk and re-embed only what changed, not the whole corpus every night.
- Delete what's gone. This is the most often forgotten step. A deleted or replaced document must leave the index, or its old rules keep answering.
- Version with dates. Store each document's effective date on its chunks. When two versions conflict, prefer the newest, and show the date in the answer: "per the PTO policy, updated 3 March".
- Re-sync permissions in the same pass, as lesson 16 required.
Cost and latency, stage by stage
Work out cost per question before launch, in tokens. Prices change often, so multiply by your provider's current rates yourself. A typical Ask Brightline question:
- Retrieval and reranking use small models or a search service, so they cost cents per thousand questions.
- Generation dominates. Five chunks of about 400 tokens, plus 300 tokens of instructions, is about 2,300 input tokens, plus about 300 output tokens.
- Extras add up. A query rewrite, a judge call or an agent loop each adds a full model call.
At 1,000 questions a day, that's about 2.3 million input tokens daily. Prompt caching helps with the part that repeats, your long instructions. Anthropic reports cached prefixes can cut costs by up to 90% and latency by more than half.
Latency adds up the same way. Microsoft's RAG guidance notes users expect answers in about 3-5 seconds. A plausible budget is 0.1 s for hybrid search, 0.2 s to rerank 50 candidates, 0.8 s for an optional rewrite and 2-3 s to generate. Stream the answer so the first words appear in about a second.
Cache what repeats: embeddings of unchanged chunks, and full answers to common questions, keyed by question and permission set.
Trace every request
When an answer is wrong, you need to see why in minutes. Log one trace per request. It holds four groups of fields:
- Queries. The original question and any rewrite.
- Retrieval. Retrieved and reranked chunk ids, with their scores.
- Generation. The prompt version, model and token counts.
- Result. The answer, its citation-check result and each stage's latency.
Traces hold sensitive text, so restrict them like the source documents. Store user ids rather than names where you can.
Three numbers to watch
Start the dashboard with three numbers. Add more when one of them moves and you can't explain why.
- Refusal rate. A jump means content went missing or retrieval broke. A drop to zero can mean the model started guessing.
- Negative feedback rate from thumbs-down and "this is wrong" reports. Turn each into a question in your evaluation set.
- p95 latency and cost per answer. They catch a runaway agent loop or a prompt that doubled in size.
Match
Match each symptom to its likely cause
Pick an item on the left, then its match on the right.
Predict, then reveal
Brightline's whole policy set is 300 pages, about 120,000 tokens, and changes monthly. Is a full RAG stack the cheapest design?
Worth keeping
- Freshness needs change detection, incremental updates and deletes that reach the index.
- Generation tokens dominate cost; count them per question before launch.
- Trace every request with queries, chunk ids, scores, versions and per-stage latency.
- Watch refusal rate, negative feedback, and p95 latency and cost.
Everything so far was Brightline, a company we made up. Next lesson looks at real deployments: a bank's research assistant, a support chatbot, legal research tools, and an airline that lost a tribunal case over its chatbot.
Leaderboard
Loading…
50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.
Sources for this lesson (2)
Each fact in the lesson is marked with the number of the page it comes from.