Black Friday50% off your first month on any monthly plan with EARLY50 See plans
All 29 lessons
  1. Foundations: what RAG is and a working pipeline

    1. 1. Getting started: what RAG is and when to use it
    2. 2. Types of RAG: a map from naive to agentic
    3. 3. Lab: build the smallest RAG pipeline end to end
    4. 4. How embeddings and vector search find meaning
    5. 5. Document types: what each one breaks, and how to prepare it
  2. Better retrieval: chunking and search

    1. 6. Chunking documents so answers survive
    2. 7. Chunking strategies compared: which to use when
    3. 8. Keyword search vs vector search, and fusing both
    4. 9. Retrieval algorithms compared: which to use when
    5. 10. Reranking the top results before the model reads them
    6. 11. Rewriting the question before you search
  3. Answers you can trust, measure and debug

    1. 12. Lab: grounded answers with citations and refusals
    2. 13. Measuring a RAG system with an evaluation set
    3. 14. Lab: evaluation methods and frameworks: Ragas and DeepEval
    4. 15. Troubleshooting a RAG system, symptom by symptom
  4. Production in a company

    1. 16. Access control: only retrieve what the user may see
    2. 17. Defending against prompt injection in retrieved text
    3. 18. Running RAG in production: freshness, cost and monitoring
    4. 19. Real-world RAG: what worked, what failed, and why
  5. Agentic RAG

    1. 20. Lab: from a fixed pipeline to an agent with a search tool
    2. 21. Self-correcting retrieval: grade, retry, fall back
    3. 22. GraphRAG and routing across many sources
    4. 23. Lab: numbers, tables and financial documents
  6. Frameworks: LangChain, LangGraph and LangSmith

    1. 24. The RAG toolbox: what each library does and why you import it
    2. 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
    3. 26. Lab: Ask Brightline in LangChain
    4. 27. Lab: agentic RAG as a LangGraph graph
    5. 28. Lab: tracing, evaluating and monitoring with LangSmith
    6. 29. Wrap-up: your production agentic RAG checklist
See the leaderboard

15 min

Running RAG in production: freshness, cost and monitoring

Keep the index fresh, control cost and latency, and watch the system so failures surface before users report them.

By the end of this lesson you can

  • Design an ingestion pipeline that keeps the index fresh, including deletes
  • Estimate cost and latency per question, stage by stage
  • Choose what to trace and which numbers to watch on a dashboard

From earlier lessons · 1 of 2

A wiki page tells the model to send users to an outside email address. Which defence catches it after the model obeys?

Pick one answer.

From earlier lessons · 2 of 2

Why should actions with side effects need human approval?

Pick one answer.

HR updates the PTO policy on Monday: carry-over drops from 10 days to 5. On Tuesday, Ask Brightline still tells 200 employees they can carry over 10. Nothing crashed. Every metric looks normal.

Keep the index fresh

A RAG system is only as current as its index. The demo indexed documents once. Production needs an ingestion pipeline that runs continuously:

  1. Detect changes with modified timestamps, change feeds or webhooks from the source system.
  2. Re-index incrementally. Re-chunk and re-embed only what changed, not the whole corpus every night.
  3. Delete what's gone. This is the most often forgotten step. A deleted or replaced document must leave the index, or its old rules keep answering.
  4. Version with dates. Store each document's effective date on its chunks. When two versions conflict, prefer the newest, and show the date in the answer: "per the PTO policy, updated 3 March".
  5. Re-sync permissions in the same pass, as lesson 16 required.
Rendering diagram…
Every change flows to the index within minutes, and deletions take their own path so old rules disappear.

Cost and latency, stage by stage

Work out cost per question before launch, in tokens. Prices change often, so multiply by your provider's current rates yourself. A typical Ask Brightline question:

  • Retrieval and reranking use small models or a search service, so they cost cents per thousand questions.
  • Generation dominates. Five chunks of about 400 tokens, plus 300 tokens of instructions, is about 2,300 input tokens, plus about 300 output tokens.
  • Extras add up. A query rewrite, a judge call or an agent loop each adds a full model call.

At 1,000 questions a day, that's about 2.3 million input tokens daily. Prompt caching helps with the part that repeats, your long instructions. Anthropic reports cached prefixes can cut costs by up to 90% and latency by more than half.

[2]

Latency adds up the same way. Microsoft's RAG guidance notes users expect answers in about 3-5 seconds. A plausible budget is 0.1 s for hybrid search, 0.2 s to rerank 50 candidates, 0.8 s for an optional rewrite and 2-3 s to generate. Stream the answer so the first words appear in about a second.

Cache what repeats: embeddings of unchanged chunks, and full answers to common questions, keyed by question and permission set.

[14]

Trace every request

When an answer is wrong, you need to see why in minutes. Log one trace per request. It holds four groups of fields:

  • Queries. The original question and any rewrite.
  • Retrieval. Retrieved and reranked chunk ids, with their scores.
  • Generation. The prompt version, model and token counts.
  • Result. The answer, its citation-check result and each stage's latency.

Traces hold sensitive text, so restrict them like the source documents. Store user ids rather than names where you can.

Three numbers to watch

Start the dashboard with three numbers. Add more when one of them moves and you can't explain why.

  1. Refusal rate. A jump means content went missing or retrieval broke. A drop to zero can mean the model started guessing.
  2. Negative feedback rate from thumbs-down and "this is wrong" reports. Turn each into a question in your evaluation set.
  3. p95 latency and cost per answer. They catch a runaway agent loop or a prompt that doubled in size.

Match

Match each symptom to its likely cause

Pick an item on the left, then its match on the right.

Predict, then reveal

Brightline's whole policy set is 300 pages, about 120,000 tokens, and changes monthly. Is a full RAG stack the cheapest design?

Worth keeping

  1. Freshness needs change detection, incremental updates and deletes that reach the index.
  2. Generation tokens dominate cost; count them per question before launch.
  3. Trace every request with queries, chunk ids, scores, versions and per-stage latency.
  4. Watch refusal rate, negative feedback, and p95 latency and cost.

Everything so far was Brightline, a company we made up. Next lesson looks at real deployments: a bank's research assistant, a support chatbot, legal research tools, and an airline that lost a tribunal case over its chatbot.

Leaderboard

Loading…

50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.

Sources for this lesson (2)

Each fact in the lesson is marked with the number of the page it comes from.

  1. [2]Anthropic (2024), Introducing Contextual Retrieval
  2. [14]Microsoft Learn: RAG in Azure AI Search