Black Friday50% off your first month on any monthly plan with EARLY50 See plans
All 29 lessons
  1. Foundations: what RAG is and a working pipeline

    1. 1. Getting started: what RAG is and when to use it
    2. 2. Types of RAG: a map from naive to agentic
    3. 3. Lab: build the smallest RAG pipeline end to end
    4. 4. How embeddings and vector search find meaning
    5. 5. Document types: what each one breaks, and how to prepare it
  2. Better retrieval: chunking and search

    1. 6. Chunking documents so answers survive
    2. 7. Chunking strategies compared: which to use when
    3. 8. Keyword search vs vector search, and fusing both
    4. 9. Retrieval algorithms compared: which to use when
    5. 10. Reranking the top results before the model reads them
    6. 11. Rewriting the question before you search
  3. Answers you can trust, measure and debug

    1. 12. Lab: grounded answers with citations and refusals
    2. 13. Measuring a RAG system with an evaluation set
    3. 14. Lab: evaluation methods and frameworks: Ragas and DeepEval
    4. 15. Troubleshooting a RAG system, symptom by symptom
  4. Production in a company

    1. 16. Access control: only retrieve what the user may see
    2. 17. Defending against prompt injection in retrieved text
    3. 18. Running RAG in production: freshness, cost and monitoring
    4. 19. Real-world RAG: what worked, what failed, and why
  5. Agentic RAG

    1. 20. Lab: from a fixed pipeline to an agent with a search tool
    2. 21. Self-correcting retrieval: grade, retry, fall back
    3. 22. GraphRAG and routing across many sources
    4. 23. Lab: numbers, tables and financial documents
  6. Frameworks: LangChain, LangGraph and LangSmith

    1. 24. The RAG toolbox: what each library does and why you import it
    2. 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
    3. 26. Lab: Ask Brightline in LangChain
    4. 27. Lab: agentic RAG as a LangGraph graph
    5. 28. Lab: tracing, evaluating and monitoring with LangSmith
    6. 29. Wrap-up: your production agentic RAG checklist
See the leaderboard

15 min

Real-world RAG: what worked, what failed, and why

Study verified deployments and failures in finance, customer support and law, and draw the design lessons from each.

By the end of this lesson you can

  • Describe real RAG deployments in finance, support and engineering, with their published results
  • Explain what went wrong in a well-known chatbot failure and a legal-tools study
  • Turn each case into a design rule for your own system

From earlier lessons · 1 of 2

A policy changed yesterday and the assistant still quotes the old one. What failed?

Pick one answer.

From earlier lessons · 2 of 2

Which three numbers would you put on a RAG dashboard first?

Pick one answer.

In 2022 a grieving traveller asked an airline's website chatbot about bereavement fares, and followed its advice. The airline later argued in a tribunal that its chatbot was a separate legal entity, responsible for its own actions. The tribunal didn't agree.

Case 1: the chatbot answer that became the company's

After his grandmother died, Jake Moffatt asked Air Canada's website chatbot about bereavement fares. It told him he could apply for the reduced fare after booking. The airline's own bereavement policy page said otherwise, and his claim was refused.

In Moffatt v. Air Canada (2024 BCCRT 149), British Columbia's Civil Resolution Tribunal rejected the airline's argument that the chatbot was responsible for its own statements. It held Air Canada liable for negligent misrepresentation. It said a customer shouldn't have to cross-check one part of a website against another. The award was CAD $650.88 in damages, about $812 with interest and fees.

The rule: your assistant's answer is your company's statement. Ground it in the current policy, cite it, and refuse rather than improvise, as in lessons 12 and 21.

[81]

Stanford researchers (Magesh et al., 2024) tested commercial legal research tools built on RAG, with about 200 legal questions. The tools hallucinated between 17% and 33% of the time. That was less than a general chatbot, and more than the vendors' claims implied.

  • Lexis+ AI answered about 65% accurately and hallucinated about 1 in 6 times.
  • Westlaw AI-Assisted Research answered about 42% accurately and hallucinated about one-third of the time.
  • Ask Practical Law AI rarely hallucinated, but 62% of its answers were incomplete, so it was often just unhelpful.

The rules: measure hallucination yourself, with your own questions (lessons 13 and 14). Count incomplete answers as failures too, and never market a system as hallucination-free.

[82]

Cases 3-6: deployments that worked, and why

LinkedIn, customer service. Instead of chunking past support tickets as flat text, the team built a knowledge graph that kept each ticket's structure and the links between tickets. Their paper reports a 77.6% better MRR than the baseline. Deployed with the customer service team for about six months, it cut median resolution time per issue by 28.6%. Rule: keep the structure your documents already have (lessons 5 and 22).

[80]

Uber, security and privacy policy questions. Uber's Enhanced Agentic-RAG worked over more than 40 policy documents. A query-optimiser agent and a source-identifier agent ran before retrieval, then BM25 plus vector hybrid search, then a post-processing agent. An LLM judge scored answers from 0 to 5. On a golden set of 100+ questions, acceptable answers rose 27% and incorrect advice fell 60%, both relative. Rule: agentic steps pay off when you can measure them on a fixed question set (lessons 13 and 20).

Uber, Genie. An internal on-call copilot over engineering wikis and Q&A. Uber reports it answered over 70,000 questions in 154 Slack channels, and estimates 13,000 engineering hours saved. Its helpfulness rate was 48.9%: about half of answers were rated helpful. Rule: a useful system can still miss half the time, so track helpfulness and keep a route to a person.

[85][86]

Deutsche Bank, DB Lumina. A research assistant using a RAG architecture over internal research, SEC filings and other repositories. Deutsche Bank and Google describe it as live since September 2024, with about 5,000 users. It saves 30 to 45 minutes when preparing earnings note templates, and up to two hours on research reports.

Morgan Stanley Wealth Management. An assistant built with OpenAI gives financial advisors answers from the firm's research library of around 100,000 documents. Every use was evaluated before rollout: advisors and prompt engineers graded summaries for accuracy and coherence. Rule: in finance, RAG is used for finding and summarising documents, with evaluation first. The numbers come from structured data, as lesson 23 shows.

[78][79][84]

DoorDash, delivery-driver support. A RAG chatbot over the support knowledge base, with two safety layers. A guardrail checks each answer before it's sent, using a cheap similarity check against the retrieved articles first and an LLM check as backup. An LLM judge reviews answers offline for retrieval correctness, accuracy, grammar, coherence and relevance. Rule: put a cheap check on every answer, and a thorough judge on a sample (lessons 21 and 28).

[83]

What the cases have in common

  1. Evaluation came first. Morgan Stanley, Uber and DoorDash all built evaluation before or alongside launch.
  2. Structure beat flat text. LinkedIn's graph, Uber's hybrid search and Deutsche Bank's mix of sources all kept the shape of their data.
  3. Checks on every answer. Guardrails, judges and citations decide what users see.
  4. Honest limits. A 49% helpfulness rate and 17-33% hallucination rates are what real systems look like. Plan for the misses, with refusals and a route to a person.

Match

Match each case to its main lesson

Pick an item on the left, then its match on the right.

Worth keeping

  1. Your assistant's answers carry your company's liability; ground, cite and refuse.
  2. Even good RAG systems miss: measure hallucination and incompleteness yourself.
  3. Successful deployments evaluate first, keep data structure, and check every answer.
  4. In finance, RAG finds and summarises documents; numbers come from structured data.

Every case used RAG on text. Deutsche Bank's analysts also need numbers from earnings tables. Next part of the course goes agentic, and its last lab shows why a basic pipeline gets that maths wrong.

Leaderboard

Loading…

50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.

Sources for this lesson (9)

Each fact in the lesson is marked with the number of the page it comes from.

  1. [78]OpenAI customer story: Morgan Stanley
  2. [79]Morgan Stanley press release (March 2023)
  3. [80]Xu et al. (2024), RAG with Knowledge Graphs for Customer Service QA (LinkedIn)
  4. [81]Moffatt v. Air Canada, 2024 BCCRT 149
  5. [82]Magesh et al. (2024), Hallucination-Free? Assessing legal AI research tools
  6. [83]DoorDash Engineering: LLM-based Dasher support automation
  7. [84]Google Cloud blog (2025): Deutsche Bank DB Lumina
  8. [85]Uber Engineering: Genie on-call copilot
  9. [86]Uber Engineering: Enhanced Agentic-RAG