All 29 lessons
Foundations: what RAG is and a working pipeline
Better retrieval: chunking and search
Answers you can trust, measure and debug
Production in a company
Agentic RAG
Frameworks: LangChain, LangGraph and LangSmith
- 24. The RAG toolbox: what each library does and why you import it
- 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
- 26. Lab: Ask Brightline in LangChain
- 27. Lab: agentic RAG as a LangGraph graph
- 28. Lab: tracing, evaluating and monitoring with LangSmith
- 29. Wrap-up: your production agentic RAG checklist
22 min
Lab: numbers, tables and financial documents
See why basic RAG gets financial maths wrong, then fix it with structured extraction, a code tool and checks on every number.
By the end of this lesson you can
- Explain why basic RAG gets financial arithmetic wrong, with benchmark evidence
- Extract table values into structured rows and compute with SQL and code
- Let a model write the calculation while code does the arithmetic, and check every number
From earlier lessons · 1 of 2
Where should "How many VPN tickets did Denver open in October?" go?
Pick one answer.
From earlier lessons · 2 of 2
Why does plain RAG fail on corpus-wide questions?
Pick one answer.
Brightline's CFO asks: "By what percentage did revenue grow from Q1 to Q2?" Ask Brightline retrieves both quarterly reports and replies with a confident figure, 4.7%. The real answer is 4.47%. Next time the Q1 operating loss is in parentheses, and it reports a profit.
What the research shows
This isn't a Brightline quirk. FinanceBench (Islam et al., 2023) asked questions about real public-company filings. GPT-4-Turbo with a shared vector store over all filings answered only 19% correctly, and was wrong or refused on 81%. With the right filing in a long context window it reached 79%, and with the exact evidence pages handed to it, 85%. Retrieval was the biggest gap, but even perfect evidence left errors.
Hybrid table-and-text questions are hard for a second reason: arithmetic. On FinQA (Chen et al., 2021), financial experts reached 91% execution accuracy, against 61% for the best model in the paper. TAT-QA's human experts scored 84 exact match against the authors' model's 50.
Three things go wrong when a model does maths from retrieved text:
- Retrieval misses. Embeddings of numbers rank poorly. Q4 Inc., which builds tools for investor relations, reported exactly this on AWS's blog, and moved numeric questions to SQL.
- Formatting traps. Thousands versus millions, commas, and accounting negatives in parentheses, like (412.0).
- Mental arithmetic. A language model predicts digits rather than calculating them.
The fix for all three is the same: numbers live in structured data, and code does the arithmetic.
Lab: numbers the right way
You'll take two quarterly table rows as a parser would extract them (lesson 5), load them into SQLite, and compute from there. Everything except step 3's model call runs locally.
Step 1: parse amounts, and the parentheses trap
def parse_amount(s):s = s.replace("$", "").replace(",", "").strip()negative = s.startswith("(") and s.endswith(")") # accounting style: (412.0) means -412.0value = float(s.strip("()"))return -value if negative else valueprint([parse_amount(x) for x in ["$1,284.6", "(412.0)", "57.4"]])
Step 2: load the rows, compute in SQL and Python
import sqlite3ROWS = [ # (quarter, line item, amount in $ thousands), as extracted from the reports("2026-Q1", "Revenue", "$1,284.6"), ("2026-Q2", "Revenue", "$1,342.0"),("2026-Q1", "Operating income", "(412.0)"), ("2026-Q2", "Operating income", "57.4"),]db = sqlite3.connect(":memory:")db.execute("CREATE TABLE financials (quarter TEXT, item TEXT, amount REAL)")db.executemany("INSERT INTO financials VALUES (?, ?, ?)", [(q, i, parse_amount(a)) for q, i, a in ROWS])def amount(item, quarter):return db.execute("SELECT amount FROM financials WHERE item = ? AND quarter = ?",(item, quarter)).fetchone()[0]q1, q2 = amount("Revenue", "2026-Q1"), amount("Revenue", "2026-Q2")print(q1, q2, round((q2 - q1) / q1 * 100, 2))
Step 3: the model writes the formula, code runs it
Questions vary, so you can't hand-write every formula. Program-aided generation lets the model write the calculation, and code evaluates it. In the Program of Thoughts paper, this lifted few-shot accuracy on FinQA from 40.4% to 64.5% over chain-of-thought reasoning. PAL describes it as offloading the solution step to an interpreter.
Never run model-written code directly in your process. Evaluate only arithmetic, as below, or use a sandbox such as Claude's code execution tool.
import ast, operatorOPS = {ast.Add: operator.add, ast.Sub: operator.sub, ast.Mult: operator.mul,ast.Div: operator.truediv, ast.USub: operator.neg}def calc(expr): # numbers and + - * / onlydef ev(n):if isinstance(n, ast.Constant) and isinstance(n.value, (int, float)): return n.valueif isinstance(n, ast.BinOp) and type(n.op) in OPS: return OPS[type(n.op)](ev(n.left), ev(n.right))if isinstance(n, ast.UnaryOp) and type(n.op) in OPS: return OPS[type(n.op)](ev(n.operand))raise ValueError("only numbers and + - * / are allowed")return ev(ast.parse(expr, mode="eval").body)expr = llm(f"Q1 revenue {q1}, Q2 revenue {q2}. Write one arithmetic expression for percent growth. Reply with the expression only.")print(expr, "=", round(calc(expr), 2))
Step 4: check every number before showing the answer
import redef unsupported_numbers(answer, allowed):found = {float(x.replace(",", "")) for x in re.findall(r"(?<![A-Za-z])-?\d[\d,]*\.?\d*", answer)}ok = {round(v, 2) for v in allowed}return sorted(v for v in found if round(v, 2) not in ok)answer = "Revenue grew 4.47% from $1,284.6K in Q1 to $1,342.0K in Q2, and margin rose 9.1 points."print(unsupported_numbers(answer, [q1, q2, 4.47]))
This check is a design choice rather than a published standard. It follows directly from the evidence above: if the model shouldn't compute numbers, it shouldn't invent them either. Pair it with citations to the exact table and page. Claude's citations feature, for example, returns the source passages and page ranges behind each claim.
How financial teams actually do it
- Structured data first. NYSE's parent ICE built a text-to-SQL system with Databricks for its data products. On 48 test queries, it reported 96% execution match, and found descriptive table and column metadata highly important.
- Filings as data. For US public companies, the SEC's EDGAR APIs serve XBRL financial-statement data as JSON, such as all reported facts for a company, with no API key. That's more reliable than parsing PDFs.
- RAG for the words. Deutsche Bank and Morgan Stanley (lesson 19) use RAG to find and summarise research and filings. The numbers come from data systems.
So yes, financial institutions use RAG heavily, but not as a calculator. Retrieval finds the passage; databases and code produce the figures; citations prove both.
Match
Match each finance question to its source of truth
Pick an item on the left, then its match on the right.
Worth keeping
- Basic RAG gets finance wrong at retrieval, formatting and arithmetic.
- Numbers belong in structured data; compute with SQL or code, never in the model's head.
- Let the model write the formula, and evaluate it safely in code.
- Check that every number in an answer comes from data or a computation.
You've now used numpy, sentence-transformers, rank-bm25, sqlite3 and anthropic, and the next labs add LangChain, LangGraph and LangSmith. What does each of those libraries actually install, and do you need them all? Next lesson opens the toolbox.
Leaderboard
Loading…
50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.
Sources for this lesson (10)
Each fact in the lesson is marked with the number of the page it comes from.
- [55]Islam et al. (2023), FinanceBench
- [56]Chen et al. (2022), Program of Thoughts Prompting
- [57]Gao et al. (2022), PAL: Program-aided Language Models
- [58]Zhu et al. (2021), TAT-QA
- [59]Chen et al. (2021), FinQA
- [60]AWS ML Blog (2023): How Q4 Inc. handled numerical data in its RAG chatbot
- [61]Databricks blog: Financial insights with NYSE/ICE
- [62]SEC: EDGAR APIs (XBRL financial data)
- [63]Claude docs: Code execution tool
- [64]Claude docs: Citations