All 29 lessons
Foundations: what RAG is and a working pipeline
Better retrieval: chunking and search
Answers you can trust, measure and debug
Production in a company
Agentic RAG
Frameworks: LangChain, LangGraph and LangSmith
- 24. The RAG toolbox: what each library does and why you import it
- 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
- 26. Lab: Ask Brightline in LangChain
- 27. Lab: agentic RAG as a LangGraph graph
- 28. Lab: tracing, evaluating and monitoring with LangSmith
- 29. Wrap-up: your production agentic RAG checklist
15 min
Keyword search vs vector search, and fusing both
Compare BM25 and vector search on the same questions and combine them with reciprocal rank fusion.
By the end of this lesson you can
- Explain how BM25 scores a chunk in plain words
- Pick keyword, vector or hybrid search for a type of question
- Fuse two rankings with reciprocal rank fusion
From earlier lessons · 1 of 2
Why embed each chunk with its title and section?
Pick one answer.
From earlier lessons · 2 of 2
What does overlap protect against?
Pick one answer.
Brightline's IT index has three error codes: VPN-0x80, NET-0x80 and PRN-0x81. Ask vector search about VPN-0x80 and the Wi-Fi chunk scores 0.38, not far below the right one at 0.49. Add a few more codes and the wrong one can win.
The decision you'll face
Every retrieval system picks how to match a question to chunks. Vector search matches meaning. Lexical search matches the actual words. Most production systems run both. To see why, we'll send the same questions through each.
We add two IT snippets to lab 1's six, so the index has the three error codes. DOCS is lab 1's list.
import reimport numpy as npfrom rank_bm25 import BM25Okapi # pip install rank_bm25from rag import embedDOCS += [{"id": "it-wifi", "text": "Wi-Fi error NET-0x80 means your network certificate expired. Reconnect to BL-Secure to renew it."},{"id": "it-printer", "text": "Printer error PRN-0x81 means the toner is low. Replace the cartridge from the supply room."},]tokenize = lambda t: re.findall(r"[a-z0-9-]+", t.lower())bm25 = BM25Okapi([tokenize(d["text"]) for d in DOCS])vecs = embed([d["text"] for d in DOCS])
BM25 in plain words
BM25 is the standard lexical ranking. It gives a chunk points for each question word it contains, with three adjustments:
- Rare words count more. A match on "vpn-0x80" is worth far more than a match on "the".
- Repeats have diminishing returns. Saying "hotel" ten times doesn't make a chunk ten times better.
- Long chunks are discounted. Otherwise they'd win just by containing more words.
Same questions, both searches
def keyword_top(q, k=2):s = bm25.get_scores(tokenize(q))return [(DOCS[i]["id"], round(float(s[i]), 2)) for i in np.argsort(-s)[:k]]def vector_top(q, k=2):s = vecs @ embed([q])[0]return [(DOCS[i]["id"], round(float(s[i]), 2)) for i in np.argsort(-s)[:k]]for q in ["What does VPN-0x80 mean?", "Can I stay home on Fridays?"]:print(q, "\n keyword:", keyword_top(q), "\n vector: ", vector_top(q))
What does VPN-0x80 mean?keyword: [('it-vpn', 1.49), ('travel-hotel', 0.0)]vector: [('it-vpn', 0.49), ('it-wifi', 0.38)]Can I stay home on Fridays?keyword: [('travel-meals', 1.65), ('travel-hotel', 0.0)]vector: [('hr-remote', 0.24), ('travel-hotel', 0.21)]
Each search fails where the other succeeds. Keyword search found the one chunk containing "vpn-0x80" and gave everything else zero. For the home question, its only match was the word "on", so it returned the meals policy. Vector search found remote work from meaning alone, but only just: 0.24 against 0.21 for the hotel policy. For the VPN code it ranked the right chunk first, with the Wi-Fi code close behind. That margin shrinks as the index grows.
When to use each:
- Exact strings such as error codes, part numbers, names and dates. Keyword search wins.
- Paraphrase and synonyms, such as "stay home" for "work remotely". Vector search wins.
- Typos and abbreviations. Vector search usually copes; keyword search misses unless you add fuzzy matching.
- Mixed real traffic. Use both, which is called hybrid search.
Microsoft's search documentation makes the same split. Keyword search is better at product codes, jargon, dates and names, while vectors find conceptually similar text.
Take a guess
You fuse both rankings. Chunk A is 1st in keyword search and 5th in vector search. Chunk B is 2nd in both. Which comes first after fusion?
No pressure. Guess now, and the lesson will show you if you were right.
Reciprocal rank fusion
Why not just add the scores? Because they're on different scales. BM25 scores have no upper limit, while cosine similarity sits between about 0 and 1. A sum would let one search drown out the other.
Reciprocal rank fusion (RRF) ignores raw scores and uses only positions. Each chunk earns 1 ÷ (k + rank) from every list it appears in, and the totals are sorted. The constant k is commonly 60. It stops the top position in one list from dominating. Azure AI Search uses RRF automatically for hybrid queries.
def rrf(rankings, k=60):scores = {}for ranking in rankings:for rank, doc_id in enumerate(ranking, 1):scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)return sorted(scores, key=scores.get, reverse=True)print(rrf([["A", "B", "C", "D", "E"], ["X", "B", "Y", "Z", "A"]])[:3])
Hybrid search is measurably better in published tests. In Anthropic's 2024 experiments, contextual embeddings alone cut failed retrievals by 35%. Adding BM25 on contextualised chunks raised the cut to 49%.
Match
Match each question to the search that should carry it
Pick an item on the left, then its match on the right.
Worth keeping
- Keyword search wins on exact strings; vector search wins on meaning.
- Never add scores from different searches; fuse their ranks.
- RRF gives each chunk 1/(k + rank) per list, with k commonly 60.
- Default to hybrid search for company documents, then measure.
Hybrid search works on 14 snippets. Brightline's full index will hold 10 million chunks, and exact search over all of them takes 15 GB of memory. Next lesson maps the retrieval algorithms and indexes, and when to use each.
Leaderboard
Loading…
50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.
Sources for this lesson (2)
Each fact in the lesson is marked with the number of the page it comes from.