All 29 lessons
Foundations: what RAG is and a working pipeline
Better retrieval: chunking and search
Answers you can trust, measure and debug
Production in a company
Agentic RAG
Frameworks: LangChain, LangGraph and LangSmith
- 24. The RAG toolbox: what each library does and why you import it
- 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
- 26. Lab: Ask Brightline in LangChain
- 27. Lab: agentic RAG as a LangGraph graph
- 28. Lab: tracing, evaluating and monitoring with LangSmith
- 29. Wrap-up: your production agentic RAG checklist
15 min
Reranking the top results before the model reads them
Add a cross-encoder reranker that rescores the top candidates, and order context so the model uses it.
By the end of this lesson you can
- Explain why a cross-encoder ranks better than a bi-encoder but can't search a whole corpus
- Add a retrieve-wide, rerank, keep-narrow stage to the pipeline
- Order passages in the prompt so the model uses the best one
From earlier lessons · 1 of 2
Why does reciprocal rank fusion use ranks instead of raw scores?
Pick one answer.
From earlier lessons · 2 of 2
Which search finds error code PRN-0x81 most reliably?
Pick one answer.
Ask Brightline's index now has 14 snippets, including a few near-misses about hotels. Someone asks who has to approve a pricier hotel. Vector search ranks the loyalty-points rule first. The approval rule comes second, and with k=1 it never reaches the model.
Where we are, shown failing
Real indexes are full of near-misses: chunks that share a topic but not the answer. We add six to lab 1's list, then rebuild the vectors so retrieve() from lab 1 sees them.
DOCS += [{"id": "travel-loyalty", "text": "Hotel loyalty points earned on business stays belong to the employee. Do not choose a hotel for its points."},{"id": "travel-booking", "text": "Book hotels and flights through the travel portal at least 7 days before the trip."},{"id": "travel-conf", "text": "Conference hotels booked through the event organiser are exempt from nightly limits."},{"id": "fin-approval", "text": "Expense reports above $1,000 need a director's approval in the finance system before payment."},{"id": "office-denver", "text": "The Denver office is open 7am to 7pm. Visitors sign in at reception on the ground floor."},{"id": "office-boston", "text": "The Boston office has 40 hot desks. Book a desk in the Brightline app before you arrive."},]vectors = embed([d["text"] for d in DOCS]) # lab 1's retrieve() reads thisprint([(d["id"], round(s, 2)) for d, s in retrieve("Who has to approve a pricier hotel?", k=3)])
The right chunk is there, a close second. The first stage did its job, which is recall: getting the answer into the shortlist. It's weak at precision: putting the best item at the very top. A reranker fixes precision.
Two kinds of model
The embedding model you've used so far is a bi-encoder. It encodes the question and each chunk separately, so chunk vectors can be computed once and searched in milliseconds. But the question and the chunk never meet inside the model.
A cross-encoder reads the question and one chunk together, as a pair, and outputs a relevance score. It can notice that "approve" in the question matches "written approval" for hotels, not loyalty points. The cost is one model pass per pair, with nothing precomputed. Scoring a million chunks per question is out of reach, but scoring 50 is cheap.
The Sentence Transformers documentation describes exactly this split. Cross-encoders are usually more accurate and slower, so they're used to rerank a bi-encoder's top results.
Add the reranking layer
from sentence_transformers import CrossEncoderreranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2") # ~90 MB, runs on CPUdef rerank(question, hits, keep=3):ranked = reranker.rank(question, [doc["text"] for doc, _ in hits])return [hits[r["corpus_id"]] for r in ranked[:keep]]q = "Who has to approve a pricier hotel?"print([doc["id"] for doc, _ in rerank(q, retrieve(q, k=10))])
Our run gave the hotel rule a score of −7.03, against −9.87 and −10.24. These models output raw scores, not 0-to-1 probabilities, so only the order matters. Scoring 10 pairs took about 130 ms on a laptop CPU, including a first-call warm-up. Hosted rerankers and GPUs are faster; measure your own.
Re-run a harder case
q = "Can I go over the hotel limit if someone senior says yes?"print([doc["id"] for doc, _ in rerank(q, retrieve(q, k=10))])
The reranker isn't perfect. It ranks the conference exemption first, which is a fair reading of "go over the limit". But the approval rule is still second. That's why you keep 3-5 chunks rather than 1. The reranker improves the order, and the model reads the rest.
Put the best passage where the model looks
Order inside the prompt matters too. In the Lost in the Middle study (Liu et al., 2023), models used information best at the start or end of a long context. Accuracy dropped when the key passage sat in the middle, even for long-context models.
So put the top-ranked chunk first. Keep the context short enough that nothing important sits deep in the middle. Reranking gives you that order for free.
How much does all this buy? In Anthropic's 2024 tests, contextual embeddings plus contextual BM25 cut failed retrievals by 49%. Adding a reranker raised that to 67%, measured as the share of answers missing from the top 20 chunks. They reranked the top 150 candidates down to 20. Treat these as one vendor's results on its datasets, and measure on yours.
Predict, then reveal
Your hybrid search returns 100 candidates and the reranker keeps 5. Answers are often incomplete, but the right chunk is usually ranked 6th to 8th. What do you change first?
Worth keeping
- First-stage search is for recall; the reranker is for precision.
- Cross-encoders read question and chunk together, so they rank better but can't scan a corpus.
- Retrieve 50-150, rerank, keep 5-20; tune the counts with your evaluation set.
- Put the best chunk first; models use the middle of long context least.
The reranker can only reorder what the first search found. When an employee types "and in Denver?", the first search has nothing to work with. Next lesson fixes the question itself.
Leaderboard
Loading…
50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.
Sources for this lesson (3)
Each fact in the lesson is marked with the number of the page it comes from.