1 Why basic RAG isn't enough
Imagine you ask a friend to fetch books from a library for your homework, but they can only grab whatever has a similar title to your question. If you ask a fuzzy question, they bring back the wrong books. If you ask a big question, they bring back books that each answer only a tiny part. And even when they grab the right book, if they hand you a giant stack, you might not notice the one useful page buried in the middle. Basic RAG is exactly this helpful-but-clumsy friend. Today we teach the friend some tricks.
Basic RAG follows one rigid recipe: take the question exactly as typed, find the most similar chunks, stuff them into the prompt, generate. The problem is that each of those steps can fail quietly, and when retrieval brings back the wrong context, the model confidently produces a wrong answer. Remember from Session 1: the model is a next-token predictor — give it bad context and it'll happily build a fluent, wrong answer on top of it. Garbage in, garbage out.
Where naive "retrieve-then-generate" breaks
| Failure | What goes wrong | Why it happens |
|---|---|---|
| Bad chunks | The right information was split across two chunks, or a chunk is too big and dilutes its meaning, or too small and loses context. | Chunking is done blindly offline, before anyone knows what will be asked. |
| Ambiguous / messy queries | "How do I reset it?" — reset what? Typos, slang, and missing context make the question's embedding point at the wrong neighbourhood. | Retrieval matches the question's wording, not the user's true intent. |
| Vocabulary mismatch | User asks about "car insurance"; the document says "auto policy coverage." Different words, same meaning — but the vectors may not line up. | Questions and answers are written in different styles. |
| Multi-part questions | "Compare our 2023 and 2024 revenue and explain the change." One search can't grab both years and the explanation well. | A single embedding tries to represent several distinct information needs at once. |
| Lost-in-the-middle | You retrieve 15 chunks; the crucial one sits at position 8. The model pays most attention to the start and end and skims the middle — so it misses it. | A real, measured weakness of long-context attention. |
| Irrelevant context | You retrieve 10 chunks "just in case"; 7 are off-topic. They distract the model and waste tokens. | More retrieved ≠ more accurate. Noise hurts. |
Every fix in this session adds extra steps — more LLM calls, more searches, more re-ordering. Each step makes retrieval more accurate but also slower and more expensive (more tokens, more latency). The art of advanced RAG is choosing which tricks are worth their cost for your problem. There's no free lunch — only better trade-offs.
A tempting fix is to retrieve 20 or 50 chunks so the answer is "definitely in there somewhere." This usually makes things worse: it triggers lost-in-the-middle, buries the signal in noise, and burns tokens (which cost money — recall token economics from Session 1). Better retrieval beats more retrieval.
A user types "whats the return window" into a shopping assistant. The
relevant document section is titled "Refund & Exchange Eligibility Period." Because the
question is short, lowercase, has a typo, and uses different vocabulary ("return window" vs "refund
eligibility period"), the closest chunks by embedding are actually about shipping windows.
The model dutifully answers with shipping info — fluent, confident, and wrong. Nothing
"errored." That's the danger: basic RAG fails silently.
2 Query rewriting
Imagine asking a librarian, "where's the thingy about the war?" A good librarian doesn't run off confused — they first say back to you, "Ah, you mean a book about World War II for a school project?" and then go find it. Query rewriting is the AI doing that helpful tidy-up: before searching, it cleans up and clarifies your messy question so the search finds better stuff.
The first and cheapest upgrade to basic RAG is to not search the raw question. Instead, we pass the user's question to an LLM first and ask it to produce a cleaner, search-friendly version. This is query rewriting (also called query reformulation or query expansion). The rewritten query is what actually goes to the vector database.
What rewriting fixes
- Typos & grammar — "recieve refnd" becomes "receive a refund," so the embedding lands in the right place.
- Missing context from the chat — turns a follow-up like "and the second one?" into a standalone question by pulling in earlier turns. (Crucial because retrieval has no memory of the conversation on its own.)
- Disambiguation — "reset it" becomes "reset the wireless router to factory settings," resolving what "it" refers to.
- Vocabulary expansion — adds synonyms and the formal terms documents actually use ("car insurance" → "car insurance auto policy coverage premium"), bridging the vocabulary-mismatch gap from Topic 1.
- Style matching — rephrases a casual question into the kind of declarative wording your documents are written in, so their embeddings line up better.
Rewriting is a small extra LLM call that happens before retrieval: user question → LLM rewrite → search the rewrite → retrieve → generate. It's cheap (a short prompt and a short output) and often gives the biggest accuracy bump per dollar of any technique here, which is why it's usually the first thing teams add.
Conversation so far: the user has been chatting about a Dell XPS 13 laptop.
Raw user question: "can i upgrade the ram on it later"
Search this directly and the embedding has no idea what "it" is — it might match generic RAM articles. So we rewrite:
Rewritten query: "Can the RAM (memory) on the Dell XPS 13
laptop be upgraded after purchase? Is the memory soldered or replaceable?"
Now the search is specific, spells out "memory" as a synonym, and even anticipates the real answer's vocabulary ("soldered"). Retrieval gets the exact spec-sheet chunk it needs.
An over-eager rewrite can change the meaning of the question or add a wrong assumption, sending retrieval off in the wrong direction. Keep the rewrite prompt tight ("preserve the user's intent; only clarify and add synonyms; do not answer the question"). And remember: it's one more LLM call, so it adds a little latency to every query.
3 Sub-query enhancement / query decomposition
If your mom says "clean your room, do your homework, and feed the dog," you don't try to do all three in one confusing motion — you split it into three little jobs and do each one. A big, complex question is the same. Instead of one giant search that does everything badly, the AI breaks the question into smaller questions, searches for each one separately, and then puts the pieces back together.
Query rewriting cleaned up one question. But some questions are really several questions wearing a trench coat. A single embedding can't faithfully represent "compare A and B and explain why" — it gets pulled in three directions and retrieves muddy results. The fix is query decomposition (also called sub-query enhancement): use an LLM to split the question into smaller, independent sub-queries, retrieve for each one separately, then combine all the retrieved context for the final answer.
The flow
Why this helps: each sub-query is simple and focused, so its embedding points cleanly at the right chunks. You're effectively giving retrieval several sharp, single-topic searches instead of one blurry, multi-topic one.
They're cousins. Rewriting turns one question into one better question. Decomposition turns one question into several questions. You often do both: decompose first, then rewrite each sub-query. Decomposition shines for "compare," "and," "list all," and multi-hop questions where the answer depends on combining separate facts.
User question: "Which of our products had higher customer
satisfaction in 2024, the Pro plan or the Team plan, and what was the main complaint about the
loser?"
One search can't do this well. Decompose into:
- Sub-query A:
"Pro plan customer satisfaction score 2024" - Sub-query B:
"Team plan customer satisfaction score 2024" - Sub-query C:
"Most common customer complaints about the Pro plan 2024" - Sub-query D:
"Most common customer complaints about the Team plan 2024"
Each search returns clean, on-topic chunks. The model then compares A vs B to find the "loser," and uses C or D (whichever matches the loser) for the complaint. Notice sub-queries C and D are a little speculative — the system retrieves both because it doesn't yet know which plan lost. That extra search is the cost of being thorough.
Four sub-queries means four searches and more retrieved chunks to stuff into the prompt — more latency, more tokens, and a higher chance of pulling in noise. Use decomposition when a question is genuinely multi-part; for a simple question it's overkill. (And after combining, you'll often want re-ranking — our next-but-one topic — to trim the merged pile back down.)
4 HyDE — Hypothetical Document Embeddings
Imagine you're looking for a specific page in a giant pile of pages, but you only have a short question. It's hard to match a tiny question to a big page. So instead, you first guess what the answer page might say — you scribble a pretend answer — and then go find the real page that looks most like your scribble. Answers look like answers, so matching answer-to-answer works better than matching question-to-answer. That clever guess-first trick is HyDE.
HyDE (Hypothetical Document Embeddings) tackles a deep mismatch: questions and answers look different. A question is short and interrogative ("What causes a 500 error?"); a document chunk is long and declarative ("A 500 Internal Server Error occurs when the server encounters an unexpected condition…"). Their embeddings can sit surprisingly far apart even though one answers the other. HyDE's fix: don't embed the question — embed a fake answer to it.
The flow
The key insight is in step 2–3: the hypothetical answer is written in the same style and vocabulary as real documents, so its embedding lands in "answer territory," right next to the genuine chunks that actually answer the question. We then throw the fake answer away and only keep what it helped us find.
It doesn't matter if the LLM hallucinates details in the hypothetical answer. We never show it to the user. We only use it as a search probe — a decoy shaped like a real answer to attract the right neighbours in vector space. The real, trustworthy facts come from the actual chunks we retrieve and then generate from.
Question: "Why is my sourdough not rising?"
Step 2 — LLM imagines an answer: "Sourdough may fail to rise if the starter is inactive or under-fed, the dough is proofed at too cool a temperature, the gluten is under-developed, or fermentation time is too short. An active starter should double in size within 4–8 hours…"
Even if a detail here is slightly off, this paragraph is packed with the exact words real baking articles use — "starter," "proof," "fermentation," "gluten." We embed this, search with it, and pull back the genuine, correct baking-guide chunks — which the original 6-word question would likely have missed. Then we generate the real answer from those chunks.
HyDE costs an extra LLM generation before you even search, adding noticeable latency. It shines when questions are short and very different in style from your documents; it helps less when questions already resemble the documents. As always, measure whether it earns its keep.
5 Re-ranking
Picture hiring for a job. First you do a super-fast skim of 100 résumés to grab the 20 that look okay — quick but rough. Then you carefully read those 20 properly and pick the best 3 to actually interview. Re-ranking is that two-step move: a cheap, fast first pass to gather lots of maybes, then a slow, careful judge to pick the true best few.
Vector search is fast but approximate. It's great at quickly fetching things that are "roughly relevant," but it often gets the exact ordering wrong — the genuinely best chunk might come back ranked 9th. Re-ranking adds a second, more accurate stage: retrieve many candidates cheaply, then re-score them with a smarter (slower) model and keep only the top few.
Why two stages? Bi-encoder vs cross-encoder
The reason this works comes down to two kinds of models. The first stage uses a bi-encoder; the re-ranker is a cross-encoder.
| Aspect | Bi-encoder (first-stage retrieval) | Cross-encoder (re-ranker) |
|---|---|---|
| How it reads them | Encodes the query and each document separately into vectors, then compares the vectors. | Reads the query and a document together, as one pair, and outputs a direct relevance score. |
| Accuracy | Lower — meaning is squashed into one vector before comparison, losing nuance. | Higher — it sees the query and document interacting word-by-word. |
| Speed | Very fast — document vectors are computed once, offline and reused. | Slow — must run the model fresh for every query–document pair, every time. |
| Scales to… | Millions of documents (just compare vectors). | Only tens of documents (too slow for millions). |
| Role | Cast a wide, cheap net: get the top ~50 candidates. | Carefully re-order those ~50 and keep the top ~5. |
Use the fast-but-rough bi-encoder to narrow millions down to a few dozen, then the slow-but-sharp cross-encoder to pick the true winners from that short list. You get cross-encoder quality at near bi-encoder cost — because the expensive model only ever looks at a handful of candidates. This directly fixes the "lost-in-the-middle" and "irrelevant context" problems from Topic 1: you hand the generator 5 truly relevant chunks instead of 50 noisy ones.
Query: "How do I cancel my subscription on mobile?"
Stage 1 (bi-encoder): fetches 50 candidate chunks. They include the right answer, but also chunks about upgrading on mobile, cancelling on desktop, and general billing — all "roughly similar." The true best chunk happens to land at rank 11.
Stage 2 (cross-encoder): re-reads each of the 50 chunks paired with the question and scores true relevance. The "cancel on mobile" chunk now jumps to rank 1; the "upgrade" and "desktop" chunks sink. We keep the top 5 and discard the rest.
Result: the generator sees 5 sharp chunks led by the perfect one, instead of 50 chunks with the answer buried at position 11.
Re-ranking is one of the highest-value upgrades to basic RAG and is widely used in production. Dedicated re-ranker models (e.g. Cohere Rerank, BGE, and similar cross-encoders) take your query plus a list of candidates and return them re-scored. You can also use an LLM itself as the re-ranker — which leads directly into our next topic.
The cross-encoder adds latency proportional to how many candidates you re-rank. Re-rank too many and you lose the speed advantage; re-rank too few and the right chunk might not be in the shortlist. A common balance: retrieve ~50, re-rank, keep ~5.
6 LLM-as-a-judge
Imagine one robot does the homework, and a second robot acts like a teacher and grades it: "Did you actually answer the question? Did you use the right facts? No making stuff up?" Using one AI to check another AI's work — that's LLM-as-a-judge. It's how we measure whether our fancy RAG is actually any good, and how the system can catch its own mistakes.
So far we've made retrieval smarter. But how do we know it's working? And can the system notice when it has retrieved junk? Enter LLM-as-a-judge: using an LLM to evaluate text — either the retrieved chunks or the final answer — against criteria, and return a score or verdict. It's both an evaluation tool (offline, for measuring quality) and a runtime component (online, inside self-correcting loops like CRAG in the next topic).
What a judge typically scores
| Criterion | Question it answers | What it catches |
|---|---|---|
| Relevance | Does this retrieved chunk actually relate to the question? | Off-topic chunks that slipped through retrieval. |
| Faithfulness / Groundedness | Is every claim in the answer supported by the retrieved context? | Hallucinations — the model inventing facts not in the chunks. |
| Answer relevance | Does the final answer actually address what the user asked? | Fluent answers that dodge the real question. |
| Completeness | Did the answer cover all parts of a multi-part question? | Half-answers that drop a sub-question. |
The whole point of RAG is to keep the model honest by grounding it in real documents. A faithfulness judge enforces that: it checks each statement in the answer against the supplied chunks and flags anything unsupported. This is your main defence against the hallucinations we first met in Session 1 — now with an automatic detector.
How you actually use it
You write a careful judge prompt that gives the criterion, the inputs (question, chunks,
and/or answer), and asks for a structured verdict — typically a score (say 1–5) plus a short
justification, or a simple PASS / FAIL. Two main uses:
- Offline evaluation — run the judge over a test set to compare two RAG pipelines ("does adding re-ranking improve faithfulness?"). This is exactly the prompt-testing idea from Session 1, scaled up: instead of grading outputs by hand, an LLM grades them.
- Online / corrective loops — grade chunks during a live query and, if they're poor, trigger a fix (re-search, rewrite, or refuse). That's CRAG, our next topic.
Judge prompt (faithfulness): "You are grading an answer. CONTEXT: [the 3 retrieved chunks]. ANSWER: [the generated answer]. Is every factual claim in the ANSWER directly supported by the CONTEXT? Reply with a score 1–5 and list any unsupported claims."
Verdict: Score: 2/5. Unsupported claim: "offers a 90-day
warranty" — the context only mentions a 30-day return window; no warranty is stated.
The judge just caught a hallucination automatically. In an eval, this lowers the pipeline's score; in a live system, it can trigger a correction or a "I'm not sure" response.
The judge is itself an LLM — it can be wrong, inconsistent, or biased (e.g. favouring longer or more confident-sounding answers). Use a low temperature (Session 1) for repeatable grades, give it crisp criteria and examples, and for important evals, spot-check the judge against human ratings. A judge is a helpful estimate of quality, not absolute truth — and every judge call costs tokens and time.
7 Corrective RAG (CRAG)
Imagine your helper fetches some books, then pauses and checks: "Wait — are these actually useful?" If yes, great, use them. If they're rubbish, the helper doesn't just shrug and answer anyway — it goes and looks somewhere else (like searching the internet) or asks the question differently. Corrective RAG is RAG that checks its own homework before answering, and fixes itself when the retrieval was bad.
Basic RAG always trusts whatever it retrieved, even when retrieval failed. That's the silent failure from Topic 1. Corrective RAG (CRAG) adds a checkpoint: after retrieving, it grades the documents (using an LLM-as-a-judge, Topic 6) and, based on the grade, takes corrective action instead of blindly generating.
The flow
The three branches (step 3 → 4)
| Grade | Meaning | Corrective action |
|---|---|---|
| Correct / relevant | The chunks clearly answer the question. | Use them as-is (maybe after a light cleanup). Proceed to generate. |
| Ambiguous | Partly relevant, but not confident or not complete. | Combine what's good with a fresh source — e.g. rewrite the query and re-search, or add a web search, then generate. |
| Incorrect / irrelevant | The retrieved chunks miss the point entirely. | Discard them. Take a different path: rewrite the query, do a web search for outside knowledge, or — if nothing is found — honestly say "I don't know" rather than hallucinate. |
Basic RAG is a straight line (retrieve → generate). CRAG introduces a feedback loop: if retrieval was bad, the system loops back and tries again differently before answering. This is the first step toward agentic RAG — systems that reason about their own retrieval and decide what to do next, which we'll build on in the LangGraph session.
Question: "What's the warranty on the 2025 model?" —
but your knowledge base only contains 2023–2024 docs.
- Retrieve: returns 2024 warranty chunks (closest available).
- Grade: the judge notices the chunks are about 2024, not 2025 → grade = incorrect/irrelevant for this question.
- Correct: trigger a web search for "2025 model warranty," which finds the manufacturer's current page.
- Generate: answer from the fresh web result, grounded and correct — instead of confidently giving outdated 2024 terms.
Without CRAG, basic RAG would have answered with 2024 info as if it were 2025. The grading checkpoint caught the gap and fixed it.
CRAG adds a grading LLM call to every query, and corrective branches add even more steps (extra searches, extra generations) — so it's slower and pricier, and the loop needs a cap so it can't retry forever. The payoff is far fewer confident-but-wrong answers, which is often worth it for high-stakes domains (medical, legal, finance).
8 Context-window & chunk optimization / token management
Your AI has a small desk (remember the desk from Session 1?). You can only fit so many papers on it, and you pay for every paper you put down. So you don't dump your whole filing cabinet on the desk — you carefully choose only the best, most useful pages, remove duplicates, and maybe shrink long ones into short notes. Packing the desk wisely is what this last topic is about.
All those retrieved chunks have to fit into the model's context window — the fixed-size "desk" from Session 1 (prompt + retrieved context + the answer, all together). And from token economics (also Session 1): every token in that window costs money and time. So the final skill of advanced RAG is fitting the best context, not the most context.
Why not just stuff everything in?
Modern models have huge context windows (hundreds of thousands of tokens), so it's tempting to dump in 50 chunks and let the model sort it out. Three reasons not to:
- Cost — you pay per input token. Stuffing 40 useless chunks into every query multiplies your bill (and remember from Session 1, history is re-sent every turn).
- Latency — more input tokens = a longer, slower prefill = the user waits longer.
- Accuracy — lost-in-the-middle (Topic 1) means a fat context can lower answer quality. Noise crowds out signal.
Techniques to pack the window well
| Technique | What it does |
|---|---|
| Keep top-K only | After re-ranking (Topic 5), pass only the best few chunks (e.g. top 3–5), not everything retrieved. |
| Deduplication | Drop near-identical chunks (overlapping passages, repeated boilerplate) so you don't pay twice for the same fact. |
| Contextual compression | Use an LLM to trim each chunk down to just the sentences relevant to the question before inserting it — keeping the gold, cutting the filler. |
| Summarization | Replace several long chunks with a shorter LLM-written summary that preserves the key facts, saving tokens. |
| Strategic ordering | Counter lost-in-the-middle by placing the most important chunks at the start and end of the context, not buried in the middle. |
| Right-sized chunking | Fix it at the source: tune chunk size/overlap offline so chunks are self-contained but not bloated (ties back to the "bad chunks" failure in Topic 1). |
Treat the context window as precious, paid real estate. Your job isn't to fill it — it's to fit the highest-signal, lowest-redundancy context that answers the question, in the best order. Quality and density of context beats quantity, every time.
Suppose input costs $5 per 1,000,000 tokens. A naive pipeline stuffs 40 chunks (~20,000 tokens) into every query:
- Naive: 20,000 ÷ 1,000,000 × $5 = $0.10 per query on input alone — plus slow prefill, plus lost-in-the-middle hurting accuracy.
- Optimized: re-rank to the top 5, dedupe and compress to ~2,500 tokens: 2,500 ÷ 1,000,000 × $5 = $0.0125 per query — 8× cheaper, faster, and more accurate because the model sees only the good stuff.
Across a million queries that's $100,000 vs $12,500. Context optimization isn't a nicety — it's where RAG quality and RAG economics meet.
Compression and summarization are themselves LLM calls — so you're trading generation cost to save input cost. They also risk dropping a fact that turns out to matter. Compress aggressively for cheap, high-volume cases; be gentler when completeness is critical.
★ Putting it all together
You started this session with a clumsy retrieve-then-generate pipeline and ended with a smart, self-checking one. Here's the one-paragraph story that connects all 8 topics:
Basic RAG fails silently — bad chunks, ambiguous queries, lost-in-the-middle, and noise — and every fix trades speed for accuracy. So before searching, we rewrite the question to clean it up, and decompose complex questions into focused sub-queries. When questions don't look like answers, HyDE embeds a hypothetical answer as a search probe. After retrieving a wide net cheaply with a bi-encoder, we re-rank with a sharp cross-encoder to keep only the truly best chunks. An LLM-as-a-judge scores relevance and faithfulness — both to evaluate the pipeline and to power Corrective RAG (CRAG), which grades retrieved docs and loops back (rewrite, web search, or admit uncertainty) when they're poor. Finally, we optimize the context window — top-K, dedupe, compress, order well — so we fit the best context, not the most, respecting the token economics from Session 1. Together these turn RAG from a brittle line into an accurate, self-correcting system.
Quick self-check
Why is "just retrieve 50 chunks to be safe" usually a bad idea?
It triggers lost-in-the-middle (the model skims the middle and misses the key chunk), buries the useful signal in irrelevant noise, and burns tokens — costing more money and latency for worse accuracy. Better retrieval beats more retrieval.
In HyDE, does it matter if the hypothetical answer is factually wrong?
No. The fake answer is only a search probe — we embed it to find real chunks that look like answers, then throw it away. The actual facts come from the real chunks we retrieve and generate from, never from the hypothetical answer itself.
What's the difference between a bi-encoder and a cross-encoder, and why use both?
A bi-encoder encodes the query and documents separately into vectors (fast but rough, scales to millions). A cross-encoder reads each query–document pair together for a direct relevance score (accurate but slow, only tens of items). You use the bi-encoder to grab ~50 candidates, then the cross-encoder to re-rank them to the top ~5 — accuracy at low cost.
What does a "faithfulness" judge check, and which Session 1 problem does it fight?
It checks that every claim in the answer is supported by the retrieved context. It's an automatic hallucination detector — fighting the "confidently wrong" problem from Session 1 by flagging any statement the chunks don't back up.
How does CRAG change the shape of the RAG pipeline?
It turns the straight line (retrieve → generate) into a feedback loop: grade the retrieved docs and, if they're poor, correct course (rewrite the query, do a web search, or admit "I don't know") before answering. This is the first step toward agentic RAG.
Models have huge context windows now — why still bother optimizing context?
Because every token costs money and latency, and over-stuffing triggers lost-in-the-middle, which lowers accuracy. Keeping only the best top-K chunks (deduped, compressed, well-ordered) is cheaper, faster, and more accurate. Best context beats most context.
📚 References & Further Reading
Class material
- 📄 Original course notes / handout (source sheet) — open the shared GenAI class material for this session.
- Class handout: "RAG 2 — Advanced Retrieval Strategies".
Papers, docs & deep dives
- 📄 HyDE: Precise Zero-Shot Dense Retrieval without Relevance Labels — the paper behind hypothetical-document embeddings (Topic 4).
- 📄 Corrective Retrieval Augmented Generation (CRAG) — grading retrieved docs and self-correcting (Topic 7).
- 📘 Cohere Rerank documentation — a production cross-encoder re-ranker (Topic 5).
- 📘 LlamaIndex docs — advanced RAG & retrieval — query transforms, re-ranking, and context optimization in practice.