1 What is RAG & why it exists
Imagine a really smart friend taking a quiz. They've read tons of books, but they can't remember everything, and some books came out after they finished studying. Now imagine you let them take the quiz with the textbook open. Before answering each question, they flip to the right page, read it, and then answer. They're still the same smart friend — but now they look things up instead of guessing. RAG turns a closed-book AI into an open-book AI.
RAG stands for Retrieval-Augmented Generation. The name is literally the recipe, in order:
- Retrieval — first, go find relevant information from an outside source (your documents, a database, the web).
- Augmented — take that found information and add it into the prompt you send to the LLM.
- Generation — the LLM then generates its answer using that fresh information, instead of relying only on what it memorised during training.
The three problems RAG solves
Recall from Session 1 that an LLM is a frozen, pre-trained model doing inference. That creates three painful limitations — RAG was invented to fix all of them:
| Problem (from Session 1) | What goes wrong | How RAG fixes it |
|---|---|---|
| Knowledge cutoff | The model only knows facts up to its training date. Ask about last week and it can't know. | Retrieve up-to-date documents at query time and feed them in. |
| No private knowledge | The model never saw your company wiki, contracts, or emails — they weren't in its training data. | Retrieve from your own private files and inject them into the prompt. |
| Hallucination | When unsure, the model predicts plausible-sounding text that may be wrong (Session 1). | Give it the real source text so it answers from evidence, and can even cite it. |
RAG doesn't change the model's brain at all. It changes the model's prompt. Right before the LLM answers, RAG quietly slips the most relevant text into the context window (Session 1) so the model has the facts sitting right in front of it. The model still just predicts the next token — but now it's predicting from your documents instead of from fuzzy memory.
The open-book exam analogy, in full
A closed-book exam tests what's in your head — that's a plain LLM. An open-book exam lets you bring the textbook: you're not memorising every fact, you're skilled at finding the right passage and using it. RAG makes the LLM an open-book exam taker:
- The "textbook" = your collection of documents (the knowledge base).
- "Flipping to the right page" = the retrieval step.
- "Reading the page, then answering in your own words" = the generation step.
Your company has a 400-page internal HR handbook updated every month. An employee asks the chatbot:
"How many vacation days do I get after 3 years?"
- Without RAG: the base model never saw your handbook. It guesses "probably 15–20" — a hallucination. Wrong and dangerous.
- With RAG: the system searches the handbook, finds the exact paragraph ("Employees with 3+ years of service accrue 22 days…"), pastes that paragraph into the prompt, and the model answers "22 days, per the current handbook" — correct, current, and grounded in a real source.
Grounding means tying the model's answer to real, retrieved evidence instead of its own memory. A grounded answer can quote and cite its sources — which is why RAG is the go-to technique for trustworthy, factual AI apps.
2 RAG vs fine-tuning — a decision framework
Say you want your friend to be more helpful. There are two very different ways. One: send them back to school to learn new habits and a new way of talking — that's slow and expensive, but it changes how they behave. Two: just hand them the right notes right before they answer — fast and cheap, and it changes what they know right now. The first is fine-tuning. The second is RAG. They solve different problems.
We met fine-tuning briefly in Session 1: it's a small, extra round of training on top of a pre-trained model to specialise it. RAG, by contrast, never touches the model — it works purely at inference time by editing the prompt. Beginners constantly ask "do I need to fine-tune?" — usually the answer is "no, you need RAG." Here's how to decide properly.
What each one actually does
- Fine-tuning changes the model's weights (the parameters from Session 1). It bakes new behaviour, style, tone, or output format into the model itself. It's good at teaching the model how to respond.
- RAG changes the model's input. It supplies fresh facts/knowledge at query time. It's good at teaching the model what to talk about right now.
Use RAG for knowledge (facts that change, are private, or must be current). Use fine-tuning for behaviour (a consistent style, tone, format, or skill the model should always exhibit). Many serious systems use both: fine-tune for how it talks, RAG for what it knows.
Side-by-side comparison
| Aspect | RAG | Fine-tuning |
|---|---|---|
| What it changes | The prompt (model untouched) | The model's parameters |
| Best for | Facts, knowledge, documents | Behaviour, style, tone, output format |
| Update speed | Instant — just add/edit a document | Slow — re-run a training job |
| Cost to set up | Low — build a search index | Higher — needs labelled data + GPU training |
| Cost per query | Slightly higher (extra retrieved tokens in the prompt) | Same as normal inference |
| Keeps current? | Yes — re-index new data anytime | No — fixed until you retrain |
| Can cite sources? | Yes — it has the real text | No — knowledge is baked in invisibly |
| Handles private data? | Yes, easily | Yes, but the data is absorbed into weights (harder to remove) |
| Risk | Bad retrieval → irrelevant context | Overfitting; expensive mistakes; data hard to update |
A simple decision flow
- "Answer questions about our constantly-changing product catalogue." → RAG. The catalogue changes weekly; you can't retrain weekly. Retrieve the current product page.
- "Always reply in our brand's playful, emoji-light voice, in this exact JSON schema." → Fine-tuning. That's behaviour/format you want baked in permanently.
- "A support bot that sounds on-brand AND quotes the latest policy docs." → Both. Fine-tune the voice, use RAG for the policies.
People try to fine-tune to teach the model facts ("just train it on our docs!"). This works poorly: the model may still hallucinate, can't cite sources, and goes stale the moment a doc changes — forcing an expensive retrain. For knowledge, reach for RAG first.
3 Embeddings for retrieval
Imagine every sentence gets a colour based on what it means. Sentences about dogs are shades of blue; sentences about cooking are shades of red. Now when you ask "how do I feed a puppy?", the computer turns your question into a colour too — a bluish one — and grabs the sentences with the closest colour. It never matched the exact words; it matched the meaning. Those "meaning colours" are called embeddings.
In Session 1 you met embeddings: a token's meaning turned into a list of numbers (a vector) where similar meanings sit close together ("king − man + woman ≈ queen"). RAG uses the exact same idea, but for whole pieces of text — a sentence, a paragraph, a chunk of a document. Each piece gets one embedding vector that summarises its meaning.
Why embeddings, and not keyword search?
Old-school search matches words. If a document says "automobile" and you search "car," plain keyword search misses it. Embeddings match meaning, so "car," "automobile," and "vehicle" all land near each other in meaning-space. This is called semantic search — search by sense, not by spelling.
Convert both your documents and the user's question into embedding vectors using the same embedding model. Then a question and a relevant chunk will have nearby vectors — even if they share no words. Retrieval is just "find the chunks whose vectors are closest to the question's vector."
Measuring "closeness": cosine similarity
How do we tell if two vectors are "close"? The most common measure in RAG is cosine similarity: it looks at the angle between two vectors rather than how long they are. The intuition:
- Vectors pointing in the same direction (small angle) → very similar meaning → cosine similarity near 1.0.
- Vectors at a right angle → unrelated → cosine similarity near 0.
- Vectors pointing opposite → opposite meaning → near −1.
You'll also hear cosine distance, which is just 1 −
cosine similarity — a smaller distance means closer/more similar. The retrieval system computes
this between your question and every candidate chunk, then keeps the closest ones.
Your knowledge base has this chunk:
"Felines often purr when content."
A user asks: "Why do cats make that rumbling sound when happy?"
There's almost no word overlap ("felines" vs "cats", "purr" vs "rumbling sound", "content" vs "happy"). Keyword search would fail. But because the embedding model understands that felines≈cats and purr≈rumbling sound and content≈happy, the two vectors point in nearly the same direction — a high cosine similarity — so the chunk gets retrieved. That's the whole magic.
The model that creates embeddings (e.g. an "embedding model" like OpenAI's
text-embedding-3) is usually a separate, smaller model from the chat LLM that
writes the final answer. Crucial rule: you must embed your documents and your queries with the
same embedding model, or their vectors won't live in the same meaning-space and
comparisons become meaningless.
4 Vector databases
A normal filing cabinet sorts things alphabetically — great if you know the exact name. But we don't want "find the file named X"; we want "find the files that feel similar to this one." A vector database is a special filing cabinet that organises things by meaning, so when you walk in holding one note, it instantly hands you the most similar notes.
Once every chunk is an embedding vector, you need somewhere to store all those vectors and search them fast. That's a vector database (or "vector store"): a database purpose-built to hold high-dimensional vectors and quickly find the ones nearest to a given query vector.
Why won't a normal database work?
Regular databases (like a SQL table) are built to match exact values or ranges:
"find rows where name = 'Alice'" or "price < 100." They have no efficient
notion of "find the rows whose 1,536-number vector points most similarly to this vector."
Doing that naively means comparing your query against every single stored vector — fine for a
hundred, hopeless for ten million. Vector databases solve exactly this:
- They store vectors efficiently (often thousands of numbers each).
- They build a special index (next topic) so search doesn't have to scan everything.
- They return the top-k nearest vectors — the k most similar chunks — in milliseconds.
- They store metadata alongside each vector (source file, date, author) so you can filter, e.g. "only docs from 2026."
Three vector databases you'll hear about
| Tool | Hosting | Good first impression |
|---|---|---|
| Chroma | Local / open-source | Tiny and beginner-friendly; runs on your laptop with a few lines of Python. Perfect for learning and prototypes. |
| Pinecone | Managed cloud (SaaS) | Fully hosted — you don't run servers; it scales to billions of vectors. Popular for production apps that want zero ops. |
| Qdrant | Open-source or managed cloud | Run it yourself or use their cloud. Strong filtering on metadata and good performance; a nice middle ground. |
Local (Chroma, self-hosted Qdrant): free, private, great for prototyping, but you handle scaling, backups, and uptime. Managed (Pinecone, Qdrant Cloud): you pay, but they handle the hard infrastructure and it scales effortlessly. A very common journey: start on Chroma locally while building, then move to a managed store for production.
For one paragraph of your handbook, the vector DB holds a record roughly like:
{
"id": "handbook-p042",
"vector": [0.013, -0.221, 0.087, ... 1533 more numbers],
"text": "Employees with 3+ years of service accrue 22 days...",
"metadata": { "source": "hr_handbook.pdf", "page": 42, "updated": "2026-05" }
}
At query time you hand the DB your question's vector and say "give me the 4 closest records." It returns the text + metadata of those 4 — ready to drop into the prompt.
5 Indexing
Imagine a library with a million books dumped in a giant pile. To find one, you'd check every single book — forever. So librarians make a card catalogue: a clever organisation system that points you straight to roughly the right shelf. Indexing is building that catalogue for your vectors, so searching is fast instead of checking every vector one by one.
Indexing is the process of preparing your documents so they can be searched quickly. It has two parts: (1) turning documents into stored, searchable vectors, and (2) organising those vectors so finding the nearest ones is fast.
Part A — How documents become a searchable index
Indexing happens ahead of time (before any user asks anything), and follows the same early steps we'll formalise in the pipeline topic:
The result — all your chunk-vectors organised for fast lookup inside the vector database — is "the index." You build it once (and update it whenever documents change). Searching it later is cheap and instant.
Part B — Nearest-neighbour search and the ANN trick
The job at query time is nearest-neighbour search: given the question's vector, find the closest chunk-vectors. The honest, perfect way is to compare against every stored vector and sort — called exact or "brute-force" search. It's accurate but slow at scale (millions of comparisons per query).
Real systems use ANN — Approximate Nearest Neighbour search. Instead of guaranteeing the perfect closest matches, ANN cleverly organises vectors (into graphs or clusters) so it can find almost certainly the closest ones while checking only a tiny fraction of them. You trade a sliver of accuracy for an enormous speed-up — usually well worth it.
You don't have to implement ANN yourself — it's exactly what vector databases (Topic 4) do under the hood. You'll sometimes see algorithm names like HNSW (a popular graph-based ANN method); for now, just know "ANN = fast, near-perfect similarity search."
Say you have 10,000,000 chunks. Brute-force search compares the query to all 10 million vectors every query — painfully slow. A good ANN index might only inspect a few thousand candidates and still return the right top-k in a few milliseconds. Same answers in practice, thousands of times faster.
An index is a snapshot. If your documents change (new policy, deleted page) but you don't re-index, retrieval will serve stale or missing info — quietly defeating the whole point of RAG. Plan to re-embed and update the index whenever the source data changes.
6 Chunking strategies
You can't shove a whole giant book into the AI at once — it only has a small desk (the context window from Session 1). So you cut the book into bite-sized pages first. But how you cut matters: chop in the middle of a sentence and the piece stops making sense. Chunking is the art of slicing documents into pieces that are small enough to handle but big enough to still mean something.
Chunking is splitting your documents into smaller pieces ("chunks") before embedding them. We chunk because (a) each chunk must fit limits and be small enough to embed cleanly, and (b) retrieval should return a focused passage, not an entire 400-page PDF. The way you chunk hugely affects how good your RAG system is.
The four common strategies
| Strategy | How it splits | Pros | Cons |
|---|---|---|---|
| Fixed-size | Every N characters or tokens (e.g. every 500 tokens), regardless of meaning. | Dead simple; predictable sizes. | Can slice mid-sentence or mid-idea, breaking meaning. |
| Sentence-based | Split on sentence boundaries; group a few sentences per chunk. | Each chunk is grammatically whole; natural. | Sentence lengths vary, so chunk sizes are uneven. |
| Semantic | Split where the topic shifts (using embeddings to detect meaning changes). | Chunks are topically coherent — great retrieval. | More complex and compute-heavy to produce. |
| Recursive | Try to split on big separators first (paragraphs), then smaller (sentences, words) until chunks fit the target size. | Respects document structure; a robust default. | Needs sensible separator rules; still size-driven. |
Most beginners start with recursive chunking (it's the default in tools like LangChain's text splitter). It tries paragraph breaks first, falls back to sentences, then words — keeping pieces as meaningful as possible while still hitting your size target. It's a great balance of simple and smart.
Chunk size & overlap — the two dials that matter most
Chunk size is how big each piece is (e.g. 500 tokens). It's a balance:
- Too small → each chunk lacks context; the answer might be split across several chunks and you miss the full picture.
- Too large → chunks contain lots of irrelevant text, so the embedding is "blurry" (averages many topics) and retrieval gets less precise; you also waste prompt tokens.
Overlap means letting consecutive chunks share some text at their edges (e.g. the last 50 tokens of one chunk also start the next). This stops an idea that straddles a boundary from being cut in half and lost. A small overlap (often 10–20% of chunk size) is common.
Source paragraph: "To reset your password, go to Settings. Then click Security.
Finally, choose 'Reset password' and confirm via email."
Fixed-size, no overlap, naive cut:
- Chunk A:
"To reset your password, go to Settings. Then click Secu" - Chunk B:
"rity. Finally, choose 'Reset password' and confirm via email."
"Security" got split — a query about "the security tab" might match neither chunk well.
Sentence-based with overlap:
- Chunk A:
"To reset your password, go to Settings. Then click Security." - Chunk B:
"Then click Security. Finally, choose 'Reset password' and confirm via email."
Now each chunk is whole, and the overlapped sentence ("Then click Security.") bridges the two so no step is orphaned.
It depends on your content. Dense legal text, chatty FAQs, and code all chunk differently. Treat chunk size and overlap as things you tune and test (just like prompts in Session 1) — measure retrieval quality and adjust.
7 The basic RAG pipeline
Two days, two jobs. Job one (do it once, ahead of time): take all your books, cut them into pages, and file them in the magic meaning-cabinet. Job two (every time someone asks): turn their question into a "meaning note," pull the few most similar pages from the cabinet, staple those pages to the question, and hand the whole bundle to the AI to read and answer. That's the entire RAG pipeline.
RAG has two phases. The ingestion phase (also called indexing) happens once up front; the query phase happens every time a user asks something. Every topic in this session is one piece of this pipeline — now we connect them all.
Phase 1 — Ingestion (build the knowledge base, once)
Phase 2 — Query (answer a question, every time)
Step 7 is the heart of "augmented." We build a prompt that stuffs the retrieved chunks into the context window alongside the user's question — typically with an instruction like "Answer using only the context below." This grounding is what makes the answer factual and citable.
Top-k is simply how many of the closest chunks you retrieve — e.g.
k = 4 fetches the 4 most similar chunks. Too few and you might miss the
answer; too many and you flood the prompt with noise (and burn tokens — Session 1's token economics).
A small k like 3–5 is a common starting point.
Ingestion (done last week): the 400-page HR handbook is split into ~800 chunks, each embedded and stored in Chroma with metadata (page number, last-updated date).
Now an employee asks: "How many vacation days after 3 years?"
- Embed question → the query becomes a vector with the same embedding model used for the chunks.
- Retrieve top-k (k=3) → the vector DB returns the 3 closest chunks. The top hit is the paragraph: "Employees with 3+ years of service accrue 22 days of paid leave annually."
- Augment → we build the prompt:
System: Answer using ONLY the context. Cite the page. Context: [p.42] "Employees with 3+ years of service accrue 22 days of paid leave annually." [p.41] "Leave accrues monthly and carries over..." [p.43] "Requests must be submitted two weeks in advance." Question: How many vacation days after 3 years?
- Generate → the LLM reads that context and replies: "After 3 years of service you accrue 22 days of paid leave annually (HR Handbook, p.42)."
Correct, current, grounded, and cited — exactly what a plain LLM couldn't do.
RAG adds zero new ability to the model itself — it's still the same next-token predictor from Session 1. All the intelligence of RAG lives in the plumbing: chunk well, embed well, retrieve the right top-k, and stuff it into the prompt cleanly. Get the retrieval right and the generation almost takes care of itself. (Session 5 dives into making retrieval much smarter.)
★ Putting it all together
You just learned how to give an AI an open book. Here's the one-paragraph story that connects all 7 topics:
RAG exists because a plain LLM has a knowledge cutoff, can't see private data, and hallucinates (all from Session 1). Unlike fine-tuning — which retrains the model to change its behaviour — RAG changes the prompt to supply knowledge at query time. To do that, we turn text into embeddings (meaning vectors) and find matches by cosine similarity, storing them in a vector database (Chroma, Pinecone, Qdrant) because normal databases can't search by meaning. We index the vectors so ANN search is fast, after first chunking documents (fixed/sentence/semantic/recursive, tuning size & overlap). Finally the full pipeline runs: ingest → chunk → embed → store, then embed the question → retrieve top-k → stuff the chunks into the context window → generate a grounded, citable answer.
Quick self-check
Your data changes every day. Do you reach for RAG or fine-tuning, and why?
RAG. It supplies knowledge at query time, so you just re-index the new data — no retraining. Fine-tuning bakes facts into the weights and goes stale immediately, forcing expensive retrains. (RAG for knowledge, fine-tuning for behaviour.)
A user asks "why do cats purr?" but the doc says "felines purr when content." Why does RAG still find it?
Because retrieval matches by meaning, not keywords. The embedding vectors for the question and the chunk point in nearly the same direction (high cosine similarity), so the chunk is retrieved despite sharing almost no words — that's semantic search.
Why can't you just use a normal SQL database to store embeddings for RAG?
Normal databases match exact values or ranges; they have no efficient way to find the vectors "closest in meaning" to a query vector. Vector databases build special (ANN) indexes to do nearest-neighbour search fast, which a SQL table can't.
What is chunk "overlap" and why does it help?
Overlap means consecutive chunks share some text at their edges. It prevents an idea or sentence that straddles a chunk boundary from being cut in half and lost, so retrieval still surfaces the complete thought.
In the RAG pipeline, what does "top-k" control, and what's the risk of making it too large?
Top-k is how many of the closest chunks you retrieve and inject into the prompt. Too large floods the context with irrelevant text (hurting answer quality) and wastes tokens/money. A small k (≈3–5) is a common starting point.
📚 References & Further Reading
Class material
- 📄 Original course notes / handout (source sheet) — open the shared GenAI class material for this session.
- Class handout: "RAG 1 — embeddings, vector databases, chunking & the basic pipeline".
- 📝 Assignment 03 — Google NotebookLM RAG — the assignment shared for this session.
Papers, docs & deep dives
- "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (Lewis et al., 2020) — the original RAG paper.
- Pinecone — "What is a Vector Database?" — clear primer on vector stores and similarity search.
- LangChain — RAG tutorial — hands-on walkthrough of building a retrieval pipeline end to end.
- Chroma documentation — docs for a popular open-source embedding/vector database.