1 Why agents need memory
Imagine a friend who is brilliant but has the worst memory in the world. Every single time you talk to them, they forget everything the moment the sentence ends — your name, what you just said, that it's your birthday. To have a real conversation with them, you'd have to re-tell them the whole story every time you speak. That's exactly what an AI model is like. A "memory system" is the notebook we keep on the side so the friend can read up on who you are before answering.
Let's recall the single most important fact from Session 1: an LLM is stateless. After it generates an answer, it keeps nothing. The billions of parameters inside it are frozen during inference — they don't change when you chat, so the model literally cannot store what you told it. Each request starts from a blank slate.
On top of being stateless, the model can only "see" a limited amount of text at once: its context window (the desk-space analogy from Session 1). Even if you wanted to paste your entire life history into every message, it wouldn't fit — and it would be ruinously expensive (Session 1's token economics).
The model has no memory of its own. So "AI memory" is never inside the model — it's a system around the model that decides what information to feed into the context window each time, so the model appears to remember. Memory is an engineering layer, not a model feature.
What "an agent" adds to the problem
An agent is an LLM wired up to take multiple steps, use tools, and hold long, multi-turn interactions toward a goal (we built up to this in earlier sessions). Agents make the memory problem much worse than a single question-and-answer:
- They run for many turns, so the conversation grows past what fits on the desk.
- They span multiple sessions — you come back tomorrow, or next week, and expect continuity.
- They make decisions and observations mid-task that later steps need to recall ("I already tried that API and it failed").
You're using a travel-planning agent.
- Turn 1: "I'm vegetarian and I hate flying — I prefer trains."
- Turn 2: "Plan me a 3-day trip to Italy."
Without a memory system, by Turn 2 the model has no idea you're vegetarian or that you prefer trains — that was a separate request and the model is stateless. It happily books you a steak restaurant and three flights. With even basic memory, Turn 1's facts are carried forward into Turn 2's context, and the agent plans vegetarian meals and rail travel. Same model — the difference is entirely the memory layer.
The whole field borrows the brain's vocabulary. Humans have short-term memory (what you're holding in mind right now) and long-term memory (things you learned years ago). AI agents copy this split exactly — that's the next two topics.
2 Short-term memory
Short-term memory is like the scrap of paper on your desk where you jot down what's happening right now in this conversation. "They said hi, I said hi, they asked for pizza." As long as the paper has room, you can glance at it and remember the chat. But the paper is small — when it fills up, the oldest scribbles at the top get rubbed out to make room. That's short-term memory: handy, immediate, and small.
Short-term memory (also called working memory or conversational memory) is simply the running conversation — the back-and-forth of the current session — kept inside the context window. In Session 8 we called this living transcript a thread: the ordered list of messages (user, assistant, tool results) that makes up one continuous interaction.
How it actually works
Remember from Session 1 how ChatGPT fakes memory? It resends the entire conversation history with every new message. That resent history is the short-term memory. There's no magic store — the application keeps the thread's messages in a list, and on each new turn it stitches them all together and sends the whole stack to the stateless model:
So short-term memory is "free" in the sense that you get it just by keeping and resending the transcript. But it has two hard limits baked in from Session 1.
The two limits: size and cost
| Limit | What goes wrong | Why (from Session 1) |
|---|---|---|
| Size | A long enough chat won't fit; the oldest turns fall off the desk and are forgotten. | The context window is a fixed maximum number of tokens. |
| Cost | Each turn gets more expensive, because you re-pay to send the whole growing history every time. | You're billed per input token, and the whole transcript is re-sent each turn (token economics). |
When the thread overflows the context window, the model doesn't warn you — the earliest messages simply aren't sent anymore, and it quietly "forgets" them. Your agent can suddenly contradict something you established an hour ago. This is the number-one cause of "the AI lost the plot" in long chats.
Coping when short-term memory overflows
Because the window is finite, applications actively manage the thread instead of letting it grow forever. The two classic techniques:
- Trimming (a.k.a. a buffer/sliding window) — keep only the most recent N messages (or the last N tokens) and drop the oldest. Simple and cheap, but you genuinely lose the old information.
- Summarisation — periodically ask the model to compress the older part of the conversation into a short summary, then keep that summary plus the recent raw messages. You retain the gist of old turns at a fraction of the token cost.
A 60-message support chat won't fit. Two strategies:
- Trimming: send only the last 10 messages. Fast and cheap — but if the customer's order number was mentioned in message 3, it's gone, and the agent has to ask again.
- Summarising: condense messages 1–50 into "Customer Jane, order #4471, reported a cracked screen, wants a replacement not a refund," then send that summary + the last 10 raw messages. The order number survives, and you've spent a fraction of the tokens.
Trimming protects you from the context-window size limit. Summarising protects you from both size and cost while preserving meaning. Many real agents combine them: summarise the far past, keep the recent past verbatim.
Short-term memory is just the live thread sitting in the context window — powerful but temporary. It dies when the session ends, and it's capped by window size and cost. To remember things across sessions, or to remember more than fits on the desk, you need something more durable: long-term memory.
3 Long-term memory
Long-term memory is the big diary you keep in a drawer. You don't carry it around all day, but important things go in it: "My friend is allergic to peanuts." Months later, before making them a snack, you open the drawer, find that one page, and read it. You don't read the whole diary — just the page that matters right now. AI long-term memory works the same way: a big store you save important facts into, and pull the relevant page out of when needed.
Long-term memory is persistent knowledge that survives across sessions. Where short-term memory dies when you close the chat, long-term memory is saved somewhere durable — a database, a file, a vector store — so the agent can recall it days or weeks later, in a brand-new conversation, even after the original thread is long gone.
What kinds of things go in long-term memory?
- User facts & preferences — "prefers metric units," "is vegetarian," "works in finance," "calls me by my first name."
- Past decisions & outcomes — "we already decided to use Postgres," "that approach failed last time."
- Distilled knowledge — summaries of previous conversations, key takeaways, project state.
Long-term memory lives outside the model, and only the relevant bits are pulled into the context window when they're needed. This is exactly the RAG (Retrieval-Augmented Generation) pattern from Session 4 — except instead of retrieving from documents, you retrieve from the agent's own remembered facts.
How it works: write now, retrieve later
Long-term memory has two distinct moments, and it's vital to keep them separate in your head:
| Moment | What happens | The hard question |
|---|---|---|
| Writing (saving) | During or after a chat, the agent decides a fact is worth keeping and stores it in the durable memory. | What is worth remembering, and how should it be phrased? |
| Reading (retrieving) | In a later turn or session, the agent searches the store and pulls back only the entries relevant to the current request. | When and which memories should be loaded? |
The retrieval step almost always uses semantic search over a vector store — the same machinery from Sessions 4 & 5. Recall the idea: every memory is turned into an embedding (a meaning-vector, Session 1), and at retrieval time the current request is also embedded and compared by similarity, so you fetch the memories whose meaning is closest — even if the exact words differ.
Week 1, Monday. You tell a coding assistant: "By the way, I always want type hints in
my Python." The agent decides this is a durable preference and writes it to long-term memory as:
user_pref: "include type hints in Python code". The session ends. The thread
(short-term memory) is gone.
Week 3, Thursday. Brand-new conversation, no shared thread. You say: "Write me a
function to parse a CSV." Before answering, the agent embeds your request and searches its memory store;
the type-hints preference comes back as relevant, gets loaded into the context window, and the agent
produces def parse_csv(path: str) -> list[dict]: — with type hints — without
you ever repeating yourself. That is long-term memory at work.
Because it still has to pass through the context window to reach the model, and that window is finite and costs money per token. Storing is cheap and unlimited; loading is the scarce, expensive step. So you store generously but retrieve selectively — which is the central tension of the next topic.
Long-term memory persists — including mistakes. If you save "user lives in London" and they move to Paris, a naive system will keep recalling the wrong city forever. Real systems need ways to update, expire, or overwrite memories, not just append them.
4 Memory vs context constraints
You have a giant toy box (everything you could remember) but a tiny desk (the context window) where you can only play with a few toys at a time. You can't dump the whole box onto the desk — there's no room, and you'd be charged for every toy you put out. So before each game you have to choose: which few toys do I actually need right now? Picking the right ones is the whole art of memory.
Here is the central tension of this entire session: you may have a huge amount of memory, but everything the model uses must fit inside the context window — and from Session 1 we know that window is (a) a fixed maximum size and (b) something you pay for per token. You therefore cannot load everything. Every turn, a memory system must make a choice: of all the things I could load, which small subset goes on the desk this time?
Memory is abundant and cheap to store; context is scarce and expensive to use. The job of a memory system is selection — deciding what to load into the limited, costly context window for this particular request.
The cost side, made concrete
Recall Session 1's token economics: every token you load is an input token you pay for, and it eats space the model's answer also needs. Stuffing the window full of "just in case" memories is wasteful twice over: it costs more money and leaves less room for the actual answer. So selectivity isn't just about fitting — it's about money and answer quality too.
Strategies for choosing what to load
| Strategy | How it picks | Best for | Weakness |
|---|---|---|---|
| Recency | Load the most recent messages/memories; drop the oldest. | Ongoing conversation flow, "what were we just saying?" | Forgets important older facts just because they're old. |
| Relevance (semantic retrieval) | Embed the current request and load the memories most similar in meaning (vector search, Session 4/5). | Pulling the right fact out of a huge store, regardless of age. | Might miss recent context that isn't textually "similar"; needs a vector store. |
| Summarisation | Compress many old items into one short summary, load that instead. | Keeping the gist of a long history cheaply. | Loses fine detail; the summary can drop or distort facts. |
| Combination (hybrid) | e.g. recent raw turns + a running summary + the top-k semantically relevant long-term facts. | Real production agents — gets continuity and relevance and thrift. | More moving parts to build and tune. |
A customer-support agent gets a new message: "Is my replacement shipping yet?" It has a 16,000-token window. Instead of dumping the whole 3-week history, it assembles:
- Recency: the last 6 messages of this chat (so it knows the immediate thread).
- Summary: a 2-sentence summary of everything earlier ("cracked screen, replacement approved").
- Relevance: a semantic search for "shipping/replacement" pulls in the saved fact "order #4471, replacement dispatched 2026-06-18, tracking ZX99."
Total: a few hundred tokens of carefully chosen context instead of tens of thousands of raw history. The agent answers correctly, cheaply, and fast. That hybrid assembly is what real memory systems do every turn.
You'll hear the term context engineering: the discipline of deciding exactly what goes into the context window for each call. Memory systems are a big part of it — they're the machinery that turns a giant pile of stored knowledge into the few hundred well-chosen tokens the model actually sees.
It's tempting to load as much as fits. But cramming the window with marginally relevant memories can actually hurt answers — the model gets distracted, important details get buried, and you pay more for worse results. Selective beats stuffed.
5 Storage methodologies
There are different kinds of "memory drawers," and each is good at a different thing. One drawer just keeps the whole conversation on a roll of paper. Another keeps a short summary on a sticky note. Another is a magic drawer that finds things by meaning ("show me anything about food allergies"). And another is a neat label-maker drawer with one fact per labelled slot ("favourite_colour = blue"). Picking the right drawer for each kind of memory is what this last topic is about.
"Where and how is memory stored?" has several standard answers, each with different trade-offs. They differ in two ways we've already met: how you write to them (what gets saved) and how you read from them (how you find the right thing again). Here are the main methodologies.
The four common methodologies
- Conversation buffer — store the raw messages in order, exactly as they happened. This is short-term memory's natural home. Reading = "give me the last N messages." Dead simple, perfectly faithful, but grows without bound and has no notion of relevance.
- Summary memory — instead of raw turns, store a compressed summary that gets updated as the chat grows. Tiny and cheap to load, but lossy — fine details are gone.
- Vector store (semantic memory) — store each memory as an embedding and retrieve by meaning-similarity (Sessions 4 & 5). Brilliant for "find the relevant fact in a huge pile," scales to lots of memories, but it's fuzzy (similarity, not exact match) and needs extra infrastructure.
- Key-value / structured store — store discrete facts in labelled slots, like
user.diet = "vegetarian", in a normal database or key-value store. Exact, easy to update or delete a single fact, but you must know the key to look it up — it can't answer fuzzy "anything related to X?" questions.
Trade-offs at a glance
| Methodology | What it stores | How you read it | Strength | Weakness |
|---|---|---|---|---|
| Conversation buffer | Raw messages in order | Take the last N / recent window | Faithful, trivial to build | Grows forever; no relevance ranking |
| Summary memory | A running compressed summary | Load the whole (small) summary | Cheap, fits easily | Lossy — drops fine detail |
| Vector store (semantic) | Embeddings of memories/text | Semantic similarity search (top-k) | Finds relevant items in huge stores | Fuzzy; needs infra; no exact-key lookup |
| Key-value / structured | Discrete labelled facts | Look up by key / query fields | Exact, easy to update one fact | Must know the key; no fuzzy search |
Every methodology forces two design decisions. On write: what do you save, and in what form? (A raw turn? A summary? A clean structured fact?) On read: when do you fetch, and how do you find the right entry? (Recent N? Semantic top-k? Exact key?) A good memory system picks the methodology whose write/read style matches the kind of memory it's holding.
A well-built personal assistant rarely uses just one. Typically:
- Conversation buffer for the live chat (short-term memory).
- Key-value store for crisp profile facts:
name,timezone,diet— exact and easy to update. - Vector store for the fuzzy long-term pile: past conversations, notes, "things the user mentioned," retrieved by meaning when relevant.
- Summary memory to keep the cost of the long buffer down.
Each kind of memory lives in the store that fits it best, and the context-assembly step (Topic 4) pulls the right pieces from each.
There's a fifth, more powerful methodology we'll explore next time: storing memories as a graph of connected facts — people, places, and things linked by relationships ("Jane → works_at → Acme → located_in → Paris"). Vector stores find things that are similar; graphs let you follow connections, answering questions like "who else at Jane's company have I talked to?" Session 10 covers graph databases and relational memory in depth.
★ Putting it all together
This session had fewer headline topics, but they all hang off one thread from Session 1. Here's the one-paragraph story that connects them:
The model is stateless and its context window is finite (Session 1), so on its own an agent forgets everything between turns and across sessions. "Memory" is therefore an external layer that feeds the right text into the window each time. Short-term memory is the live conversation thread (Session 8) held in the window and resent every turn — immediate but capped by size and cost, so we trim or summarise when it overflows. Long-term memory is durable knowledge stored outside the model (user facts, past decisions) and pulled back in by retrieval — the RAG pattern from Session 4, usually via semantic search over a vector store (Session 5). Because you can't load everything into the costly, finite window (token economics, Session 1), a memory system must select what to load using recency, relevance, summarisation, or a hybrid. And it physically stores all this using buffers, summaries, vector stores, or key-value/structured stores — with graph-based memory coming in Session 10.
Quick self-check
Why do AI agents need a memory system at all if the model is so smart?
Because the model is stateless (Session 1) — it keeps nothing between requests — and its context window is finite. Without an external memory layer the agent forgets everything between turns and across sessions.
What exactly is short-term memory, and what kills it?
It's the current conversation thread held in the context window and resent each turn. It's killed by the window's size limit (old turns fall off) and token cost, and it vanishes entirely when the session ends.
How does an agent remember your preference from three weeks ago in a brand-new chat?
Long-term memory: the preference was written to a durable store (e.g. a vector store). In the new session the agent retrieves it by semantic similarity — the RAG pattern from Session 4 — and loads just that fact into the context window.
Why can't we just load all of the agent's memory into the context window every time?
The context window is a fixed maximum size and you pay per input token (Session 1). Loading everything won't fit, costs too much, and can even degrade the answer. So memory systems select what to load via recency, relevance, summarisation, or a hybrid.
You need to store the exact fact "user's diet = vegetarian" and update it easily later. Which storage methodology fits best, and why not a vector store?
A key-value / structured store — it holds a discrete labelled fact you can look up exactly and overwrite cleanly. A vector store retrieves by fuzzy meaning-similarity and isn't built for exact, single-fact updates.
📚 References & Further Reading
Class material
- 📄 Original course notes / handout (source sheet) — open the shared GenAI class material for this session.
- Class handout: "Memory Systems in AI Agents".
Papers, docs & deep dives
- 📝 Lilian Weng — LLM-Powered Autonomous Agents — short-term vs long-term memory and retrieval in agent design.
- 📄 MemGPT: Towards LLMs as Operating Systems — paging memory in and out of a finite context window (Topics 4–5).
- 📘 LangChain — memory concepts — buffers, summaries, and managing conversation history in practice.