1 What is GPT?
Imagine a robot that has read almost every book, website, and story in the world. It doesn't understand things like you do, but it's amazing at guessing what word comes next. You say "Once upon a…" and it instantly knows "time." GPT is that super-reader-and-guesser. You give it some words, and it keeps adding the next most likely word, one at a time, until it has written a whole answer.
GPT stands for Generative Pre-trained Transformer. That's three words doing three jobs. Let's unpack each one, because the name literally tells you how it works.
Breaking down the name
| Word | What it means | In plain English |
|---|---|---|
| Generative | It generates (creates) new text rather than just picking from a fixed list of answers. | It writes things, one word at a time. |
| Pre-trained | Before you ever use it, it was already trained on a giant pile of text. The "pre" means "done ahead of time." | It studied for years before meeting you. |
| Transformer | The specific type of neural-network design it's built on (invented by Google in 2017). It's the engine. | The brain's blueprint. |
At its heart, GPT does exactly one thing: given some text, it predicts the next chunk of text. Everything else — answering questions, writing code, translating, summarising — is just that one trick repeated over and over.
So what is a "Large Language Model"?
You'll constantly hear the term LLM (Large Language Model). GPT is one famous family of LLMs. The words mean:
- Language Model — a program whose job is to predict text. ("Model" here just means a mathematical system that learned patterns from data.)
- Large — it has a huge number of internal dials called parameters (often billions). More on parameters below.
Think of a giant mixing board with billions of tiny volume sliders. Each slider is a number the model learned during training. Together, these numbers store everything the model "knows." When people say "GPT-4 has hundreds of billions of parameters," they mean hundreds of billions of these tiny learned numbers. You don't set them — training did.
A quick history so the names make sense
- 2017 — The "Transformer" design is published (the famous paper Attention Is All You Need, our next topic).
- 2018–2020 — OpenAI builds GPT-1, GPT-2, GPT-3, each bigger than the last.
- 2022 — ChatGPT launches and the world notices.
- Today — Many providers (OpenAI's GPT, Anthropic's Claude, Google's Gemini, Meta's Llama). They differ in details but share the same core transformer idea.
You type: The capital of France is
GPT has seen this pattern millions of times in its training text, so the next word it
predicts with highest confidence is Paris. It's not "looking up" a
fact in a database — it's predicting the most likely next word based on patterns it
absorbed. (This is also why it can sometimes confidently say wrong things — called a
hallucination — because it's predicting, not fact-checking.)
2 "Attention Is All You Need" — the intuition
Read this sentence: "The animal didn't cross the road because it was too tired." What does "it" mean — the animal or the road? You instantly know it's the animal, because your brain pays attention to the right earlier words. "Attention" is the AI's ability to do the same: when reading each word, it looks back and decides which other words matter most for understanding this one.
In 2017 a research paper titled "Attention Is All You Need" introduced the attention mechanism and the Transformer. It's arguably the most important AI paper of the last decade — every modern chatbot is built on it. We'll stay at the intuition level (no scary math), exactly as this session intends.
The problem attention solved
Older AI read text strictly left-to-right, one word at a time, trying to cram the meaning of everything it had read so far into a single "memory." Like trying to remember a long phone number — by the end it forgot the start. Long sentences broke it.
Instead of reading word-by-word and forgetting, the Transformer looks at all the words at once and lets every word "ask" every other word: "How relevant are you to me?" The relevant words get more weight. That weighting is called attention.
How attention works, intuitively
For each word, the model builds three things. The classic analogy is a library search:
| Name | Library analogy | Job |
|---|---|---|
| Query (Q) | The question you ask the librarian | "What am I looking for?" |
| Key (K) | The label on each book's spine | "What do I contain?" |
| Value (V) | The actual content inside the book | "Here's my information." |
Each word's Query is compared against every other word's Key. Good matches score high. Those scores decide how much of each word's Value gets blended into the current word's new, context-aware meaning. So after attention, the word "it" in our example has soaked up meaning from "animal."
The word "bank" alone is ambiguous. Attention lets it look around:
- "I sat by the river bank" → "bank" pays attention to "river" → means riverside.
- "I deposited cash at the bank" → "bank" pays attention to "cash/deposited" → means a financial bank.
Same word, different meaning, figured out purely by looking at the neighbours that matter.
Two more terms you'll hear
- Self-attention — words in the same sentence attending to each other (the example above). This is the main engine.
- Multi-head attention — instead of paying attention one way, the model does it several times in parallel ("heads"), each focusing on a different kind of relationship (one head tracks grammar, another tracks subject–object links, etc.). Then it combines them.
"Attention Is All You Need" claimed you can throw away the older, slower machinery and the attention mechanism alone is enough to build powerful language models. It turned out to be true — hence the bold title.
3 How transformers predict the next token
It's like the world's best autocomplete. You start typing "I love peanut butter and…" and it suggests "jelly." The AI does this but for everything: it looks at all the words so far, and picks the most likely next little piece of text. Then it adds that piece and does it again. And again. Like building a LEGO sentence one brick at a time.
A quick word first: AI doesn't work with whole words, it works with tokens (small chunks of text — we cover these fully in Topic 5). For now, just read "token" as "a little piece of a word."
The step-by-step loop
The crucial insight: the model predicts only one token at a time. After picking it, that token is added to the input, and the whole process runs again to get the next one. This is called autoregressive generation ("auto" = self, "regressive" = feeding its own output back in).
Prompt: The sky is
- Round 1 → input "The sky is" → predicts "blue" → text is now "The sky is blue"
- Round 2 → input "The sky is blue" → predicts "and" → "The sky is blue and"
- Round 3 → input "The sky is blue and" → predicts "the" → …continues…
- Eventually it predicts a special "stop" token and ends.
This is also why ChatGPT's answers appear to "type out" word by word — you're literally watching the loop run in real time.
What "scoring" actually produces
In step 3, the model doesn't pick a word directly. It produces a probability for every single token in its vocabulary (tens of thousands of options). For "The sky is" it might give:
| Candidate next token | Probability |
|---|---|
| blue | 78% |
| clear | 9% |
| falling | 4% |
| grey | 3% |
| … thousands more … | tiny amounts each |
These raw scores are turned into clean percentages by a function called softmax, and how the model then picks from this list is controlled by settings like temperature and top-p — all covered in Topic 7.
GPT has no plan and no idea how its sentence will end. It genuinely makes it up one token at a time, each step guided only by everything written so far. The "intelligence" emerges from doing this incredibly well, billions of parameters deep.
4 The end-to-end LLM request → response flow
Think of ordering food. You tell the waiter what you want (your message). The kitchen chops it into ingredients (tokens), the chef cooks (the AI thinks), they plate it up (turn tokens back into words), and the waiter brings it to your table (the answer on your screen). Let's follow your message through that whole kitchen.
Here's what happens between you hitting "Send" and the answer appearing. Understanding this whole pipeline is the goal of Session 1, because every later topic plugs into one of these stages.
The full journey
| # | Stage | What happens |
|---|---|---|
| 1 | You write a prompt | Your text (the prompt) is sent to the model, often with a hidden system prompt setting the AI's behaviour. |
| 2 | Tokenisation | Your text is chopped into tokens and each token becomes a number (its ID). (Topic 5) |
| 3 | Embedding | Each token ID becomes a list of numbers capturing meaning, plus positional info so order is preserved. (Topic 6) |
| 4 | Transformer layers | The model runs attention across all tokens, building rich context-aware understanding. (Topics 2 & 3) |
| 5 | Next-token prediction | It produces probabilities for the next token. (Topic 3) |
| 6 | Decoding / sampling | It picks a token using temperature, top-p, etc. (Topic 7) |
| 7 | Loop | Append the token, repeat steps 4–6 until a stop token or max_tokens is hit. |
| 8 | Detokenise | The generated tokens are stitched back into readable text. |
| 9 | Response | The text streams back to your screen, word by word. |
Steps 1–4 are sometimes called the prefill (the model reads your whole prompt at once). Steps 5–7 are the decode phase (it generates new tokens one by one). Prefill is fast and parallel; decode is the slower, one-at-a-time part — which is why long answers take longer than long questions.
The "context window" — the model's desk space
Everything in steps 1–7 has to fit inside the model's context window: the maximum number of tokens it can consider at once (your prompt plus its answer). Think of it as the size of the desk it can lay papers on. If the conversation gets longer than the desk, the oldest papers fall off the edge — the model "forgets" the earliest parts.
If a model has an 8,000-token context window and your prompt uses 7,500 tokens, only ~500 tokens are left for the answer. Ask for a long essay and it'll get cut off. This is why very long chats eventually "lose track" of what you said at the start — it literally scrolled off the desk.
An LLM is stateless — it has no memory between requests. ChatGPT feels like it remembers because the app secretly resends the entire conversation with every new message. The model re-reads the whole thing each time. This matters hugely for cost (Topic 5) and for agents and memory systems in later sessions.
5 Tokenisation & token economics
The AI can't read letters or words the way you do. Before it reads anything, the words get chopped into little puzzle pieces called tokens. Sometimes a piece is a whole word, sometimes just part of one. And here's the grown-up part: you pay money for each piece, both the ones you send in and the ones the AI sends back. So tokens are like little coins.
What is a token?
A token is the basic unit of text an LLM reads and writes. It's usually not a whole word. Tokenisers split text into common chunks. Rough rules of thumb for English:
- 1 token ≈ 4 characters of English text.
- 1 token ≈ ¾ of a word, so ~100 tokens ≈ 75 words.
- Common words are often one token; rare/long words split into several.
- Spaces and punctuation usually count too (a leading space is often part of the token).
"Tokenization is fascinating!" might split into:
["Token", "ization", " is", " fascinating", "!"] — that's 5 tokens
for 3 words, because "Tokenization" is uncommon and gets broken up, while " is" and " fascinating"
(with their leading spaces) are single tokens.
Meanwhile "the cat sat" is just 3 tokens — three common words, one each.
The common method is Byte-Pair Encoding (BPE). The idea: keep frequent pieces whole, break rare ones into smaller known pieces. This means the model can handle any word — even one it never saw, like a typo or a new brand name — by assembling it from smaller token pieces. It's a balance between "one token per letter" (too many tokens) and "one token per word" (can't handle new words).
Why tokens behave in surprising ways
- Numbers can split oddly: "2024" might be one token or several.
- Other languages often use more tokens than English for the same meaning, because tokenisers are usually optimised for English.
- Whitespace and code (indentation, brackets) eat tokens, which matters for programming prompts.
- This is the famous reason early models struggled to count the letters in "strawberry" — they
see tokens like
["straw", "berry"], not individual letters.
Token economics — why this is about money
Token economics means: AI providers charge per token, and they charge separately for two kinds:
| Type | What it is | Note |
|---|---|---|
| Input tokens | Everything you send: your prompt, system prompt, and the entire chat history. | Usually cheaper per token. |
| Output tokens | Everything the model generates back. | Usually more expensive per token. |
Say a model charges $5 per 1,000,000 input tokens and $15 per 1,000,000 output tokens. You send a 1,000-token prompt and get a 500-token answer:
- Input: 1,000 ÷ 1,000,000 × $5 = $0.005
- Output: 500 ÷ 1,000,000 × $15 = $0.0075
- Total ≈ $0.0125 for that one exchange.
Tiny — until you multiply by millions of users, or by long conversations where the whole history is resent every turn (remember: the model is stateless, Topic 4). That's how costs balloon.
Because the full conversation is resent each turn, a long chat costs more per message as it grows — you re-pay for the whole history every time. Managing this (trimming, summarising) is a real skill we'll revisit in RAG and memory sessions.
6 Input embeddings & positional encoding
Computers only understand numbers, not words. So every token gets turned into a list of numbers — like giving each word its own secret code. And here's the clever bit: words that mean similar things get similar codes. "King" and "queen" sit close together; "king" and "banana" sit far apart. That number-code is called an embedding.
Part A — Input embeddings (turning meaning into numbers)
After tokenisation gives each token an ID number, that's not enough — the ID is just a label, like a jersey number. The model converts each token ID into an embedding: a long list of numbers (a vector), often hundreds or thousands of numbers long, that captures the token's meaning.
Embeddings place words in a giant invisible "meaning space." Words with related meanings end up near each other. Distance and direction in this space encode relationships — this is what lets the model reason about similarity.
Because embeddings capture relationships as directions, you can do arithmetic with meaning: take the vector for "king," subtract "man," add "woman," and you land very close to "queen." The model learned the concept of royalty and the concept of gender as directions in number-space — nobody programmed that in. (You'll meet embeddings again, in a big way, in the RAG sessions, where they power search over documents.)
Embeddings aren't hand-written; they're learned during training. Early in the pipeline they carry general meaning; as the token passes through attention layers, its representation gets enriched with context (recall "bank" from Topic 2).
Part B — Positional encoding (teaching the model about order)
Here's a subtle problem. Remember from Topic 2 that the transformer looks at all tokens at once, in parallel. That's fast — but it means the model has no built-in sense of order. To it, a bag of tokens has no "first" or "last." Yet order is everything in language:
"Dog bites man" vs "Man bites dog" — same words, completely different meaning (one is normal, one is newsworthy!). If the model ignored order, these would look identical to it.
The fix is positional encoding: before the tokens enter the transformer, the model adds extra numbers to each embedding that encode where the token sits in the sequence (position 1, 2, 3…). Now each token's vector carries both "what I mean" (embedding) and "where I am" (position).
Imagine the same musical note played at different beats in a bar. The note (meaning) is the same, but its position in the rhythm changes the music. Positional encoding is the "beat number" stamped onto each token so the model can hear the rhythm of the sentence.
Two things get combined for every token before the thinking starts:
Embedding (what the token means) + Positional encoding (where it sits).
Together they give the model meaning and order — both essential for understanding language.
7 Softmax, Temperature, top-p & max_tokens
Remember the AI guesses the next word and gives each option a score? These are the "knobs" that decide how it picks. One knob (temperature) decides how adventurous it is — low means "play it safe and boring," high means "be creative and surprising." Another knob (max_tokens) is just how long it's allowed to talk before it has to stop.
From Topic 3, the model produces a score for every possible next token. These four concepts control how raw scores become an actual chosen word. As an LLM user, these are the dials you'll actually adjust, so it's worth knowing each well.
Softmax — turning scores into probabilities
The model's raw output scores are called logits — messy numbers that can be negative or huge. Softmax is a function that squashes them into clean probabilities that are all positive and add up to 100%.
Raw logits like [2.0, 1.0, 0.1] become probabilities like
[65%, 24%, 11%] after softmax. Notice the biggest score stays biggest, but now
everything is a tidy percentage you can sample from. Softmax happens automatically —
you don't control it directly, but temperature works through it.
Temperature — the creativity dial
Temperature controls how "flat" or "peaky" those probabilities are before picking.
| Temperature | Effect | Use it for |
|---|---|---|
| Low (≈0–0.3) | Sharpens toward the top choice. Nearly always picks the most likely token. Predictable, repeatable. | Facts, code, math, extraction — when you want correct and consistent. |
| Medium (≈0.7) | A balanced mix of likely and slightly less likely words. | General chat, everyday writing. |
| High (≈1.0–1.5+) | Flattens the odds so rarer words get a real chance. Creative, varied — but can wander or get weird. | Brainstorming, poetry, story ideas. |
Prompt: "Write a name for a coffee shop:"
- Temp 0 → "The Coffee House" (safe, obvious, same answer every time)
- Temp 1.3 → "Bean Voyage" / "Grumpy Mug Collective" (creative, different each run)
Higher temperature = more creative and more likely to make mistakes or hallucinate. For anything where accuracy matters, turn it down.
Top-p — nucleus sampling
Top-p (also called nucleus sampling) is another way to control randomness. Instead of touching the temperature, it limits which tokens are even eligible to be picked. It keeps only the smallest set of top tokens whose probabilities add up to p, and ignores the long tail of unlikely options.
With top_p = 0.9: the model lines up candidates from most to least likely
and keeps adding them until their combined probability reaches 90%. Everything below that cutoff is
discarded, then it samples from what remains. This prevents bizarre, super-unlikely words from
sneaking in, while still allowing variety. top_p = 1.0 means "consider
everything"; lower values mean "only the most confident options."
Both control randomness but differently: temperature reshapes all the probabilities; top-p cuts off the unlikely tail. People usually tune one or the other, not both hard at once. A common safe combo: low temperature for factual tasks; moderate top-p to keep things sane.
max_tokens — the length limit
max_tokens sets the maximum number of tokens the model is allowed to generate in its reply. It's a hard cap / safety brake.
- If the model finishes naturally before the limit, great — it stops at a stop token.
- If it hits
max_tokensfirst, it gets cut off mid-sentence. - It controls cost (output tokens cost money — Topic 5) and prevents runaway responses.
- Remember it shares the context window with your input (Topic 4): input + max_tokens must fit.
Set max_tokens = 50 and ask for a 1,000-word essay → you'll get an answer
that abruptly stops after about 35–40 words. Set it generously for long answers, but not so high
that you risk cost or overflowing the context window.
8 Inference vs Training, & Prompt versioning/testing
Training is like a child going to school for years to learn — slow, expensive, and it changes what's in their head. Inference is that grown-up now answering your question using what they already learned — fast, and it doesn't change their brain. Every time you use ChatGPT, you're doing inference. The "school" part happened long ago.
Part A — Training vs Inference
These are the two completely different phases of an LLM's life. Mixing them up is the most common beginner confusion, so let's nail it.
| Aspect | Training | Inference |
|---|---|---|
| What it is | Teaching the model by adjusting its billions of parameters on huge text data. | Using the finished model to generate answers. |
| When | Once (or occasionally), ahead of time — the "Pre-trained" in GPT. | Every single time you send a prompt. |
| Does it learn? | Yes — parameters change. | No — parameters are frozen. It does not remember your chat afterwards. |
| Cost & time | Enormous — months, thousands of GPUs, millions of dollars. | Small & fast — fractions of a second to seconds. |
| Who does it | The AI lab (OpenAI, Anthropic, Google…). | You, every time you chat. |
1. Knowledge cutoff. Because the model learned during training, it only knows things
up to its training date. Ask about yesterday's news and it can't know — its "school" ended before then.
(Fixing this is exactly what agents and RAG in later sessions are for.)
2. No memory. Since inference doesn't change parameters, the model doesn't actually
"remember" your conversation. As we saw in Topic 4, the app fakes memory by resending history.
Fine-tuning is a small, extra bit of training done on top of a pre-trained model to specialise it (e.g. for legal or medical writing). It's "training," not inference — and it's a different, more expensive path than just writing a good prompt. We compare fine-tuning vs prompting vs RAG properly in Session 4.
Part B — Prompt versioning & testing
Since you can't easily change the model (that's training), the main way you steer it is the prompt. And because the same model can give very different results from slightly different wording, prompts need to be treated like real work products — written, saved, versioned, and tested.
Exactly like saving versions of code or a document: you keep track of each version of a prompt (v1, v2, v3…), what changed, and how well each performed. So if v3 is worse than v2, you can roll back. Don't just overwrite your prompt and lose the good one.
- v1: "Summarise this." → too vague, inconsistent length.
- v2: "Summarise this in 3 bullet points." → better, but too formal.
- v3: "Summarise this in 3 short, friendly bullet points a beginner can understand." → just right.
Each version is a deliberate, recorded experiment — not random fiddling.
Why test prompts?
Because LLMs can be unpredictable (especially at higher temperature), a prompt that works on one example might fail on another. Prompt testing means running your prompt across many example inputs and checking the outputs are good — before you ship it to real users.
- Build a small set of test inputs (including tricky edge cases).
- Run each prompt version against all of them.
- Judge the outputs (by hand, by rules, or even using another LLM as a judge — see Session 5).
- Keep the version that performs best, and record why.
For consistent testing, use a low temperature so results are repeatable — otherwise the same prompt gives different outputs each run and you can't tell if your prompt improved or you just got lucky.
★ Putting it all together
You just learned the entire foundation of GenAI. Here's the one-paragraph story that connects all 8 topics:
GPT is a generative, pre-trained transformer — an LLM whose only skill is predicting the next token. When you send a prompt, it's chopped into tokens (tokenisation, which is also what you pay for), each turned into a meaning-rich embedding plus a positional encoding for order. The transformer uses attention to let every token consider every other, then predicts a probability for the next token (softmax), and picks one using temperature and top-p, looping until a stop token or max_tokens. All of this is inference on a frozen model that learned everything during training — which is why it has a knowledge cutoff and no real memory, and why we steer it with carefully versioned and tested prompts.
Quick self-check
What does the "T" in GPT stand for, and why does it matter?
Transformer — the neural-network architecture (from "Attention Is All You Need") that uses attention to process all tokens at once. It's the engine behind every modern LLM.
Why does ChatGPT seem to remember your conversation if the model is stateless?
It doesn't truly remember. The app resends the entire conversation history with every new message, so the model re-reads it all each time (which also drives up token cost).
You want factual, repeatable answers. Set temperature high or low?
Low (≈0). High temperature adds creativity and randomness — and more mistakes.
Why can't a base model tell you today's news?
Knowledge cutoff. It only learned from data up to its training date; inference doesn't teach it anything new. Agents and RAG (later sessions) solve this.
What's the difference between a token and a word?
A token is a sub-word chunk (≈4 characters / ¾ of a word). Common words may be one token; rare or long words split into several. The model reads/writes and is billed in tokens, not words.
📚 References & Further Reading
Class material
- 📄 Original course notes / handout (source sheet) — open the shared GenAI class material for this session.
- Class handout: "Introduction to GenAI — GPT, transformers, tokens & embeddings".
Papers, docs & deep dives
- "Attention Is All You Need" (Vaswani et al., 2017) — the paper that introduced the transformer architecture behind GPT.
- Jay Alammar — "The Illustrated Transformer" — visual walkthrough of self-attention and the transformer block.
- Jay Alammar — "The Illustrated GPT-2" — how a decoder-only model predicts the next token step by step.
- OpenAI Tokenizer — interactive tool to see how text splits into tokens and token economics in practice.