1 Pretraining & fine-tuning — the paradigm
Imagine a child who spends years reading every book in a giant library. They don't have a job yet — they're just soaking up how language works, what usually comes next, how ideas connect. That's pretraining. Later, you give that well-read child a specific job, like sorting letters into "happy" and "angry," and they pick it up almost instantly because they already understand language. That quick job-specific lesson is fine-tuning. The hard, slow part (reading the whole library) was already done.
In Session 15 we built the Transformer architecture. But an architecture is just an empty engine — its millions of weights start out random and meaningless. The breakthrough that made transformers so powerful isn't only the architecture; it's the two-stage training recipe wrapped around it.
Stage 1 — Pretraining
Pretraining means training a model on a huge pile of unlabeled text (think: most of the public internet, books, Wikipedia) on a generic task that needs no human labels. The model isn't told "this sentence is positive" or "this is spam." Instead it plays a self-made guessing game — predict a hidden word, predict the next word — over billions of sentences. Because the text itself supplies the right answers, we call this self-supervised learning.
Labeling data by hand is slow and expensive — someone has to read each example and tag it. But the next word in a sentence is a free label: the text already tells us the answer. So we can train on essentially unlimited data without paying anyone to label it. This is the single trick that unlocked giant models.
Stage 2 — Fine-tuning
A pretrained model understands language in general, but it doesn't know your task yet. Fine-tuning takes that pretrained model and trains it a little more on a small, labeled dataset for one specific job — sentiment analysis, spam detection, medical-question answering, whatever you need. You keep almost everything the model already learned and just nudge the weights toward your task.
Why does it work so well?
This recipe is an example of transfer learning — knowledge learned on one task transfers to another. It works because language has deep, reusable structure: grammar, word meaning, world facts, reasoning patterns. A model that learned all that to play the "guess the word" game already holds 95% of what it needs for your task. Fine-tuning just teaches the last 5% — the mapping from "understanding" to "your specific output."
| Aspect | Pretraining | Fine-tuning |
|---|---|---|
| Data | Massive, unlabeled (billions of words) | Small, labeled (hundreds–thousands of examples) |
| Labels | Self-supervised (text is its own label) | Human-provided (the task's true answers) |
| Cost | Enormous — weeks, many GPUs, $millions | Cheap — minutes to hours on one GPU |
| Who does it | Big labs (Google, OpenAI, Meta…) | You, on your own data |
| Goal | General language understanding | One specific downstream task |
Suppose you want to flag angry customer reviews. The old way: collect a million labeled reviews and train a model from scratch — months of work, and it still wouldn't understand sarcasm.
The pretrain/fine-tune way:
- Download a model (e.g. BERT) that already pretrained on billions of words. Free, done.
- Collect just ~2,000 labeled reviews (positive / negative).
- Fine-tune for ~10 minutes on one GPU.
- Result: a classifier that already "gets" English — including tone and context — and now also knows your task. Accuracy that used to need a million examples now needs a couple thousand.
Very large models like GPT can often do a task with no fine-tuning at all — you just describe the task in the prompt (zero-shot) or give a few examples in the prompt (few-shot). That's the application/engineering view, covered in the GenAI track. Here we keep the ML lens: how the model is built and trained.
2 BERT — the bidirectional understander
It's like a fill-in-the-blank puzzle. You see the sentence "I poured milk into my ___ of cereal" and you guess "bowl" — but to guess it, you used the words on both sides of the blank ("poured milk" before, "of cereal" after). BERT learns by playing exactly this game, millions of times. Because it always looks left and right, it becomes great at understanding what a sentence means.
BERT stands for Bidirectional Encoder Representations from Transformers (Google, 2018). The name tells you almost everything once you unpack it.
Encoder-only
Recall from Session 15 that the original Transformer had two halves: an encoder (which reads and understands the input) and a decoder (which generates output). BERT throws away the decoder and keeps only a stack of encoder layers. It's not built to write text — it's built to read and produce a rich, context-aware representation (a set of vectors) for every token.
Bidirectional
This is the key word. When BERT processes a word, its self-attention can look at every other word in the sentence at once — both before and after. That's bidirectional context. (Contrast this with GPT in the next section, which can only look backward.) Seeing the full sentence makes BERT excellent at understanding meaning.
How BERT pretrains: two clever games
BERT can't just predict the next word — if it could see the whole sentence including the next word, the answer would be sitting right there ("peeking"). So it pretrains on two tasks:
1. Masked Language Modeling (MLM). Randomly hide about 15% of the tokens with a
special [MASK] token, then make the model guess the hidden words using both
sides. This is the fill-in-the-blank game from the ELI5.
Original: the cat sat on the mat
Masked input: the cat [MASK] on the [MASK]
BERT must predict: position 3 → "sat", position 6 → "mat".
To get "sat" it leans on "cat … on"; to get "mat" it leans on "sat on the". It needs both directions — exactly the skill that makes it a strong understander.
2. Next Sentence Prediction (NSP). Show BERT two sentences and ask: did sentence B actually follow sentence A in the original text, or is it a random impostor? This teaches it about relationships between sentences (useful for question answering and inference). Later research (e.g. RoBERTa) showed NSP wasn't very helpful and dropped it, but the original BERT used it.
The special tokens
BERT wraps its input in two helper tokens you'll see everywhere:
[CLS]— a "classification" token placed at the very start. After all the attention layers, the vector sitting at[CLS]is treated as a summary of the whole input. To classify a sentence, you attach a small classifier on top of just this one vector.[SEP]— a "separator" placed between two sentences (and at the end) so BERT knows where one piece of text stops and the next begins.
Take pretrained BERT, feed it [CLS] this movie was fantastic [SEP], grab
the output vector at the [CLS] position, and run it through a tiny added
layer that outputs two numbers (positive / negative). Fine-tune on a few thousand labeled reviews.
The big BERT body barely changes; the little classifier on top learns the task. The flow:
What BERT is good at
Because it reads the whole input and builds deep understanding, BERT shines at understanding tasks rather than generation:
- Text classification — sentiment, topic, spam, intent.
- Named-entity recognition — tagging each word as a person, place, date, etc.
- Extractive question answering — pointing to the answer span inside a passage.
- Sentence similarity / search ranking — turning text into meaning vectors.
What it's not for: writing free-form text. It can't naturally "continue" a sentence, because it was never trained to generate — that's GPT's job.
BERT = encoder-only + bidirectional + masked-language-model pretraining. Seeing the whole sentence
at once makes it a powerful reader. You fine-tune it by adding a small head on top of the
[CLS] vector for classification, or on every token for tagging tasks.
[CLS] token
summarizes the input for classification. It's the go-to model for understanding tasks — classification,
NER, extractive QA — not for generating text.
3 GPT — the autoregressive generator
GPT is the world's most well-read autocomplete. You give it "Once upon a…" and it says "time," then reads "Once upon a time" and says "there," then "was," and so on — adding one word at a time until it has written a whole story. It only ever looks backward at what it has written so far, never peeking ahead (it can't — the future words don't exist yet!).
GPT stands for Generative Pre-trained Transformer (OpenAI). Where BERT is built to read, GPT is built to write.
Decoder-only
GPT keeps only the Transformer's decoder stack (minus the part that attended to an encoder). A decoder is designed to generate a sequence one token at a time, so it's the natural choice for text generation.
Autoregressive & the causal mask
GPT pretrains on a beautifully simple task: predict the next token, given all the previous ones. This is called autoregressive generation ("auto" = self, "regressive" = feeding its own output back in). Because it must not cheat by looking at future words, its attention uses a causal mask (also called a look-ahead mask): each position can attend only to itself and earlier positions. This is the one-directional, left-to-right contrast with BERT's bidirectional reading.
Prompt: The sky is
- Step 1 → reads "The sky is" → predicts "blue" → text is now "The sky is blue"
- Step 2 → reads "The sky is blue" → predicts "and" → "The sky is blue and"
- Step 3 → reads "The sky is blue and" → predicts "the" → …continues…
- Eventually it predicts a special end-of-text token and stops.
This is the same one-token-at-a-time loop you may have seen in GenAI Session 1 — the ML reason it works is the causal-masked decoder trained on next-token prediction.
Next-token pretraining is shockingly general
The famous GPT-2 paper was titled "Language Models are Unsupervised Multitask Learners." The insight: if a model gets good enough at predicting the next word across the whole internet, it implicitly learns translation, question answering, summarization, and more — because all of those appear as text patterns. Just predicting the next token, at scale, produces a model that can do many tasks without task-specific training.
What GPT is good at
- Text generation — stories, articles, emails, dialogue.
- Code generation — writing programs token by token.
- Summarization & translation — by generating the rewritten text.
- Open-ended chat & reasoning — the foundation of ChatGPT and friends.
BERT vs GPT — the comparison you must know
| Dimension | BERT | GPT |
|---|---|---|
| Part of Transformer used | Encoder only | Decoder only |
| Direction of attention | Bidirectional (sees left & right) | Causal / left-to-right (sees only the past) |
| Pretraining task | Masked Language Modeling (fill the blank) + NSP | Next-token prediction (autoregressive) |
| Core strength | Understanding / analysis | Generation |
| Typical tasks | Classification, NER, extractive QA, search | Writing, chat, code, summarization, translation |
| Can it generate free text? | No (not trained to) | Yes — that's the whole point |
| How you adapt it | Fine-tune a head on top | Fine-tune, or just prompt (zero/few-shot) |
| Released by | Google (2018) | OpenAI (2018 onward) |
Encoder = reader, decoder = writer. BERT is all reader. GPT is all writer. (There's also a third family — encoder-decoder models like T5 and the original Transformer — that read one text and write another, great for translation. We touched on the full architecture in Session 15.)
The choice is task-driven: if you need to understand or label existing text, reach for a BERT-style encoder. If you need to produce text, reach for a GPT-style decoder. Same Transformer building blocks (Session 15) — different half, different masking, different training game.
4 Tokenization — BPE & WordPiece
A computer can't read letters or words like you do — it only understands numbers. So before any text goes into the model, it's chopped into little puzzle pieces called tokens, and each piece gets a number. Sometimes a piece is a whole word ("cat"), sometimes just part of one ("ization"). The trick is choosing the puzzle pieces wisely so we can build any word — even one the model has never seen — out of pieces it knows.
A token is the basic unit a transformer reads and writes. The component that does the chopping is the tokenizer. Picking the right size of piece is a real design decision, so let's see the two bad extremes first.
Why not just use words? Why not just characters?
| Approach | Problem |
|---|---|
| One token per word | The vocabulary explodes — there are millions of words, plus typos, names, and new words. Worse, any word the tokenizer never saw becomes an out-of-vocabulary (OOV) "unknown" — the model literally can't represent it. |
| One token per character | No OOV problem (only ~100 characters), but sequences get extremely long — "internationalization" becomes 20 tokens. The model wastes effort and its context window fills up fast, and it has to relearn spelling from scratch. |
| Subword units (the sweet spot) | Keep common words whole, break rare words into reusable pieces. Small-ish vocabulary, short-ish sequences, and no OOV — any word can be built from subword pieces (down to single characters if needed). |
Modern tokenizers use subword units: frequent words stay as one token, rare words split into smaller known pieces. This balances a manageable vocabulary against short sequences, and guarantees any text can be represented.
Byte-Pair Encoding (BPE)
Byte-Pair Encoding (BPE) (Sennrich et al., 2016 for NLP) builds the vocabulary with a simple, greedy "merge the most common pair" algorithm. GPT models use a BPE-style tokenizer.
- Start with a vocabulary of individual characters.
- Scan the training text and find the most frequent adjacent pair of symbols.
- Merge that pair into a single new token and add it to the vocabulary.
- Repeat thousands of times, until you reach a target vocabulary size (often ~30k–50k tokens).
Suppose our tiny corpus is the words low, low, lower, newest, widest.
Start as characters. The pair e + s (in "newest", "widest") is very common,
so merge it → es. Now es + t is common → merge to
est. Then l + o → lo, then
lo + w → low. After a few merges our vocabulary
contains useful chunks like low and est.
Now a brand-new word lowest — never seen as a whole — tokenizes cleanly as
["low", "est"]. No OOV, built from known pieces!
WordPiece
WordPiece (used by BERT) is very similar to BPE — it also merges subword
pieces — but instead of merging the most frequent pair, it merges the pair that most increases
the likelihood of the training data (a probability-based score rather than a raw count).
In practice the results look alike. WordPiece marks continuation pieces with a ##
prefix.
BERT's WordPiece tokenizer turns "playing unhappily" into roughly:
["play", "##ing", "un", "##happ", "##ily"]
The ## means "this piece glues onto the previous one with no space." So
"play" + "##ing" = "playing". A word like "play" stays whole (it's common), while rarer "unhappily"
breaks into reusable morpheme-like chunks. Detokenizing just strips the ## and
rejoins.
Surprising consequences (cross-reference: token economics)
Because text is billed and measured in tokens, subword splitting has real-world effects — this is the token economics covered in GenAI Session 1. A few highlights:
- Rough rule of thumb (English): 1 token ≈ 4 characters ≈ ¾ of a word, so 100 tokens ≈ 75 words.
- Rare/long words cost more tokens than common ones — "the" is 1 token, "antidisestablishmentarianism" is many.
- Other languages often need more tokens per meaning, since tokenizers are usually optimized for English.
- The "strawberry" puzzle: early models struggled to count the r's in "strawberry"
because they see chunks like
["straw", "berry"], not individual letters.
GPT-2 onward use byte-level BPE: they start from raw bytes instead of characters, so the tokenizer can represent any Unicode text — emoji, any language, weird symbols — with truly zero OOV. It's the same merge algorithm, just operating on bytes.
##. Because we count and pay per token, splitting choices
drive cost (see GenAI Session 1).
5 Using transformers in practice — the HuggingFace ecosystem
Imagine a giant free app store, but for trained AI brains. Instead of building and training a model yourself (which costs millions), you just go to the store, pick a model someone already trained, and download it. With a couple of lines of code it's running on your laptop. That store is HuggingFace.
You almost never train a transformer from scratch — the pretraining (Topic 1) is far too expensive.
Instead you grab a pretrained one and fine-tune or just use it. The center of this world is
HuggingFace, especially its transformers Python
library and the Model Hub — a public repository of hundreds of thousands of pretrained
models (BERT, GPT-2, and many more).
Two things always travel together
Every model on the Hub comes with its matching tokenizer. You must use the same tokenizer the model was trained with — feeding text split a different way would be like handing someone a book in the wrong alphabet. So you load both.
The pipeline helper bundles tokenizer + model + post-processing into one
call. Great for getting started:
from transformers import pipeline # Downloads a fine-tuned model + its tokenizer, then runs it clf = pipeline("sentiment-analysis") result = clf("Transformers make this so easy!") print(result) # [{'label': 'POSITIVE', 'score': 0.9998}]
When you want control, load the tokenizer and model explicitly. Here's BERT for classification:
from transformers import AutoTokenizer, AutoModelForSequenceClassification import torch name = "distilbert-base-uncased-finetuned-sst-2-english" # 1. Load the matching tokenizer and the pretrained model tokenizer = AutoTokenizer.from_pretrained(name) model = AutoModelForSequenceClassification.from_pretrained(name) # 2. Tokenize: text -> token IDs (note [CLS]/[SEP] added automatically) inputs = tokenizer("This movie was fantastic", return_tensors="pt") print(inputs["input_ids"]) # e.g. tensor([[ 101, 2023, 3185, 2001, 10392, 102]]) # 3. Run the model (inference: no gradients needed) with torch.no_grad(): logits = model(**inputs).logits # 4. Softmax the logits into probabilities, pick the top label probs = torch.softmax(logits, dim=-1) label = model.config.id2label[probs.argmax().item()] print(label, probs.max().item()) # POSITIVE 0.9994
Notice the pieces from earlier topics all show up: the tokenizer adds 101
([CLS]) and 102 ([SEP]), the
model is a pretrained BERT-style encoder, and softmax (recall it from the
decoding math) turns raw logits into probabilities.
The "Auto" classes
You'll see AutoTokenizer, AutoModel,
AutoModelForSequenceClassification, AutoModelForCausalLM
(for GPT-style generation), and more. The Auto prefix means "look at the model's
config and load the right specific class automatically" — so the same code works whether you point it at
BERT, GPT-2, or something else.
| Task | Auto class to use |
|---|---|
| Sentence/text classification | AutoModelForSequenceClassification |
| Token tagging (NER) | AutoModelForTokenClassification |
| Text generation (GPT-style) | AutoModelForCausalLM |
| Fill-in-the-blank (BERT MLM) | AutoModelForMaskedLM |
| Just the raw embeddings | AutoModel |
from transformers import AutoTokenizer, AutoModelForCausalLM tok = AutoTokenizer.from_pretrained("gpt2") model = AutoModelForCausalLM.from_pretrained("gpt2") inputs = tok("In the future, AI will", return_tensors="pt") out = model.generate(**inputs, max_new_tokens=20) print(tok.decode(out[0], skip_special_tokens=True))
The autoregressive loop from Topic 3 is hidden inside .generate() — it
predicts one token, appends it, and repeats up to max_new_tokens.
Always pair a model with its own tokenizer (use the same name for
both). And remember: a downloaded model carries the biases and knowledge cutoff of its pretraining
data — fine-tuning steers it but doesn't erase what it learned.
This session is the ML/architecture view. For the application and engineering side — calling hosted LLM APIs, prompt design, cost and context windows — see the GenAI track (it picks up right where the token economics in Topic 4 left off).
pipeline(); the controlled path is
AutoTokenizer + an AutoModelFor… class chosen for your
task. Tokenizer and model always travel as a pair, and the topics from this session (special tokens,
softmax, the autoregressive loop) all show up in the code.
★ Putting it all together
You now know how the Transformer from Session 15 turns into the tools people actually use. Here's the one-paragraph story connecting all five topics:
Transformers get their power from a two-stage recipe: pretrain on huge unlabeled text with a self-supervised guessing game, then fine-tune on a small labeled dataset for your task (transfer learning). Two famous families split by which half of the Transformer they keep and which game they play. BERT is encoder-only and bidirectional, pretrained by filling in masked words — a powerful understander for classification, NER, and QA. GPT is decoder-only and autoregressive, pretrained to predict the next token with a causal mask — a powerful generator for writing, chat, and code. Both read text that's been chopped into subword tokens by BPE (GPT) or WordPiece (BERT), which keeps vocabularies small, sequences short, and out-of-vocabulary words impossible — and drives the token economics you pay for. In practice you grab a pretrained model and its matching tokenizer from HuggingFace and run it in a few lines, with the same building blocks (special tokens, softmax, the generation loop) showing up right in your code.
Quick self-check
Why is pretraining "self-supervised," and why does that matter?
The training labels come free from the text itself (the next word, or the hidden word), so no humans have to label anything. That lets us train on essentially unlimited data — the key to building giant models.
You need to classify support tickets by topic. BERT or GPT?
BERT (or a BERT-style encoder). Classification is an understanding task, and BERT's
bidirectional encoder with a head on the [CLS] token is built for exactly this.
What stops GPT from "cheating" by looking at future words during training?
The causal (look-ahead) mask: each position can attend only to itself and earlier positions, never to tokens that come after it. That's what makes it autoregressive and left-to-right.
Why use subword tokens instead of whole words or single characters?
Whole words give a giant vocabulary and out-of-vocabulary failures; single characters give very long sequences. Subwords keep common words whole and split rare ones into known pieces — small vocabulary, short sequences, and no OOV (any word can be assembled).
How do BPE and WordPiece differ?
Both build subword vocabularies by merging pieces. BPE (used by GPT) merges the
most frequent adjacent pair. WordPiece (used by BERT) merges the pair that most increases the
training-data likelihood, and marks continuation pieces with ##.
In HuggingFace, why must you load the tokenizer that matches the model?
Each model was trained on text split a specific way, with specific token IDs and special tokens. A different tokenizer would feed the model unfamiliar IDs — like reading a book in the wrong alphabet — and the output would be garbage.
📚 References & Further Reading
Class material
- SST Deep Learning handout (Session 16) — your course handout for this session.
Papers, docs & deep dives
- SST Deep Learning — Neural Nets course page — the companion online notes and resources for this course.
- BERT: Pre-training of Deep Bidirectional Transformers (Devlin et al., 2018) — the original BERT paper; masked-LM and NSP explained by its authors.
- Language Models are Unsupervised Multitask Learners (GPT-2, Radford et al., 2019) — why scaling next-token prediction yields a general-purpose model.
- Neural Machine Translation of Rare Words with Subword Units (Sennrich et al., 2016) — the paper that brought Byte-Pair Encoding to NLP.
- HuggingFace Transformers documentation — official docs for loading models, tokenizers, pipelines, and fine-tuning.