Transformers in Practice — BERT & GPT

1 The pre-training paradigm

• Before 2018: every NLP task trained its own model from scratch. After 2018: pre-train once, fine-tune for any task. (As big a shift for NLP as ImageNet was for vision.)
• Pre-training: train a big model on massive unlabeled text (the internet, billions of words — free) to learn grammar, word meanings, world facts.
• Fine-tuning: adapt that model to a specific task with a small labeled dataset (labels are expensive + scarce).
• Why it works: language knowledge transfers — it's reusable across sentiment, NER, QA, translation, etc.
• Analogy: medical school (long, expensive, once) → residency (short, cheap, per specialty).
• Key insight: "If you can fill in blanks correctly, you understand language." Google → BERT, OpenAI → GPT (same insight, two designs).

2 Two philosophies, one Transformer

• BERT (Google): see everything, understand deeply → bidirectional. Like an open-book exam.
• GPT (OpenAI): see the past, predict the future → left-to-right. Like writing an essay.
• Same Transformer — different attention masks. BERT = better at understanding; GPT = better at generating.

3 BERT — Bidirectional Encoder Representations

• Uses only the Transformer encoder (no decoder).
• Key idea: to understand a word you need both sides of context.
• Disambiguation example: "The bank was flooded" → river bank (needs "flooded" on the right); "The bank approved the loan" → financial (needs "loan" on the right). Without right context you cannot disambiguate.

4 Masked Language Modeling (MLM)

• Training task: randomly mask 15% of tokens, predict them from surrounding context. Forces bidirectional understanding (a fill-in-the-blank exam — read the whole sentence to answer).

• Why not 100% [MASK]? At fine-tuning time there are no masks — a 100%-mask model would see a train/test mismatch.
• Why 15%? Balance between enough training signal and enough visible context.

5 BERT architecture & fine-tuning

• Pre-trained on Wikipedia + BookCorpus; ~4 days on 64 TPUs — you do not train it yourself, you download it (HuggingFace).
• Also pre-trained with Next Sentence Prediction (NSP) — later shown to be less important (RoBERTa dropped it).
• Fine-tuning: add a small task head (linear + softmax) on top of pre-trained BERT (frozen or fine-tuned).
  • Classification: [CLS] token → linear → softmax.
  • NER: each token → label.
  • QA: predict start & end positions of the answer span.
• Typical recipe: learning rate 2e-5, 3–5 epochs, 1 GPU, a few hours.
• The revolution: one expensive pre-train → cheap adaptation. ~1000 labeled examples + 3 epochs → 85%+ accuracy; from scratch would need ~100× more data.

6 GPT — Generative Pre-trained Transformer

• Uses only the Transformer decoder.
• Autoregressive: predict the next token given all previous tokens.
• Causal mask: position i can only attend to positions 1 … i (no peeking ahead). "GPT wears blinders."
• Core idea: like your phone's autocomplete — predict the next word.

7 Autoregressive generation

• Training: predict next token at every position (teacher forcing).
• Generation loop: (1) predict next-token distribution → (2) sample → (3) append token → (4) repeat until a stop token. Every ChatGPT word is produced this way — one token at a time.
• Temperature softmax(zᵢ / T): low (0.1) = peaked, deterministic, safe; high (1.5) = flat, creative, risky.
• Top-k / Top-p: sample only from the most likely candidates → prevents nonsense.

8 Scaling & few-shot learning

• Emergent abilities: capabilities appear at scale that weren't explicitly trained — nobody programmed GPT-3 to do arithmetic or translate; it just learned.
• Few-shot / in-context learning (GPT-3): give examples in the prompt → it follows the pattern, with no weight updates.
  • Zero-shot: "Translate English to French: Hello →"
  • One-shot: "Hello → Bonjour. Goodbye →"
  • Few-shot: 5 examples → new query.
• This is why prompt engineering matters — you're programming with examples, purely in context.

9 BERT vs GPT — head to head

10 Tokenization — text to numbers

• Models need numbers, not text. Three strategies:
  • Word-level: "unhappiness" = 1 token. Problem: vocabulary explodes.
  • Character-level: 11 separate tokens. Problem: sequences too long.
  • Subword (BPE): "un" + "happi" + "ness" — best of both worlds.
• Goldilocks rule: common words stay whole; rare words split into reusable pieces ("un-", "-ness" appear in hundreds of words).

Byte Pair Encoding (BPE):

1. Start with a character vocabulary
2. Count all adjacent character pairs in the corpus
3. Merge the most frequent pair into a new token
4. Repeat N times  (N = desired vocab size - initial chars)

Example merges:  l,o,w,e,r -> "lo" -> "low" -> "lower"

• Result: common words stay whole, rare words split into frequent subwords. GPT-2/3/4 use BPE with ~50K vocab.
• WordPiece (BERT's tokenizer): similar to BPE but uses likelihood-based merging (BPE is frequency-based). ## prefix marks continuation tokens: "playing" → ["play", "##ing"]. ~30K vocab for BERT. Different algorithm, same idea — in practice, similar results.

★ Likely exam questions