The Transformer & Self-Attention

1 The problem RNNs have

• RNNs process one token at a time → cannot parallelize across the sequence.
• Even LSTMs struggle beyond ~200 tokens (long-range info fades).
• Attention helped, but earlier it was still bolted on top of RNNs.
• The question: "What if we used ONLY attention?" → Vaswani et al., 2017, "Attention Is All You Need."

2 Self-attention — the core idea

• Self-attention: every token attends to every other token simultaneously — all positions computed in parallel, no sequential chain.
• Each token asks: "which other tokens are relevant to me?"
• RNN: token n must wait for tokens 1…n−1. Self-attention: all tokens at once.
• Example — "The animal didn't cross the street because it was too tired." → "it" attends most strongly to "animal" (high weight). Patterns are learned from data, not hand-coded coreference.

3 Queries, Keys, Values

• Each token embedding xᵢ is projected into three vectors via separate learned weight matrices:
• Query (q) — "What am I looking for?" · Key (k) — "What do I offer?" · Value (v) — "What info do I carry?"

qᵢ = W_Q · xᵢ
kᵢ = W_K · xᵢ
vᵢ = W_V · xᵢ

• Library analogy: query = your question, keys = book titles, values = book contents. Match query→titles, read the best-matching content.
• Q, K, V come from the same input but different matrices W_Q, W_K, W_V — that's what makes it "self".

4 Scaled dot-product attention

Attention(Q, K, V) = softmax( QKᵀ / √dₖ ) V

• 1. QKᵀ — pairwise similarity scores (dot products between every query and key).
• 2. ÷ √dₖ — scale to prevent softmax saturation.
• 3. softmax — normalize each row into a probability distribution (weights sum to 1).
• 4. × V — weighted combination of value vectors. "Match queries to keys, then read the corresponding values."

Shapes (n tokens, dₖ = dᵥ = 4): Q is n×dₖ; QKᵀ → n×n score matrix; softmax each row; ×V → n×dᵥ output. (e.g. 3×4 · 4×3 = 3×3 scores → ·3×4 = 3×4 output.)

5 Why divide by √dₖ

• Without scaling: dot products grow proportionally to dₖ → large values make softmax very "peaked" → gradients near zero (saturation).
• With √dₖ scaling: normalizes variance back to ≈ 1 → softmax stays in a learnable regime → healthy gradient flow.
• Same principle as Xavier initialization: control the scale of activations to keep gradients healthy.

6 Multi-head attention

• A single head learns one type of relationship; language has many (syntax, semantics, coreference, proximity).
• Solution: run h attention heads in parallel, each with its own smaller W_Q, W_K, W_V.
• How it works: (1) Split dₘₒдₑₗ into h heads, dₖ = dₘₒдₑₗ / h; (2) each head attends independently; (3) concatenate all head outputs; (4) project with W_O back to dₘₒдₑₗ.
• Example: 8 heads × 64 dims = 512 total → same parameter count / total compute as one big 512-dim head, but 8 different "perspectives."
• Specialization emerges from training: e.g. head 1 = subject–verb agreement, head 2 = adjective–noun proximity, head 3 = pronoun→antecedent, head 4 = punctuation boundaries.

7 Positional encoding

• The position problem: self-attention is permutation-invariant — it treats the input as a set. Without position info, "Dog bites man" = "Man bites dog." RNNs got order for free; transformers must explicitly inject it.
• Sinusoidal encoding: add a position-specific vector to each token embedding.

PE(pos, 2i)   = sin( pos / 10000^(2i/d) )
PE(pos, 2i+1) = cos( pos / 10000^(2i/d) )

• Each position gets a unique "fingerprint"; relative positions have consistent patterns; generalizes to sequences longer than seen in training.
• Different frequencies per dimension (low-freq early dims, high-freq later dims). Nearby positions → similar encodings (high dot product); distant → different. Model learns to extract relative position from these signals.

8 Encoder & decoder blocks

Encoder block (stack N = 6):

• Multi-head self-attention → Add & LayerNorm → Feed-Forward Network → Add & LayerNorm.
• Residual connections around each sub-layer (same gradient-flow reason as ResNet).

Decoder block (three sub-layers, not two):

• Masked self-attention — can't see future tokens. Set future positions to −∞ before softmax → softmax(−∞)=0. Position 5 cannot peek at position 6.
• Cross-attention — decoder provides queries; encoder output provides keys & values. "Decoder asks questions, encoder has answers."
• Feed-Forward Network — per-position transformation.

Full architecture: Encoder (×6, input + pos. enc.) → passes K,V → Decoder (×6, masked + cross + FFN, output + pos. enc.) → Linear + Softmax. Encoder reads input; decoder generates output one token at a time.

9 Why transformers beat RNNs (complexity)

• Self-attention complexity: O(n²·d) time. Per-layer: self-attention O(n²d) vs recurrent O(nd²) → attention cheaper when n < d, costlier for very long sequences.
• n=512 manageable (standard BERT); n=2048 (GPT-3 context); n=10000+ problematic → motivates efficient-attention research.
• Trade-off: O(n²) memory, but GPUs handle it for typical lengths — RNN's n sequential steps are slow on GPU, transformer is 1 parallel step. Parallelism wins.
• Original "Attention Is All You Need" base model: 65M params, 8 GPUs, ~12 hours (2017).

★ Likely exam questions