NN Exam · Seq2Seq & Attention

1 The seq→seq problem

Translation: variable-length input → variable-length output, different lengths, different word orders.
One-to-many and many-to-one word mappings — need any length in → any length out.
What changed (2014→2016): phrase-based statistical MT (choppy) → Neural MT (fluent) via encoder-decoder + attention.

2 Encoder-decoder architecture

Key idea: compress the entire input into a single context vector c, then decode from it.
Encoder (RNN/LSTM/GRU): processes tokens left-to-right, h_t = RNN(x_t, h_{t-1}). Final state h_T = "summary" of the whole input. Produces no output — just a representation.
Decoder: receives c as initial hidden state, generates one token at a time y_1, y_2, … until <END>. Autoregressive — each output depends on all previous outputs.
Teacher forcing: in training, feed the correct previous token (stabilizes early training); at inference, feed the model's own predictions.
Loss = cross-entropy on output tokens. Whole pipeline trained end-to-end (Sutskever et al., 2014).

3 The information bottleneck problem

Entire input is squeezed into ONE fixed-size vector (e.g. 256-dim).
Fine for short sequences; for long ones information is lost — early words get "forgotten" by the time encoding finishes.
Analogy: summarize a 50-page book into one Post-it, then recreate the book from it.
Evidence: BLEU drops sharply with sentence length for basic seq2seq; with attention it stays high regardless of length.

Must-know for exam The basic decoder only sees the encoder's final hidden state — a single fixed-capacity vector. That fixed capacity can't hold all the info in a long input, so early words fade → translation quality degrades with length.

4 The attention mechanism

Key insight: instead of one context vector, give the decoder access to ALL encoder hidden states and let it choose which matter at each step. "Open-book exam instead of closed-book."

At each decoder step t (4 steps):

1. Score:     e_ti = score(s_t, h_i)        for each encoder state h_i
2. Normalize: α_ti = softmax(e_ti)         → attention weights (sum to 1)
3. Context:   c_t  = Σ_i α_ti · h_i         → weighted sum of encoder states
4. Predict:   use [c_t ; s_t] to generate the output token

Must-know for exam A fresh context vector c_t at EVERY decoding step — no more single bottleneck. Inputs: decoder state s_t + all encoder states h_i; output: context c_t. Soft attention (continuous weighted sum) is differentiable → trainable by backprop; hard attention (argmax one position) is not → needs REINFORCE/RL.

Alignment heatmap: the matrix of α_ti shows learned word alignment automatically (no supervision). Diagonal = monotonic; off-diagonal = word reordering between languages.

5 Scoring functions

Bahdanau (additive), 2015: a tiny 1-hidden-layer net; "additive" because it adds transformed states. W_s, W_h, v are learnable. More expressive, slower, better for small dims.
```
score(s, h) = vᵀ · tanh(W_s · s + W_h · h)
```
Luong, 2015: simpler variants that work nearly as well.

Name	Formula	Params
Dot-product	`score = sᵀh`	None
General	`score = sᵀ W h`	`W` matrix
Concat (= Bahdanau)	`score = vᵀ tanh(W[s; h])`	`W, v`

	Dot-product	Additive (Bahdanau)
Params	None	Learnable
Speed	Very fast (matrix multiply)	Slower (extra layer)
Scaling	Scales to large dims; used in Transformers	More expressive, better for small dims

In practice, dot-product wins for modern model sizes. Preview: Transformers use scaled dot-product sᵀh / √d_k.

6 Attention as Query-Key-Value lookup

Query (decoder s_t): what am I looking for?
Keys (encoder h_i): what does each position advertise?
Values (encoder h_i): what gets retrieved?
A differentiable "soft lookup": compare Q to K → similarity → softmax → weighted sum of V. Exactly what Transformers use.

Attn = softmax( Q Kᵀ / √d_k ) · V

7 Impact: before vs. after attention

	Without Attention	With Attention
Context	Single fixed vector	Fresh vector per step
Long sequences	Degrades badly	Stays strong
Interpretability	Black box	Can visualize alignment
BLEU (long)	Drops sharply	Maintains quality

Attention doesn't just improve accuracy — it makes the model interpretable.

★ Likely exam questions

Q1. In a basic encoder-decoder, why does the decoder struggle with long inputs?

It only receives the encoder's final hidden state — one fixed-size context vector. That limited capacity can't hold all the information, and early words are "far away" in the computation graph, so their info fades (the information bottleneck).

Q2. State the steps of attention at decoder step t, with inputs and output.

(1) Score e_ti = score(s_t, h_i) for all encoder states. (2) Normalize α_ti = softmax(e_ti). (3) Weighted sum c_t = Σ_i α_ti h_i. Inputs: decoder state + all encoder states. Output: context vector c_t (then combined with s_t to predict).

Q3. Why is soft attention trainable by backprop but hard attention is not?

Soft attention is a continuous weighted sum → differentiable. Hard attention uses argmax to pick one position → not differentiable, so it needs REINFORCE / RL methods.

Q4. Bahdanau vs. Luong attention — difference and when to use each?

Bahdanau (additive): vᵀ tanh(W_s s + W_h h) — learnable, more expressive, slower. Luong dot-product: sᵀh — no params, faster, scales to large dims. Prefer additive for small models, dot-product for large.

Q5. Map Q, K, V to seq2seq translation.

Query = current decoder state s_t; Keys = encoder hidden states h_i (what each position advertises); Values = also encoder states h_i (retrieved info). Attention weights = similarity(Q, K) via softmax.

Q6. What does a non-diagonal attention heatmap indicate?

Word reordering between source and target languages. The model learns alignment automatically with no supervision — diagonal means monotonic alignment, off-diagonal means reordering.