Teaching a network to translate — and to look back

1 The sequence-to-sequence problem

So far our networks did things like "read a sentence, output one label" (is this review positive or negative?) or "read numbers, predict the next number." But a huge family of real problems is different: the input is a sequence and the output is a sequence — and the two can be different lengths. This is the sequence-to-sequence (often written seq2seq) problem.

Where it shows up

Notice the key wrinkle: "I am hungry" is 3 words but "J'ai faim" is 2. The input and output don't line up one-to-one, and we don't know the output length in advance. A plain RNN that emits one output per input step can't handle that. We need an architecture that reads the whole input first, then generates an output of whatever length it needs.

The encoder–decoder architecture

The classic 2014 solution (Sutskever et al., and Cho et al.) splits the job into two RNNs:

• The encoder — an RNN (in practice usually an LSTM or GRU, recall Session 13) that reads the input sequence one token at a time and squeezes everything it understood into a single fixed-size summary vector.
• The context vector — that final summary, often written c. It's the encoder's last hidden state: a fixed-length list of numbers that's supposed to capture the meaning of the whole input.
• The decoder — a second RNN that starts from that context vector and generates the output sequence one token at a time, feeding each word it just produced back in as the next input (just like the autoregressive loop you've seen for language models).

The flow, step by step

Two special tokens make this work. <EOS> ("end of sequence") tells the encoder where the input ends, and is the signal the decoder emits when it's done. A start token, often <SOS> or <BOS>, is the decoder's very first input to kick off generation.

Let's write the math compactly. The encoder runs its recurrence over input tokens x₁ … xₜ, producing hidden states h₁ … hₜ:

class Seq2Seq(nn.Module):
    def __init__(self, vocab_in, vocab_out, emb, hid):
        self.enc_emb = nn.Embedding(vocab_in, emb)
        self.encoder = nn.LSTM(emb, hid)        # reads the source
        self.dec_emb = nn.Embedding(vocab_out, emb)
        self.decoder = nn.LSTM(emb, hid)        # writes the target
        self.out = nn.Linear(hid, vocab_out)    # hidden -> vocab scores

    def forward(self, src, tgt):
        # 1) ENCODE: read whole source, keep only final state
        _, (h, c) = self.encoder(self.enc_emb(src))
        # 'h' is THE context vector — the bottleneck (Topic 2)

        # 2) DECODE: start decoder from the encoder's final state
        dec_in = self.dec_emb(tgt)              # <SOS> J'ai faim ...
        dec_out, _ = self.decoder(dec_in, (h, c))
        return self.out(dec_out)            # scores for each output step

2 The information bottleneck

Look back at the encoder–decoder. Everything the decoder ever sees about the input is squeezed through that single fixed-length context vector c. Whether the input is 3 words or 300, it all has to fit into the same number of slots (say 512 numbers). This is the information bottleneck, and it's the core limitation of plain seq2seq.

Why long sequences suffer most

• Compression loss. 300 words of meaning forced into 512 numbers means the encoder must throw information away. The decoder simply never receives it.
• Recency bias. Because the context is the last hidden state, the words read most recently (the end of the input) are best represented; the beginning has been overwritten through many recurrence steps — the classic vanishing-gradient / forgetting problem from Session 13, now hurting us at the architecture level.
• No way to revisit. Once encoding is done, the original words are gone. If the decoder, halfway through, really needs that one noun from the start of the sentence, it has no way to go back and look — it only has the blurry summary.

3 Attention: letting the decoder look back

Bahdanau et al. (2014) asked: why throw away all the encoder's per-word hidden states and keep only the last one? Instead, keep all of them — h₁ … hₜ, one per input word — and let the decoder build a fresh context vector at every output step, focusing on whichever input words are relevant for the word it's about to produce.

The three ingredients

At decoder step t, with the decoder's current state sᵖ₋₁, attention does three things:

That step-specific cᵖ is then combined with the decoder state to predict the output word: sᵖ = g(sᵖ₋₁, yᵖ₋₁, cᵖ), then P(yᵖ) = softmax(W·[sᵖ; cᵖ]). Crucially, because the weights αᵖᵢ change every step, the decoder reads a different mix of the source for every word it writes. The bottleneck is gone: there's no longer a single vector that has to hold everything.

The flow with attention

# enc_states: (T, hid) — one row per input word
# s_prev:    (hid,)   — decoder state from the last step
scores = enc_states @ W_a @ s_prev      # e_{t,i}: relevance of each input word
alpha  = softmax(scores)                # weights, sum to 1
c_t    = (alpha.unsqueeze(1) * enc_states).sum(0)   # weighted blend
out    = predict(s_prev, y_prev, c_t)   # produce this output word

4 Attention variants & intuition

The key choice in attention is how you compute the alignment score eᵖᵢ = score(sᵖ₋₁, hᵢ). Two famous families appeared within a year of each other.

Additive (Bahdanau) attention

Additive attention, from Bahdanau et al. (2014), feeds the decoder state and an encoder state through a tiny one-hidden-layer neural network and reads off a score:

e_ti = vᵀ · tanh( W₁·s_{t-1} + W₂·h_i )

It's called "additive" because the two inputs are added inside the tanh. The learned vectors W₁, W₂, v let the model figure out, flexibly, what "relevant" means. It's expressive and works even when the encoder and decoder states have different sizes — but the little network makes it slower.

Multiplicative (Luong) attention

Multiplicative attention (also called dot-product attention), from Luong et al. (2015), scores relevance with a simple dot product — essentially asking "how aligned are these two vectors?":

Because a dot product is just one big matrix multiply — and GPUs love matrix multiplies — this is much faster than the additive recipe. That speed is precisely why the dot-product style won out and sits at the heart of the Transformer.

Side by side

Reading the weights: attention as an X-ray

Because the attention weights αᵖᵢ form a tidy grid (output words down one side, input words across the top, each cell a number 0–1), you can plot them as a heatmap and literally see what the model looked at.

★ Putting it all together

Today's session is one tight story about reading and writing sequences — and the single idea that unlocked modern NLP. Here's the whole arc in a paragraph:

Quick self-check

📚 References & Further Reading

Class material

• SST Deep Learning handout (Session 14) — your course handout for this session.

Papers, docs & deep dives

• SST Deep Learning handout (online) — the companion notes for this session's material.
• Sutskever, Vinyals & Le — "Sequence to Sequence Learning with Neural Networks" (2014) — the paper that introduced the LSTM encoder–decoder and the input-reversal trick.
• Bahdanau, Cho & Bengio — "Neural Machine Translation by Jointly Learning to Align and Translate" (2014) — the original attention mechanism (additive); read it for where attention came from.
• Luong, Pham & Manning — "Effective Approaches to Attention-based NMT" (2015) — introduces multiplicative / dot-product and global vs local attention.
• Vaswani et al. — "Attention Is All You Need" (2017) — where attention goes next: the Transformer (preview of Session 15).