LSTMs, GRUs & Gating

1 The problem & the fix

• Vanilla RNNs suffer vanishing gradients — can't learn dependencies beyond ~20 steps.
• Cause: backprop multiplies by Whh repeatedly → gradient decays to ~0.
• Need: an architecture where gradients flow without repeated multiplication by W_hh.
• Roadmap: Vanilla RNN (broken) → LSTM (3 gates) → GRU (2 gates) → Bidirectional → Deep/Stacked.

2 Cell state: the conveyor belt

• Cell state C_t: a "conveyor belt" carrying memory across time. Info rides it unchanged unless a gate intervenes.
• Gates are sigmoid networks: output 0 (closed) to 1 (open).
• Only element-wise operations on the belt — no matrix multiplication (this is the key to gradient flow).
• LSTM keeps two states: hidden h and cell C (vs RNN's single h).

3 The three LSTM gates

Forget gate — what to erase from old memory:

f_t = σ( W_f · [h_{t-1}, x_t] + b_f )

• 0 = completely forget, 1 = completely keep; applied element-wise to Ct-1.
• Detail: bf initialized to 1 → biased toward remembering by default.

Input gate — what new info to write (two parts):

i_t = σ( W_i · [h_{t-1}, x_t] + b_i )      (which dims to update)
C~_t = tanh( W_C · [h_{t-1}, x_t] + b_C )   (candidate values, in [-1,1])

• it (sigmoid) picks which dimensions to write; C~t (tanh) is the candidate. Together it ⊙ C~t = new info to add.

Output gate — what to reveal as this step's hidden state:

o_t = σ( W_o · [h_{t-1}, x_t] + b_o )
h_t = o_t ⊙ tanh( C_t )

• ht is a filtered view of the cell state — used for predictions AND passed to next timestep.
• 3 sigmoid activations per cell (forget, input, output); tanh used for candidate and on Ct.

4 Cell-state update equation

C_t = f_t ⊙ C_{t-1} + i_t ⊙ C~_t

• Left term: old memory × forget gate (some stuff erased).
• Right term: new candidate × input gate (some stuff written).
• The combine is ADDITION, not multiplication → gradients flow freely.

5 Why LSTMs fix vanishing gradients

∂C_t / ∂C_{t-1} = f_t       (just the forget gate!)

• If ft ≈ 1: gradient flows perfectly — no decay.
• No W_hh multiplication in the cell-state path; the forget gate learns when to let gradients through.
• Same principle as ResNet skip connections (invented 18 years earlier).

Result: handles sequences 10–100× longer than vanilla RNN, at ~4× the parameters (4 weight matrices vs 1).

6 GRU: fewer gates

• Two simplifications: (1) merge cell & hidden state into one state h; (2) use 2 gates instead of 3 — the update gate does both forget + input.

z_t  = σ( W_z · [h_{t-1}, x_t] )          (update gate)
r_t  = σ( W_r · [h_{t-1}, x_t] )          (reset gate)
h~_t = tanh( W · [r_t ⊙ h_{t-1}, x_t] )   (candidate)
h_t  = (1 - z_t) ⊙ h_{t-1} + z_t ⊙ h~_t   (final state)

• Update gate zt does both in one shot: (1-zt) keeps from old state (like forget), zt takes from candidate (like input).
• Reset gate rt controls how much past state feeds the candidate.
• When zt = 0 → h_t = h_{t-1}: perfect memory, gradient flows unchanged (identity mapping).

7 LSTM vs GRU

• Rule of thumb: start with GRU (simpler, faster). Switch to LSTM for very long dependencies, when GRU underperforms, or with lots of data/compute. In practice the difference is often negligible.

8 Bidirectional RNNs

• Problem: standard RNN/LSTM sees only past context at each position. Sometimes future context helps ("Apple released a phone" — fruit or company?).
• Run two RNNs: one forward →, one backward ←. Output at each position = concatenation of both hidden states → output dim = 2 × hidden_size.

Mnemonic: bidirectional = you already have the whole input; unidirectional = generating one token at a time.

9 Deep / stacked RNNs

• Stack layers: layer-1 output → layer-2 input → ... Each layer learns a different level of abstraction (syntax → semantics → discourse).
• PyTorch: nn.LSTM(input_size, hidden_size, num_layers=3).
• 2015–2017 NLP standard: 2-layer bidirectional LSTM with dropout.

★ Likely exam questions