Sequence Modeling: RNNs

1 Why MLPs fail on sequences

• Text, audio, stock prices, video, DNA all have a temporal/order dimension — shuffling destroys meaning. CNNs assume spatial locality; sequences are temporal.
• Three fatal MLP problems:
  1. Fixed input size — MLP needs a predetermined number of input neurons.
  2. No notion of order — position 1 and position 100 are treated as unrelated features.
  3. Parameter explosion — 500 words × 300-dim = 150,000 input neurons.
• Flattening a sentence into one long vector loses all temporal structure.
• What makes sequences special: variable length, order matters ("dog bites man" ≠ "man bites dog"), and long-range dependencies (e.g. "The students who came from France ___ French" → answer "speak" depends on "students" 6 words back).
• Key insight — weight sharing across time: apply the same function at every timestep. Same weights handle word 1 and word 1000 → handles any length with constant parameters.

2 The vanilla RNN & recurrence

• An RNN is a network with memory: it takes the previous hidden state h_t-1 plus input x_t and produces a new hidden state h_t.

h_t = tanh(W_xh · x_t  +  W_hh · h_t-1  +  b_h)

• Unrolling: the folded loop = compact notation; the unrolled chain = what actually happens. It is the same network copied across time with identical weights at every step.
• Three weight matrices (the only learned params, shared across all timesteps):

• Parameter count for length T: MLP is O(T·d²) (grows with length); RNN is O(d²) — constant regardless of length.
• PyTorch: nn.RNN(input_size, hidden_size, batch_first=True); input shape (batch, seq_len, features). Forward returns output (hidden states at all timesteps) and h_n (final hidden state only).

3 Hidden state as memory

• h_t summarizes everything from x₁ to x_t in a fixed-size vector — its job is to compress all past information.
• It is a lossy compression: you can't reconstruct all past inputs. The network learns what to remember during training.
• Karpathy's char-level RNN: hidden units spontaneously learn to track quote open/close, line position, inside-comment-vs-code, indentation — without explicit supervision.
• The bottleneck: a fixed-size vector (e.g. 256 dims) must encode everything. Older info gets overwritten by newer info; finite memory means early tokens get "forgotten" on long sequences — this leads directly to vanishing gradients.

4 Backpropagation Through Time (BPTT)

• The big idea (3 steps): (1) unroll the RNN across all T timesteps; (2) the unrolled RNN looks like a very deep feedforward network (a 100-token sequence ≈ a 100-layer net); (3) apply standard backprop on this unrolled graph. Gradients flow backward through time.
• The chain rule through time — gradient of loss w.r.t. W_hh is a product of Jacobians across timesteps:

∂L/∂W_hh  ∝  ∏ (t=k..T)  ∂h_t / ∂h_t-1

each factor:  ∂h_t/∂h_t-1 = W_hh^T · diag(tanh'(·))

• Truncated BPTT: full BPTT on 10,000 tokens = backprop through 10,000 layers — impractical. Instead, only unroll K steps (common K = 20–100, e.g. 35): process the sequence in chunks, carry h forward, backprop only within each chunk. Trade-off: can't learn dependencies longer than K steps.

5 Vanishing & exploding gradients

• Vanishing intuition: multiply a number < 1 by itself repeatedly → 0. e.g. 0.9¹⁰ ≈ 0.349, 0.9⁵⁰ ≈ 0.005, 0.9¹⁰⁰ ≈ 0.00003. The gradient from the loss barely reaches early timesteps.
• Consequence: the network cannot learn that an input 50 steps ago matters. In practice vanilla RNNs fail on dependencies beyond ~20 tokens.
• Exploding: symptoms are loss → NaN, params → ∞. Fix = gradient clipping: if ‖∇‖ > threshold, scale it down. torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0).
• Clipping fixes exploding, NOT vanishing. Vanishing needs a new architecture → LSTMs/GRUs (gates create "gradient highways" through time).

6 RNN task patterns

★ Likely exam questions