Teaching a network to remember: how RNNs read things in order

1 Why sequences need a new model

So far in this course we built two big kinds of network. A Multi-Layer Perceptron (MLP) takes a fixed list of numbers and produces an answer. A Convolutional Neural Network (CNN) (Sessions 8–11) takes a fixed-size image and produces an answer. Both share two limits that turn out to matter a lot:

• They expect a fixed-size input. An MLP built for 10 numbers cannot read 7 numbers or 3,000 numbers. But a sentence can be 3 words or 300.
• They have no memory of order. Each input is processed on its own, from scratch. Nothing from the previous input carries over.

Why "order matters" is the whole game

A sequence is just data where the order of the items carries meaning. Shuffle it and you destroy the meaning. Three huge examples:

The core new idea: process step by step, and keep a memory

Instead of swallowing the whole input at once, a Recurrent Neural Network (RNN) reads a sequence one element at a time. After each step it updates a little internal summary of "everything I've seen so far," then carries that summary into the next step. That carried-forward summary is called the hidden state, and it is the network's memory.

2 The RNN cell — hidden state as memory

An RNN is built from one small reusable piece called the RNN cell. The same cell — the same set of weights — is applied at every time step. At each step it takes two things in and produces an updated memory:

• x_t — the input at the current step t (e.g. the current word's vector).
• h_t-1 — the hidden state from the previous step (the memory so far).
• It outputs h_t — the updated hidden state, which is both the memory passed forward and (optionally) used to make a prediction at this step.

The recurrence formula

Here is the heart of the whole session — the recurrence relation. At each step the cell mixes the new input with the old memory and squashes the result:

ht = f( Wx · xt  +  Wh · ht-1  +  b )

Let's name every piece — none of it is mysterious:

If we also want a prediction at a step (say, the next word, or a class), we read it off the hidden state with one more small layer:

yt = softmax( Wy · ht  +  by )

Unrolling through time

It helps to unroll the loop: draw the same cell once per time step, side by side, with the hidden state flowing left to right like a conveyor belt. The picture below is the same cell three times — not three different cells.

The very first hidden state h₀ is usually just a vector of zeros — a blank notepad before reading anything.

import numpy as np

# Learned parameters (random here; training would set them)
W_x = np.random.randn(hidden_size, input_size) # input  → hidden
W_h = np.random.randn(hidden_size, hidden_size) # hidden → hidden
b   = np.zeros(hidden_size)

def rnn_step(x_t, h_prev):
    # the recurrence: mix new input + old memory, then squash
    return np.tanh(W_x @ x_t + W_h @ h_prev + b)

def rnn_forward(inputs):           # inputs: list of x_t vectors
    h = np.zeros(hidden_size)        # h_0 = blank notepad
    states = []
    for x_t in inputs:           # step through the sequence
        h = rnn_step(x_t, h)         # SAME weights every step
        states.append(h)
    return states                   # h_1, h_2, …, h_T

import torch.nn as nn

rnn = nn.RNN(input_size=50, hidden_size=128, batch_first=True)
# x shape: (batch, seq_len, input_size)
outputs, h_final = rnn(x)   # outputs = every h_t; h_final = last h_t

3 Backprop Through Time (BPTT)

Recall from earlier sessions that we train a network with backpropagation: we measure how wrong the output is (the loss), then push that error backwards through the network to compute a gradient for each weight — a number saying "nudge me this way to reduce the loss." Backpropagation Through Time (BPTT) is just ordinary backprop applied to the unrolled RNN.

How BPTT works, step by step

The twist that makes it "through time": because the same weights are used at every step (weight sharing, Topic 2), each weight gets a gradient contribution from every time step. So we compute the contribution at each step and add them all up before updating.

∂L/∂Wh  =  Σt  ( ∂L/∂hT )·( ∂hT/∂ht )·( ∂ht/∂Wh )

where  ∂hT/∂ht  =  ∏k=t+1..T  ∂hk/∂hk-1   # a long product!

Truncated BPTT — making it practical

For a very long sequence (say a whole book, or a long audio clip), unrolling all the steps is far too expensive in memory and time. The standard fix is Truncated BPTT: process the sequence in chunks of, say, 35 steps. You still carry the hidden state forward across chunks (so the memory continues), but you only backpropagate the error within each chunk, not all the way to the start.

h = zeros(...)                  # persistent memory across chunks
for chunk in split(sequence, size=35):
    h = h.detach()              # stop gradients flowing past this chunk
    outputs, h = rnn(chunk, h)  # forward: memory continues
    loss = criterion(outputs, targets_for(chunk))
    loss.backward()             # backprop only within the 35 steps
    optimizer.step()
    optimizer.zero_grad()

4 The vanishing & exploding gradient problem

Remember that long product term from Topic 3: ∂hT/∂ht = ∏ ∂hk/∂hk-1. To send error from step T back to a much earlier step t, BPTT multiplies many small matrices together. Multiplying many numbers in a chain has a dramatic effect:

Why this breaks long-range dependencies

Language is full of long-range dependencies — relationships between words far apart:

Two cures

Exploding gradients are the easy one. We use gradient clipping: if the gradient's total size exceeds a threshold, we scale it back down before the update. It caps the step so the weights can't jump off a cliff.

loss.backward()
# rescale gradients so their combined norm ≤ 5.0
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0)
optimizer.step()

Vanishing gradients are the hard one — clipping doesn't help a signal that's already faded to zero. Partial mitigations help (using the ReLU family instead of tanh, careful weight initialisation), but the real fix is a better cell design with a protected "memory highway" that lets information flow across many steps without being squashed each time.

5 Sequence task types

The flexibility of having a hidden state means we can wire an RNN up in several shapes depending on the task. The shapes are named by inputs-to-outputs:

Many-to-one — sentiment analysis

Read the whole review one word at a time; ignore the intermediate outputs; take only the final hidden state (which has "seen everything") and classify it.

review = ["the", "food", "was", "amazing"]
outputs, h_final = rnn(embed(review))   # h_final = summary of WHOLE review
prediction = classifier(h_final)        # → 👍 positive / 👎 negative

One-to-many — generation

Start from one input (a topic, a starting note, or even nothing) and generate a sequence by feeding each output back in as the next input — the autoregressive trick.

h = seed_state
char = "T"
for _ in range(100):
    out, h = rnn_step(embed(char), h)
    char = sample(out)        # pick next char from the probabilities
    print(char, end="")  # …feed it back in as next input

Many-to-many (seq2seq) — translation

When input and output lengths differ (English "I am hungry" → French "j'ai faim" — different word counts), we use the sequence-to-sequence (seq2seq) design: an encoder RNN reads the whole input into a single summary vector (the context vector), then a decoder RNN generates the output sequence from that summary.

★ Putting it all together

You just learned how a network gains a memory and reads things in order. Here's the one-paragraph story that connects all five topics:

Quick self-check

📚 References & Further Reading

Class material

• SST Deep Learning handout (Session 12) — your course handout for this session, available via the class portal at priyanshsaxena.com/sst-deep-learning-nn.

Papers, docs & deep dives

• Andrej Karpathy — "The Unreasonable Effectiveness of Recurrent Neural Networks" — the classic, fun blog post that shows char-level RNNs generating text; the best intuition-builder for this whole session.
• Stanford CS231n — course notes (RNN / sequence material) — clear, authoritative lecture notes covering RNNs, BPTT, and sequence tasks with diagrams.
• Jeffrey L. Elman — "Finding Structure in Time" (1990) — the foundational paper introducing the recurrent ("Elman") network and the idea of hidden state as memory.
• Goodfellow, Bengio & Courville — Deep Learning, Ch. 10 "Sequence Modeling" — the rigorous textbook treatment of RNNs, BPTT, and the vanishing/exploding gradient analysis.
• Sutskever, Vinyals & Le — "Sequence to Sequence Learning with Neural Networks" (2014) — the seq2seq paper behind the encoder–decoder translation design in Topic 5.
• Hochreiter & Schmidhuber — "Long Short-Term Memory" (1997) — the original LSTM paper that fixes vanishing gradients; the bridge into Session 13.