Backpropagation & Computational Graphs

1 The learning problem

• Goal: minimize the loss L. The loss depends on each weight indirectly, through a chain: w → z (pre-act) → h (activation) → … → ŷ → L.
• We need ∂L/∂w for every weight. ∂L/∂w = "if I wiggle this weight a tiny bit, how much does the loss change?" → gives the direction + magnitude of steepest change.
• The tool that computes it: the chain rule. Backprop is just the chain rule applied efficiently to a whole network.

2 The chain rule

• Single variable: if y = f(g(x)) with u = g(x):

dy/dx = (dy/du) · (du/dx)

• Derivative of a composition = product of derivatives along the chain.
• Worked example f(x) = sin(x²): let u = x² → df/du = cos(u), du/dx = 2x ⟹ df/dx = 2x·cos(x²). At x=1 ≈ 1.081 (matches numerical).
• Multi-variable chain (the backprop spine): nudge w → z changes → h changes → ŷ changes → L changes. Multiply all the local effects:

∂L/∂w = (∂L/∂ŷ) · (∂ŷ/∂h) · (∂h/∂z) · (∂z/∂w)

• Each factor is one local gradient — one easy derivative. One hard problem → many easy ones.

3 Computational graphs

• Computational graph = a directed acyclic graph (DAG) representing a computation. Nodes = operations (+, ×, σ, ReLU); edges = data flow.
• Forward pass: evaluate left → right. Backward pass: propagate gradients right → left.
• Why: makes the chain rule mechanical — no staring at giant formulas, just follow the arrows. Every DL framework (PyTorch, TF, JAX) builds & differentiates these.
• Example f(x,y) = (x+y)·(x−y): with a = x+y, b = x−y, f = a·b. At x=3, y=1 → a=4, b=2, f=8.

4 Local gradients (memorize these)

Each node knows how its output depends on its inputs. Know these and you can backprop through anything.

Other useful pieces: ∂z/∂w = x, and for MSE output ∂L/∂ŷ = −(y − ŷ).

5 Forward & backward pass

• Forward: compute values and store all intermediates (e.g. a, b) — they are needed for the backward pass (because a·b gradients swap inputs).
• Backward — the core update rule at each node:

downstream gradient = upstream gradient × local gradient

• Seed: start at the output with ∂f/∂f = 1, then propagate backward node by node. If a variable reaches the output by multiple paths, sum the paths.
• Worked example f=(x+y)(x−y) at x=3, y=1: through × → ∂f/∂a=b=2, ∂f/∂b=a=4; then ∂f/∂x = 2·1 + 4·1 = 6, ∂f/∂y = 2·1 + 4·(−1) = −2. Verify: f = x²−y² → 2x=6, −2y=−2 ✓.

6 The backpropagation algorithm

Four steps, same loop for every architecture:

1. Forward pass — compute all activations & prediction ŷ.
2. Compute loss — e.g. L = ½(y − ŷ)².
3. Backward pass — compute ∂L/∂w for every weight.
4. Update weights — w_new = w_old − η·(∂L/∂w).

"Forward: how wrong are you? Backward: who's responsible? Update: fix it." Repeat thousands of times → converges.

• Worked 2-2-1 MLP (x=[0.5,0.8], y=1.0): forward gives ŷ=0.594, loss L=0.082. Backward starts at output: ∂L/∂ŷ = −0.406, ∂ŷ/∂z₂ = 0.241 → ∂L/∂z₂ = −0.098, then ∂L/∂W₂ = (∂L/∂z₂)·hᵀ.
• All gradients came out negative → since ŷ is below target, weights should increase to reduce loss (gradient descent steps opposite the gradient). After one update ŷ ≈ 0.601 (closer).
• Backward order: the layer nearest the output (W₂) gets its gradient first; gradients flow backward toward the input.
• One forward+backward on one example = an iteration; a full pass over all data = an epoch (typical: 50–100 epochs).

7 Gradient flow: vanishing & exploding

• The chain rule multiplies local gradients across layers: ∂L/∂w = ∏ gᵢ.
• Vanishing: if each gᵢ < 1, the product shrinks to ~0 → early layers stop learning (stuck at random init).
• Exploding: if each gᵢ > 1, the product blows up → loss → ∞/NaN, training diverges. Same root cause, opposite direction.
• Sigmoid problem: max gradient = 0.25. After n layers: 0.25ⁿ → 0.25¹⁰ ≈ 10⁻⁶, 0.25²⁰ ≈ 10⁻¹². Deep sigmoid nets couldn't be trained.
• ReLU fix: gradient = 1 for positive inputs → 1ⁿ = 1, signal stays full strength. ReLU enabled deep learning. (Downside: gradient = 0 for negative inputs → "dead neurons", manageable in practice.)
• Exploding-gradient fixes (Week 3 preview): gradient clipping, careful initialization (Xavier/He), batch normalization.

8 Automatic differentiation (PyTorch autograd)

• How it works: (1) create tensors with requires_grad=True; (2) run operations → PyTorch builds the computational graph; (3) call loss.backward() → traverses the graph in reverse topological order; (4) gradients stored in each tensor's .grad.
• You define the forward pass; PyTorch gives you the backward pass for free — even through if-statements and loops. Each op records its own backward function; .backward() chains them.

output = model(input)
loss = criterion(output, target)
loss.backward()     # compute all gradients
optimizer.step()    # update all weights
optimizer.zero_grad()  # reset for next step

• Everything in this topic happens inside those four lines. Number of gradients = number of parameters (e.g. 109,386 params → 109,386 gradients; scales linearly).

★ Likely exam questions