How a neural network actually learns

1 How a network learns — the big loop

Recall from Session 3 that a network is just a stack of layers, each holding a pile of numbers called weights (and biases). Learning means finding good values for those weights so the network's outputs match the right answers. Training is a loop that repeats this four-step cycle millions of times:

Step 1 — the forward pass

The forward pass is just running the network normally: feed in the input, let each layer compute its outputs, and read off the final prediction. We did this in Session 3. Nothing learns yet — we're just guessing.

Step 2 — the loss

The loss (also called the cost or error) is a single number that says how bad the guess was. Big loss = very wrong; loss of zero = perfect. A simple one is the squared error: take the difference between the prediction and the true answer, and square it. We'll explore many loss functions in Session 5; for now just think "one number measuring wrongness."

Step 3 — the backward pass

This is the heart of this whole session. The backward pass (a.k.a. backpropagation) answers one question for every single weight in the network: "if I nudge you up a little, does the loss go up or down, and by how much?" That "how much" is called a gradient.

Step 4 — the update (gradient descent)

Now we take the downhill step. Gradient descent updates each weight by moving it a tiny bit in the direction that reduces the loss — i.e. opposite to the gradient (because the gradient points uphill):

# the single most important formula in deep learning
w_new = w_old - learning_rate * (∂Loss / ∂w)

The learning rate is the size of your step. Too small and you crawl downhill forever; too big and you leap right over the valley and bounce around. (We tune it properly in Session 5.)

w_new = 0.80 - 0.1 × (+2.0)
      = 0.80 - 0.20
      = 0.60

2 The chain rule & computational graphs

A computation is a graph

Any calculation can be drawn as a computational graph: a diagram of nodes (operations like + or ×) connected by arrows that carry values. Inputs flow in on the left; the final result comes out on the right. A neural network is just a very big computational graph.

Let's build a tiny one. Consider the function:

f = (a + b) × c

We break it into two simple steps, giving the intermediate a name:

d = a + b      # node 1: an addition
f = d × c      # node 2: a multiplication

Local gradients — each node minds its own business

The beautiful trick: each node only needs to know how its own output changes when its own inputs wiggle. These are called local gradients, and they're always simple, even when the whole function is huge. The standard derivative rules are all you need:

The chain rule composes them

To find how the final output f changes when an early input a changes, we can't read it off directly — a doesn't touch f directly, it goes through d. The chain rule says: multiply the local gradients along the path.

∂f/∂a = (∂f/∂d) × (∂d/∂a)

Read it as: "how f reacts to d" times "how d reacts to a." The intermediate d cancels conceptually, leaving the total effect of a on f. This is the entire mathematical engine of backpropagation.

d = a + b = 2 + 3 = 5
f = d × c = 5 × 4 = 20

∂f/∂d = c = 4
∂f/∂c = d = 5
∂d/∂a = 1
∂d/∂b = 1

∂f/∂a = (∂f/∂d)(∂d/∂a) = 4 × 1 = 4
∂f/∂b = (∂f/∂d)(∂d/∂b) = 4 × 1 = 4
∂f/∂c = 5                              # direct, one node away

3 Backpropagation, step by step

Now we do the real thing: a complete forward and backward pass on a tiny 2-layer network, with every number worked out. Take it slowly — this is the most important worked example in the whole course.

The network

One input x, one hidden neuron, one output neuron, and we'll use the sigmoid activation σ(z) = 1/(1+e⁻ᶻ) on each (recall sigmoids from Session 3). Squared-error loss against a target y.

# Layer 1 (hidden)
z1 = w1 · x  + b1
h  = σ(z1)

# Layer 2 (output)
z2 = w2 · h  + b2
o  = σ(z2)

# Loss (½ makes the derivative clean)
L  = ½ (o - y)²

A handy fact we'll lean on: the derivative of sigmoid is σ'(z) = σ(z)·(1 − σ(z)). So if a neuron's output is h, then ∂h/∂z1 = h·(1−h). Neat — no exponentials needed once you already have the output.

Forward pass (pick concrete numbers)

z1 = w1·x + b1 = 0.5·1.0 + 0 = 0.500
h  = σ(0.500)        = 0.6225        # 1/(1+e^-0.5)

z2 = w2·h + b2 = 0.5·0.6225 + 0 = 0.31124
o  = σ(0.31124)     = 0.5772        # the prediction

L  = ½(o - y)² = ½(0.5772 - 0)² = ½(0.33316) = 0.16658

Backward pass — propagate the gradient right-to-left

We walk backward through the graph, applying the chain rule at each node. At every step we keep a running gradient "of the loss with respect to this value," then multiply by the next local gradient as we move left.

∂L/∂o = (o - y) = 0.5772 - 0 = 0.5772

∂o/∂z2 = o(1-o) = 0.5772 × (1 - 0.5772) = 0.5772 × 0.4228 = 0.2440
∂L/∂z2 = ∂L/∂o × ∂o/∂z2 = 0.5772 × 0.2440 = 0.14086

∂L/∂w2 = δ2 × h = 0.14086 × 0.6225 = 0.08769
∂L/∂b2 = δ2 × 1 = 0.14086

∂L/∂h = δ2 × w2 = 0.14086 × 0.5 = 0.07043

∂h/∂z1 = h(1-h) = 0.6225 × (1 - 0.6225) = 0.6225 × 0.3775 = 0.23499
∂L/∂z1 = ∂L/∂h × ∂h/∂z1 = 0.07043 × 0.23499 = 0.01655

∂L/∂w1 = δ1 × x = 0.01655 × 1.0 = 0.01655
∂L/∂b1 = δ1 × 1 = 0.01655

We now have a gradient for every parameter. Notice the pattern: at each layer the blame δ is "current running gradient × local sigmoid slope," then we spread it onto that layer's weights (× the input feeding the weight) and pass a slimmer version further back (× the weight). That repeating shape is why it's called back-propagation.

The update step

w2 ← 0.5 - 1.0 × 0.08769 = 0.41231
b2 ← 0.0 - 1.0 × 0.14086 = -0.14086
w1 ← 0.5 - 1.0 × 0.01655 = 0.48345
b1 ← 0.0 - 1.0 × 0.01655 = -0.01655

The same thing in code

# Backprop by hand in plain Python — the example above
import math

def sigmoid(z): return 1 / (1 + math.exp(-z))

x, y = 1.0, 0.0
w1, b1, w2, b2 = 0.5, 0.0, 0.5, 0.0

# --- forward ---
z1 = w1 * x + b1
h  = sigmoid(z1)
z2 = w2 * h + b2
o  = sigmoid(z2)
L  = 0.5 * (o - y) ** 2

# --- backward (chain rule, right to left) ---
dL_do  = (o - y)                 # 0.5772
dL_dz2 = dL_do * o * (1 - o)     # 0.14086  (= delta2)
dL_dw2 = dL_dz2 * h              # 0.08769
dL_db2 = dL_dz2                  # 0.14086
dL_dh  = dL_dz2 * w2             # 0.07043
dL_dz1 = dL_dh * h * (1 - h)      # 0.01655  (= delta1)
dL_dw1 = dL_dz1 * x              # 0.01655
dL_db1 = dL_dz1                  # 0.01655

# --- update (gradient descent, lr = 1.0) ---
lr = 1.0
w2 -= lr * dL_dw2;  b2 -= lr * dL_db2
w1 -= lr * dL_dw1;  b1 -= lr * dL_db1

4 Automatic differentiation

Automatic differentiation ("autodiff" or "autograd") is how frameworks like PyTorch and TensorFlow compute exact gradients with no hand-derived formulas from you. It is not the same as the two things people confuse it with:

It records the graph as you compute

When you run a forward pass in PyTorch, it secretly builds the same computational graph from Topic 2, node by node, remembering each operation and the local gradient it will need later. This recorded graph is sometimes called the tape (TensorFlow even names its tool GradientTape). When you ask for gradients, it replays the tape backward — exactly the right-to-left walk we did by hand in Topic 3.

Forward-mode vs reverse-mode

There are two directions you can apply the chain rule along the graph:

PyTorch autograd — our example, automatically

Here's the very same 2-layer computation from Topic 3, but now PyTorch computes all the gradients for us. We just mark which numbers need gradients with requires_grad=True, run the forward pass, and call .backward().

import torch

# parameters we want gradients for (requires_grad=True)
w1 = torch.tensor(0.5, requires_grad=True)
b1 = torch.tensor(0.0, requires_grad=True)
w2 = torch.tensor(0.5, requires_grad=True)
b2 = torch.tensor(0.0, requires_grad=True)
x  = torch.tensor(1.0)
y  = torch.tensor(0.0)

# forward pass — autograd records the graph as this runs
h = torch.sigmoid(w1 * x + b1)
o = torch.sigmoid(w2 * h + b2)
L = 0.5 * (o - y) ** 2

# one call walks the tape backward and fills every .grad
L.backward()

print(w1.grad, b1.grad, w2.grad, b2.grad)
# tensor(0.0166) tensor(0.0166) tensor(0.0877) tensor(0.1409)
# — identical to the numbers we derived by hand!  🎉

The printed gradients match our hand calculation exactly: ∂L/∂w1 = 0.0166, ∂L/∂w2 = 0.0877, ∂L/∂b2 = 0.1409. Autograd did in one line what took us six steps — and it would do the same for a network with a billion weights.

★ Putting it all together

You now understand the single most important algorithm in deep learning. Here's the one-paragraph story that ties all four topics together:

Quick self-check

📚 References & Further Reading

Class material

• SST Deep Learning handout (Session 4) — your course handout for this session.

Papers, docs & deep dives

• Rumelhart, Hinton & Williams (1986) — "Learning representations by back-propagating errors" — the landmark paper that put backpropagation on the map; remarkably readable.
• Stanford CS231n — Backpropagation notes — the clearest gateway explanation, built entirely around computational graphs and local gradients.
• Andrej Karpathy — "Yes you should understand backprop" — why treating .backward() as a black box will burn you, with vivid failure cases.
• PyTorch — Autograd basics tutorial — the official intro to requires_grad and .backward() used above.
• Karpathy — "The spelled-out intro to neural networks and backpropagation" (micrograd) — builds a tiny autodiff engine from scratch; the perfect companion to Topic 4.