How a neural net measures its mistakes — and fixes them

1 What a loss function is

A loss function (also called a cost function or objective) is a formula that takes the network's prediction and the true answer and returns a single number: the loss. The bigger the number, the more wrong the network is. Training has exactly one goal — make this number as small as possible.

Why a single number?

Recall from Session 4 that backpropagation needs a starting point to compute gradients — a quantity it can differentiate. The loss L is that quantity. Because it's one number, we can ask the crisp question "if I nudge this weight a tiny bit, does L go up or down?" That answer is the gradient, and the whole training machine runs on it.

Loss vs. cost vs. objective — same family

• Loss — usually the error on one training example.
• Cost — usually the average loss over a batch or the whole dataset.
• Objective — the thing we optimize overall (often the cost, sometimes plus extra penalty terms like the regularization we'll meet in Session 6).

People use these words loosely and often interchangeably. Don't lose sleep over it — they all mean "the number we're trying to shrink."

What makes a good loss function?

• Small when right, large when wrong — so minimizing it actually improves predictions.
• Differentiable (smooth) — backprop needs a slope at every point. A loss with no gradient gives the network no direction to move.
• Matched to the task — a loss for predicting house prices looks different from a loss for sorting photos into "cat / dog / bird." We unpack that in Topic 2.

# prediction and truth, in thousands of dollars
y_pred = 300
y_true = 250

error   = y_pred - y_true        # 50
loss    = error ** 2          # 2500

2 Common losses: MSE & cross-entropy

The two questions are the two great families of machine-learning tasks:

• Regression — predict a continuous number (price, temperature, age). Use Mean Squared Error.
• Classification — pick a category from a fixed set (cat/dog/bird, spam/not-spam). Use Cross-Entropy.

Mean Squared Error (MSE) — for regression

Mean Squared Error averages the squared difference between each prediction and its true value. For n examples:

# MSE formula
MSE = (1/n) * Σ (y_pred_i - y_true_i)²

Two design choices make it work:

• Squaring makes every error positive (so +5 and −5 don't cancel out) and punishes large errors disproportionately — a miss of 10 costs 100, a miss of 20 costs 400.
• Averaging (the 1/n) makes the loss comparable whether you have 10 examples or 10,000.

# average of the squared errors
MSE = (2500 + 400 + 100) / 3 = 1000

Cross-Entropy (log loss) — for classification

For classification the network outputs probabilities — e.g. "I'm 70% sure it's a cat, 20% dog, 10% bird." Cross-entropy (also called log loss) measures how far those predicted probabilities are from the truth. The truth is a one-hot vector: 1 for the correct class, 0 for the rest (e.g. cat = [1, 0, 0]).

# Cross-entropy for one example, C classes
CE = - Σ  y_true_c * log(y_pred_c)

# Because y_true is one-hot, only the correct class survives:
CE = - log(y_pred[correct_class])

So cross-entropy boils down to: take the probability the network assigned to the right answer, and penalise it with −log. Predict 1.0 (perfectly confident and correct) → loss is −log(1) = 0. Predict 0.01 for the right class (confidently wrong) → loss is −log(0.01) ≈ 4.6, a big penalty. The −log shape punishes confident mistakes savagely, which is exactly what we want.

Softmax — turning scores into probabilities

The network's raw outputs are logits — arbitrary numbers, possibly negative. Softmax squashes them into a valid probability distribution (all positive, summing to 1) before cross-entropy looks at them:

# softmax over logits z
softmax(z_i) = exp(z_i) / Σ_j exp(z_j)

In practice the two are glued together (e.g. PyTorch's CrossEntropyLoss applies softmax internally) because computing them jointly is more numerically stable.

# 1) softmax the logits
exp([2.0, 1.0, 0.1]) = [7.39, 2.72, 1.11]
sum                       = 11.22
probs = [0.659, 0.242, 0.099]   # cat=66%, dog=24%, bird=10%

# 2) cross-entropy = -log(prob of correct class = cat)
CE = -log(0.659) = 0.417

import torch
import torch.nn as nn

# regression
mse = nn.MSELoss()
loss_r = mse(pred, target)            # pred, target are floats

# classification (softmax applied INSIDE the loss)
ce = nn.CrossEntropyLoss()
logits = torch.tensor([[2.0, 1.0, 0.1]])
target = torch.tensor([0])   # class index 0 = cat
loss_c = ce(logits, target)           # ≈ 0.417

When to use which

3 Gradient descent variants

From Topic 1, the loss is a landscape and we want the lowest valley. The gradient points in the direction of steepest uphill, so we step the opposite way. That's the core update rule, repeated thousands of times:

# the heart of all training
w = w - learning_rate * gradient_of_loss_wrt_w

The learning rate (often written η, "eta") is the step size. The three variants differ only in how much data they look at before each step.

The three variants

Batch gradient descent

Batch gradient descent computes the gradient over the entire training set before taking one step. Every step points in the true "average downhill" direction, so the path is smooth and stable. The catch: one step requires processing all your data — painfully slow and memory-hungry for large datasets (millions of examples → one tiny step).

Stochastic gradient descent (SGD)

Stochastic gradient descent (SGD) goes to the other extreme: it estimates the gradient from a single random example and steps immediately. ("Stochastic" just means "random.") Steps are lightning-fast and you make many of them, but each step is noisy — one example is a rough guess of the true slope, so the path zig-zags. Surprisingly, that noise is often helpful: the jitter can bounce the network out of shallow bad valleys.

Mini-batch gradient descent

Mini-batch gradient descent is the practical sweet spot used by virtually all modern deep learning. It computes the gradient over a small batch of examples (commonly 32, 64, 128, or 256) before stepping. You get most of the stability of batch GD and most of the speed of SGD, plus it maps perfectly onto GPUs, which love crunching a batch of numbers in parallel.

The speed / noise trade-off

for epoch in range(num_epochs):
    for x_batch, y_batch in dataloader:   # yields batches of, say, 64
        preds = model(x_batch)
        loss  = loss_fn(preds, y_batch)

        optimizer.zero_grad()             # clear old gradients
        loss.backward()                   # backprop (Session 4): fill .grad
        optimizer.step()                  # w = w - lr * grad

Learning-rate intuition — too big vs. too small

The learning rate is the single most important knob in all of training. Get it wrong and nothing else matters.

4 Advanced optimizers

Plain SGD has two weaknesses: it can crawl through long shallow valleys, and it uses the same learning rate for every weight. These optimizers each fix a specific problem. They're all drop-in replacements for the optimizer in our training loop.

Momentum — keep rolling

Momentum keeps a running average of past gradients (a "velocity") and steps in that accumulated direction. Like a heavy ball, it powers through small bumps and flat patches and dampens the zig-zag of SGD.

# Momentum (β ≈ 0.9)
v = β * v + gradient
w = w - η * v

Fixes: slow progress in long valleys and noisy zig-zagging — old gradients in the same direction reinforce; conflicting ones cancel out.

Nesterov momentum — look before you leap

Nesterov accelerated gradient is a smarter momentum: it first jumps ahead in the direction of the existing velocity, then measures the gradient at that lookahead point. By "peeking" where it's about to land, it corrects course earlier and overshoots less.

# Nesterov: gradient evaluated at the lookahead position (w + β·v)
v = β * v + gradient_at(w + β * v)
w = w - η * v

Fixes: momentum's tendency to overshoot the bottom of a valley — the lookahead acts like an early brake.

RMSProp — a custom step size per weight

RMSProp keeps a running average of the squared gradients for each weight, then divides the step by the square root of that. Weights with consistently big gradients get smaller steps; weights with tiny gradients get bigger ones. Every parameter gets its own adaptive learning rate.

# RMSProp (β ≈ 0.9, ε tiny, e.g. 1e-8)
s = β * s + (1 - β) * gradient²
w = w - η * gradient / (sqrt(s) + ε)

Fixes: one global learning rate being wrong for differently-scaled weights; great on the noisy, non-stationary objectives common in deep nets.

Adam — momentum + RMSProp together

Adam (Adaptive Moment Estimation) combines momentum (a running average of gradients, the "first moment") and RMSProp (a running average of squared gradients, the "second moment"). It's the default optimizer for most deep learning today because it just works with little tuning.

Fixes: essentially everything above at once — directional smoothing and per-weight step sizes — plus a bias correction step so the early updates (when the running averages start at zero) aren't artificially tiny.

# Defaults: lr=0.001, β1=0.9, β2=0.999, ε=1e-8
m = 0;  v = 0;  t = 0          # m: momentum, v: RMS term

for each batch:
    t = t + 1
    g = gradient_of_loss

    m = β1 * m + (1 - β1) * g       # 1st moment (mean of grads)
    v = β2 * v + (1 - β2) * g * g   # 2nd moment (mean of squared grads)

    m_hat = m / (1 - β1 ** t)       # bias correction
    v_hat = v / (1 - β2 ** t)

    w = w - lr * m_hat / (sqrt(v_hat) + ε)

Comparison table

5 Learning-rate schedules

From Topic 3 we know a fixed learning rate forces a compromise: big enough to make progress, small enough not to diverge. A learning-rate schedule escapes that compromise by changing the learning rate over time — typically starting larger and decaying it.

Step decay

Step decay cuts the learning rate by a fixed factor every so many epochs — e.g. multiply by 0.1 every 30 epochs. Simple, classic, and easy to reason about (it shows up as clean "stairs" in a learning-rate plot).

# step decay: drop by 10× every 30 epochs
lr = initial_lr * (0.1 ** (epoch // 30))
# epoch 0–29  -> 0.1
# epoch 30–59 -> 0.01
# epoch 60+   -> 0.001

Cosine annealing

Cosine annealing smoothly lowers the learning rate following a cosine curve from the initial value down toward zero over training — fast at first, gentle near the end. No abrupt jumps, and it's a favourite for modern models.

# cosine annealing from lr_max to ~0 over T total steps
lr = lr_max * 0.5 * (1 + cos(π * step / T))

Warmup

Warmup does the opposite at the very start: it ramps the learning rate up from near zero over the first few hundred or thousand steps, then hands off to decay. Why? At the very beginning the weights are random and gradients are erratic; a big step immediately can destabilise training (especially with Adam, whose moment estimates haven't warmed up yet). Warmup eases in gently. It's nearly universal when training Transformers, and often paired with cosine decay ("warmup + cosine").

import torch

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

# cosine annealing over 100 epochs
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
    optimizer, T_max=100)

for epoch in range(100):
    train_one_epoch(model, optimizer)
    scheduler.step()                 # nudge the lr along the cosine curve
    print(epoch, scheduler.get_last_lr())

★ Putting it all together

These five topics are the complete "learning engine" of a neural network. Here's the one-paragraph story that connects them:

Quick self-check

📚 References & Further Reading

Class material

• Your course handout: "SST Deep Learning handout (Session 5)" — the notes that accompany this session, available through your class at priyanshsaxena.com/sst-deep-learning-nn.

Papers, docs & deep dives

• Adam: A Method for Stochastic Optimization (Kingma & Ba, 2014) — the original Adam paper; surprisingly readable and the source of the pseudocode in Topic 4.
• An overview of gradient descent optimization algorithms (Sebastian Ruder) — the single best tour of SGD, momentum, Nesterov, RMSProp, Adam and friends, with clear math and intuition.
• CS231n — Optimization notes (Stanford) — Stanford's classic course notes on loss landscapes and gradient descent; excellent visual intuition.
• Deep Learning Book — Ch. 8: Optimization (Goodfellow, Bengio, Courville) — the authoritative textbook chapter on training deep models, free online.
• PyTorch torch.optim documentation — official reference for every optimizer and learning-rate scheduler used in this session's code.