Regularization — Fighting Overfitting

1 Overfitting vs underfitting — the bias–variance trade-off

When we train a network we always have (at least) two piles of data:

• The training set — the examples the model learns from (its "practice book").
• The validation set (and later a held-out test set) — examples it has never seen, used to check whether it truly learned the pattern.

What we actually care about is generalization: doing well on data the model has never seen. Doing well on the training data alone is worthless — that's just memorising.

The two failure modes

The bias–variance trade-off

These two failure modes have formal names that you'll hear constantly:

• Bias — error from the model being too simple to represent the truth. High bias → underfitting. (Think: trying to fit a straight line through a curve.)
• Variance — error from the model being too sensitive to the exact training data, so it changes wildly if you'd trained on a slightly different sample. High variance → overfitting.

Total expected error on new data decomposes (roughly) as:

# Expected test error
Error = Bias²  +  Variance  +  Irreducible noise
        #  ↑ too simple   ↑ too sensitive   ↑ can't fix

You can't remove the irreducible noise (the world is messy). You can trade between bias and variance, and that's the whole game.

How you actually see overfitting: the learning curves

The classic diagnostic is to plot training loss and validation loss against training time (epochs). Recall from Session 5 that loss is the number the optimizer is trying to push down. Here's the tell-tale diagram:

That growing gap between the two curves is the visual signature of overfitting. An underfitting diagram looks different: both curves plateau high and close together — the model never got good even on training data.

2 Weight decay — L1 & L2 regularization

Recall from Session 5 that training means minimising a loss function — say the prediction error L(w), where w is the giant pile of weights. Weight decay adds a penalty term that grows as the weights grow, so the optimizer is now told: "fit the data and keep the weights small."

L2 regularization (the most common one)

L2 regularization adds the sum of squared weights to the loss:

# L2-regularized loss
L_total(w) = L_data(w)  +  λ · Σ wᵢ²
             #   ↑ fit the data   ↑ penalty for big weights

The knob λ (lambda) is the regularization strength — a hyperparameter you choose. Bigger λ = more pressure to shrink weights = simpler model.

• λ = 0 → no regularization at all (free to overfit).
• λ too large → weights crushed toward zero → the model can't fit → underfit.
• Goldilocks λ → just enough pressure to stay near the sweet spot.

w ← w − η·(∂L_data/∂w)  −  η·2λ·w
  = (1 − 2ηλ)·w  −  η·(∂L_data/∂w)
    #  ↑ every step, w is multiplied by a number < 1 — it DECAYS toward zero

L1 regularization

L1 regularization penalises the sum of absolute values instead of squares:

# L1-regularized loss
L_total(w) = L_data(w)  +  λ · Σ |wᵢ|

This small change has a big consequence: L1 tends to drive many weights to exactly zero, producing a sparse model (lots of dead connections). L2 instead makes all weights small but rarely exactly zero.

# PyTorch: L2 weight decay is built into the optimizer
optimizer = torch.optim.SGD(model.parameters(),
                            lr=0.1,
                            weight_decay=1e-4)  # this IS λ for L2

# L1 has no built-in flag — you add it to the loss by hand:
l1 = 0.0
for p in model.parameters():
    l1 = l1 + p.abs().sum()
loss = data_loss + 1e-5 * l1

3 Dropout

Dropout is one of the most effective and beloved regularizers, introduced by Srivastava et al. in 2014. The idea is almost absurdly simple: during training, on every forward pass, randomly set each neuron's output to zero with probability p (a typical p is 0.5 for hidden layers). The surviving neurons are scaled up to keep the overall signal level steady.

Why on earth does turning off neurons help?

• It breaks co-adaptation. Co-adaptation is when neurons form fragile little cliques that only work if their specific partners are present — a way of memorising. Random dropping forces each neuron to be useful on its own, building redundant, robust features.
• It's like training a huge ensemble for free. Each random pattern of dropped neurons is effectively a different thinned network. Over training you implicitly train an astronomical number of these sub-networks that share weights. At test time, using all neurons approximates averaging all those networks — and ensembles (averaging many models) are a classic way to reduce variance.

The crucial difference between training and test time

This trips everyone up, so go slow. Dropout behaves differently in the two modes:

Because of that scaling, at test time you do nothing special — just run the network normally. This is why frameworks make you flip a switch (model.train() vs model.eval()): it tells dropout which mode to use. Forgetting to call .eval() is a classic bug that makes test predictions randomly jittery.

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Dropout(p=0.5),   # drop 50% of these activations during training
    nn.Linear(256, 10),
)

# TRAINING: dropout is ACTIVE
model.train()
out = model(x_batch)        # random neurons zeroed each call

# EVALUATION: dropout is OFF, full network used
model.eval()
with torch.no_grad():
    preds = model(x_test)   # deterministic, all neurons on

Manually, "inverted dropout" on a layer's activations a looks like:

# Training forward pass, p = prob of DROPPING
mask = (torch.rand_like(a) > p).float()   # 1 = keep, 0 = drop
a = a * mask / (1.0 - p)               # scale up survivors
# Test time: just use `a` as-is, no mask, no scaling

4 Early stopping & data augmentation

Part A — Early stopping

From Topic 1, validation loss falls, bottoms out, then starts rising as overfitting kicks in. The obvious move: stop training at the bottom. Early stopping watches validation performance and halts training once it stops improving — keeping the model from the point where it was best at generalising.

Because validation loss can wobble up and down a little, we use a patience counter: keep going for a few more epochs after the best score, and only stop if it doesn't beat the best. We also save the best weights, not just the last ones.

best_val = float('inf')
patience, wait = 5, 0

for epoch in range(max_epochs):
    train_one_epoch(model, train_loader)
    val_loss = evaluate(model, val_loader)

    if val_loss < best_val:
        best_val = val_loss
        torch.save(model.state_dict(), "best.pt")  # remember the best
        wait = 0
    else:
        wait += 1
        if wait >= patience:        # no improvement for 5 epochs
            print("Early stopping!")
            break

model.load_state_dict(torch.load("best.pt"))  # roll back to the best

Part B — Data augmentation

The deepest cure for overfitting is simply more data — a model can't memorise what it can't keep up with. But collecting and labelling data is expensive. Data augmentation is a clever cheat: create new, slightly-altered copies of your existing examples that are still valid. The model sees more variety and learns the invariances that actually matter.

For images, common augmentations include:

• Horizontal flip — mirror left↔right. A flipped cat is still a cat.
• Random crop — take a random sub-region (often after padding). Teaches the model that the object can be off-centre or partly out of frame.
• Color jitter — randomly nudge brightness, contrast, saturation, hue. A cat under warm lamp light vs cool daylight is still a cat.
• Small rotations, scaling, translation, adding a little noise, random erasing of patches, and modern mixes like Cutout, Mixup and CutMix.

from torchvision import transforms

train_tf = transforms.Compose([
    transforms.RandomCrop(32, padding=4),   # shift the object around
    transforms.RandomHorizontalFlip(),         # mirror 50% of the time
    transforms.ColorJitter(brightness=0.2,
                           contrast=0.2,
                           saturation=0.2),    # lighting changes
    transforms.ToTensor(),
])

# Validation/test: NO random augmentation — just resize & tensor-ise
test_tf = transforms.Compose([transforms.ToTensor()])

5 Batch Normalization

As signals pass through many layers, their scale can drift — getting huge or tiny — which makes training slow and unstable (we'll dig into why deeper, along with weight initialization, in Session 7). Batch Normalization (Ioffe & Szegedy, 2015) tackles this by normalising the activations of a layer across each mini-batch.

What it actually computes

Recall from Session 5 that we train on mini-batches — small groups of examples at a time. For a given activation, BatchNorm looks at its values across all examples in the current mini-batch and standardises them:

# For each feature, over the B examples in the mini-batch:
μ  = mean(x)                 # batch mean
σ² = var(x)                  # batch variance
x̂  = (x − μ) / sqrt(σ² + ε)  # normalize → mean 0, variance 1  (ε avoids /0)
y  = γ · x̂ + β               # scale & shift with LEARNED γ, β

The first three lines force the activation to have mean 0 and variance 1. The last line is the clever part: γ (gamma) and β (beta) are learnable parameters that let the network undo the normalization if that's actually better. So BatchNorm doesn't force a rigid distribution — it gives the network a well-behaved starting point and the freedom to adjust.

Why it speeds up and stabilises training

• No more exploding/vanishing scale between layers → gradients (Session 5) stay well-behaved, so updates are reliable.
• Smoother optimization → you can crank the learning rate up and converge in fewer epochs.
• Mild regularization → because each example's normalization depends on the random other examples in its batch, the network sees slightly noisy activations — a bit like dropout. This is why heavy dropout is often unnecessary when BatchNorm is used.

The train vs inference difference (very important)

Just like dropout, BatchNorm behaves differently in the two modes:

So during training, BatchNorm quietly keeps a moving average of the means and variances it sees; at inference it switches to those frozen statistics. Once again this is governed by model.train() vs model.eval() — forget .eval() and your test predictions will wrongly depend on whatever else happens to be in the batch.

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 256),
    nn.BatchNorm1d(256),   # normalize the 256 activations across the batch
    nn.ReLU(),                # normalize BEFORE the nonlinearity is common
    nn.Linear(256, 10),
)

model.train()   # BN uses this batch's mean/var + updates running stats
loss = criterion(model(x_batch), y_batch)

model.eval()    # BN now uses the stored running mean/var → deterministic
with torch.no_grad():
    preds = model(x_test)

★ Putting it all together

You now have a complete toolkit for the central problem of deep learning. Here's the one-paragraph story tying all five topics together:

Quick self-check

📚 References & Further Reading

Class material

• SST Deep Learning handout (Session 6) — your course handout for this session, available through the class at priyanshsaxena.com/sst-deep-learning-nn.

Papers, docs & deep dives

• Dropout: A Simple Way to Prevent Neural Networks from Overfitting (Srivastava et al., 2014) — the original dropout paper; clear and very readable on the ensemble/co-adaptation intuition.
• Batch Normalization (Ioffe & Szegedy, 2015) — the paper that introduced BatchNorm and showed how it speeds and stabilises training.
• CS231n: Neural Networks Part 2 (Stanford) — an excellent, beginner-friendly walkthrough of regularization, dropout, and data preprocessing.
• Deep Learning Book, Ch. 7: Regularization (Goodfellow, Bengio & Courville) — the definitive, free textbook chapter covering L1/L2, early stopping, dropout and more.
• PyTorch docs: nn.Dropout & nn.BatchNorm — the official API reference for the layers used in this session's code.