Regularization

1 Why deep nets overfit

• Overfitting: train loss → 0 while val loss diverges upward. The network memorizes training data (incl. noise) instead of learning patterns.
• Params ≫ data → enough capacity to memorize anything. ResNet-18 = 11M params on CIFAR-10 (50K) = 220 params per sample.
• Zhang et al. 2017: a deep net hit zero training loss on randomly shuffled labels → it can fit pure noise. Overfitting is the default behavior; regularization constrains it to learn patterns.
• Key diagnostic: the gap between train and val loss. Earliest sign of overfitting = val loss stops decreasing while train loss keeps falling. The minimum val loss = the "sweet spot".

2 L2 regularization / weight decay

Penalize large weights — add a squared-weight penalty to the loss:

L_total = L_original + λ · Σᵢ wᵢ²

• λ controls penalty strength, typically 10⁻⁴ to 10⁻². Small λ = mild; large λ = aggressive (risk underfitting).
• Same idea as Ridge regression from ML — now applied to network weights.
• Intuition (budget constraint): without L2 a few weights blow up → network "bets heavily" on specific features → memorizes. With L2 all weights stay moderate, many features used → robust, smoother decision boundary.

Weight decay in SGD — the penalty turns into a shrink factor:

Standard SGD:  w ← w − η · ∇L
With decay:    w ← (1 − ηλ) · w − η · ∇L
                    └─ decay factor ─┘

• Each step weights are multiplied by a number slightly < 1 → they literally decay toward zero (pull the weight vector toward the origin). Hence "weight decay".
• PyTorch: optim.SGD(params, lr=0.01, weight_decay=0.01).

3 Dropout

• Dropout: during each training forward pass, randomly zero each neuron's output with probability p. A different random mask every pass.
• Why it works — prevents co-adaptation: neurons can't rely on specific others being present, so each feature must be independently useful. (Team-project analogy: if random members skip every meeting, everyone must understand the whole task → no single point of failure.)
• Dropout = implicit ensemble: each mask is a different sub-network. With n neurons and p=0.5 there are 2ⁿ possible sub-networks trained simultaneously; inference with all neurons ≈ averaging all of them (same principle as Random Forests, but free inside one net).

Training vs inference (inverted dropout):

• Why scale by 1/(1−p)? If p=0.5 half the neurons are zeroed → output magnitude halves; multiply survivors by 2 so expected output stays the same. (Scaling done at train time = "inverted dropout".)
• Common bug: forgetting model.eval() at test time → dropout stays active → noisy, non-deterministic predictions.

Practical rates: hidden layers p=0.5 (default); input layer p=0.2 (don't drop too much input); after conv p=0.25 or skip; usually don't combine with BatchNorm. ResNets use dropout sparingly (BN handles regularization); Transformers apply it on attention weights + residuals. Too-high p → underfitting.

4 Early stopping

• Monitor validation loss; stop when val loss starts rising (and keep the best checkpoint). Stops training at the sweet spot before overfitting kicks in.
• Essentially free; the simplest regularizer. Use "patience" (wait N epochs of no improvement) to avoid stopping on noise. PyTorch Lightning has an EarlyStopping callback.

5 Data augmentation

• Generate free synthetic training data by applying label-preserving transforms → effectively more data, less memorization.
• Image transforms: RandomHorizontalFlip, RandomCrop(32, padding=4), ColorJitter (via torchvision.transforms / Albumentations).
• Often the single most effective technique on image tasks — listed first in the recommended fix order.

6 Batch normalization

• BatchNorm: normalize each layer's pre-activations over the mini-batch (zero mean, unit variance), then rescale/shift with learnable γ, β.
• Speeds and stabilizes training; the per-batch noise has a mild regularizing effect → less need for dropout (why ResNets lean on BN instead).
• Like dropout, it behaves differently in train() vs eval() (uses running statistics at inference) — always switch modes correctly.

7 The toolkit at a glance

99% train / 72% val? Apply in this order:

1. Data augmentation (flips + crops)
2. Weight decay (AdamW, λ=10⁻⁴)
3. Early stopping
4. Dropout (p=0.3–0.5)

Still overfitting? → get more data or use a smaller model. Combining techniques usually beats any single one.

★ Likely exam questions