Why training a deep net sometimes just… fails (and how to fix it)

1 Why initialization matters

When you create a neural network, every connection has a weight — a number the network will adjust during training. Before training begins you must give those weights some starting values. That choice is called weight initialization, and beginners usually assume it's harmless. It isn't. A bad init can make a perfectly good architecture fail to learn at all.

There are two classic ways init goes wrong. Let's name them.

Problem A — the symmetry problem (why not all zeros?)

The tempting idea is "just set every weight to 0, that's neutral." It's a disaster. If every weight in a layer is identical, then every neuron in that layer computes the same output, receives the same gradient during backpropagation (the algorithm from Session 3 that pushes error backwards to update weights), and therefore gets updated identically. They stay clones of each other forever.

Note: biases can safely start at 0, because each neuron's randomly-different weights already break the symmetry. It's the weights that must be random.

Problem B — the scale problem (how big should the random numbers be?)

Okay, random — but random how big? This is the subtle part. Each layer multiplies its input by its weights and sums them up. If the weights are too large, those sums grow bigger and bigger as the signal passes through layer after layer (the values explode). If the weights are too small, the sums shrink toward zero layer after layer (the signal vanishes). Either way, by the time the signal reaches the end — or the gradient travels back to the start — it's useless.

import numpy as np

x = np.random.randn(512)            # input vector, ~unit variance

# Case 1: weights too big (std = 0.1, but 512 of them add up)
a = x.copy()
for i in range(50):
    W = np.random.randn(512, 512) * 0.1
    a = W @ a
print("big   :", a.std())   # -> astronomically large (explodes)

# Case 2: weights too small (std = 0.001)
b = x.copy()
for i in range(50):
    W = np.random.randn(512, 512) * 0.001
    b = W @ b
print("small :", b.std())   # -> basically 0 (vanishes)

2 Vanishing & exploding gradients

Recall from Session 3 that we train with backpropagation: we compute the loss at the output, then use the chain rule to figure out how much each weight contributed to that loss — the gradient. The chain rule works backwards layer by layer, and crucially it multiplies a term for every layer it passes through.

Why repeated multiplication is dangerous

Roughly, the gradient that reaches an early layer is a long product of per-layer factors:

grad_early  ≈  f₁ · f₂ · f₃ · … · f_L

         (one factor f per layer, L layers deep)

Now think about what happens to a product of many numbers:

• If each factor is less than 1 (say 0.8), then 0.8⁵⁰ ≈ 0.00001 — the gradient vanishes. Early layers get almost no update signal, so they barely learn.
• If each factor is greater than 1 (say 1.5), then 1.5⁵⁰ ≈ 640,000,000 — the gradient explodes. Updates become huge, weights jump wildly, and the loss often turns into NaN.

Which activations make it worse

One of those per-layer factors is the derivative of the activation function. So your choice of activation (Session 2) directly controls whether gradients survive.

The problem with sigmoid and tanh is saturation: when their input is large (positive or negative), the curve flattens, so their slope (derivative) is nearly 0. A nearly-0 factor, multiplied through many layers, guarantees vanishing gradients. ReLU avoids this on its positive side — its slope is a clean 1 — which is exactly why modern deep nets lean on ReLU and its cousins (recall Session 2's discussion of activations).

import numpy as np

def sigmoid(z):      return 1 / (1 + np.exp(-z))
def sig_deriv(z):    s = sigmoid(z); return s * (1 - s)

# Suppose each layer's activation derivative averages ~0.2
factor = 0.2
for depth in [5, 10, 30]:
    print(depth, "layers -> gradient scaled by", factor ** depth)

# 5  layers -> 0.00032
# 10 layers -> 1.0e-07
# 30 layers -> 1.07e-21   (effectively zero)

Symptoms — how to recognize it in the wild

3 Smart initialization: Xavier & He

Topic 1 told us weights must be random and the right scale. Topic 2 told us "right scale" means keeping per-layer factors near 1. Smart initialization turns that into an exact formula. The guiding principle has a name:

Here's the intuition for the math. A neuron computes a weighted sum of fan-in inputs (the number of inputs feeding into it, written n_in). When you add up n_in independent random products, the variance of the sum grows roughly proportional to n_in. To cancel that growth, we shrink each weight's variance by a factor of 1 / n_in. That single observation gives us both famous initialization schemes.

Xavier / Glorot initialization (for sigmoid & tanh)

Xavier initialization (also called Glorot initialization, from Glorot & Bengio, 2010) balances both the forward signal and the backward gradient, so it uses both fan-in and fan-out (number of outputs). Draw weights with variance:

Var(W) = 2 / (n_in + n_out)

  Normal form : W ~ Normal(0, sqrt(2 / (n_in + n_out)))
  Uniform form: W ~ Uniform(-limit, +limit),
                limit = sqrt(6 / (n_in + n_out))

Xavier assumes the activation is roughly linear around 0 and symmetric — which is true for tanh and sigmoid. Use Xavier with those.

He initialization (for ReLU & friends)

He initialization (from He et al., 2015 — the same group behind ResNet) fixes Xavier for ReLU. The catch: ReLU zeroes out all negative inputs, so on average it throws away half the signal's variance. He compensates by doubling the scale:

Var(W) = 2 / n_in

  Normal form : W ~ Normal(0, sqrt(2 / n_in))
  Uniform form: W ~ Uniform(-limit, +limit),
                limit = sqrt(6 / n_in)

That extra factor of 2 exactly undoes ReLU's halving. Use He with ReLU, Leaky ReLU, and similar.

Which to use with which activation

import numpy as np

def relu(z): return np.maximum(0, z)

x = np.random.randn(512)
a = x.copy()
for i in range(50):
    n_in = 512
    W = np.random.randn(512, 512) * np.sqrt(2.0 / n_in)  # He init
    a = relu(W @ a)
print("He init, 50 ReLU layers -> std =", a.std())
# prints a healthy value of roughly the same order as the input

import torch.nn as nn

layer = nn.Linear(512, 512)
nn.init.kaiming_normal_(layer.weight, nonlinearity="relu")  # He
nn.init.zeros_(layer.bias)

# For tanh/sigmoid layers instead:
nn.init.xavier_uniform_(layer.weight)                            # Xavier

4 Normalization techniques

Good initialization keeps the signal balanced at the start. Normalization keeps it balanced throughout training by actively re-centering and re-scaling the values flowing between layers. The core operation is the same for every variant: take a set of numbers, subtract their mean, divide by their standard deviation (so they have mean 0, variance 1), then let the network re-scale and re-shift them with two learnable parameters, γ (gamma, scale) and β (beta, shift):

normalized = (x - mean) / sqrt(variance + ε)
output     = γ · normalized + β        # γ, β are learned

The tiny ε (epsilon, e.g. 1e-5) just avoids dividing by zero. The only question that separates the variants is: which numbers do we compute the mean and variance over?

Recap: Batch Normalization (from a previous session)

Batch Normalization (BatchNorm), which you met earlier in the course, normalizes across the batch: for each feature, it computes the mean and variance over all the examples in the current mini-batch. So if your batch has 32 images, BatchNorm computes statistics for feature #1 across all 32 images, feature #2 across all 32, and so on.

• It stabilizes training, often lets you use a higher learning rate, and adds a mild regularizing effect (recall regularization from Session 6).
• Catch: it depends on the batch. With tiny batches the statistics are noisy, and at inference time (one example at a time) there's no batch — so BatchNorm keeps a running average of training statistics to use instead. This train/inference difference is a common source of bugs (see Topic 5).

New: Layer Normalization

Layer Normalization (LayerNorm) flips the axis: instead of normalizing each feature across the batch, it normalizes each example across its own features. For a single example with 512 features, LayerNorm computes one mean and one variance over those 512 numbers and normalizes them.

That independence from batch size is exactly why LayerNorm is the standard in Transformers (the architecture behind modern language models). Transformers process variable-length sequences where batch statistics are awkward and inference is often one sequence at a time — LayerNorm sidesteps all of that. As a rule of thumb: BatchNorm dominates in convolutional vision networks (CNNs); LayerNorm dominates in Transformers and other sequence/NLP models.

Why normalization helps gradient flow

Recall Topic 2: gradients explode or vanish because per-layer factors drift away from 1, and activations saturate when their inputs wander into the flat tails. By forcing each layer's inputs back to a tidy mean-0, variance-1 distribution, normalization keeps activations in their responsive, non-saturated region — where derivatives are healthy and gradients pass through cleanly. It smooths the loss landscape so gradient descent can take bigger, more confident steps. It is, in effect, a continuous version of what good init does once.

import torch
import torch.nn as nn

x = torch.randn(8, 512)   # batch of 8 examples, 512 features each

bn = nn.BatchNorm1d(512)   # stats over the 8 examples, per feature
ln = nn.LayerNorm(512)     # stats over the 512 features, per example

print(bn(x).shape, ln(x).shape)   # both -> torch.Size([8, 512])

# A Transformer block uses LayerNorm; a CNN block uses BatchNorm:
transformer_block = nn.Sequential(
    nn.Linear(512, 512),
    nn.LayerNorm(512),     # <- the Transformer choice
    nn.ReLU(),
)

5 Debugging neural nets

A neural net can fail silently: no error message, it just doesn't learn. So we lean on deliberate checks and habits. Here are the essential ones.

Check 1 — Overfit a tiny batch (the single best sanity check)

Before training on your full dataset, take a handful of examples (say 2–10) and try to make the network memorize them perfectly. A correctly-wired network should be able to drive the loss on 5 examples to nearly zero — it has way more than enough capacity. If it can't, your model, loss, or data pipeline has a real bug, and there's no point training on the full set.

# Grab a tiny fixed batch and overfit it on purpose.
x_tiny, y_tiny = x_train[:5], y_train[:5]

for step in range(500):
    optimizer.zero_grad()
    loss = loss_fn(model(x_tiny), y_tiny)
    loss.backward()
    optimizer.step()
    if step % 100 == 0:
        print(step, loss.item())

# GOOD: loss marches toward ~0  -> wiring is sound.
# BAD : loss stalls / is NaN     -> bug in model, loss, or data.

Check 2 — Gradient checking

Gradient checking verifies that your backprop is computing correct gradients, by comparing them to a brute-force numerical estimate. The numerical gradient nudges one weight by a tiny h and measures how the loss changes:

numerical_grad ≈ ( L(w + h) − L(w − h) ) / (2h)     # h ~ 1e-5

If your analytic (backprop) gradient and this numerical estimate agree to several decimal places, your gradients are correct. This was essential when people hand-wrote backprop; today frameworks like PyTorch compute gradients automatically (autograd), so you'll rarely do it by hand — but it's the gold-standard way to validate any custom layer or loss you write yourself.

Check 3 — Watch the loss curves

Plot training and validation loss over time. The shape of the curve is a diagnostic dashboard:

Check 4 — Hunt the common bugs

A few mistakes account for a huge share of "my net won't train" cases:

• Wrong learning rate. The #1 culprit. Too high → divergence/NaN; too low → painfully slow or stuck. Always sweep a few LRs (e.g. 1e-2, 1e-3, 1e-4).
• Data leakage. Information from the validation/test set sneaks into training — e.g. normalizing using statistics computed over the whole dataset, or shuffling before splitting time-series. Symptom: suspiciously great validation scores that collapse in production. Fix: compute all preprocessing stats on the training set only.
• Label mismatch / misalignment. Inputs and labels get shuffled out of sync, or class indices are off by one, or one-hot vs index labels are mixed up. The model is being told the wrong answers. Symptom: loss won't drop even on the tiny-batch test.
• Forgetting model.train() / model.eval(). Dropout and BatchNorm behave differently in the two modes (Topic 4). Forgetting to switch causes weird train/eval gaps.
• Forgetting optimizer.zero_grad(). In PyTorch, gradients accumulate by default; skip the reset and your updates are garbage.
• Unnormalized inputs. Feeding raw pixel values 0–255 instead of scaling them — instantly destabilizes training.

★ Putting it all together

These five topics are really one story about keeping the signal healthy as it flows through a deep network — forwards and backwards. Here's the connected version:

Quick self-check

📚 References & Further Reading

Class material

• SST Deep Learning handout (Session 7) — your course handout for this session, covering initialization, normalization, and debugging.

Papers, docs & deep dives

• SST Deep Learning handout (Session 7) — online — the official class handout page for this session.
• Glorot & Bengio (2010): Understanding the difficulty of training deep feedforward networks — the original paper introducing Xavier/Glorot initialization and the variance-preserving argument.
• He et al. (2015): Delving Deep into Rectifiers — introduces He initialization, tailored to ReLU, and the ×2 variance correction.
• Karpathy: A Recipe for Training Neural Networks — a famous, practical playbook for debugging nets, including the "overfit one batch" advice.
• Ioffe & Szegedy (2015): Batch Normalization — the paper that introduced BatchNorm and the idea of normalizing layer inputs during training.
• Ba, Kiros & Hinton (2016): Layer Normalization — introduces LayerNorm, the normalization used throughout Transformers.