Loss Functions & Optimization

1 Why the loss function matters

• The loss defines the optimization landscape: different losses → different gradient shapes → different training dynamics.
• A poorly chosen loss can make training impossible.
• Regression → MSE (penalizes errors quadratically). Classification → Cross-Entropy (penalizes confident wrong answers logarithmically).
• "The loss is the exam rubric — it decides what counts as a mistake and how harshly to punish it."

2 Mean Squared Error (MSE)

L = (1/n) Σ (yᵢ − ŷᵢ)²        ∂L/∂ŷ = −(2/n)(y − ŷ)

• Default for regression. Penalizes large errors quadratically → outliers dominate.
• Key property: gradient ∝ error → gradient shrinks as prediction nears target → learning slows near the answer (fine for regression).
• Worked (y=1.0): ŷ=0.20 → grad −1.60; ŷ=0.50 → −1.00; ŷ=0.80 → −0.40; ŷ=0.95 → −0.10. Closer ⇒ smaller gradient.

3 Cross-Entropy Loss (binary)

L = −[ y·log ŷ + (1−y)·log(1−ŷ) ]
when y = 1:  L = −log ŷ        ∂L/∂ŷ = −y/ŷ + (1−y)/(1−ŷ)

• Default for classification. Catastrophic penalty for confident wrong answers.
• Logarithmic penalty (y=1): ŷ=0.9 → L=0.105; ŷ=0.1 → L=2.303; ŷ=0.01 → L=4.605. Being confidently wrong is devastatingly expensive.

4 MSE vs Cross-Entropy & the sigmoid trap

• CE gradients are much larger when wrong → faster learning. Even at ŷ=0.95, CE gradient is 10× MSE.

The gradient problem (sigmoid output ŷ = σ(z)):

MSE w.r.t. z:  ∂L/∂z = −2(y − ŷ)·σ'(z)   ← σ'(z) ≤ 0.25, vanishes when saturated
CE  w.r.t. z:  ∂L/∂z = ŷ − y             ← σ'(z) cancels out!

• MSE + sigmoid → weakest signal exactly when most wrong (saturation). CE + sigmoid → strong signal when most wrong.

5 Multi-class cross-entropy with softmax

Softmax:  ŷₖ = e^(zₖ) / Σⱼ e^(zⱼ)
CE loss:  L = −Σₖ yₖ log ŷₖ
Gradient: ∂L/∂zₖ = ŷₖ − yₖ      ← softmax + log derivatives cancel

• Example, logits [2.0, 1.0, 0.5], target class 1 → softmax [0.506, 0.265, 0.229]; L = −log(0.265) = 1.328; gradient [0.506, −0.735, 0.229] (push true class up, others down).
• PyTorch tip: nn.CrossEntropyLoss expects raw logits — it applies softmax internally.

6 Gradient descent variants

Update rule:  w ← w − η·∇L(w)        (η = learning rate)

Three knobs: how many examples (batch/SGD/mini-batch), learning rate (fixed/adaptive/scheduled), extras (momentum/adaptation).

• Common batch sizes: 32 (small/limited GPU), 64 (default), 128–256 (big GPUs). Powers of 2 align with GPU memory.
• Why not batch size 1? You lose GPU utilization, stable gradient estimates, and batch-norm needs batches.

7 SGD with momentum

• Problem: real loss surfaces are elongated valleys → plain SGD oscillates across the narrow dimension and crawls along the long one.

vₜ = β·vₜ₋₁ + ∇L(wₜ)
wₜ₊₁ = wₜ − η·vₜ
β = 0.9 typically  (effective window ≈ 1/(1−β) = 10 steps)

• Velocity accumulates gradient history: consistent directions build speed, oscillating directions cancel out.
• "A ball rolling downhill, not a person stopping at every step."

8 Adaptive optimizers (AdaGrad, RMSProp, Adam)

• Why: one global η is always a compromise — different parameters need different rates. Solution: per-parameter rates from gradient history.

AdaGrad (2011):  Gₜ = Gₜ₋₁ + gₜ²        wₜ₊₁ = wₜ − η/√(Gₜ+ε) · gₜ
   Flaw: Gₜ only grows → η/√Gₜ → 0 → learning stops before converging.

RMSProp (2012):  sₜ = γ·sₜ₋₁ + (1−γ)·gₜ²   (γ=0.9)   wₜ₊₁ = wₜ − η/√(sₜ+ε) · gₜ
   EMA of squared gradients (old fade) → lr recovers when gradients shrink.

Adam (2015) = momentum + RMSProp + bias correction:
   mₜ = β₁·mₜ₋₁ + (1−β₁)gₜ        v̂ₜ = vₜ/(1−β₂ᵗ)
   vₜ = β₂·vₜ₋₁ + (1−β₂)gₜ²       m̂ₜ = mₜ/(1−β₁ᵗ)
   wₜ₊₁ = wₜ − η · m̂ₜ/(√v̂ₜ + ε)
   Defaults: β₁=0.9, β₂=0.999, ε=1e−8, η=0.001

9 Learning-rate schedules

• Even Adam benefits: early = large lr to explore; late = small lr to settle. Constant lr overshoots the minimum late in training.
• Step decay: drop lr ×0.1 at fixed epochs. Simple, but must choose when.
• Cosine annealing: smooth half-cosine decay; no extra hyperparameters. Used in ResNets, Transformers.
• Warmup + decay: start small, ramp up (1–5% of training), then decay. Early gradients are unreliable (random weights) — large lr amplifies noise. Critical for Transformers (BERT, GPT) and large-batch training.

10 The practical recipe

• Classification → Cross-Entropy; Regression → MSE (or Huber).
• Optimizer → Adam, η=0.001; Schedule → cosine annealing; Batch size → largest power of 2 on GPU.
• Training stalls → reduce lr. Training diverges → reduce lr. Underfitting → bigger model. Overfitting → regularization (next session).
• MNIST case study (784→128→64→10): Adam converges fastest; all reach ~97%. On simple problems the optimizer affects speed more than final accuracy.

★ Likely exam questions