NN Exam · CNN Architectures

1 The ImageNet timeline

Top-5 error = % of images whose correct label is not in the model's top 5 predictions. Lower is better; human ≈ 5%.
In 3 years CNNs went from barely competing to surpassing humans: Traditional 2011 = 26% → AlexNet 16% → VGG 7.3% → GoogLeNet 6.7% → ResNet 2015 = 3.6%.
Each architecture solved a specific limitation: Scale+GPU (AlexNet) → Deeper (VGG) → Smarter (Inception) → Skip connections (ResNet).

2 LeNet-5 (1998)

Yann LeCun, for handwritten digit recognition (32×32 input, 10 classes).
~60K parameters, 5 layers, sigmoid/tanh activations.
Established the Conv → Pool → Conv → Pool → FC pattern.
The "funnel": spatial shrinks 32→28→14→10→5; channels grow 1→6→16.
Core principles: local connectivity (each neuron sees a small patch), weight sharing (same filter everywhere), subsampling (pooling shrinks spatial size).
Limitation: too small for natural 224×224 images; sigmoid/tanh cause vanishing gradients when deepened.

Key innovation First working CNN — proved local connectivity + weight sharing + subsampling. Set the template every later net builds on.

3 AlexNet (2012)

Won ImageNet 2012 with 16.4% top-5 error (runner-up non-DL: 26.2%) — the "Big Bang" that convinced everyone NNs work at scale.
5 conv + 3 FC layers (8 deep), 60M parameters, split across 2 GPUs.
What it changed (the combination is the lesson, not any one trick):

ReLU — ~6× faster training than tanh, no saturation.
Dropout (0.5) in FC layers — regularization without more data.
Data augmentation — random crops, flips, color jitter.
GPU training — made 60M params feasible.
Local Response Normalization (now obsolete, replaced by BatchNorm).

Key innovation Same recipe as LeNet, 1000× bigger kitchen: ReLU + Dropout + Augmentation + GPU together unlocked deep learning at scale.

4 VGGNet (2014)

Oxford VGG. Core idea: use only 3×3 convolutions, stacked deep. VGG-16 = 13 conv + 3 FC; VGG-19 = 16 conv + 3 FC. 7.3% top-5.
Receptive-field equivalence: two 3×3 = one 5×5; three 3×3 = one 7×7.
Three 3×3 vs one 7×7 (same RF): 45% fewer params + 3× more non-linearity → more expressive.
Design rule: halve spatial, double channels each stage → keeps per-layer compute roughly constant.
Downside: 138M params, of which ~124M live in the FC layers — very wasteful.

One 7×7 filter:   7 × 7 × C²        = 49C²   (1 non-linearity)
Three 3×3 filters: 3 × (3 × 3 × C²)  = 27C²   (3 non-linearities)
Spatial: 224 → 112 → 56 → 28 → 14 → 7
Channels:  64 → 128 → 256 → 512 → 512

Key innovation Small filters + depth beat large filters. Stacking 3×3 gives the same receptive field with fewer params and more non-linearity — underpins all modern CNN design.

5 Inception / GoogLeNet (2014)

Problem (going wider): run 1×1, 3×3, 5×5 filters in parallel and concatenate — but channels explode (e.g. 28×28×256 → 28×28×768, and it keeps growing).
1×1 convolution = a fully-connected layer applied at each pixel; mixes channels without touching spatial dims. Use it to reduce channels cheaply (256 → 64) before the expensive 3×3/5×5 — the bottleneck trick (squeeze then process).
Inception module: parallel 1×1 / 3×3 / 5×5 / pool paths, each preceded by a 1×1 reduce, concatenated depth-wise. "Don't choose the filter size — use them ALL and let the network decide."
Result: 22 layers, only 5M params (28× fewer than VGG), 6.7% top-5, won ImageNet 2014. Smarter > bigger.
Global Average Pooling: instead of flattening into a huge FC layer, average each channel's spatial map into one number → eliminates millions of params (no large FC layers).

Key innovation 1×1 conv bottlenecks (cheap channel reduction) + parallel multi-scale filters + global average pooling → far fewer params at higher accuracy.

6 ResNet + skip connections (2015)

Degradation problem: a 56-layer plain net has higher training error than a 20-layer one. This is NOT overfitting (training error itself is worse) — it's an optimization problem: deep plain nets can't even learn the identity in the extra layers.
Residual learning: instead of learning H(x) directly, learn the residual F(x) = H(x) − x; output is F(x) + x (the skip / shortcut).
If the optimal map ≈ identity, learning F(x)=0 (push weights to zero, easy with weight decay) is much easier than learning H(x)=x directly.
Why skips work: (1) gradient highway — gradients flow straight through the skip, no multiplication chain → no vanishing; (2) easy identity baseline — worst case a layer does nothing, no harm; (3) ensemble effect — behaves like many shallow paths.
Basic block (ResNet-18/34): 3×3 → 3×3 + skip. Bottleneck block (ResNet-50/101/152): 1×1 reduce → 3×3 → 1×1 expand + skip (same 1×1 idea as Inception).
ResNet-152: 3.57% top-5 — superhuman. Won ImageNet + COCO detection + COCO segmentation. "Deeper IS better — when you have skip connections."

Output = F(x) + x        F(x) = H(x) − x  (the residual)
ResNet-18  →  11M  → 10.9%      ResNet-101 → 44M → 6.0%
ResNet-34  →  21M  →  7.8%      ResNet-152 → 60M → 3.6%
ResNet-50  →  25M  →  6.7%

Must-know for exam Skip connection y = F(x) + x solves the degradation problem. Plain deep nets fail to optimize (not overfit); the shortcut gives a gradient highway and makes the identity trivial, enabling 152+ layers.

7 Comparison table

Architecture	Year	Depth	Params	Top-5 err	Key innovation
LeNet-5	1998	5	60K	—	First CNN: conv+pool+FC, weight sharing
AlexNet	2012	8	60M	16.4%	ReLU + Dropout + augmentation + GPU
VGG-16	2014	16	138M	7.3%	Only 3×3 convs, stacked deep
GoogLeNet	2014	22	5M	6.7%	Inception module + 1×1 bottleneck + GAP
ResNet-152	2015	152	60M	3.6%	Skip connections (residual learning)

LeNet→AlexNet: same recipe, 1000× scale. Note GoogLeNet beats VGG with 28× fewer params (5M vs 138M) — smarter, not bigger.

8 Architecture design rules

Use small filters (3×3) stacked deep — more non-linearity, fewer params.
Halve spatial, double channels at each stage — keeps compute balanced.
Use 1×1 convs for channel reduction (bottleneck).
Add skip connections when going beyond ~20 layers.
Global Average Pooling instead of large FC layers at the end.

Conv params: each filter = K × K × C_in weights + 1 bias; a layer with C_out filters = C_out × (K × K × C_in + 1) total.

★ Likely exam questions

Q1. Why do three stacked 3×3 convs have fewer params than one 7×7, despite the same receptive field?

Three 3×3: 3×(3×3×C²) = 27C². One 7×7: 7×7×C² = 49C². That's 45% fewer, plus 3 ReLU non-linearities instead of 1, making the function more expressive.

Q2. A 56-layer plain net has higher training error than a 20-layer one. Why is this NOT overfitting?

Overfitting = low training error but high test error. Here training error itself is higher — the net can't even fit the training data. It's a degradation/optimization problem: deep plain nets can't learn the identity mapping in extra layers. ResNet skip connections fix it.

Q3. What is the role of 1×1 convolutions in the Inception module?

They reduce channel count cheaply (e.g. 256→64) before expensive 3×3/5×5 convs. Without them, concatenating parallel paths makes channels explode at each module, making computation infeasible. A 1×1 conv mixes channels per pixel without touching spatial dims.

Q4. In a residual block, if the optimal map ≈ identity, why is learning F(x)=0 easier than learning H(x)=x?

Pushing all weights toward zero (F(x)=0) is trivial with weight decay/L2. Learning H(x)=x in a multi-layer block requires specific non-trivial weight patterns — much harder for the optimizer.

Q5. GoogLeNet has 5M params vs VGG-16's 138M, yet has better accuracy. Name two reasons.

(1) Global Average Pooling instead of large FC layers eliminates 100M+ params. (2) 1×1 bottlenecks before expensive convs cut computation per Inception module dramatically.

Q6. Why couldn't LeNet-5 be applied directly to ImageNet, and what single innovation most sped up AlexNet's training?

60K params can't model 1000 classes on 224×224 images — too little capacity. ReLU had the biggest impact on training speed (no saturation, ~6× faster than tanh).

Q7. What is the difference between a Basic block and a Bottleneck block in ResNet?

Basic (ResNet-18/34): two 3×3 convs + skip. Bottleneck (ResNet-50/101/152): 1×1 reduce → 3×3 → 1×1 expand + skip — uses 1×1 convs (same idea as Inception) to keep computation manageable in very deep nets.