The famous CNNs, from a tiny digit reader to 152 layers deep

1 The evolution of CNNs — why architectures matter

From Session 9 you know a convolution slides a small filter over an image to detect patterns (edges, then textures, then shapes). A CNN (Convolutional Neural Network) is just many of these convolution layers stacked, usually with pooling layers (which shrink the image) and a few ordinary fully-connected layers at the end to make the final decision.

So here's the natural question this whole session answers: if a CNN is "just layers stacked," why do we have famous named networks? Why not pick layers at random?

The competition that drove everything: ImageNet

Most of this story is powered by one contest. ImageNet is a giant dataset of ~1.2 million photos labelled into 1,000 categories (dogs, cars, mushrooms…). From 2010–2017 teams competed each year in the ILSVRC challenge to classify these images with the lowest error. Each architecture below was, in its year, the network that smashed the previous record.

A timeline you can hold in your head

2 LeNet & AlexNet — the pioneer and the breakthrough

LeNet-5 (1998) — the pioneer

Long before the AI boom, Yann LeCun built LeNet-5 to read handwritten digits (zip codes, cheque amounts). It's tiny by today's standards — about 60,000 parameters — but it already had every idea a modern CNN uses: convolutions to find patterns, pooling to shrink, and fully-connected layers to decide.

Its layout, on a 32×32 grayscale digit image, was:

Notice the timeless shape: as you go deeper, the image gets smaller (pooling) but richer (more filters / channels). LeNet used the smooth tanh activation and average pooling — choices later networks would improve on.

AlexNet (2012) — the ImageNet breakthrough

AlexNet (by Krizhevsky, Sutskever & Hinton) is essentially "LeNet, but much bigger and trained on ImageNet's 1.2M photos." It has ~60 million parameters (1,000× LeNet) and 8 weight layers: 5 convolutional + 3 fully-connected. When it won ILSVRC 2012 by a huge margin, it kicked off the entire modern deep-learning era. But its size alone wasn't the win — three new ingredients made training such a big net actually possible.

import torch.nn as nn

# One convolutional block, AlexNet-style
block = nn.Sequential(
    nn.Conv2d(3, 96, kernel_size=11, stride=4),  # 3 input colours -> 96 filters
    nn.ReLU(inplace=True),                          # the speed-up trick: max(0, x)
    nn.MaxPool2d(kernel_size=3, stride=2),       # shrink the feature map
)

# The classifier head, with dropout to fight overfitting
head = nn.Sequential(
    nn.Dropout(p=0.5),       # randomly drop 50% of neurons while training
    nn.Linear(9216, 4096),
    nn.ReLU(inplace=True),
    nn.Linear(4096, 1000),  # 1000 ImageNet classes
)

3 VGG & Inception — go deeper, or go wider

VGG (2014) — beautiful simplicity through depth

VGG (from Oxford's Visual Geometry Group) made one elegant bet: use only 3×3 convolutions, stacked deep. No fancy filter sizes — every conv is the same small 3×3, and you just stack 16 or 19 weight layers (VGG-16, VGG-19). Its uniform design made it the easiest architecture to understand and copy.

Inception / GoogLeNet (2014) — go wide and cheap

The same year, Google's GoogLeNet (built from Inception modules) took the opposite tack. Instead of choosing one filter size, why not run several sizes in parallel within the same layer and let the network keep whatever's useful? A single Inception module runs 1×1, 3×3, and 5×5 convolutions plus a pooling path side-by-side, then concatenates (glues together) all their outputs.

This "look at multiple scales at once" design is the multi-scale idea. A small dog and a large dog in different photos can both be caught because some path's filter size matches.

The secret weapon: 1×1 convolutions

The genius trick that makes Inception affordable is the 1×1 convolution. A 1×1 filter doesn't look at neighbours at all (it's a single pixel wide) — so what's the point? It mixes across channels at each pixel, and crucially it can reduce the number of channels cheaply before an expensive 3×3 or 5×5 runs.

4 ResNet & skip connections — making "deeper" work again

The mystery: deeper got worse

By 2015, everyone believed "deeper = better." But researchers found something baffling: a 56-layer plain network performed worse than a 20-layer one — and not just on test data, on the training data too. This is the degradation problem. It's not overfitting (that would show as good training, bad test). The deeper network simply couldn't be trained well at all.

The fix: residual blocks with identity shortcuts

ResNet (by He et al.) introduced the residual block. The idea: instead of asking a few layers to learn the full desired output H(x), ask them to learn only the change (the "residual") F(x) = H(x) − x, then add the original input back: output = F(x) + x. That + x is an identity shortcut (or skip connection) — the input is copied forward and added to the block's output.

import torch.nn as nn

class ResidualBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1)
        self.bn1   = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1)
        self.bn2   = nn.BatchNorm2d(channels)
        self.relu  = nn.ReLU(inplace=True)

    def forward(self, x):
        identity = x                  # save the input for the shortcut
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += identity               # ★ the skip connection: F(x) + x
        return self.relu(out)

5 Design principles & comparison

Now that you've met all five, let's pull out the recurring design principles — the levers every architect pulls — and then compare the networks side by side.

The four levers

The full comparison

Approximate figures for the canonical ImageNet versions (top-5 error, lower is better):

★ Putting it all together

You just walked through the entire history of CNN architectures. Here's the one-paragraph story that connects all five:

Quick self-check

📚 References & Further Reading

Class material

• SST Deep Learning handout (Session 10) — your course handout for this session, covering CNN architectures from LeNet to ResNet.

Papers, docs & deep dives

• SST Deep Learning (course site) — the companion site for this course's notes and materials.
• AlexNet — "ImageNet Classification with Deep CNNs" (Krizhevsky et al., 2012) — the paper that started the modern deep-learning era; see ReLU, dropout, and GPU training in context.
• VGG — "Very Deep Convolutional Networks" (Simonyan & Zisserman, 2014) — the case for stacking small 3×3 convolutions deep; clean and very readable.
• GoogLeNet / Inception — "Going Deeper with Convolutions" (Szegedy et al., 2014) — introduces the multi-scale Inception module and 1×1 bottlenecks.
• ResNet — "Deep Residual Learning for Image Recognition" (He et al., 2015) — the degradation problem and the residual block / skip connection that fixed it.