1 Why not just use an MLP for images?
Imagine you had to describe a photo of a cat to a friend, but the only thing you're allowed to say is the brightness of every single dot, one by one, in order: "dot 1 is grey, dot 2 is grey, dot 3 is orange…" for a million dots. That's exhausting, and worse — if the cat slides one step to the left, every dot changes and your whole description is useless! That's exactly the trouble a plain network has with pictures. We need a smarter way to look that notices shapes, not just dots-in-a-row.
Recall from the earlier sessions that an MLP (multi-layer perceptron) is a stack of fully-connected layers: every neuron in a layer is wired to every value coming in. That's fine for, say, 10 features describing a house. But an image is a grid of numbers, and the moment we flatten that grid into one long list to feed an MLP, three serious problems appear.
An image is a grid of numbers
A grayscale image is a 2-D grid of pixels, each a brightness value
(often 0–255). A colour image is three such grids stacked — one each for Red, Green, Blue — which we
call channels. So a colour image has shape
height × width × channels. A small phone photo might be
224 × 224 × 3.
Problem 1 — parameter explosion
To feed an image into an MLP we must flatten it into one long vector. A
224 × 224 × 3 image flattens to 150,528 numbers. If the first
hidden layer has just 1,000 neurons, every one of them connects to all 150,528 inputs:
# Parameters in the FIRST layer alone inputs = 224 * 224 * 3 # = 150,528 pixel values neurons = 1000 weights = inputs * neurons # = 150,528,000 (~150 million) biases = neurons # = 1,000 total = weights + biases # > 150 MILLION weights — for ONE layer
Over 150 million parameters before we've done anything useful. That's slow to train, hungry for memory, and almost guaranteed to overfit (memorise the training images instead of learning general patterns — see Session 6).
Problem 2 — it throws away spatial structure
Flattening destroys geometry. In the grid, two pixels that sit side by side are neighbours; once flattened, that pixel and the one directly below it end up 224 slots apart in the list. The MLP has no idea they were ever close. But in vision, nearby pixels are everything — edges, corners, and textures are all local patterns. An MLP starts blind to locality.
Problem 3 — it isn't translation-invariant
If the network learned to spot a cat's ear in the top-left corner, an MLP has learned a separate set of weights for that exact location. Move the same ear to the bottom-right and those weights are useless — it must learn the "ear detector" all over again for every position. We say the MLP is not translation-invariant: shifting the object around the frame breaks it. Real objects move around, so this is fatal.
Train an MLP on handwritten digits that always sit dead-centre, and it does fine. Now show it a "7" shoved into the corner. To the MLP, almost every input value is different, so it may confidently call it a "1" or nonsense — even though you instantly see it's still a 7. The shape is the same; only the position changed. The MLP never learned "a 7 is a 7 wherever it is."
What if, instead of one neuron staring at the whole image, we used a small detector that looks at one little patch at a time and then slides across the whole image reusing the same weights everywhere? That detector would be tiny (few parameters), respect locality (it sees neighbours), and find its pattern anywhere (translation-invariant). That detector is a convolution — the subject of this entire session.
Not at all. We still use fully-connected layers at the end of a vision network to make the final decision. The trick is to let convolutions do the heavy "looking" first, and let an MLP do the small "deciding" last. We'll see exactly this layering in Session 10 on CNN architectures.
2 The convolution operation
Imagine a little magnifying glass that can only see a 3×3 square at a time. You slide it across a picture, square by square, and at every stop it asks one question: "How much does this little patch look like the thing I'm hunting for?" — say, a vertical line. It writes down a score at each stop. When you've covered the whole picture, your scores form a new little map showing where the vertical lines are. That magnifying glass is a filter, and sliding it around is a convolution.
Filters / kernels
A filter (also called a kernel) is a small grid of learnable weights — commonly 3×3 or 5×5. Each filter is a pattern detector. One might respond strongly to vertical edges, another to a blob of colour, another to a corner. Crucially, the network learns these weights during training (Session 5's backpropagation) — we don't hand-design them.
The operation: slide, multiply, sum
At every position the filter visits, we do a dot product: multiply each filter weight by the pixel it's sitting on, then add all those products into a single number. That number is one output value. Slide the filter over by one position and repeat. The grid of all these output numbers is called a feature map (or activation map): a map of "how strongly was this pattern found here?"
The math, written out
For an input image I and a kernel K of size
k × k, the output value at row i, column
j is the sum of element-wise products over the patch:
# Output at position (i, j) — a single dot product
S(i, j) = sum over m, n of: I(i + m, j + n) * K(m, n)
where m, n range from 0 .. k-1
Strictly, true mathematical convolution flips the kernel first. What deep-learning libraries actually compute (and what we show here) is cross-correlation — the same thing without the flip. Since the kernel weights are learned anyway, the flip makes no practical difference, so everyone just calls it "convolution." Don't let the terminology trip you up.
A tiny numeric example, fully worked
Let's convolve a 4×4 image with a 3×3 vertical-edge detector (a famous classic kernel). With no padding and a step of 1, a 3×3 filter fits in a 4×4 image in a 2×2 set of positions, so the output is 2×2.
# INPUT image (4x4) # KERNEL (3x3): vertical-edge detector 3 0 1 2 1 0 -1 5 4 2 1 1 0 -1 1 2 3 4 1 0 -1 0 1 2 3 # --- Top-left position: overlay kernel on the top-left 3x3 patch --- patch = 3 0 1 kernel = 1 0 -1 5 4 2 1 0 -1 1 2 3 1 0 -1 # dot product = (1*3 + 0*0 + -1*1) = 3 + 0 - 1 = 2 # + (1*5 + 0*4 + -1*2) = 5 + 0 - 2 = 3 # + (1*1 + 0*2 + -1*3) = 1 + 0 - 3 = -2 # total at (0,0) = 2 + 3 + (-2) = 3 # Sliding the kernel to all 2x2 valid positions gives the feature map: OUTPUT (2x2) = 3 -2 6 -2
Notice the kernel's left column is +1 and right column is
-1. So it lights up (a big positive or negative number) wherever the left side of
the patch is brighter than the right — i.e. wherever there's a vertical brightness change,
which is exactly an edge. Flat regions (left ≈ right) give values near zero. The filter is an edge
detector, purely from its weights.
Here is the entire operation written out by hand so you can see there's no magic — just loops, multiplies, and adds:
import numpy as np image = np.array([ [3, 0, 1, 2], [5, 4, 2, 1], [1, 2, 3, 4], [0, 1, 2, 3], ]) kernel = np.array([ [1, 0, -1], [1, 0, -1], [1, 0, -1], ]) k = kernel.shape[0] # 3 out_h = image.shape[0] - k + 1 # 4 - 3 + 1 = 2 out_w = image.shape[1] - k + 1 # 2 out = np.zeros((out_h, out_w)) for i in range(out_h): for j in range(out_w): patch = image[i:i+k, j:j+k] # the 3x3 window out[i, j] = np.sum(patch * kernel) # multiply + sum = dot product print(out) # [[ 3. -2.] # [ 6. -2.]] <-- matches our hand calculation above
A convolution is just one tiny operation — overlay, multiply, sum — repeated at every location as the filter slides. The output, a feature map, records where the filter's pattern was found. Stack many filters and you get many feature maps, one per pattern the layer hunts for.
3 Conv layer hyperparameters
When you use your sliding magnifying glass, you get to choose a few things: How big is the glass (do you peek at a 3×3 patch or a 5×5 one)? How far do you hop between peeks (every step, or skip some)? And do you put a little cardboard frame around the picture so the glass can still reach the very edges? Those choices are the settings of a convolution, and they decide how big your new score-map turns out.
A convolutional layer has a handful of hyperparameters — knobs you set before training (unlike weights, which the network learns). Get comfortable with these four; you'll choose them in every CNN you ever build.
Kernel size
The kernel size is the filter's width and height, e.g. 3×3 or 5×5. Small kernels (3×3) see a tiny area but are cheap and, when stacked, can cover large regions — which is why modern networks favour them. Larger kernels see more at once but cost more parameters.
Stride
The stride is how many pixels the filter jumps each step. A stride of 1 slides one pixel at a time (lots of overlap, big detailed output). A stride of 2 hops two pixels at a time, skipping positions — this downsamples, roughly halving the output's height and width and cutting computation.
Padding (valid vs same)
Notice in Topic 2 that a 4×4 image gave only a 2×2 output — the image shrank, and edge pixels were visited fewer times than central ones. Padding fixes this by adding a border of zeros around the image so the filter can reach the edges:
| Padding mode | What it does | Effect on output size |
|---|---|---|
| valid | No padding at all — only "valid" full overlaps are used. | Output shrinks (the 4×4 → 2×2 case). |
| same | Add just enough zero-border so the output is the same H×W as the input (for stride 1). | Output keeps the input's size. |
Depth / number of channels
Two "depth" ideas live here, so let's be precise:
- Input depth (channels in): each filter must be as deep as its input. For an RGB
image (3 channels), a 3×3 filter is really
3 × 3 × 3and sums across all three colour channels into one feature map. - Output depth (channels out): a layer has many filters — say 32. Each produces its own feature map, so the layer's output is a stack of 32 maps. That stack's depth is the number of filters.
The output-size formula
This is the formula you'll reach for constantly. For an input of size W (one
dimension), kernel size K, padding P (zeros added on
each side), and stride S:
# Output side length along one dimension
out = floor( (W - K + 2P) / S ) + 1
Input 32×32, kernel 5×5, with two common setups:
# Case A: valid padding (P=0), stride 1 out = floor((32 - 5 + 0) / 1) + 1 = 27 + 1 = 28 # → 28x28 (shrank) # Case B: same padding (P=2), stride 1 out = floor((32 - 5 + 4) / 1) + 1 = 31 + 1 = 32 # → 32x32 (preserved) # Case C: same kernel, stride 2 (downsampling), P=2 out = floor((32 - 5 + 4) / 2) + 1 = floor(15.5) + 1 = 15 + 1 = 16 # → 16x16
Tip: for a k×k kernel with stride 1, "same" padding is
P = (k-1)/2 — so 1 for a 3×3 kernel, 2 for a 5×5. That's why odd kernel sizes
are popular: the padding comes out symmetric.
Receptive fields
The receptive field of an output value is the region of the original input that influenced it. One 3×3 conv has a 3×3 receptive field. But stack a second 3×3 conv on top and each new value depends on a 3×3 patch of the first map — which itself each came from 3×3 of the input — so the second layer "sees" a 5×5 region. Stack a third and it sees 7×7.
This is the secret to depth: stacking small kernels grows the receptive field cheaply. Early layers (small receptive field) catch tiny features like edges; deeper layers (large receptive field) combine them into bigger concepts like eyes, then faces. Two 3×3 convs see a 5×5 region using far fewer parameters than one 5×5 conv — a key insight we'll see exploited by real architectures in Session 10.
floor((W − K + 2P)/S) + 1. Stacking small kernels grows the
receptive field, letting deep layers see larger patterns.
5 Pooling & a full conv block
Imagine you have a huge, detailed map and you want a smaller summary. You chop it into little 2×2 tiles and, for each tile, keep just the most important fact — say, the tallest building. Now your map is a quarter of the size but still tells you where the big stuff is. That shrinking-while-keeping-the- gist trick is pooling. It makes the picture smaller and cheaper to work with, and it stops the network from fussing over the exact pixel a feature sat on.
What pooling does
Pooling (also called subsampling or downsampling) shrinks a feature map's height and width by summarising small regions. It has no learnable weights — it's a fixed rule. The two common kinds:
- Max pooling — take the maximum value in each window. Keeps the strongest activation, i.e. "was the feature present anywhere in this region?" Most popular.
- Average pooling — take the mean of the window. Smoother; common near the end of networks as global average pooling.
# INPUT feature map (4x4) 1 3 | 2 4 5 6 | 1 0 -----+----- 2 1 | 0 3 7 8 | 4 2 # Split into four non-overlapping 2x2 windows (stride 2): # top-left {1,3,5,6} top-right {2,4,1,0} # bot-left {2,1,7,8} bot-right {0,3,4,2} MAX POOL (2x2): AVERAGE POOL (2x2): 6 4 3.75 1.75 8 4 4.50 2.25 # max of {1,3,5,6}=6 # mean of {1,3,5,6}=3.75
The 4×4 map became 2×2 — a 4× reduction in values — while max pooling kept the strongest signal in each region.
Three reasons: (1) it reduces computation for later layers; (2) it gives a little extra translation tolerance — if the feature shifts by one pixel, the max of the window often doesn't change; (3) it gradually grows the receptive field so deeper layers see more of the image. Note: many modern networks instead use strided convolutions to downsample, but pooling remains a clear, classic tool you must know.
A typical conv block: Conv → ReLU → Pool
Convolutions are linear (just multiplies and adds). To learn non-linear patterns we add an
activation function after each conv — almost always ReLU
(from Session 3): ReLU(x) = max(0, x), which simply zeros out negatives. The
classic building block of a CNN is therefore:
Stack several of these blocks and the spatial size keeps shrinking while the number of channels grows — the network trades "where" (resolution) for "what" (rich features). At the very end, the small feature stack is flattened and fed to a couple of fully-connected layers (the MLP returns!) to produce the final class scores. That whole pipeline is a Convolutional Neural Network (CNN).
Here's a real, runnable conv block using nn.Conv2d, with the shapes printed
so you can trace the math from Topic 3:
import torch import torch.nn as nn # A fake batch: 1 image, 3 channels (RGB), 32x32 pixels x = torch.randn(1, 3, 32, 32) # shape (N, C, H, W) # One conv block: Conv -> ReLU -> MaxPool conv = nn.Conv2d( in_channels=3, # RGB input out_channels=32, # 32 filters -> 32 feature maps kernel_size=3, stride=1, padding=1, # "same" padding for a 3x3 kernel -> H,W preserved ) relu = nn.ReLU() pool = nn.MaxPool2d(kernel_size=2, stride=2) # halves H and W out = pool(relu(conv(x))) print("after conv:", conv(x).shape) # [1, 32, 32, 32] (padding kept 32x32) print("after pool:", out.shape) # [1, 32, 16, 16] (pool halved to 16x16) # Learnable parameters in this conv layer: # 32 filters * (3 channels * 3 * 3) + 32 biases = 864 + 32 = 896 print("params:", sum(p.numel() for p in conv.parameters())) # 896
Note PyTorch's tensor order is (N, C, H, W) — batch, channels, height,
width. The output went from 3×32×32 to 32×16×16: more
channels, smaller spatial size — exactly the "trade where for what" pattern.
nn.Conv2d +
nn.ReLU + nn.MaxPool2d builds exactly this block.
★ Putting it all together
You just built the entire foundation of computer vision with neural networks. Here's the one-paragraph story that connects all five topics:
A plain MLP drowns on images — it has a parameter explosion, it
loses spatial structure when you flatten the grid, and it's not
translation-invariant. The cure is the convolution: a small learned
filter/kernel that slides over the image taking a dot product at each
stop, producing a feature map of where its pattern appears. We tune it with
kernel size, stride, padding (valid vs same) and
depth, predicting the output with
floor((W−K+2P)/S)+1, while stacking layers grows the
receptive field. Because of local connectivity and weight
sharing, a conv layer has a tiny, image-size-independent parameter count yet finds features
anywhere. We add ReLU for non-linearity and pooling to downsample,
giving the classic Conv → ReLU → Pool block that — stacked and finished with a small MLP
head — forms a CNN. Next session we'll assemble these blocks into the famous
architectures.
Quick self-check
Name the three reasons a plain MLP struggles with images.
(1) Parameter explosion — flattening a big image gives millions of weights in layer one; (2) it loses spatial structure — flattening hides which pixels are neighbours; (3) it's not translation-invariant — it must relearn a pattern for each position.
What exactly is computed at each position when a filter slides over an image?
A dot product: multiply each kernel weight by the pixel beneath it and sum all the products into a single number. Collecting these numbers over all positions gives a feature map.
An input is 28×28, kernel 3×3, padding 1, stride 1. What is the output size?
floor((28 − 3 + 2×1)/1) + 1 = floor(27) + 1 = 28. So 28×28 — "same" padding preserved the size (P = (k−1)/2 = 1 for a 3×3 kernel).
Why does a conv layer have far fewer parameters than a fully-connected layer?
Weight sharing: the same small filter is reused at every position instead of a unique weight per connection. A conv layer's parameter count depends only on kernel size and channel counts, not on the image's height and width.
What does max pooling do, and what are its benefits?
It takes the maximum value in each small window, downsampling the feature map (e.g. 2×2 stride 2 quarters the values). Benefits: less computation later, mild translation tolerance, and a growing receptive field. It has no learnable weights.
What are the layers in a classic CNN building block, and why is each there?
Conv (detect patterns with learned filters) → ReLU (add non-linearity so the network can model complex shapes) → Pool (downsample, cut cost, add shift tolerance). Stacked repeatedly, then a fully-connected head makes the final decision.
📚 References & Further Reading
Class material
- SST Deep Learning handout (Session 9) — your course handout for this session (covers convolutions, conv hyperparameters, weight sharing, and pooling).
Papers, docs & deep dives
- SST Deep Learning handout (online) — the class handout for Session 9, hosted online for quick reference.
- CS231n — Convolutional Neural Networks — Stanford's gold-standard notes; the clearest explanation of filters, stride, padding, the output-size formula, and pooling anywhere.
- A guide to convolution arithmetic for deep learning (Dumoulin & Visin, 2016) — the definitive, diagram-rich reference for exactly how kernel size, stride, and padding determine output shapes.
- LeNet (LeCun et al.) — the original convolutional network for digit recognition; the historical origin of the Conv→Pool→FC design we built here.
- PyTorch
nn.Conv2ddocumentation — official reference for every argument (in_channels, out_channels, kernel_size, stride, padding) used in our code example.