Convolutions

1 Why FC layers fail on images

• A 224×224×3 image = 150,528 inputs. One FC layer to 1000 neurons = ~150.5M weights — and that's ONE layer.
• Wasteful, overfits instantly, slow, memory-hungry.
• Core issue: FC treats every pixel as equally related to every neuron → it ignores spatial structure.
• Key fact: nearby pixels are correlated (sky is blue in patches); distant pixels are usually unrelated.

2 Local connectivity

• Solution: connect each neuron to only a small local patch of the input, not the whole image.
• Same job, drastically fewer parameters — each neuron sees only a small region.
• Intuition (Where's Waldo): your eye scans local patches for a pattern, then recognises it anywhere in the image.

3 Filters / kernels

• A filter (kernel) is a small grid of learnable weights (e.g. 3×3).
• Operation = slide, multiply, sum, repeat: at each position, element-wise multiply with the patch then sum → one output value.
• A filter responds (large output) where it finds its pattern; uniform regions → output ≈ 0; matching edges → large output.
• Examples: vertical-edge filter [1 0 -1; 1 0 -1; 1 0 -1]; horizontal-edge filter [1 1 1; 0 0 0; -1 -1 -1].

4 Feature maps & dimensions

• 1 filter → 1 feature map (detects one pattern). N filters → N feature maps. Typical: 32, 64, 128, 256 per layer.
• Filter depth always equals the number of input channels.

e.g. 224×224×3 with 64 filters → 224×224×64.

5 Padding & stride

• Shrinking problem: each conv layer shrinks spatial dims (5×5 → 3×3 → 1×1 → gone). Fix = padding.
• Valid padding (no pad): output shrinks, e.g. 5×5 with 3×3 → 3×3.
• Same padding (pad=1 for 3×3): ring of zeros around the border → output = input size (5×5 → 5×5). Most common.
• Stride = how far the filter jumps. Stride 1 = every position (default); stride 2 = skip every other → output roughly halves.
• Stride > 1 is used for downsampling — a modern alternative to pooling.

6 Output-size formula

Output = (W - F + 2P) / S + 1

Rule of thumb: for "same" padding with stride 1, set P = ⌊F/2⌋.

7 Parameter sharing

• Parameter sharing: the same filter weights are used at every spatial location.
• Why: a vertical edge looks the same top-left or bottom-right → no need to learn a separate detector per position.
• Result: one 3×3 filter = 9 parameters, reused at all positions; FC equivalent ≈ 150M.
• Makes sense for images (a pattern can appear anywhere) but NOT for tabular data, where each feature has a distinct meaning.

8 Receptive field

• Receptive field (RF): the region of the original input that one neuron "sees".
• Stacking 3×3 convs (stride 1): 1 layer → 3×3, 2 layers → 5×5, 3 layers → 7×7. Each layer adds context.
• Formula: RF = 1 + L·(F - 1) (L = layers, F = filter size).
• Hierarchy of features: early layers → edges/colors (small RF); middle → textures/parts; deep → objects/scenes (large RF). This is why we stack conv layers.

9 Pooling

• Pooling downsamples and summarizes. Max pooling (2×2, stride 2): take the max in each window → 4×4 becomes 2×2.
• Idea: "is there a strong feature somewhere in this region? I don't care exactly where."
• Three wins: (1) reduces spatial size / compute, (2) translation invariance (small shifts → same output), (3) increases receptive field.
• Types: max pooling (classic), average pooling, global average pool (end of network).

★ Likely exam questions