Standing on the shoulders of giant models

1 Transfer learning — borrowing a trained brain

In Session 10 we saw that training a deep CNN like ResNet on a giant dataset takes enormous data and compute. Transfer learning is the trick of reusing a model that was already pre-trained on a big dataset and adapting it to your own, usually much smaller, task.

The standard pre-training dataset: ImageNet

The most common starting point is a model pre-trained on ImageNet — a dataset of about 1.2 million labelled images across 1,000 categories (dogs, cars, mushrooms, guitars…). A model that has learned to tell those 1,000 classes apart has, as a side effect, learned to recognise an enormous vocabulary of visual patterns. We reuse that.

Why early features are general

Recall from Session 10 that a CNN learns features in layers, and they get more abstract the deeper you go. This is the key insight that makes transfer learning work:

This isn't just hand-waving — Yosinski et al. (2014) measured it directly, showing that features in the first layers transfer almost perfectly between tasks, while the last layers are specialised and transfer poorly. The deeper you go, the less general the features.

Two flavours: feature extraction vs fine-tuning

There are two main ways to do transfer learning, differing in how much of the borrowed model you let change.

A simple recipe for choosing

import torch
import torch.nn as nn
from torchvision import models

# 1. Load a model pre-trained on ImageNet
model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)

# 2. FEATURE EXTRACTION: freeze every borrowed parameter
for param in model.parameters():
    param.requires_grad = False

# 3. Replace the head. resnet18's classifier is `model.fc`,
#    a Linear(512 -> 1000). Swap in a fresh Linear(512 -> 5).
#    New layers default to requires_grad=True, so ONLY these train.
num_features = model.fc.in_features         # 512
model.fc = nn.Linear(num_features, 5)   # 5 flower classes

# 4. Only the new head's params reach the optimizer
optimizer = torch.optim.Adam(model.fc.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

# 5. Train as usual — but it's fast: few params, few epochs.
for images, labels in train_loader:
    optimizer.zero_grad()
    loss = criterion(model(images), labels)
    loss.backward()
    optimizer.step()

2 Object detection — not just "what," but "where"

So far our CNNs answered "which single class is this image?" Object detection goes further: it finds multiple objects in one image and, for each, predicts both a class label and a bounding box — a rectangle (x, y, width, height) that frames the object.

Two families of detectors

Almost every detector falls into one of two camps, trading accuracy vs speed.

How do we measure "is this box correct?" — IoU

A predicted box is rarely pixel-perfect. IoU (Intersection over Union) measures overlap between the predicted box and the true box:

IoU = area(overlap)  /  area(union)

• IoU = 1.0 — perfect overlap (boxes identical).
• IoU = 0 — no overlap at all.
• A common rule: a detection "counts as correct" if its IoU with a true box is ≥ 0.5.

def iou(boxA, boxB):  # boxes as (x1, y1, x2, y2)
    # intersection rectangle
    xi1, yi1 = max(boxA[0], boxB[0]), max(boxA[1], boxB[1])
    xi2, yi2 = min(boxA[2], boxB[2]), min(boxA[3], boxB[3])
    inter = max(0, xi2 - xi1) * max(0, yi2 - yi1)
    areaA = (boxA[2]-boxA[0]) * (boxA[3]-boxA[1])
    areaB = (boxB[2]-boxB[0]) * (boxB[3]-boxB[1])
    return inter / (areaA + areaB - inter)   # overlap / union

Scoring a whole detector — mAP

To grade a detector across a whole dataset we use mAP (mean Average Precision). At a high level: for each class you plot how precise (few false boxes) and how complete (found all objects) the detector is, summarise that curve into one number called Average Precision, then average it across all classes. Higher mAP = better detector. You'll see scores like "mAP@0.5" meaning "AP computed using the IoU ≥ 0.5 rule."

3 Segmentation — labelling every single pixel

Detection gives you boxes; segmentation goes finer and assigns a label to every pixel. There are two important kinds:

The encoder–decoder shape (and U-Net)

Classification CNNs keep shrinking the image down to a tiny feature map, then output one label. But segmentation needs an output that's the same size as the input (one label per pixel). So we use an encoder–decoder architecture: first squeeze the image down to learn what's there, then expand it back up to recover where exactly.

The classic example is U-Net (Ronneberger et al., 2015), born in medical imaging. Drawn out, it makes a U shape: the encoder going down the left, the decoder climbing the right. Its signature trick is skip connections that hand the sharp, high-resolution details from each encoder level straight across to the matching decoder level — so fine edges aren't lost during the squeeze.

  input
    │  ┌──────────────── skip ───────────────┐
    ▼  │                                      ▼
 [enc1] ──────────── skip ──────────────► [dec1] ─► output map
    │  ┌──────── skip ────────┐              ▲
    ▼  │                       ▼             │
 [enc2] ──── skip ────► [dec2]───────────────┘
    │                      ▲
    ▼                      │
 [enc3] ──────────────► [dec3]
    │                      ▲
    └──── bottleneck ──────┘
   (encoder down)     (decoder up)
            the "U" shape

Instance segmentation: Mask R-CNN

For instance segmentation, the most famous model is Mask R-CNN (He et al., 2017). The idea is beautifully simple: take Faster R-CNN (the two-stage detector from Topic 2) and add a third output branch that, for each detected box, predicts a small binary mask — a per-pixel "object / not object" map inside that box.

import torch
from torchvision.models.detection import maskrcnn_resnet50_fpn

# Pretrained on COCO (80 everyday object classes)
model = maskrcnn_resnet50_fpn(weights="DEFAULT")
model.eval()                              # inference mode

with torch.no_grad():
    out = model([image_tensor])[0]   # one image in, dict out

# out["boxes"]  -> bounding boxes  (N, 4)
# out["labels"] -> class id per object
# out["scores"] -> confidence per object
# out["masks"]  -> per-pixel masks (N, 1, H, W)
keep = out["scores"] > 0.7      # keep confident detections

4 A tour of other computer-vision applications

The same convolutional machinery — often via transfer learning from a big pretrained backbone — powers a huge range of real applications. Here's a quick guided tour of four major areas.

Face recognition

The trick here isn't to classify "which of 1,000 fixed people is this" — you can't retrain every time a new person joins. Instead, a CNN turns each face into an embedding: a vector of numbers where the same person's faces sit close together and different people sit far apart (recall embeddings give similar things similar codes). Recognition then becomes "is this new face's vector close to a stored one?"

Pose estimation

Pose estimation locates a person's keypoints — joints like elbows, knees, wrists, and shoulders — and connects them into a stick-figure skeleton. The model predicts a heatmap for each joint (where is it most likely?), then links them up.

Medical imaging

CNNs (and U-Net especially, from Topic 3) read X-rays, CT and MRI scans, and microscope slides — detecting tumours, segmenting organs, or flagging fractures. The encoder–decoder shape is ideal because doctors often need the exact outline of a region, not just a box.

Generative vision

Generative models don't classify images — they create them. Two families dominate: GANs (Generative Adversarial Networks), where a "generator" tries to fool a "discriminator" that judges real vs fake, and diffusion models, which learn to turn random noise into an image step by step (the engine behind tools like Stable Diffusion).

★ Putting it all together

This session was about using the CNNs you built in Session 10 — not training from scratch, and not just classifying. Here's the one-paragraph story connecting all four topics:

Quick self-check

📚 References & Further Reading

Class material

• SST Deep Learning handout (Session 11) — your course handout for this session.

Papers, docs & deep dives

• SST Deep Learning handout (online) — the companion notes for this course, covering transfer learning and CNN applications.
• Yosinski et al. — "How transferable are features in deep neural networks?" — the paper that measured exactly why early layers transfer and late layers don't.
• PyTorch transfer learning tutorial — hands-on official walkthrough of feature extraction and fine-tuning, mirroring the code in Topic 1.
• Redmon et al. — "You Only Look Once" (YOLO) — the original one-stage, real-time object detection paper.
• Ronneberger et al. — "U-Net" — the encoder–decoder with skip connections that defined modern segmentation.
• He et al. — "Mask R-CNN" — adds a per-object mask branch to Faster R-CNN for instance segmentation.