Why Deep Learning?

1 The classical ML pipeline

• Flow: Raw Data → Hand-Crafted Features → ML Model → Prediction. The features step is the bottleneck.
• Model quality is bounded by feature quality — a great model on bad features still fails.
• Feature design needs deep domain expertise + months of iteration.
• Every new task = build the feature pipeline from scratch (digit vs cat photo vs chest X-ray each need totally different expertise).
• Examples: images → pixel counts per region, color histograms, hand-coded edge detectors; text/audio → word counts, keyword checks, pitch/loudness.

2 Where hand-crafted features break

• Brittleness: features tuned for one setting fail when conditions change even slightly.
• No invariance: shift or rotate an object → many classical features break.
• No transfer: features for one task/domain (medical, satellite, audio) don't carry over — each needs entirely new features.
• As data complexity grows, the feature engineer becomes the bottleneck, not the algorithm.

3 The scaling problem

• Classical ML plateaus as data grows — fixed features can't absorb more information.
• Deep learning keeps scaling — more data → better learned representations.
• Curves cross: classical wins on small data, deep learning overtakes once data is large.

4 What if the model learned its own features?

• Flow becomes: Raw Data → Neural Network (learns features + mapping) → Prediction.
• No hand-crafted features — the network discovers what matters.
• Same architecture works across images, text, audio; features improve automatically with more data.
• Open question this raises: can a network really learn any function we need? → leads to UAT.

5 Universal Approximation Theorem (UAT)

• Plain English: give it any smooth input→output relationship, a one-hidden-layer net can get arbitrarily close — but may need a lot of neurons.
• NNs are universal function approximators: in principle they represent any mapping.
• LEGO analogy: each neuron = one brick producing one simple bump/step; stack enough → build any shape. More neurons = finer detail. The network learns which bricks to place and where.
• UAT in action: 1 neuron (line-ish) → 5 → 20 → 100 neurons progressively fit a wavy target. More neurons = better fit.
• Caveat: the theorem guarantees existence only — it says nothing about how to find the right configuration.

6 The catch & the 23-year gap

What UAT does NOT tell you:

• How many neurons (could be astronomically large).
• How to train — no learning algorithm is guaranteed.
• How to generalize — fitting training data ≠ working on new data.
• How to choose the architecture (depth, width, activation).

7 The big idea: representation learning

• Deep nets build their own feature pipeline, raw pixels → high-level concepts, automatically.
• Vision hierarchy (simple → complex): L1 edges/gradients → L2 textures/patterns → L3 parts (eyes, wheels, wings) → L4–5 objects (faces, cars, birds).
• Audio analogy: phonemes → syllables → words → meaning. Deep networks discover these hierarchical levels on their own.

8 What a CNN actually learns (VGG16)

• First conv layer: 64 filters of 3×3 pixels; detect edges, color gradients, simple textures. Discovered, not programmed — they resemble hand-crafted Gabor filters but the net found them itself.
• Depth progression: Layer 0 = raw pixels (no abstraction) → Layer 7 = corners/contours → Layer 14 = textures/repeating patterns → Layer 28 = class-specific parts (eyes, wheels, feathers).

9 The power of learned features

• Same data; only difference is better features.
• A simple classifier on learned features beats a complex classifier on raw data — that's representation learning.

★ Likely exam questions