1 The ML recap & the problem with hand-crafted features
Imagine you want to teach a friend to spot a cat in a photo, but your friend can only look at numbers you hand them β never the picture itself. So you have to stare at the photo and write down clues: "has pointy ears," "has whiskers," "is furry." Your friend just learns the rule "if these clues are present, say cat." The hard, tiring work is you inventing the right clues. That clue-writing is exactly what made old-school AI such a slog β a human had to think up every clue by hand.
Before we can say why deep learning is special, we need a quick picture of what came before it. The umbrella term is machine learning (ML): instead of a programmer writing explicit rules, we show a program lots of examples and it learns the rule from data. That much is still true in deep learning. The difference is in what the human has to provide.
The classical ML pipeline
In classical (or "shallow") machine learning β think logistic regression, decision trees, support vector machines β the model itself is fairly simple. The heavy lifting happens before the model ever sees the data, in a step called feature engineering: a human decides which measurable properties (the features) the model should look at.
Notice where the intelligence really lives: step 2. The model in step 3 is only as good as the features a person dreamed up in step 2. Give it bad clues and it cannot recover, no matter how clever the algorithm.
For email spam detection, hand-crafted features are actually pretty natural. A human can list useful clues, and a simple model learns a weight for each:
# A hand-crafted feature vector for one email features = { "contains_word_free": 1, # yes "num_exclamation_marks": 7, "all_caps_ratio": 0.42, "sender_in_contacts": 0, # no "contains_link": 1, } # A simple model just learns: score = w1*f1 + w2*f2 + ... + bias # If score > threshold -> "spam"
This works because a human can easily name the clues for spam. The features are obvious and few. Classical ML shines exactly here: tabular data where good features are known and cheap to compute.
Where hand-crafted features fall apart
Now try the cat photo. The raw data is a grid of pixels β maybe 224Γ224Γ3 β 150,000 numbers, each just a brightness value. There is no pixel called "has pointy ears." The clue you actually care about is buried in the relationships between thousands of pixels, and those relationships shift with lighting, angle, breed, background, and occlusion.
For decades, computer-vision researchers tried to hand-design these clues anyway. They built elaborate feature extractors with names like SIFT, HOG, and SURF β clever hand-written recipes for detecting edges, corners, and gradients. Speech researchers did the same with MFCCs for audio. Each one was a PhD's worth of effort, and each worked only so-so.
For rich, high-dimensional data (images, audio, natural language), good features are not obvious, not few, and not transferable:
- Not obvious β what single number captures "looks like a cat's face"? Nobody can write it down.
- Brittle β a feature tuned for daytime photos fails at night; one tuned for one accent fails on another.
- Expensive β each new task (cats β tumors β road signs) needs a fresh army of experts inventing fresh features.
- Lossy β humans throw away subtle information they didn't think was important.
This bottleneck β humans must invent the features β is the single biggest reason progress on vision, speech, and language stalled for years.
In classical ML, the model is dumb and the human is the smart feature designer. That is fine when good features are obvious (spam, simple tables) and a disaster when they aren't (pixels, sound waves, raw text). Deep learning's whole pitch is: what if the machine could invent the features itself?
2 Representation learning β features that learn themselves
Remember how you had to write all the cat-clues by hand? Imagine instead a stack of helpers. The first helper only notices tiny things β little lines and edges. It passes notes to the next helper, who combines those lines into shapes like circles and triangles. That helper passes notes up to one who sees "two pointy ear shapes and round eyes," and the one at the very top says "that's a cat!" Nobody told the helpers what to look for β they figured out their own clues just by practising on lots of pictures. That stack of self-teaching helpers is a deep neural network.
This is the heart of the whole field. Deep learning is machine learning with neural networks that have many layers stacked one after another ("deep" = many layers). And the magic those layers buy us has a name: representation learning β the network learns the features itself, directly from raw data, instead of waiting for a human to hand them over.
The new pipeline
Compare this to the classical pipeline from Topic 1. The human-feature box is gone. The network swallows raw pixels and produces both the features and the prediction:
The feature hierarchy
The deepest, most beautiful result of this is that the learned features arrange themselves into a hierarchy of representations, from simple to abstract, exactly like the helper stack in the story. When researchers peek inside a trained image network, they reliably find this:
| Layer depth | What it learns to detect | Built from⦠|
|---|---|---|
| Early layers | Edges, color blobs, simple gradients | Raw pixels |
| Middle layers | Corners, textures, simple shapes (circles, stripes) | Combinations of edges |
| Later layers | Object parts β an eye, a wheel, a nose, an ear | Combinations of shapes |
| Final layers | Whole objects / concepts β "cat," "car," "face" | Combinations of parts |
Each layer builds its features out of the layer below it. Crucially, nobody designed any of these detectors β they emerged automatically as the network adjusted its internal numbers (its weights) to get better at the final task. The human only supplied raw data and labels; the network discovered that "edges β shapes β parts β objects" is a useful way to see the world.
Don't engineer features β learn them. A deep network turns the painful, human-driven feature-engineering step into something the model does for itself, layer by layer, building simple features into ever more abstract ones. This is why one architecture can be pointed at cats, tumors, road signs, or speech and learn the right features for each.
You don't need the full machinery yet (that's the next several sessions), but here's the shape of it. A layer is just: multiply inputs by learned weights, add a bias, then bend the result through a non-linearity (a function like ReLU). Stack two layers and the second one gets to build features out of the first one's outputs:
import numpy as np def relu(x): # the non-linearity that lets layers build on each other return np.maximum(0, x) # x = raw input (e.g. flattened pixels). W1, W2, b1, b2 are LEARNED, not designed. def forward(x, W1, b1, W2, b2): h = relu(x @ W1 + b1) # layer 1: learns low-level features (edges) y = h @ W2 + b2 # layer 2: builds higher features from h, then predicts return y # Training nudges W1, W2, b1, b2 so the network discovers good features ON ITS OWN.
The entries of W1 and W2 start as random
noise. Training (which we cover in a later session) slowly tunes them until
h becomes a genuinely useful set of learned features. The
features live inside those weight matrices.
Why bend each layer through ReLU? Because without a non-linearity, stacking layers is pointless: two matrix multiplies in a row collapse into a single one (Wβ(Wβx) = (WβWβ)x), so a "deep" net would be no more powerful than one layer. The non-linearity is what lets later layers build genuinely new, richer features out of earlier ones. Keep this in your back pocket β it returns in the next topic.
3 Universal approximation β what a network can represent
Imagine drawing any wiggly line you like on paper. Now I give you a big box of tiny straight LEGO bricks. With enough little bricks, you can trace any wiggle as closely as you want β steps so small they look smooth. A neural network is like that LEGO box: with enough little pieces (neurons), it can shape itself to match almost any pattern. But here's the catch grown-ups forget: having enough bricks doesn't mean you'll actually figure out how to arrange them. That's a different, harder problem.
Topic 2 told us deep networks learn features. A natural worry follows: are neural networks even powerful enough to capture the patterns we care about? The reassuring answer is a famous result called the Universal Approximation Theorem (UAT).
The theorem, intuitively
A neural network with just one hidden layer and a non-linear activation, given enough neurons, can approximate any reasonable (continuous) function to any desired accuracy, over a bounded region.
This was proved for sigmoid-like activations by George Cybenko in 1989, and generalized to essentially any non-polynomial activation by Kurt Hornik in 1991. "Approximate any function" is a big deal: a function is just a mapping from inputs to outputs, and that's what every prediction task is β pixels β "cat," audio β words, board position β best move. If a network can approximate any function, it can in principle represent any of these tasks.
The LEGO intuition is precise: each neuron, after its non-linearity, contributes a little bump or step. Add up enough bumps of the right heights and positions and you can trace any curve. More neurons = smaller, more numerous bumps = a closer fit.
Suppose we want a network to match the smooth target f(x) = sin(x)
on the range 0 to 2Ο. A single hidden layer of ReLU neurons builds the output as a sum of
"ramp" functions β and more neurons give a tighter fit:
import numpy as np def relu(x): return np.maximum(0, x) # A 1-hidden-layer net: output = sum over neurons of v_j * relu(w_j*x + b_j) def net(x, w, b, v): # w,b,v are arrays of length N (the number of neurons) bumps = v * relu(np.outer(x, w) + b) # each neuron makes one ramp return bumps.sum(axis=1) # add the ramps together # With N = 4 neurons -> a rough, jagged approximation of sin(x) # With N = 50 neurons -> visually indistinguishable from sin(x) # UAT says: as N -> large, the approximation error -> 0.
The takeaway: there is no shape this network is fundamentally incapable of matching. Width (more neurons) buys you accuracy. That's the existence guarantee.
What it does NOT promise β existence vs. learnability
Here is where beginners (and a lot of hype) go wrong. The UAT is a statement about existence, not attainability. Read the small print:
| The theorem promises⦠| The theorem says NOTHING about⦠|
|---|---|
| A good set of weights exists. | Whether training will ever find those weights. |
| One hidden layer is enough in principle. | How many neurons β it could need an astronomically huge number. |
| You can fit the data you have. | Whether the network will generalize to new, unseen data. |
| An approximation is possible. | How long, how much data, or how much compute it takes. |
UAT guarantees a perfect arrangement of LEGO bricks exists. It does not guarantee you can find it by training, that it fits in reasonable memory, or that it works on data it hasn't seen. Learnability β actually discovering good weights via an algorithm, from finite data, in finite time β is a wholly separate problem, and it's what the rest of this course is really about (loss functions, gradient descent, backpropagation, regularization).
If one wide layer is enough in theory, why stack many? Because "enough neurons" can mean an impractically gigantic single layer, whereas depth often represents the same function far more efficiently β fewer total neurons, reused features (Topic 2), and patterns that are much easier to learn in practice. Theory says one layer can; practice says many layers do it better. We'll see exactly how in later sessions.
4 When & why deep learning wins (and when it doesn't)
Think of two students. One has a small brain but can only read a few books β past a point, extra books don't help them much. The other has a huge brain and just keeps getting smarter the more books you give them, as long as they have time and energy to read. Old-style ML is the first student. Deep learning is the second. So when you have a mountain of books (data), fast reading glasses (GPUs), and a big brain (a deep network), the second student wins by a mile. With only a handful of books, though, the small-brain student is often just fine β and cheaper to feed.
We've seen what deep learning does (learn features) and what it can represent (almost anything). The practical question is: when does it actually beat classical ML? The honest answer is a recipe of three ingredients β and they all have to show up together.
The three ingredients: data + compute + depth
None of the three is new on its own β neural nets and backpropagation existed in the 1980s. What changed around 2012 was that all three arrived at once:
- Data β the internet produced enormous labeled datasets (e.g. ImageNet, ~1.2 million labeled images across 1,000 categories).
- Compute β GPUs (graphics processing units), originally built for video games, turned out to be perfect for the massive parallel matrix multiplications a neural net needs. They made training 10β100Γ faster.
- Depth & ideas β better architectures, activations (ReLU), and training tricks let very deep networks actually train without falling apart.
In 2012 a deep network called AlexNet won the ImageNet competition by a stunning margin, slashing the error rate well below every hand-crafted-feature system. It ran on GPUs and trained on a million-plus images. That single result is widely seen as the moment deep learning "won" computer vision and the modern era began. Notice it needed all three ingredients β big data (ImageNet), compute (GPUs), and depth (a deep convolutional net).
Why scale flips the result: the "more data" curve
Here's the deepest reason DL wins on hard problems. Classical models tend to plateau: past a certain amount of data, feeding them more barely helps, because their hand-crafted features can only capture so much. Large deep networks keep improving as data grows, because they can keep learning richer features. So the two curves cross:
| Amount of data | Classical ML (good features) | Deep learning |
|---|---|---|
| Tiny (hundreds) | Often better β and far cheaper | Overfits; not enough to learn features |
| Medium (thousands) | Competitive | Catching up |
| Large (millions+) | Plateaus | Pulls clearly ahead |
Where deep learning does NOT win
Deep learning is not a hammer for every nail. It's worth being honest about its weak spots β this saves you from over-engineering:
| Situation | Better choice & why |
|---|---|
| Small datasets (a few hundred rows) | Classical ML β DL overfits and has nothing to learn features from. |
| Clean tabular / structured data | Gradient-boosted trees (XGBoost, LightGBM) frequently beat DL here, and train faster. |
| You need to explain every decision | Simple, interpretable models β DL is a "black box" that's hard to audit. |
| Tight compute / battery / latency budget | Lightweight classical models β DL can be heavy and power-hungry. |
| Good features are already obvious | Classical ML β the main DL advantage (learning features) buys you little. |
You're at a bank with two projects. Which calls for deep learning?
- Project A: predict loan default from a clean spreadsheet of 8,000 rows with 25 well-understood columns (income, age, credit scoreβ¦). β Classical ML / gradient-boosted trees. Small, tabular, features already meaningful, and you must explain rejections to regulators.
- Project B: automatically read and classify 5 million scanned check images. β Deep learning. Huge data, raw pixels with no obvious features, GPUs available β exactly the data + compute + depth sweet spot.
Same company, opposite answers. The choice is driven by data size, data type, and constraints β not by which technique is "cooler."
Deep learning wins when (1) the data is big, (2) it's raw and high-dimensional (images/audio/text) so good features aren't obvious, and (3) you have the compute to train depth. Remove any one ingredient β too little data, plain tabular data with known features, or no GPUs / a need for interpretability β and classical ML is often the smarter, cheaper, faster choice.
β Putting it all together
You just learned why deep learning exists and why it took over. Here's the one-paragraph story that ties all four topics together:
Classical machine learning needed a human to hand-craft features, which works for tidy problems like spam but collapses on raw pixels, audio, and text (Topic 1). Deep learning's answer is representation learning: stacked layers that discover their own features, self-organizing into a hierarchy from edges to shapes to parts to whole objects (Topic 2). We trust these networks are powerful enough because the Universal Approximation Theorem says a network can represent almost any function β though that's only an existence promise, not a guarantee we can learn it (Topic 3). And in practice deep learning wins when data, compute, and depth all show up together β big, raw, high-dimensional data with GPUs to spare β while classical ML stays the better pick for small or tabular problems (Topic 4). The "how" of actually finding good weights β neurons, loss, gradient descent, backprop β is the journey of the sessions ahead.
Quick self-check
What is the single biggest difference between classical ML and deep learning?
Who designs the features. In classical ML a human hand-crafts them; in deep learning the network learns them automatically from raw data (representation learning).
Why do hand-crafted features fail badly on images but work okay on spam email?
For spam, good clues ("contains the word free," "lots of !!!") are obvious and easy to name. For images the meaningful clues live in relationships among thousands of pixels β non-obvious, brittle, and impossible to write down by hand.
In a trained image network, what do early layers vs. later layers tend to detect?
Early layers detect simple things β edges and color blobs. Middle layers detect shapes and textures. Later layers detect parts (eyes, wheels) and then whole objects. It's a hierarchy, each layer built from the one below.
What does the Universal Approximation Theorem guarantee β and what does it NOT?
It guarantees a network with enough neurons can represent almost any continuous function (existence). It does NOT promise that training will find those weights, how many neurons are needed, or that the result will generalize. Existence β learnability.
Name the three ingredients that must combine for deep learning to win.
Big data, lots of compute (GPUs/TPUs), and depth (many-layer networks with good architectures). All three together β as happened around 2012 with ImageNet and AlexNet.
You have 5,000 rows of clean tabular data and must explain every prediction. Deep learning?
Probably not. Small, tabular, and interpretability-critical β classical ML (e.g. gradient-boosted trees) is usually more accurate, faster, cheaper, and explainable here.
π References & Further Reading
Class material
- SST Deep Learning handout (Session 1) β your course handout for this session, posted on the class site as "Post handout link-1." Read it alongside these notes for the instructor's framing.
Papers, docs & deep dives
- SST Deep Learning β course site β the official course page where the Session 1 handout link is posted; check here for the latest materials and announcements.
- LeCun, Bengio & Hinton, "Deep Learning," Nature (2015) β the landmark review by three pioneers; the best single high-level overview of why deep, representation-learning networks work.
- Goodfellow, Bengio & Courville β Deep Learning (the "Deep Learning Book") β the free, authoritative textbook; Chapter 1 covers exactly this session's representation-learning motivation.
- Cybenko (1989), "Approximation by Superpositions of a Sigmoidal Function" β the original Universal Approximation Theorem proof for sigmoid networks.
- Hornik (1991), "Approximation Capabilities of Multilayer Feedforward Networks" β generalizes UAT to essentially any non-polynomial activation.
- Krizhevsky, Sutskever & Hinton (2012), "ImageNet Classification with Deep CNNs" (AlexNet) β the result that ignited the modern deep-learning era and showed data + compute + depth winning.