1 Feature hierarchies in depth
Imagine building a castle out of LEGO. You don't start with a finished tower β you start with tiny bricks. First you click two bricks into a little wall. Then you join walls into a room. Then rooms into a whole castle. A deep network learns pictures the same way: first it notices tiny edges, then it sticks edges together into shapes, then shapes into eyes and wheels, and finally whole faces and cars. Each layer builds on the one below it.
Recall from Session 1 that a deep network is just many layers stacked on top of each other, and that we don't hand it features β it learns them. This topic is about what those learned features actually look like, and why stacking layers produces a feature hierarchy: simple features near the input, complex ones near the output, each layer composing the layer below.
What is a "feature"?
A feature is just a measurable pattern the network looks for. In an image, an early feature might be "is there a slanted edge here?" A later feature might be "is there an eye here?" Each layer (a group of neurons that transform the data) produces a new set of features from the features beneath it. The key word is compose: a layer doesn't start from raw pixels, it starts from whatever the previous layer already figured out.
Depth gives you composition. Layer 1 finds simple parts. Layer 2 combines those parts into bigger parts. Layer 3 combines those, and so on. Complex concepts are built out of simpler ones β exactly how humans understand a face as "two eyes, a nose, a mouth, arranged just so."
What CNN layers actually learn
The clearest evidence comes from a CNN (Convolutional Neural Network) β the kind of network built for images (we'll cover CNNs properly in a later session; for now just picture "a deep network for pictures"). When researchers visualise what each layer responds to, a beautiful pattern appears, going from input to output:
| Layer depth | What it detects | Built from⦠|
|---|---|---|
| Early (layer 1β2) | Oriented edges, blobs of colour, simple gradients | Raw pixels |
| Middle (layer 3β4) | Textures and simple shapes: corners, curves, stripes | Combinations of edges |
| Later (layer 5β6) | Object parts: an eye, a wheel, a door handle, a beak | Combinations of textures & shapes |
| Deepest (near output) | Whole objects / categories: a face, a car, a specific dog breed | Combinations of parts |
Nobody told the network "look for eyes." It discovered that eyes are useful building blocks for recognising faces, because faces kept appearing in the training images. This is the promise of deep learning made concrete: the hierarchy of features is learned, not programmed.
An early CNN feature is literally a small grid of numbers (a filter) slid across the image. Here's a classic vertical-edge filter and what it does to a patch of pixels (bright = high number, dark = low number):
# A 3x3 vertical-edge filter (Sobel-style) filter = [[ 1, 0, -1], [ 2, 0, -2], [ 1, 0, -1]] # An image patch: bright pixels on the LEFT, dark on the RIGHT patch = [[ 9, 9, 0], [ 9, 9, 0], [ 9, 9, 0]] # Multiply matching cells, then add them all up: score = (1*9 + 0*9 + -1*0) + (2*9 + 0*9 + -2*0) + (1*9 + 0*9 + -1*0) score = 36 # big positive number = "yes, a vertical edge is here!"
A flat patch (all 5s) would score 0 β no edge.
Layer 1 is hundreds of little filters like this. Layer 2's filters then run over
layer 1's edge-maps, so a layer-2 filter that lights up when "a vertical edge meets a
horizontal edge" is effectively a corner detector. That is composition in action.
Researchers peek inside trained networks using feature visualisation: they find the image patches that make a given neuron fire hardest, or even generate a synthetic image that maximally excites it. Famous work by Zeiler & Fergus (2013) and later the Distill.pub "Feature Visualization" article showed exactly this edges β textures β parts β objects progression. It's not a metaphor β you can literally see it.
A deep network is a feature factory with assembly lines. Each layer takes the parts made by the layer before and assembles slightly bigger parts. Shallow features (edges) are reused to build everything above them, which is wildly efficient β you don't re-learn "edge" separately for cats, cars, and faces.
2 Depth vs width β why deep beats shallow-but-wide
Imagine you have to fold a long strip of paper to make a fancy shape. You could try to do it all in one giant fold using a thousand hands at once (that's "wide"). Or you could fold it in half, then in half again, then again (that's "deep"). With folding, each fold doubles what you did before, so a few folds go a very long way. Deep networks are like folding: a few extra layers can do the work of an absurdly huge single layer.
A network can grow in two directions. Making a layer wider means adding more neurons side by side in the same layer. Making a network deeper means adding more layers stacked on top of each other. Both add capacity β but they are not equal. The central claim of this topic:
For many functions, a deep network can represent them with exponentially fewer neurons than a shallow-but-wide network. Depth buys you reuse: each layer recombines the work of the one below, so capability grows multiplicatively with depth instead of just adding up.
The intuition: reuse vs re-doing
In Topic 1 we saw that an "edge" feature gets reused to build textures, parts, and objects. A shallow network can't reuse anything β it has only one layer of features, so to recognise 1,000 objects it must learn each object more or less from scratch, in parallel, in that one layer. A deep network learns the edge once and shares it everywhere above. Sharing is why depth is so much more economical.
A useful way to measure a network's expressive power is: how many distinct linear
regions can it carve the input space into? More regions = it can draw a more complicated,
wiggly decision boundary. With the standard ReLU activation (more on
ReLU in Topic 3), the math works out roughly like this for a network with n
neurons per layer:
# Shallow: ONE hidden layer of width W regions β W # grows LINEARLY with neurons # Deep: L layers, each of width n regions β n^L # grows EXPONENTIALLY with depth # Plug in numbers: n = 10 neurons per layer 1 layer (shallow, 10 neurons) β ~10 regions 5 layers (deep, 50 neurons total) β ~10^5 = 100,000 regions
Read that again: 50 neurons arranged in 5 layers can express far more than 50 β or even 50,000 β neurons in a single layer. The shallow network would need an astronomical number of neurons to match what the deep one does with a handful. This is the depth efficiency result, made precise by researchers like MontΓΊfar (2014) and Telgarsky (2016).
You may hear about the Universal Approximation Theorem: a single hidden layer, if made wide enough, can approximate any continuous function. That is true β but it's a statement about possibility, not practicality. The catch is "wide enough" can mean an impossibly, exponentially large number of neurons. Depth lets you reach the same function with a network you can actually build and train. Possible β efficient.
| Wider (more neurons per layer) | Deeper (more layers) | |
|---|---|---|
| What it adds | More patterns detected at the same level | More levels of abstraction (composition) |
| Feature reuse | Little β features sit side by side | Heavy β each layer reuses the one below |
| Efficiency | Capability grows roughly linearly | Capability can grow exponentially |
| Risk | Bloated, parameter-hungry, can overfit | Harder to train (very deep = vanishing gradients β Session 3+) |
Deeper is more expressive, but historically very deep networks were hard to train: the learning signal would shrink to nothing as it travelled back through many layers (the vanishing gradient problem). The fixes β better activations like ReLU, smarter weight initialisation, and architectures like residual/skip connections β are a big part of why deep learning finally took off, which is exactly our next topic. We'll cover gradients and training mechanics in Session 3 and beyond.
3 The three ingredients: data, compute, algorithms
To bake a great cake you need three things: ingredients (flour, eggs), a hot enough oven, and a good recipe. If any one is missing, the cake flops. Deep learning is the same. The idea of deep networks existed for decades but the cake kept flopping β because we didn't have enough ingredients (data), a hot enough oven (computers), or a good recipe (the right tricks). Around 2012, all three finally showed up at once, and the cake came out perfect.
Neural networks are old ideas. So why did they suddenly explode in the 2010s and not the 1990s? Because deep learning needs three things together, and for a long time at least one was always missing. Those three ingredients are data, compute, and algorithms.
Ingredient 1 β Data
A deep network has millions of parameters (dials) to set. To set them well, it needs to see a lot of examples β otherwise it just memorises the few it saw (overfitting, Topic 4). The internet era gave us oceans of data. The landmark was ImageNet: a dataset of over 14 million hand-labelled images across thousands of categories, released around 2009. For the first time there was enough data to feed a hungry deep network.
Ingredient 2 β Compute
Training a deep network means doing trillions of multiply-and-add operations. On a normal CPU this took weeks or was simply infeasible. The breakthrough was realising that GPUs (Graphics Processing Units) β chips built to render video-game graphics β are perfect for this, because they do thousands of simple math operations in parallel. A training job that took weeks on a CPU could finish in days on a GPU.
Ingredient 3 β Algorithms
Even with data and GPUs, deep networks were notoriously hard to train (recall the vanishing gradient warning from Topic 2). A cluster of algorithmic improvements fixed that:
- ReLU (Rectified Linear Unit) β a dead-simple activation,
f(x) = max(0, x), that replaced the old "squashing" functions (sigmoid/tanh). Because its slope is exactly 1 for positive inputs, the learning signal no longer shrinks away through many layers. This single change made deep nets much easier to train. - Better weight initialisation β starting the network's random weights at the right scale (Xavier/Glorot 2010, He 2015) so signals neither explode nor die out as they pass through layers.
- Dropout (2012) and other regularisation tricks β randomly switching off neurons during training so the network can't lean on any one path, which fights overfitting.
- Faster optimisers and, later, normalisation layers (e.g. batch normalisation, 2015) β all making training stabler. We'll meet these in detail in later sessions.
The old sigmoid activation squashes everything into 0β1, and its
slope is tiny except near zero. When the learning signal multiplies many tiny slopes together
(once per layer), it vanishes:
# sigmoid: max slope is only 0.25 signal after 10 layers β 0.25^10 β 0.00000095 # basically zero β no learning # ReLU: slope is exactly 1 for positive inputs relu(x) = max(0, x) signal after 10 layers β 1^10 = 1 # signal survives β deep layers can learn!
That one change β swapping a squashing function for a straight line β is a huge part of why training 10, 50, or 150 layers became possible.
The "AI winters" β when ingredients were missing
An AI winter is a period when excitement (and funding) for AI collapsed because the technology under-delivered. There were two big ones, and in hindsight each was the result of missing ingredients:
| Era | What happened | What was missing |
|---|---|---|
| 1958β1969 | The perceptron (a single neuron, Session 3!) sparks huge hype. Then Minsky & Papert (1969) show one neuron can't even learn XOR. | Algorithms (no way to train multiple layers) |
| 1st winter (1970s) | Funding dries up; neural nets are written off. | β |
| 1986 | Backpropagation is popularised β finally a way to train multi-layer networks. A brief revival. | Now needed data + compute |
| 2nd winter (late 1980sβ1990s) | Nets work on toy problems but not big ones; computers too slow, datasets too small. Other methods (e.g. SVMs) win. | Data + compute |
| 2012 | AlexNet crushes the ImageNet contest (top-5 error ~16% vs ~26% for the runner-up). | Nothing β all three arrived! |
In 2012, a deep CNN called AlexNet (Krizhevsky, Sutskever & Hinton) entered the ImageNet image-recognition competition and won by a landslide β dropping the error rate by about 10 percentage points over the best non-deep method. It used all three ingredients at once: trained on ImageNet's millions of images (data), on GPUs (compute), using ReLU and dropout (algorithms). This is widely seen as the spark of the modern deep-learning era. Everything since β including the GenAI models in the sister course β descends from this moment.
A short timeline
| Year | Milestone |
|---|---|
| 1958 | Rosenblatt's perceptron β a single trainable neuron. |
| 1969 | Minsky & Papert show its limits β first AI winter. |
| 1986 | Backpropagation popularised β multi-layer nets become trainable. |
| 1989β98 | LeCun's LeNet reads handwritten digits (early CNN). |
| 2006 | Hinton et al. revive "deep" networks; the term deep learning spreads. |
| 2009 | ImageNet dataset released (data arrives). |
| 2012 | AlexNet wins ImageNet (compute + algorithms arrive). The boom begins. |
| 2015 | ResNet trains 150+ layers via skip connections; surpasses human-level on ImageNet classification. |
| 2017β | The Transformer arrives β the modern LLM / GenAI era. |
4 Limits & caveats β when deep learning isn't the answer
A bulldozer is amazing for digging a giant hole. But if you just need to plant one flower, a bulldozer is overkill β a little hand trowel is better, faster, and won't wreck your garden. Deep learning is the bulldozer: incredible for huge, messy jobs, but a clumsy, hungry, hard-to- understand choice for small or simple ones. A good engineer knows when to grab the trowel instead.
This whole session has been a cheerleader for depth. Now the honest part. Deep learning is powerful but it is not free and not always the right tool. Here are the real trade-offs, and a clear rule for when classical methods still win.
Caveat 1 β Data hunger
Those millions of parameters need a lot of examples to set correctly. With only a few hundred rows of data, a deep network has nothing to learn from and will perform poorly. Classical methods often do far better in the small-data regime.
Caveat 2 β Overfitting
Overfitting is when a model memorises the training examples instead of learning the general pattern β like a student who memorises the answer key but fails a test with new questions. Because deep nets are so flexible (recall their huge expressive power from Topic 2), they overfit easily when data is limited. That's exactly why we needed regularisers like dropout (Topic 3).
You watch two error numbers as training proceeds: error on the training set (data it learns from) and on a held-out validation set (data it never sees during training):
# Healthy fit: both errors fall together epoch 1: train_err = 40% val_err = 42% epoch 10: train_err = 12% val_err = 14% # good β generalising # Overfitting: train keeps dropping, validation turns BACK UP epoch 1: train_err = 40% val_err = 42% epoch 10: train_err = 3% val_err = 13% epoch 30: train_err = 0% val_err = 25% # memorising! val_err is rising
The fix when you see this: get more data, simplify the model, add regularisation, or stop training early (early stopping) at the point validation error was lowest.
Caveat 3 β Lack of interpretability
A deep network's "knowledge" is spread across millions of numbers. When it makes a decision, it usually can't tell you why in human terms β it's a black box. For a movie recommendation, who cares. For a loan denial, a medical diagnosis, or a self-driving decision, "the network said so" is not good enough β and may not even be legal. There's a whole field (explainable AI) working on this, but a simple model you can read is sometimes worth more than an accurate one you can't.
Caveat 4 β Cost, energy, and brittleness
- Cost & energy β training large models needs expensive hardware and a lot of electricity.
- Brittleness β deep nets can be fooled by tiny, deliberate tweaks to an input (adversarial examples) that a human wouldn't even notice.
- Confidently wrong β like the hallucinations seen in the GenAI course, a model can be very sure and very wrong.
When classical ML is still the better choice
Classical (non-deep) methods β logistic regression, decision trees, random forests, and especially gradient-boosted trees (XGBoost, LightGBM) β frequently beat deep learning in these situations:
| Situation | Better tool | Why |
|---|---|---|
| Small dataset (hundredsβfew thousand rows) | Classical ML | Deep nets starve without lots of data. |
| Tabular / spreadsheet data (rows & columns) | Gradient-boosted trees | Tree models routinely win on structured tables. |
| You must explain every decision | Linear models / trees | They're transparent and auditable. |
| Tight compute / no GPU / must run on a small device | Classical ML | Cheap to train and run. |
| Images, audio, language, huge unstructured data | Deep learning | This is exactly where depth shines. |
Reach for deep learning when you have lots of data and the input is unstructured (images, sound, text) where features are hard to hand-design. For small, tabular, or must-be-explainable problems, start with a simple classical model β it's often more accurate, far cheaper, and you can actually understand it. More power is not always more useful.
None of this means deep learning is bad β it means it's a specialised tool. The mark of a real practitioner isn't always choosing the fanciest model; it's choosing the right-sized one for the data and the stakes in front of them.
β Putting it all together
This session revisited "why deep learning?" with real depth. Here's the one-paragraph story that ties all four topics together:
Depth works because it lets a network compose features into a hierarchy β edges β textures β parts β objects β reusing simple parts to build complex ones. That reuse is why a deep network can be exponentially more efficient than a shallow-but-wide one (even though a single wide layer could approximate the same function in theory). This power was unlocked only when three ingredients arrived together β data (ImageNet), compute (GPUs), and algorithms (ReLU, good init, dropout) β ending the AI winters at the 2012 AlexNet moment. But depth is a bulldozer: data-hungry, overfit-prone, hard to interpret, so for small, tabular, or must-be-explainable problems, classical ML often still wins.
Quick self-check
In a CNN, what's the rough order of features from early to deep layers?
Edges β textures β object parts β whole objects. Each layer composes the features from the layer below it, and this hierarchy is learned, not programmed.
Why can a deep network be more efficient than a shallow-but-wide one?
Because layers reuse each other's work, expressive power can grow exponentially with depth (β nα΄Έ regions) rather than linearly with width. A shallow net would need an impractically huge number of neurons to match it.
If one wide hidden layer can approximate any function (Universal Approximation), why bother going deep?
Because "can" isn't "can practically." The required width can be astronomically large. Depth reaches the same function with a network you can actually build and train β possible β efficient.
What were the three ingredients that made deep learning take off, and which event showed all three together?
Data (ImageNet), compute (GPUs), and algorithms (ReLU, better initialisation, dropout). AlexNet winning the 2012 ImageNet competition is the moment they all came together.
Why did ReLU help train deep networks where sigmoid struggled?
Sigmoid's slope is tiny, so the learning signal vanishes when multiplied across many layers (vanishing gradient). ReLU's slope is exactly 1 for positive inputs, so the signal survives through many layers.
You have 800 rows of spreadsheet data and must explain every prediction. Deep net or not?
Not. That's small, tabular, and explainability-critical β a classic case for classical ML (e.g. logistic regression or gradient-boosted trees), which will likely be more accurate, cheaper, and transparent.
π References & Further Reading
Class material
- SST Deep Learning handout (Session 2) β your course handout for this session (priyanshsaxena.com/sst-deep-learning-nn).
Papers, docs & deep dives
- LeCun, Bengio & Hinton β "Deep Learning" (Nature, 2015) β the definitive overview of why feature hierarchies and depth work, written by three pioneers. The single best survey to read after this session.
- Goodfellow, Bengio & Courville β Deep Learning (the textbook, free online) β the standard reference; Chapter 6 covers depth, and Chapter 5 covers overfitting and classical ML trade-offs.
- Krizhevsky, Sutskever & Hinton β AlexNet (NeurIPS, 2012) β the ImageNet-winning paper that started the modern era; uses GPUs, ReLU, and dropout all at once.
- Distill β "Feature Visualization" β gorgeous, interactive visualisations of what each layer of a CNN actually learns (edges β textures β parts β objects).
- He et al. β "Deep Residual Learning" / ResNet (2015) β how skip connections finally made very deep (150+ layer) networks trainable, addressing the depth-vs-trainability trade-off from Topic 2.