1 The perceptron — a single artificial neuron
Imagine a tiny doorman deciding whether to let you into a party. He has a checklist: "Are you on the guest list?", "Did you bring a gift?", "Are you wearing shoes?" Some questions matter a lot to him, some barely matter. He adds up the answers — giving more weight to the ones he cares about — and if the total is high enough, he opens the door. That little yes/no doorman is exactly what a perceptron is.
A perceptron is the simplest possible artificial neuron — and the building block of every neural network. It was invented by Frank Rosenblatt in 1958, inspired by how a real brain cell works. Let's build it up piece by piece, because once you understand this one tiny unit, the whole rest of the course is just "lots of these, stacked together."
The biological analogy
A real brain cell (a neuron) receives little electrical signals through branches called dendrites. If the combined signal is strong enough, the neuron "fires" and sends its own signal down its axon to the next neuron. If the signal is too weak, it stays quiet. The artificial perceptron copies this idea exactly: take in several inputs, combine them, and either "fire" (output 1) or stay quiet (output 0).
The parts of a perceptron
| Part | Symbol | What it does |
|---|---|---|
| Inputs | x₁, x₂, …, xₙ | The numbers coming in (like the doorman's questions). |
| Weights | w₁, w₂, …, wₙ | How much each input matters. A big weight means "this input is important." |
| Bias | b | A built-in nudge — how eager the neuron is to fire even before seeing any input. |
| Weighted sum | z | All inputs multiplied by their weights and added up, plus the bias. |
| Activation | step() | The final decision: fire (1) or stay quiet (0). |
The formula
First the neuron computes the weighted sum — multiply each input by its weight, add them all up, then add the bias:
# Weighted sum (also called the "pre-activation", z) z = (w₁·x₁) + (w₂·x₂) + … + (wₙ·xₙ) + b # In compact form, where ∑ means "add up all of them": z = ∑(wᵢ · xᵢ) + b
Then it passes z through a step activation
function to make the final yes/no decision:
# Step function: fire if the sum clears the threshold (here, 0) output = 1 if z >= 0 output = 0 if z < 0
The bias b shifts the threshold. With a big positive bias, the neuron
fires easily (an eager doorman). With a big negative bias, it's very hard to make it fire (a
strict doorman). Without a bias, the neuron would be forced to fire exactly when the weighted
inputs hit zero — which is rarely what we want. The bias gives it freedom to set its own
threshold.
Let's build a perceptron that outputs 1 only when both inputs are 1 (the
logical AND). We'll use weights w₁ = 1, w₂ = 1,
and bias b = -1.5. Now feed it every combination:
| x₁ | x₂ | z = x₁+x₂ − 1.5 | step(z) |
|---|---|---|---|
| 0 | 0 | 0 + 0 − 1.5 = −1.5 | 0 |
| 0 | 1 | 0 + 1 − 1.5 = −0.5 | 0 |
| 1 | 0 | 1 + 0 − 1.5 = −0.5 | 0 |
| 1 | 1 | 1 + 1 − 1.5 = 0.5 | 1 |
It fires (outputs 1) only on the last row, exactly when both inputs are 1. We just built a logic gate out of one neuron, purely by choosing the right weights and bias!
The same thing in code
def perceptron(inputs, weights, bias): # weighted sum: multiply each input by its weight, add them, add bias z = bias for x, w in zip(inputs, weights): z = z + x * w # step activation: fire if the sum reaches the threshold return 1 if z >= 0 else 0 # Our AND neuron from above print(perceptron([1, 1], [1, 1], -1.5)) # -> 1 print(perceptron([1, 0], [1, 1], -1.5)) # -> 0
A perceptron does just three things: weight its inputs, sum them (plus a bias), and fire if the total clears a threshold. "Learning" later will just mean automatically finding good weights and biases — that's the whole game.
z, then a step function turns z into a 1 or 0.
Choose the weights and bias well and a single neuron can act like a logic gate (we built AND).
2 Activation functions — adding the spark
Imagine a light dimmer switch. A plain on/off switch can only do two things — fully on or fully off. A dimmer can do anything in between: dim, medium, bright. An activation function is the dimmer switch of a neuron. It decides how the neuron turns its inner sum into an output — and choosing a smart dimmer is what lets a network learn curvy, complicated patterns instead of just straight lines.
In Topic 1 we used the simplest activation: the hard step function. But the step has a problem — it's flat everywhere and then jumps suddenly, which makes it impossible to train with the smooth math we'll meet in Session 4. So in practice we use smoother activation functions. They do two jobs: shape the output into a useful range, and — crucially — add non-linearity.
Why non-linearity is essential
This is the single most important idea in this topic, so let's be careful. A linear operation is just "multiply and add" — it can only draw straight lines (or flat planes). Here's the catch: if you stack many neurons that only multiply and add, the whole stack collapses back into one big multiply-and-add. A pile of straight lines is still just a straight line.
Without a non-linear activation between layers, a 100-layer network is mathematically identical to a single layer — useless. The non-linearity is what lets stacked neurons bend and fold the space, so the network can learn curves, corners, and complex patterns. No activation = no deep learning.
The common activation functions
Let's meet the cast. Each takes the weighted sum z and reshapes it.
Sigmoid
The sigmoid squashes any number into the range 0 to 1, smoothly. Big positive numbers → near 1; big negative → near 0; zero → exactly 0.5. Great for representing a probability.
sigmoid(z) = 1 / (1 + e^(−z))
z: −6 −2 0 2 6
σ: 0.00 0.12 0.50 0.88 1.00 # a smooth S-shaped curve
Tanh
The tanh (hyperbolic tangent) is sigmoid's cousin, but it squashes into the range −1 to +1 and is centred at 0. Being zero-centred often makes training a little easier than sigmoid.
tanh(z) = (e^z − e^(−z)) / (e^z + e^(−z))
z: −3 0 3
tanh: −0.99 0.00 0.99 # S-shaped, but spanning −1 to +1
ReLU
The ReLU (Rectified Linear Unit) is the workhorse of modern deep learning. It's brutally simple: if the input is positive, pass it through unchanged; if it's negative, output 0. Fast to compute and it dodges a big problem we'll meet in a moment.
ReLU(z) = max(0, z)
z: −3 −1 0 2 5
ReLU: 0 0 0 2 5 # a flat floor, then a straight ramp up
Leaky ReLU
Plain ReLU has one weakness: if a neuron's input is always negative, it outputs 0 forever and stops learning (a "dead neuron"). Leaky ReLU fixes this by letting a tiny slope through on the negative side instead of a hard zero.
LeakyReLU(z) = z if z > 0 LeakyReLU(z) = 0.01 * z if z <= 0 z: −3 −1 0 2 out: −0.03 −0.01 0 2 # a gentle slope instead of dead-flat zero
Softmax
The others handle one neuron at a time. Softmax is different: it takes a whole row of output scores and turns them into a set of probabilities that add up to 1. It's used at the very end of a classifier to answer "which class is this?" (You saw softmax in the GenAI course turning model scores into probabilities — same function, same job.)
softmax(zᵢ) = e^(zᵢ) / ∑ e^(zⱼ) # divide each by the total scores [2.0, 1.0, 0.1] → softmax → [0.66, 0.24, 0.10] # adds up to 1.00 = 100%
Pros and cons at a glance
| Function | Output range | Pros | Cons |
|---|---|---|---|
| Step | 0 or 1 | Simplest; pure decision. | Not smooth → can't train with gradients. |
| Sigmoid | 0 to 1 | Smooth; reads like a probability. | Saturates at the ends → vanishing gradients; not zero-centred. |
| Tanh | −1 to 1 | Smooth and zero-centred. | Still saturates → vanishing gradients. |
| ReLU | 0 to ∞ | Fast; avoids vanishing gradient for positive inputs; today's default. | "Dead neurons" if stuck negative. |
| Leaky ReLU | −∞ to ∞ | Fixes dead neurons with a small negative slope. | Extra tiny knob (the leak slope) to choose. |
| Softmax | 0 to 1 (sums to 1) | Turns scores into class probabilities. | Only for the output layer of a classifier. |
Notice that sigmoid and tanh go almost flat at their extremes. When the curve is flat, its slope is nearly zero. In Session 4 (Backpropagation) we'll learn that networks improve by following slopes — so a near-zero slope means an early layer barely gets a learning signal. In a deep network these tiny slopes multiply together and shrink to almost nothing — the vanishing-gradient problem. This is the main reason ReLU largely replaced sigmoid/tanh in deep hidden layers. Hold that thought for next session.
Suppose a neuron computes a weighted sum of z = −2. What comes out
depends entirely on its activation function:
- Step → 0 (since −2 < 0, it stays quiet)
- Sigmoid → 0.12 (a small probability)
- Tanh → −0.96 (strongly negative)
- ReLU → 0 (negatives become zero)
- Leaky ReLU → −0.02 (a tiny leak gets through)
Same input, five different "personalities." Choosing the activation is a real design decision.
3 Multi-Layer Perceptrons (MLP)
One doorman (a perceptron) can only make a simple decision. But imagine a whole office: the front-desk people each notice one small thing, pass notes to a middle team, who combine those notes and pass them to a boss who makes the final call. Each person is simple, but as a team passing messages forward they can solve hard problems. That team of neurons, organised in rows, is a Multi-Layer Perceptron.
A Multi-Layer Perceptron (MLP) is what you get when you stop using one neuron and instead organise many of them into layers. This is, finally, a real neural network. There are three kinds of layer:
- The input layer isn't really neurons — it's just your data (one slot per feature, e.g. the brightness of each pixel).
- One or more hidden layers sit in the middle. They're called "hidden" because you never see their values directly — they're the network's internal scratch work. This is where patterns get built up.
- The output layer produces the final result (one neuron for yes/no, or several with softmax for "which class").
Fully-connected layers
In a standard MLP, every neuron in one layer connects to every neuron in the next. That's called a fully-connected (or dense) layer. Each connection has its own weight, and each neuron has its own bias. A layer with 3 inputs and 4 neurons therefore has 3 × 4 = 12 weights plus 4 biases to learn.
That's the whole secret behind the word deep in "deep learning" — a network with many hidden layers stacked up is a deep network. Each layer learns slightly more abstract features than the one before (edges → shapes → objects, for an image).
The forward pass with matrix math
Running data through the network from input to output is called the forward pass. We could compute each neuron one at a time like in Topic 1, but it's far cleaner to do a whole layer at once with a matrix. For one layer:
# x = inputs (a row of numbers) # W = weight matrix (one column per neuron in this layer) # b = bias vector (one per neuron) # f = the activation function z = x · W + b # matrix multiply, then add biases (the weighted sums) a = f(z) # apply activation elementwise (the layer's output)
The output a of one layer becomes the input x of
the next. Repeat through every layer and the final a is your answer. That's
the entire forward pass.
Tiny network. Inputs x = [1, 0]. Hidden layer has 2 neurons using
ReLU; output layer has 1 neuron using sigmoid. Watch the numbers flow:
# ---- Given weights & biases ---- x = [1, 0] # the input # Hidden layer: 2 inputs -> 2 neurons W1 = [[ 0.5, -0.4], # weights from x1 to (h1, h2) [ 0.8, 0.2]] # weights from x2 to (h1, h2) b1 = [0.1, -0.1] # Output layer: 2 inputs -> 1 neuron W2 = [[1.0], [0.6]] b2 = [0.2] # ---- Step 1: hidden weighted sums z1 = x·W1 + b1 ---- h_z1 = 1*0.5 + 0*0.8 + 0.1 = 0.6 # neuron h1 h_z2 = 1*(-0.4) + 0*0.2 - 0.1 = -0.5 # neuron h2 # ---- Step 2: hidden activation (ReLU) ---- h1 = ReLU(0.6) = 0.6 h2 = ReLU(-0.5) = 0.0 # negative -> clamped to 0 h = [0.6, 0.0] # ---- Step 3: output weighted sum z2 = h·W2 + b2 ---- o_z = 0.6*1.0 + 0.0*0.6 + 0.2 = 0.8 # ---- Step 4: output activation (sigmoid) ---- output = sigmoid(0.8) = 0.69 # final answer ~ 69%
Every number above came purely from multiplying, adding, and applying an activation — exactly the perceptron from Topic 1, just done many times and chained layer to layer.
The same forward pass in code
import numpy as np def relu(z): return np.maximum(0, z) def sigmoid(z): return 1 / (1 + np.exp(-z)) x = np.array([1.0, 0.0]) W1 = np.array([[0.5, -0.4], [0.8, 0.2]]) b1 = np.array([0.1, -0.1]) W2 = np.array([[1.0], [0.6]]) b2 = np.array([0.2]) h = relu(x @ W1 + b1) # hidden layer -> [0.6, 0.0] y = sigmoid(h @ W2 + b2) # output layer -> [0.69] print(y) # -> [0.69...]
An MLP is just neurons arranged in layers, fully connected, run front-to-back. The forward pass is the same move repeated: multiply by weights, add bias, apply activation, then hand the result to the next layer. Matrices let us do a whole layer in one line.
a = f(x·W + b) at each layer,
feeding one layer's output into the next. More hidden layers = "deeper" = more abstract patterns.
4 The XOR problem — why we need hidden layers
Imagine four toys on the floor: two red ones in opposite corners, two blue ones in the other two corners. Now try to separate the reds from the blues by laying down a single straight piece of string. You can't — no matter how you angle it, a red and a blue end up on the same side! You'd need to bend the string. That bend is exactly what a hidden layer gives a neural network.
The XOR problem ("exclusive OR") is the most famous puzzle in neural network history. XOR outputs 1 when the two inputs are different, and 0 when they're the same:
| x₁ | x₂ | XOR output |
|---|---|---|
| 0 | 0 | 0 (same) |
| 0 | 1 | 1 (different) |
| 1 | 0 | 1 (different) |
| 1 | 1 | 0 (same) |
Why a single perceptron CANNOT solve XOR
Recall from Topic 1 that a perceptron fires when w₁x₁ + w₂x₂ + b >= 0.
That equation draws a single straight line through the input space, and the
perceptron says "1 on this side, 0 on that side." So a perceptron can only solve problems that are
linearly separable — ones you can split with one straight line.
Now plot the four XOR points on a grid and mark which should output 1:
x2
1 | ●(0,1)=1 ○(1,1)=0
|
|
0 | ○(0,0)=0 ●(1,0)=1
+-------------------------- x1
0 1
● = should output 1 ○ = should output 0
The two "1" points sit in opposite corners (bottom-right and top-left). The two "0" points sit in the other opposite corners. There is no single straight line that puts both ●'s on one side and both ○'s on the other. Try it — every line you draw gets at least one point wrong. XOR is not linearly separable, so one perceptron is mathematically incapable of solving it.
In 1969, Minsky and Papert's book Perceptrons proved exactly this limitation. The field lost confidence and funding dried up for years — the first "AI winter." The fix was known in principle (add a hidden layer) but a good way to train such networks didn't catch on until backpropagation became popular in the 1980s — which is precisely our next session.
Why a 2-layer MLP CAN solve XOR
The trick: use a hidden layer to first transform the problem into one that is linearly separable. Each hidden neuron draws its own line; the output neuron then combines those lines to carve out a bent boundary. A classic solution uses two hidden neurons that compute simpler gates, then combines them:
The insight in plain English: XOR means "at least one input is on, but not both." That's exactly OR minus AND — and each of those is a straight-line (linearly separable) problem a single neuron can do. The hidden layer computes OR and AND; the output layer subtracts.
Two hidden neurons (h₁ = OR, h₂ = AND) using a step activation, then an output neuron that fires for "OR is on AND AND is off." Concrete weights:
# Hidden neuron h1 (OR): fires if x1 + x2 - 0.5 >= 0 # Hidden neuron h2 (AND): fires if x1 + x2 - 1.5 >= 0 # Output (XOR): fires if h1 - h2 - 0.5 >= 0 # ( = "OR is on, but AND is not" )
Run all four inputs through it:
| x₁ | x₂ | h₁ (OR) | h₂ (AND) | h₁ − h₂ − 0.5 | Output | XOR? |
|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | −0.5 | 0 | ✓ |
| 0 | 1 | 1 | 0 | 0.5 | 1 | ✓ |
| 1 | 0 | 1 | 0 | 0.5 | 1 | ✓ |
| 1 | 1 | 1 | 1 | −0.5 | 0 | ✓ |
All four rows correct! What one perceptron couldn't do, two layers handle with ease — because the hidden layer reshaped the problem into a linearly separable one.
XOR is the textbook proof that hidden layers add real power. A single perceptron only draws one straight line, so it fails on patterns that aren't linearly separable. Stacking layers lets the network bend the boundary — and that's why every useful neural network has hidden layers.
★ Putting it all together
You just built a neural network from scratch — from one neuron all the way to a network that beats the famous XOR puzzle. Here's the one-paragraph story tying all four topics together:
A perceptron is a single artificial neuron: it takes inputs, multiplies each by
a weight, adds them up with a bias to get the weighted sum, and
fires through an activation function. Plain step activations can't be trained
smoothly and can't bend space, so we use non-linear activations (sigmoid, tanh,
ReLU, Leaky ReLU, softmax) — without them, stacked layers collapse into one. Stacking neurons into
input, hidden, and output layers gives a Multi-Layer Perceptron, run via the
forward pass a = f(x·W + b) layer after layer. And the
XOR problem shows exactly why we bother: one perceptron only draws a straight line
and fails on non-linearly-separable data, but a hidden layer reshapes the problem so the network can
solve it. Next session — backpropagation — we'll finally learn how the network
finds all those weights and biases on its own.
Quick self-check
What are the three computations a perceptron performs?
It (1) multiplies each input by its weight, (2) sums them and adds the bias to get
the weighted sum z, and (3) passes z through an
activation function to produce the output.
Why do we need a non-linear activation function between layers?
Because stacking purely linear (multiply-and-add) layers collapses into a single linear layer. The non-linearity lets the network bend space and learn curves and complex patterns — it's what makes depth meaningful.
What does the bias do in a neuron?
It shifts the firing threshold. A positive bias makes the neuron fire more easily; a negative bias makes it harder to fire. Without it, the threshold is stuck at zero.
Why can't a single perceptron solve XOR?
A perceptron can only separate data with one straight line (linearly separable problems). XOR's two "1" points sit in opposite corners, so no single line can split the 1s from the 0s.
How does a 2-layer MLP manage to solve XOR?
The hidden layer computes simpler linearly-separable functions (e.g. OR and AND), and the output layer combines them ("OR but not AND"). The hidden layer reshapes the problem into one that's linearly separable.
Why is ReLU often preferred over sigmoid in hidden layers?
Sigmoid saturates (goes flat) at its extremes, giving near-zero slopes that cause the vanishing-gradient problem in deep networks. ReLU stays linear for positive inputs, keeping gradients healthy, and it's faster to compute.
📚 References & Further Reading
Class material
- SST Deep Learning handout (Session 3) — your course handout for this session, available at priyanshsaxena.com/sst-deep-learning-nn.
Papers, docs & deep dives
- Rosenblatt, "The Perceptron" (1958) — the original paper that introduced the perceptron and the biological-neuron analogy we built on in Topic 1.
- Minsky & Papert, "Perceptrons" (1969) — the book that proved a single perceptron can't solve XOR; essential context for Topic 4.
- 3Blue1Brown — Neural Networks series — gorgeous animated explainers of neurons, layers, and the forward pass; the best visual companion to this session.
- Stanford CS231n — Neural Networks Part 1 — a rigorous, beginner-friendly write-up of neurons, activation functions (with pros/cons), and fully-connected layers.
- Goodfellow, Bengio & Courville — "Deep Learning" — the free authoritative textbook; Chapter 6 covers feedforward networks and the XOR example in depth.