Perceptron to MLP

1 The single neuron

• Borrowed idea from biology: weighted inputs → threshold → output (we model the idea, not the biology).
• A neuron does two steps:

1. Weighted sum:  z = w1·x1 + w2·x2 + b
2. Activation:    ŷ = f(z)   (f is a non-linear function)

Full form:  ŷ = f(w1·x1 + w2·x2 + b) = f(wᵀx + b)

• Weights (w) scale each input; bias (b) shifts the result.

2 Neuron = logistic regression

• A single neuron with sigmoid activation is exactly logistic regression: ŷ = σ(wᵀx + b).
• Both: weighted sum of features → sigmoid squashes to (0,1) → binary classification.
• Takeaway: you already know neural networks — a neuron is the atom they're built from.

3 Decision boundary & bias

• A 2-input neuron draws one straight line in 2D: w1·x1 + w2·x2 + b = 0 (a hyperplane in nD).
• Change weights → line rotates; change bias → line shifts.
• Without a bias the boundary must pass through the origin — bias is the intercept that lets it move to the right spot.
• Limitation: a single neuron can only make a linear boundary → cannot separate data that isn't linearly separable.

4 Why activation functions

• Stack two linear layers and substitute:

h = W1·x + b1
y = W2·h + b2 = (W2·W1)·x + (W2·b1 + b2)
              = W'·x + b'        ← still ONE linear layer

• Linear × linear = linear. No matter how many layers you stack, with no activation it collapses to a single linear map.
• Activations add the non-linear "bend" that makes depth meaningful (analogy: stacking flat glass never makes a lens — you need a curve).

5 Activation functions

Sigmoid:  σ(z) = 1 / (1 + e⁻ᶻ)
Tanh:     tanh(z) = (eᶻ − e⁻ᶻ)/(eᶻ + e⁻ᶻ),   tanh(z) = 2σ(2z) − 1
ReLU:     max(0, z)
LeakyReLU:max(αz, z),  small α (e.g. 0.01)

• Sigmoid: output (0,1), interpretable as probability, smooth. Cons: vanishing gradient for large |z|, not zero-centered, slow in hidden layers → use at output for binary.
• Tanh: output (−1,1), zero-centered, stronger gradients near 0. Still vanishes at extremes; common in RNNs.
• ReLU: cheap (just a comparison), no vanishing gradient for z>0, sparse activation. Cons: dead neuron problem (if z<0 always, gradient stays 0). Default for hidden layers.
• LeakyReLU / ELU: small slope for z<0 → fix dead neurons.

6 From neuron to layer to MLP

• A layer = many neurons in parallel; each has its own weights and draws its own boundary → many boundaries at once.
• Multi-Layer Perceptron (MLP): Input → Hidden layer(s) → Output, fully connected (every neuron connects to every neuron in the next layer).
• What depth buys you: 1 layer = straight lines, 2 layers = convex shapes, 3+ layers = arbitrary shapes. More layers → more complex boundaries → more powerful models.

7 Forward pass

• The forward pass = data flowing input → output, one layer at a time:

For each layer:  z = W·x + b   →   a = f(z)   →   feed a to next layer

• In words: multiply by weights, add bias, apply activation, repeat until the output layer.

8 The XOR problem

• XOR is not linearly separable → a single neuron (one line) cannot solve it.
• Fix: add one hidden layer with 2 neurons (a 2-2-1 MLP). Each hidden neuron learns one line; the output neuron combines them (AND/OR-like) to give correct XOR.
• This is the core reason "deep" learning is deep — depth solves problems a flat model can't.
• History: Minsky & Papert (1969) proved the perceptron can't do XOR. The field concluded neural nets were a dead end; the fix (hidden layers + backprop) wasn't developed until 1986 — a ~17-year gap.

9 MLP architecture choices

• How many hidden layers? Start with 1–2; add more only if underfitting.
• Neurons per layer? Common: 32, 64, 128, 256 (powers of 2 for GPU efficiency).
• Which activation? ReLU hidden; sigmoid (binary) / softmax (multi-class) output.
• Magic formula? No — architecture design is empirical: try, evaluate, adjust.

★ Likely exam questions