πŸ“š Study Notes / Home / Neural Nets / Cheat Sheet
Neural Nets & CV Β· 1-Page Cheat Sheet

Neural Networks & Computer Vision β€” Quick Revision

Every core idea from perceptrons to nanoGPT, condensed into one dense, scannable page.

← Full sessions

S1Β·S2 Why deep learning wins

  • Universal approximation: one hidden layer can fit any continuous function β€” but may need exponentially many neurons.
  • Depth buys exponential efficiency: deep nets reuse features, composing simple parts into complex ones.
  • Representation learning: net learns features itself (edges β†’ shapes β†’ objects) vs hand-crafted features.
  • Took over thanks to big data + GPUs + better algorithms (ReLU, dropout, Adam).
  • Hierarchy of abstraction = why "deep" beats wide-and-shallow.

S3 Perceptron β†’ MLP

  • Perceptron: y = step(wΒ·x + b) β€” only linearly separable problems.
  • XOR is not linearly separable β†’ needs a hidden layer (stacked perceptrons).
  • MLP: fully-connected layers + nonlinear activation; without nonlinearity, stacked layers collapse to one linear map.
  • Activations: sigmoid (0–1, saturates), tanh (βˆ’1–1, zero-centered), ReLU=max(0,x) (default), LeakyReLU, softmax (output probs).
  • Forward pass = repeated a = act(WΒ·x + b).

S4 Backprop & comp. graphs

  • Learning = gradient descent: w ← w βˆ’ Ξ·Β·βˆ‚L/βˆ‚w.
  • Backprop = chain rule applied over the computational graph, back-to-front.
  • Each node: local gradient Γ— upstream gradient (reuse cached forward values).
  • Autodiff (reverse mode) computes all grads in one backward pass β€” what PyTorch/TF do.
  • Forward computes outputs; backward computes βˆ‚L/βˆ‚param for every weight.

S5 Losses & optimizers

  • Losses: MSE for regression; cross-entropy for classification (pairs with softmax/sigmoid).
  • SGD: noisy mini-batch steps. Momentum: velocity smooths + accelerates.
  • Adam = momentum + per-param adaptive LR (RMSProp); robust default.
  • LR schedules: step decay, cosine, warmup β€” too high diverges, too low crawls.
  • Batch size trades gradient noise vs speed/memory.

S6 Regularization

  • Overfitting: low train loss, high val loss (memorizing not generalizing).
  • Dropout: randomly zero neurons at train time β†’ ensemble effect.
  • Weight decay / L2: penalize large weights; L1 β†’ sparsity.
  • Early stopping: halt when val loss rises.
  • Data augmentation: flips/crops/rotations expand data cheaply.
  • BatchNorm: normalize activations per batch β†’ faster, more stable training (mild regularizer).

S7 Init & debugging

  • Vanishing gradients: deep nets + saturating acts β†’ grads shrink to 0 (no learning).
  • Exploding gradients: grads blow up β†’ NaN; fix with gradient clipping.
  • Xavier/Glorot init for tanh/sigmoid; He init for ReLU β€” keeps variance stable.
  • Gradient checking: compare backprop grad to numerical (L(w+Ξ΅)βˆ’L(wβˆ’Ξ΅))/2Ξ΅.
  • Debug: overfit a tiny batch first; watch loss curves & activation stats.

S8 Build a net (NumPy & PyTorch)

  • From scratch: forward β†’ loss β†’ backward β†’ update loop, all by hand.
  • Framework: nn.Module, autograd, optimizer.step(), loss.backward(), zero_grad().
  • Training loop: epoch β†’ batches β†’ forward β†’ loss β†’ backward β†’ step.
  • Pieces assembled: layers, activations, GD, backprop, regularization, init.

S9 Convolutions

  • Filter/kernel slides over image, shares weights β†’ translation invariance + few params.
  • Padding keeps size ("same"); stride downsamples.
  • Out size: (W βˆ’ F + 2P)/S + 1.
  • Receptive field: region of input one output sees; grows with depth.
  • Pooling (max/avg) downsamples + adds invariance; channels = feature maps.

S10 CNN architectures

  • LeNet: tiny digit reader. AlexNet (2012): ReLU+dropout+GPU, won ImageNet.
  • VGG: deep, uniform 3Γ—3 convs. Inception/GoogLeNet: parallel multi-scale convs, 1Γ—1 bottlenecks.
  • ResNet (152 layers): skip/residual connections y=F(x)+x solve vanishing grads in very deep nets.
  • Trend: deeper + smarter blocks, fewer params per accuracy.

S11 Transfer learning & CV tasks

  • Transfer learning: reuse ImageNet-pretrained backbone; replace + fine-tune head.
  • Freeze early layers (generic features), train later layers for your task.
  • Detection: localize + classify (boxes) β€” R-CNN, YOLO, SSD.
  • Segmentation: per-pixel labels β€” U-Net, FCN (semantic vs instance).
  • Few labels? Transfer learning is the go-to.

S12 RNNs

  • Process sequences step-by-step, carrying a hidden state h_t = act(WΒ·x_t + UΒ·h_{t-1}).
  • Shared weights across time; handles variable length.
  • BPTT (backprop through time): unroll then backprop.
  • Vanishing gradients β†’ can't learn long-range dependencies.
  • Types: one-to-many, many-to-one, many-to-many.

S13 LSTM & GRU

  • LSTM: cell state + gates (forget, input, output) let gradients flow β†’ long memory.
  • Forget gate decides what to drop; input gate what to add; output gate what to expose.
  • GRU: simpler (reset + update gates), fewer params, often comparable.
  • Bidirectional: read sequence both ways β†’ richer context.
  • Gating = controlled remember/forget, fixing plain-RNN vanishing grads.

S14 Seq2seq & attention

  • Seq2seq: encoder compresses input into a context vector, decoder generates output.
  • Bottleneck: single fixed context loses info on long inputs.
  • Attention: decoder looks back at all encoder states, weighted by relevance.
  • Alignment weights = soft mapping between output and input tokens.
  • Used for translation; teacher forcing during training.

S15 The Transformer

  • No recurrence β€” pure attention, fully parallel.
  • Self-attention: softmax(QKα΅€/√d)Β·V with query/key/value projections.
  • Multi-head: several attention subspaces in parallel.
  • Positional encoding injects order (no recurrence to track it).
  • Block = attention + FFN + residual + LayerNorm; encoder/decoder stacks.

S16 Transformers in practice

  • BERT: encoder-only, bidirectional, masked-LM pretraining β†’ understanding/classification.
  • GPT: decoder-only, causal (left-to-right) LM β†’ generation.
  • Tokenization: BPE / WordPiece split text into subword units (handle rare words).
  • Pretrain on huge corpus β†’ fine-tune on downstream task.
  • Foundation models = pretrain once, adapt many times.

S17 Project: Neural Style Transfer

  • Combine content of one image with style of another.
  • Use pretrained VGG19 features (no training the net).
  • Content loss: match deep feature maps; style loss: match Gram matrices (feature correlations).
  • Optimize the image pixels (not weights) via gradient descent.

S18 Project: char-RNN & translator

  • Char-RNN on Shakespeare: predict next character, sample to generate text.
  • LSTM/GRU learns spelling, structure, style char-by-char.
  • Seq2seq translator: encoder-decoder + attention on sentence pairs.
  • Hands-on: data prep, teacher forcing, sampling temperature.

S19Β·S20 Project: mini-GPT & nanoGPT

  • Build a runnable mini-Transformer (decoder-only) in PyTorch from scratch.
  • Pieces: token + positional embeddings, masked self-attention, FFN blocks, LM head.
  • Train a small LM (nanoGPT-style) on text; autoregressive sampling.
  • LoRA: low-rank adapters β†’ fine-tune cheaply by training tiny extra matrices, freezing the base.
  • Capstone = full pipeline: data β†’ train β†’ generate.