Neural Networks and Backpropagation

A neural network is a stack of simple units that together can learn very complex functions. It is the foundation of image recognition, speech and language models.

A neuron

A neuron computes a weighted sum of its inputs plus a bias, then applies an activation:

a = g(\mathbf{w}^\top\mathbf{x} + b)

With a sigmoid activation, a single neuron is exactly logistic regression.

Layers

Neurons are organised in layers. A layer with weight matrix $W$ (shape out × in) transforms its input vector:

\mathbf{h} = g(W\mathbf{x} + \mathbf{b})

A multi-layer perceptron (MLP) chains layers:

flowchart LR
  subgraph Input
    x1((x1)) & x2((x2)) & x3((x3))
  end
  subgraph Hidden["Hidden layer (ReLU)"]
    h1((h1)) & h2((h2)) & h3((h3)) & h4((h4))
  end
  subgraph Output["Output (softmax)"]
    o1((ŷ1)) & o2((ŷ2))
  end
  x1 & x2 & x3 --> h1 & h2 & h3 & h4
  h1 & h2 & h3 & h4 --> o1 & o2

\mathbf{h} = \text{ReLU}(W_1\mathbf{x} + \mathbf{b}_1), \qquad \hat{\mathbf{y}} = \text{softmax}(W_2\mathbf{h} + \mathbf{b}_2)

Why non-linearity

Without activations, two layers collapse into one: $W_2(W_1\mathbf{x}) = (W_2W_1)\mathbf{x}$ , still linear. Non-linear activations let each layer bend the space, and stacking them builds complicated decision boundaries.

Activation	Formula	Notes
Sigmoid	$\frac{1}{1+e^{-z}}$	Output in (0,1); saturates, so gradients vanish; used for binary outputs
Tanh	$\frac{e^z - e^{-z}}{e^z + e^{-z}}$	Zero-centred, but still saturates
ReLU	$\max(0, z)$	Default for hidden layers: cheap, no saturation for $z > 0$ ; units can “die” at 0
Leaky ReLU	$\max(0.01z, z)$	Avoids dead units
GELU / SiLU	smooth ReLU-like	Used in transformers
Softmax	$\frac{e^{z_k}}{\sum_j e^{z_j}}$	Output layer for multi-class probabilities

Universal approximation: a network with one hidden layer and enough units can approximate any continuous function on a bounded region. In practice, depth is far more efficient than width: deep networks reuse features hierarchically (edges → textures → parts → objects).

Training: forward pass, loss, backward pass

Forward: compute activations layer by layer and store them.
Loss: e.g. cross-entropy between $\hat{\mathbf{y}}$ and the true label.
Backward (backpropagation): use the chain rule to get $\partial L / \partial W$ for every layer, from the output back to the input.
Update with SGD or Adam.

Backprop for a two-layer network

Forward: $\mathbf{z}_1 = W_1\mathbf{x} + \mathbf{b}_1$ , $\mathbf{a}_1 = \text{ReLU}(\mathbf{z}_1)$ , $\mathbf{z}_2 = W_2\mathbf{a}_1 + \mathbf{b}_2$ , $\hat{\mathbf{y}} = \text{softmax}(\mathbf{z}_2)$ , $L = -\log \hat{y}_{\text{true}}$ .

Backward (the softmax + cross-entropy gradient is famously clean):

\boldsymbol{\delta}_2 = \frac{\partial L}{\partial \mathbf{z}_2} = \hat{\mathbf{y}} - \mathbf{y}

\frac{\partial L}{\partial W_2} = \boldsymbol{\delta}_2\,\mathbf{a}_1^\top, \qquad \frac{\partial L}{\partial \mathbf{b}_2} = \boldsymbol{\delta}_2

\boldsymbol{\delta}_1 = \frac{\partial L}{\partial \mathbf{z}_1} = (W_2^\top\boldsymbol{\delta}_2) \odot \mathbb{1}[\mathbf{z}_1 > 0]

\frac{\partial L}{\partial W_1} = \boldsymbol{\delta}_1\,\mathbf{x}^\top, \qquad \frac{\partial L}{\partial \mathbf{b}_1} = \boldsymbol{\delta}_1

The pattern repeats for any depth: error signal × input to the layer gives the weight gradient, and Wᵀ × error, gated by the activation’s derivative passes the error back one layer. The cost of the backward pass is about twice the forward pass.

Parameters vs hyperparameters

Parameters: weights and biases, learned by gradient descent.
Hyperparameters: chosen by you, such as the number of layers and units, activation, learning rate, batch size, epochs, regularisation and optimiser. Tune them on validation data (grid, random or Bayesian search).