Backpropagation Algorithm

5.1The blame problem

In Module 4 you learned the rule for improving a weight: new weight = weight − learning rate × gradient. The gradient tells us how much the loss changes when that one weight changes. For a single neuron, that's easy to work out. But in a deep network, a weight in the first layer affects the output only indirectly, through every layer after it.

A slow way to find each gradient would be to nudge every weight one at a time and re-run the network to see what changes. With a million weights, that's a million extra runs per step. Backpropagation gets all the gradients in roughly the time of one extra run. That efficiency is why it transformed the field when it was popularised in 1986.

5.2The chain rule, without the scary maths

Backpropagation is built on one idea from calculus called the chain rule. Here it is with gears:

A neural network is a chain too: weight → neuron → next layer → … → output → loss. To find how much a weight affects the loss, we multiply the effects of every link in its chain. And because the later links are shared by many weights, we can calculate them once and reuse them. That's the trick that makes backpropagation fast.

5.3A worked example

Let's follow the smallest possible network: 1 input, 1 hidden neuron and 1 output neuron, both using sigmoid. The input is x = 1, the correct answer is y = 1, and the weights start at w₁ = 0.5 and w₂ = 0.8 (biases are 0 to keep it simple). The loss is ½ × (y − prediction)².

Step 1: forward pass

Step 2: backward pass

Now we go backwards, multiplying the effect of each link (the chain rule):

Step 3: update the weights

With a learning rate of 1: w₂ = 0.8 − 1 × (−0.0553) = 0.855 and w₁ = 0.5 − 1 × (−0.0167) = 0.517. Run the forward pass again and the prediction rises from 0.622 to 0.652, and the loss drops from 0.0714 to 0.0605. One step closer to the right answer.

5.4Try it: step through backpropagation

Here's the same tiny network. Press the buttons in order to run the forward pass, the backward pass and the update, and watch the loss fall with every round.

5.5Backpropagation in a full network

In a real network, the same three steps happen for every layer at once:

A neuron that sends its output to several neurons in the next layer collects blame from all of them, added together. That's why, in the Python example, the hidden layer's blame is calculated with the whole weight matrix: d_out @ W2.T.

5.6You'll never do this by hand again

Working out gradients by hand is great for understanding, but impractical for real networks. Deep learning libraries do it automatically, using a technique called automatic differentiation (autodiff).

As your network runs forward, the library quietly records every calculation in a computational graph. It knows the slope of every simple operation (multiply, add, sigmoid, ReLU…), so it can run the chain rule backwards through the whole graph for you.

So why learn it? Because understanding backpropagation explains why training sometimes fails: vanishing gradients, exploding gradients, dying ReLUs and learning rates that are too high all come from how blame flows backwards. You'll use that understanding in Module 7.

5.7Vanishing and exploding gradients, revisited

In the worked example, the blame was multiplied by a weight and an activation slope at every layer on its way back. In a deep network, that happens many times over:

Fixes include ReLU activations (Module 3), careful starting weights, batch normalisation, gradient clipping and skip connections. You'll meet all of them in Module 7.

5.8Backpropagation from scratch in Python

In Module 2, a single perceptron could never learn XOR. Let's build a 2-4-1 network in NumPy, write the forward pass, backward pass and update ourselves, and watch it solve XOR.

import numpy as np

# XOR: the problem a single perceptron could never solve (Module 2)
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([[0], [1], [1], [0]])

sigmoid = lambda z: 1 / (1 + np.exp(-z))
rng = np.random.default_rng(1)

# A 2-4-1 network: 2 inputs, 4 hidden neurons, 1 output
W1, b1 = rng.normal(0, 1, (2, 4)), np.zeros((1, 4))
W2, b2 = rng.normal(0, 1, (4, 1)), np.zeros((1, 1))
lr = 1.0

for epoch in range(5001):
    # ---- Forward pass ----
    h = sigmoid(X @ W1 + b1)            # hidden layer
    y_hat = sigmoid(h @ W2 + b2)        # output layer
    loss = np.mean((y - y_hat) ** 2)

    # ---- Backward pass (backpropagation) ----
    d_out = (y_hat - y) * y_hat * (1 - y_hat)     # blame at the output
    d_hidden = (d_out @ W2.T) * h * (1 - h)       # blame passed back to hidden

    # ---- Update every weight (gradient descent) ----
    W2 -= lr * h.T @ d_out
    b2 -= lr * d_out.sum(axis=0, keepdims=True)
    W1 -= lr * X.T @ d_hidden
    b1 -= lr * d_hidden.sum(axis=0, keepdims=True)

    if epoch in (0, 500, 1000, 2000, 5000):
        print(f"Epoch {epoch:4}: loss = {loss:.4f}")

print("\nPredictions after training:")
for inputs, pred in zip(X, y_hat):
    print(f"  {inputs} -> {pred[0]:.3f}  (rounded: {round(pred[0])})")

Epoch    0: loss = 0.2749
Epoch  500: loss = 0.0135
Epoch 1000: loss = 0.0023
Epoch 2000: loss = 0.0008
Epoch 5000: loss = 0.0002

Predictions after training:
  [0 0] -> 0.012  (rounded: 0)
  [0 1] -> 0.984  (rounded: 1)
  [1 0] -> 0.986  (rounded: 1)
  [1 1] -> 0.019  (rounded: 0)

What the output tells us

• The loss falls from 0.275 to 0.0002 over 5,000 epochs, as backpropagation steadily improves all 17 weights and biases.
• XOR is solved: the predictions are 0.012, 0.984, 0.986 and 0.019, which round to exactly 0, 1, 1, 0. A hidden layer, trained by backpropagation, did what the perceptron never could.
• d_out is the output's blame, the same "loss slope × sigmoid slope" as in the worked example.
• d_hidden passes that blame back through W2 and multiplies by the hidden sigmoid slope: the chain rule, for 4 neurons and 4 examples at once.

SummaryKey takeaways

Check your understanding

Your neural networks roadmap

1. 01
   Introduction to Neural NetworksWhat neural networks are and how they work
2. 02
   The Perceptron and the Artificial NeuronBuild a single neuron from scratch
3. 03
   Activation FunctionsSigmoid, tanh, ReLU and softmax
4. 04
   Loss Functions and Gradient DescentHow networks measure and reduce error
5. 05
   BackpropagationYou are here
6. 06
   Building Networks with KerasYour first deep learning library
7. 07
   Training Better NetworksOverfitting, dropout, batch normalisation, early stopping
8. 08
   Convolutional Neural NetworksTeaching computers to see
9. 09
   Recurrent Neural Networks and LSTMsWorking with sequences
10. 10
    Transformers and AttentionThe architecture behind ChatGPT and Claude
11. 11
    Transfer Learning and Pretrained ModelsStanding on the shoulders of giants
12. 12
    Capstone ProjectBuild and present a deep learning project