Loss Functions & Gradient Descent

4.1What is a loss function?

A network can't improve unless it knows how wrong it is. A loss function (also called a cost function) turns every mistake into a single number: the loss. A big loss means bad predictions; a loss of zero means perfect ones.

4.2Choosing a loss function

You've met most of these in the Machine Learning track. The right choice depends on the task, and it goes hand in hand with the output activation from Module 3.

Cross-entropy, in plain English

For classification, the loss only looks at the probability the network gave to the correct answer. If the correct digit is 7:

The loss is calculated as −log(probability of the correct answer). You don't need to calculate it by hand; just remember it rewards being confidently right and heavily punishes being confidently wrong.

4.3The loss landscape

Imagine every possible combination of weights as a location on a map, and the loss as the height at that location. The result is a landscape of hills and valleys. Training means finding the lowest valley.

4.4Gradient descent

The gradient is the slope of the loss for each weight: it says which direction is uphill, and how steeply. To go downhill, we step in the opposite direction:

A worked example

A weight is currently 5. The gradient of the loss for this weight is 4 (uphill to the right). With a learning rate of 0.1:

The weight moves a little to the left, downhill. Next step, we measure the new gradient and repeat. A real network does this for every weight at once. Working out all those gradients efficiently is the job of backpropagation, in Module 5.

4.5The learning rate

The learning rate sets how big each step is. It's one of the most important settings in deep learning. Try it: the ball starts on the left and wants to reach the bottom of the curve.

Common starting values are 0.001 for the Adam optimiser and 0.01 for plain gradient descent. If your loss shoots up to huge numbers or becomes nan ("not a number"), the learning rate is usually too high.

4.6Batches, mini-batches and epochs

Should the network look at every training example before taking one step, or take a step after each one? There are three options:

Mini-batches are the standard. Three words you'll see all the time:

4.7Smarter optimisers: momentum and Adam

Plain gradient descent has a weakness. In long, narrow valleys, the steepest direction points across the valley rather than along it, so it zigzags from side to side. Modern optimisers fix this.

4.8Gradient descent from scratch in Python

Let's write gradient descent ourselves in NumPy. We'll create data that follows the rule y = 3x + 2 (plus noise), start with a terrible guess of y = 0x + 0, and let gradient descent find the rule. Then we'll try three learning rates.

import numpy as np

# 1. Some data: y is roughly 3x + 2, plus noise
rng = np.random.default_rng(0)
x = rng.uniform(0, 2, 100)
y = 3 * x + 2 + rng.normal(0, 0.3, 100)

def mse(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

# 2. Gradient descent: start with a bad guess and improve it step by step
def train(learning_rate, steps=200):
    w, b = 0.0, 0.0                              # starting guess
    history = []
    for step in range(steps):
        y_pred = w * x + b                        # forward pass
        loss = mse(y, y_pred)                     # how wrong are we?
        grad_w = -2 * np.mean(x * (y - y_pred))   # slope of the loss for w
        grad_b = -2 * np.mean(y - y_pred)         # slope of the loss for b
        w -= learning_rate * grad_w               # step downhill
        b -= learning_rate * grad_b
        history.append(loss)
    return w, b, history

w, b, history = train(learning_rate=0.1)
for step in [0, 10, 50, 199]:
    print(f"Step {step:3}: loss = {history[step]:.4f}")
print(f"Learned: y = {w:.2f}x + {b:.2f}   (true rule: y = 3x + 2)")

# 3. The learning rate matters
print()
for lr in [0.001, 0.1, 0.9]:
    w, b, h = train(lr)
    print(f"learning rate {lr:<5}: final loss = {h[-1]:.4g}, w = {w:.3g}, b = {b:.3g}")

Step   0: loss = 31.0796
Step  10: loss = 0.0895
Step  50: loss = 0.0850
Step 199: loss = 0.0846
Learned: y = 2.98x + 2.00   (true rule: y = 3x + 2)

learning rate 0.001: final loss = 4.593, w = 1.77, b = 1.35
learning rate 0.1  : final loss = 0.0846, w = 2.98, b = 2
learning rate 0.9  : final loss = 3.82e+210, w = -3.33e+105, b = -2.58e+105

What the output tells us

• The loss falls fast: from 31.1 at the start to 0.09 after just 10 steps, then it levels off. It can't reach zero, because the data contains random noise.
• Gradient descent found the rule by itself: y = 2.98x + 2.00, extremely close to the true y = 3x + 2.
• A learning rate of 0.001 is too small: after 200 steps it's still far from the answer (w = 1.77).
• A learning rate of 0.9 is far too big: every step overshoots further, and the numbers explode to around 10²¹⁰. This is divergence.
• grad_w and grad_b are the slopes of the MSE loss, worked out with calculus. In real networks you never write these yourself: Keras and PyTorch calculate them automatically (Module 5).

SummaryKey takeaways

Check your understanding

Your neural networks roadmap

1. 01
   Introduction to Neural NetworksWhat neural networks are and how they work
2. 02
   The Perceptron and the Artificial NeuronBuild a single neuron from scratch
3. 03
   Activation FunctionsSigmoid, tanh, ReLU and softmax
4. 04
   Loss Functions and Gradient DescentYou are here
5. 05
   BackpropagationHow every weight learns its share of the blame
6. 06
   Building Networks with KerasYour first deep learning library
7. 07
   Training Better NetworksOverfitting, dropout, batch normalisation, early stopping
8. 08
   Convolutional Neural NetworksTeaching computers to see
9. 09
   Recurrent Neural Networks and LSTMsWorking with sequences
10. 10
    Transformers and AttentionThe architecture behind ChatGPT and Claude
11. 11
    Transfer Learning and Pretrained ModelsStanding on the shoulders of giants
12. 12
    Capstone ProjectBuild and present a deep learning project