A model that scores brilliantly on its training data but poorly on new data has overfitted: it memorised instead of learning. Keras has simple, powerful tools to prevent this. In this module you'll use early stopping, dropout, L2 regularisation and batch normalisation, and test each one on a deliberately difficult dataset.
- Level: Intermediate
- Time: about 40 minutes
- Needs: Modules 1 to 6
By the end of this module you will be able to
- Recognise overfitting from training results and curves
- Use early stopping to stop at the right moment
- Add Dropout layers and explain how they work
- Add L1 and L2 regularisation to layers
- Use BatchNormalization and place it correctly
- Compare regularisation methods fairly on a validation set
- Choose a practical regularisation recipe
7.1What is overfitting?
Neural networks have thousands or millions of weights, which gives them enough capacity to memorise their training data, including its noise and mistakes. When that happens, the model looks great on the data it trained on but performs poorly on anything new. That's overfitting.
Memorising vs understanding
Imagine a student who memorises the answers to last year's exam paper. They'd score 100% on that paper but struggle with new questions. A student who understands the topic does well on both. Regularisation is any technique that pushes a model towards understanding rather than memorising.
| Warning sign | Where you'll see it |
|---|---|
| Training accuracy much higher than validation or test accuracy | The results of fit() and evaluate() |
| Validation loss falls, then starts to rise, while training loss keeps falling | The training curves (Module 5) |
| Training loss close to zero | Often means the model has memorised every example |
Overfitting is most likely when you have a big model, little data, or noisy labels. The Python example in this module has all three.
7.2Early stopping
The simplest regularisation is to stop training at the right time: when the validation loss stops improving. You met the EarlyStopping callback in Module 5:
early_stop = keras.callbacks.EarlyStopping(
monitor="val_loss", patience=10, restore_best_weights=True)
model.fit(X, y, epochs=500, validation_split=0.2, callbacks=[early_stop])
It costs nothing, saves training time, and works with every other technique. Use it almost always, and remember restore_best_weights=True, or you'll keep the weights from the last epoch instead of the best one.
7.3Dropout
A Dropout layer randomly switches off a share of the previous layer's neurons at every training step, with a different random set each time. The rate is the share switched off: Dropout(0.5) ignores half of them.
Try it: see dropout in action
Why switching neurons off helps
Think of a team where one star player does everything. If the coach randomly benches different players at each practice, everyone has to learn to contribute. Dropout stops the network relying on a few neurons that have memorised quirks of the training data, so it learns patterns that many neurons agree on.
| Tip | Why |
|---|---|
Put Dropout after hidden Dense layers | It drops that layer's outputs |
| Common rates: 0.2 to 0.5 | Higher rates regularise more strongly but slow learning |
| Never put dropout after the output layer | You'd be deleting your predictions |
| Expect training to take more epochs | The network sees a slightly different version of itself each step |
7.4L1 and L2 regularisation
Large weights let a network react strongly to tiny details, which is a sign of memorising. L2 regularisation (also called weight decay) adds a penalty to the loss for large weights, nudging the model towards smaller, smoother ones:
from keras import regularizers
layers.Dense(128, activation="relu",
kernel_regularizer=regularizers.l2(0.001)) # or regularizers.l1(...)
| Type | Penalty on | Effect |
|---|---|---|
| L2 | Squared weights | Keeps all weights small; the usual choice |
| L1 | Absolute size of weights | Pushes many weights to exactly zero, simplifying the model |
| L1 + L2 | Both | regularizers.l1_l2(l1=..., l2=...) |
The strength matters: typical values are 0.0001 to 0.01. Too weak and it does nothing; too strong and the model can't learn at all. Because the penalty is added to the loss, your reported training loss will be a little higher than without it. That's expected.
7.5Batch normalisation
In Module 2 you scaled your inputs. BatchNormalization does the same thing inside the network: it rescales a layer's outputs for each batch, so the next layer always receives numbers in a steady range.
layers.Dense(128), # no activation here...
layers.BatchNormalization(), # ...normalise...
layers.Activation("relu"), # ...then activate
| Benefit | Why |
|---|---|
| Faster, more stable training | Layers aren't constantly adjusting to wildly changing inputs |
| Allows higher learning rates | Values are less likely to blow up |
| A mild regularising effect | Each batch's statistics add a little helpful noise |
Like dropout, it behaves differently when training and predicting: during training it uses each batch's statistics, and when predicting it uses averages it learned during training. Keras handles this automatically. Placing it between the Dense layer and its activation is the classic pattern; putting it after the activation also works well in practice.
7.6Comparing regularisation in Python
To see real overfitting, we'll make things hard on purpose: only 300 training images of digits, with 25% of their labels deliberately wrong, and a big network of over 300,000 weights. Then we'll train the same model six ways and compare the results on 1,497 clean test images.
import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
import numpy as np
import keras
from keras import layers, regularizers
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
# Digits again, but a hard setting: only 300 training images, and 25% of their
# labels are wrong (real data is often messy). Perfect conditions for overfitting.
X, y = load_digits(return_X_y=True)
X = (X / 16.0).astype("float32")
X_train, X_test, y_train, y_test = train_test_split(
X, y, train_size=300, random_state=42, stratify=y)
rng = np.random.default_rng(0)
noisy = rng.random(300) < 0.25
y_train = y_train.copy()
y_train[noisy] = rng.integers(0, 10, noisy.sum())
def make_model(dropout=0.0, l2=0.0, batch_norm=False):
reg = regularizers.l2(l2) if l2 else None
model = keras.Sequential([keras.Input(shape=(64,))])
for _ in range(2):
model.add(layers.Dense(512, kernel_regularizer=reg))
if batch_norm:
model.add(layers.BatchNormalization())
model.add(layers.Activation("relu"))
if dropout:
model.add(layers.Dropout(dropout))
model.add(layers.Dense(10, activation="softmax"))
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy",
metrics=["accuracy"])
return model
experiments = {
"No regularisation": dict(),
"Dropout(0.5)": dict(dropout=0.5),
"L2(0.001)": dict(l2=0.001),
"BatchNorm": dict(batch_norm=True),
"Early stopping": dict(),
"All four": dict(dropout=0.5, l2=0.001, batch_norm=True),
}
print(f"{'':18} epochs train acc test acc test loss")
for name, settings in experiments.items():
keras.utils.set_random_seed(0)
model = make_model(**settings)
callbacks = []
if name in ("Early stopping", "All four"):
callbacks = [keras.callbacks.EarlyStopping(monitor="val_loss", patience=10,
restore_best_weights=True)]
hist = model.fit(X_train, y_train, epochs=100, batch_size=32, verbose=0,
validation_split=0.2, callbacks=callbacks)
train_acc = model.evaluate(X_train, y_train, verbose=0)[1]
test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
print(f"{name:18} {len(hist.history['loss']):5} {train_acc:9.3f} {test_acc:8.3f} {test_loss:9.3f}")
epochs train acc test acc test loss No regularisation 100 0.933 0.798 0.833 Dropout(0.5) 100 0.937 0.828 0.615 L2(0.001) 100 0.933 0.804 0.842 BatchNorm 100 0.953 0.854 0.521 Early stopping 16 0.773 0.900 0.548 All four 57 0.927 0.833 1.079
What the output tells us
- The unregularised model memorised the wrong labels. It scores 93.3% on its training data, even though about a quarter of those labels are wrong. A model that had learned only the real patterns couldn't score much more than about 77% on them.
- Early stopping gave the best test accuracy: 90.0%, up from 79.8%. It stopped after just 16 epochs, before memorisation set in, and its training accuracy of 77.3% shows it hadn't learned the wrong labels.
- Batch normalisation (85.4%) and dropout (82.8%) also helped, and both lowered the test loss.
- L2 at 0.001 made little difference here. With 512-unit layers and this much noise, it was too weak. The strength needs tuning.
- Combining all four was not the best (83.3%, with the highest test loss). Stacking every technique doesn't automatically help: they interact, and each adds settings to tune. Test each change on your validation set and keep what works.
A practical recipe
1. Always use early stopping. 2. If the model still overfits, add Dropout(0.2 to 0.5) after hidden layers. 3. Try BatchNormalization for deeper networks. 4. Tune L2 between 0.0001 and 0.01 if needed. 5. Above all, if you can get more (and cleaner) data, that beats every technique here. For images, data augmentation creates more data for free, as you'll see in Module 8.
SummaryKey takeaways
- Overfitting means memorising the training data; watch for a gap between training and validation results.
- Early stopping with
restore_best_weights=Trueis simple, free and often the most effective. Dropout(rate)randomly switches off neurons during training only; Keras turns it off for predictions.- L2 (
kernel_regularizer=regularizers.l2(...)) penalises large weights; its strength needs tuning. BatchNormalizationkeeps values steady inside the network for faster, more stable training.- Combining techniques isn't automatically better: test each change on the validation set.
- More, cleaner data beats every regularisation trick.
Check your understanding
Your Keras roadmap
- 01Introduction to KerasWhat Keras is, setup and your first model
- 02Tensors and Data for KerasShapes, scaling and splitting data
- 03The Sequential API and Dense LayersStacking layers into a model
- 04Compiling ModelsLosses, optimisers and metrics
- 05Training and Evaluatingfit(), validation and callbacks
- 06The Functional APIMultiple inputs, outputs and branches
- 07RegularisationYou are here
- 08Image Classification with KerasCNNs and data augmentation
- 09Text and Sequences with KerasEmbeddings, LSTMs and text classification
- 10Pretrained Models, Saving and DeploymentFine-tuning and sharing your models
Next module
Image Classification with Keras