CX Learn by CadetX
Exams My Dashboard
Keras · TensorFlow · Deep Learning
KR-101 · Module 7 Intermediate about 40 minutes 7 Lessons Prereq: Modules 1 to 6

Regularisation & Optimisation

A model that scores brilliantly on its training data but poorly on new data has overfitted: it memorised instead of learning. Keras has simple, powerful tools to prevent this. In this module you'll use early stopping, dropout, L2 regularisation and batch normalisation, and test each one on a deliberately difficult dataset.

  • Level: Intermediate
  • Time: about 40 minutes
  • Needs: Modules 1 to 6

By the end of this module you will be able to

  • Recognise overfitting from training results and curves
  • Use early stopping to stop at the right moment
  • Add Dropout layers and explain how they work
  • Add L1 and L2 regularisation to layers
  • Use BatchNormalization and place it correctly
  • Compare regularisation methods fairly on a validation set
  • Choose a practical regularisation recipe

7.1What is overfitting?

Neural networks have thousands or millions of weights, which gives them enough capacity to memorise their training data, including its noise and mistakes. When that happens, the model looks great on the data it trained on but performs poorly on anything new. That's overfitting.

Memorising vs understanding

Imagine a student who memorises the answers to last year's exam paper. They'd score 100% on that paper but struggle with new questions. A student who understands the topic does well on both. Regularisation is any technique that pushes a model towards understanding rather than memorising.

Warning signWhere you'll see it
Training accuracy much higher than validation or test accuracyThe results of fit() and evaluate()
Validation loss falls, then starts to rise, while training loss keeps fallingThe training curves (Module 5)
Training loss close to zeroOften means the model has memorised every example

Overfitting is most likely when you have a big model, little data, or noisy labels. The Python example in this module has all three.

7.2Early stopping

The simplest regularisation is to stop training at the right time: when the validation loss stops improving. You met the EarlyStopping callback in Module 5:

early_stopping.py
early_stop = keras.callbacks.EarlyStopping(
    monitor="val_loss", patience=10, restore_best_weights=True)
model.fit(X, y, epochs=500, validation_split=0.2, callbacks=[early_stop])
▶ Open Google ColabPaste the code into a new Colab notebook and press Run. Keras is already installed.

It costs nothing, saves training time, and works with every other technique. Use it almost always, and remember restore_best_weights=True, or you'll keep the weights from the last epoch instead of the best one.

7.3Dropout

A Dropout layer randomly switches off a share of the previous layer's neurons at every training step, with a different random set each time. The rate is the share switched off: Dropout(0.5) ignores half of them.

Try it: see dropout in action

in your model

Why switching neurons off helps

Think of a team where one star player does everything. If the coach randomly benches different players at each practice, everyone has to learn to contribute. Dropout stops the network relying on a few neurons that have memorised quirks of the training data, so it learns patterns that many neurons agree on.

TipWhy
Put Dropout after hidden Dense layersIt drops that layer's outputs
Common rates: 0.2 to 0.5Higher rates regularise more strongly but slow learning
Never put dropout after the output layerYou'd be deleting your predictions
Expect training to take more epochsThe network sees a slightly different version of itself each step

7.4L1 and L2 regularisation

Large weights let a network react strongly to tiny details, which is a sign of memorising. L2 regularisation (also called weight decay) adds a penalty to the loss for large weights, nudging the model towards smaller, smoother ones:

total loss = normal loss + strength × (sum of every weight²)
l2.py
from keras import regularizers

layers.Dense(128, activation="relu",
             kernel_regularizer=regularizers.l2(0.001))   # or regularizers.l1(...)
▶ Open Google ColabPaste the code into a new Colab notebook and press Run. Keras is already installed.
TypePenalty onEffect
L2Squared weightsKeeps all weights small; the usual choice
L1Absolute size of weightsPushes many weights to exactly zero, simplifying the model
L1 + L2Bothregularizers.l1_l2(l1=..., l2=...)

The strength matters: typical values are 0.0001 to 0.01. Too weak and it does nothing; too strong and the model can't learn at all. Because the penalty is added to the loss, your reported training loss will be a little higher than without it. That's expected.

7.5Batch normalisation

In Module 2 you scaled your inputs. BatchNormalization does the same thing inside the network: it rescales a layer's outputs for each batch, so the next layer always receives numbers in a steady range.

batchnorm.py
layers.Dense(128),                 # no activation here...
layers.BatchNormalization(),       # ...normalise...
layers.Activation("relu"),         # ...then activate
▶ Open Google ColabPaste the code into a new Colab notebook and press Run. Keras is already installed.
BenefitWhy
Faster, more stable trainingLayers aren't constantly adjusting to wildly changing inputs
Allows higher learning ratesValues are less likely to blow up
A mild regularising effectEach batch's statistics add a little helpful noise

Like dropout, it behaves differently when training and predicting: during training it uses each batch's statistics, and when predicting it uses averages it learned during training. Keras handles this automatically. Placing it between the Dense layer and its activation is the classic pattern; putting it after the activation also works well in practice.

7.6Comparing regularisation in Python

To see real overfitting, we'll make things hard on purpose: only 300 training images of digits, with 25% of their labels deliberately wrong, and a big network of over 300,000 weights. Then we'll train the same model six ways and compare the results on 1,497 clean test images.

compare_regularisation.py
import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
import numpy as np
import keras
from keras import layers, regularizers
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

# Digits again, but a hard setting: only 300 training images, and 25% of their
# labels are wrong (real data is often messy). Perfect conditions for overfitting.
X, y = load_digits(return_X_y=True)
X = (X / 16.0).astype("float32")
X_train, X_test, y_train, y_test = train_test_split(
    X, y, train_size=300, random_state=42, stratify=y)
rng = np.random.default_rng(0)
noisy = rng.random(300) < 0.25
y_train = y_train.copy()
y_train[noisy] = rng.integers(0, 10, noisy.sum())

def make_model(dropout=0.0, l2=0.0, batch_norm=False):
    reg = regularizers.l2(l2) if l2 else None
    model = keras.Sequential([keras.Input(shape=(64,))])
    for _ in range(2):
        model.add(layers.Dense(512, kernel_regularizer=reg))
        if batch_norm:
            model.add(layers.BatchNormalization())
        model.add(layers.Activation("relu"))
        if dropout:
            model.add(layers.Dropout(dropout))
    model.add(layers.Dense(10, activation="softmax"))
    model.compile(optimizer="adam", loss="sparse_categorical_crossentropy",
                  metrics=["accuracy"])
    return model

experiments = {
    "No regularisation": dict(),
    "Dropout(0.5)":      dict(dropout=0.5),
    "L2(0.001)":         dict(l2=0.001),
    "BatchNorm":         dict(batch_norm=True),
    "Early stopping":    dict(),
    "All four":          dict(dropout=0.5, l2=0.001, batch_norm=True),
}
print(f"{'':18} epochs  train acc  test acc  test loss")
for name, settings in experiments.items():
    keras.utils.set_random_seed(0)
    model = make_model(**settings)
    callbacks = []
    if name in ("Early stopping", "All four"):
        callbacks = [keras.callbacks.EarlyStopping(monitor="val_loss", patience=10,
                                                   restore_best_weights=True)]
    hist = model.fit(X_train, y_train, epochs=100, batch_size=32, verbose=0,
                     validation_split=0.2, callbacks=callbacks)
    train_acc = model.evaluate(X_train, y_train, verbose=0)[1]
    test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
    print(f"{name:18} {len(hist.history['loss']):5}  {train_acc:9.3f}  {test_acc:8.3f}  {test_loss:9.3f}")
Output
                   epochs  train acc  test acc  test loss
No regularisation    100      0.933     0.798      0.833
Dropout(0.5)         100      0.937     0.828      0.615
L2(0.001)            100      0.933     0.804      0.842
BatchNorm            100      0.953     0.854      0.521
Early stopping        16      0.773     0.900      0.548
All four              57      0.927     0.833      1.079
▶ Open Google ColabPaste the code into a new Colab notebook and press Run. Keras is already installed.

What the output tells us

  • The unregularised model memorised the wrong labels. It scores 93.3% on its training data, even though about a quarter of those labels are wrong. A model that had learned only the real patterns couldn't score much more than about 77% on them.
  • Early stopping gave the best test accuracy: 90.0%, up from 79.8%. It stopped after just 16 epochs, before memorisation set in, and its training accuracy of 77.3% shows it hadn't learned the wrong labels.
  • Batch normalisation (85.4%) and dropout (82.8%) also helped, and both lowered the test loss.
  • L2 at 0.001 made little difference here. With 512-unit layers and this much noise, it was too weak. The strength needs tuning.
  • Combining all four was not the best (83.3%, with the highest test loss). Stacking every technique doesn't automatically help: they interact, and each adds settings to tune. Test each change on your validation set and keep what works.

A practical recipe

1. Always use early stopping. 2. If the model still overfits, add Dropout(0.2 to 0.5) after hidden layers. 3. Try BatchNormalization for deeper networks. 4. Tune L2 between 0.0001 and 0.01 if needed. 5. Above all, if you can get more (and cleaner) data, that beats every technique here. For images, data augmentation creates more data for free, as you'll see in Module 8.

SummaryKey takeaways

  • Overfitting means memorising the training data; watch for a gap between training and validation results.
  • Early stopping with restore_best_weights=True is simple, free and often the most effective.
  • Dropout(rate) randomly switches off neurons during training only; Keras turns it off for predictions.
  • L2 (kernel_regularizer=regularizers.l2(...)) penalises large weights; its strength needs tuning.
  • BatchNormalization keeps values steady inside the network for faster, more stable training.
  • Combining techniques isn't automatically better: test each change on the validation set.
  • More, cleaner data beats every regularisation trick.

Check your understanding

Your Keras roadmap

  1. 01
    Introduction to KerasWhat Keras is, setup and your first model
  2. 02
    Tensors and Data for KerasShapes, scaling and splitting data
  3. 03
    The Sequential API and Dense LayersStacking layers into a model
  4. 04
    Compiling ModelsLosses, optimisers and metrics
  5. 05
    Training and Evaluatingfit(), validation and callbacks
  6. 06
    The Functional APIMultiple inputs, outputs and branches
  7. 07
    RegularisationYou are here
  8. 08
    Image Classification with KerasCNNs and data augmentation
  9. 09
    Text and Sequences with KerasEmbeddings, LSTMs and text classification
  10. 10
    Pretrained Models, Saving and DeploymentFine-tuning and sharing your models