Training Better Networks

7.1Overfitting in neural networks

You met overfitting in the Machine Learning track: a model that scores brilliantly on its training data but poorly on new data, because it memorised instead of learning. Neural networks are especially prone to it, because they have so many weights. The network in this module's Python example has over 76,000 weights but only 200 training examples to learn from: plenty of room to memorise every one.

The techniques that fight overfitting are called regularisation. Try them in the lab below, using real training histories.

7.2See it happen: five ways to train

We trained the same big network five times on the same small, noisy dataset. Choose an experiment to see its real training and validation loss.

Without regularisation, the validation loss bottoms out around epoch 6, then climbs steadily for the rest of training, even though the training loss keeps falling towards zero. The network is becoming more and more confident about patterns that only exist in its training data.

7.3Early stopping

The simplest fix: stop training when the validation loss stops improving, and keep the weights from the best epoch.

early_stop = keras.callbacks.EarlyStopping(
    monitor="val_loss",          # watch the validation loss
    patience=15,                 # wait 15 epochs for an improvement
    restore_best_weights=True)   # then roll back to the best epoch

model.fit(X, y, epochs=500, validation_split=0.2, callbacks=[early_stop])

patience gives the network a few chances, because validation loss naturally wobbles. Early stopping also saves time: set epochs high and let it decide when to stop.

7.4Dropout

Dropout randomly switches off a share of neurons in a layer at every training step. A different random set each time.

layers.Dense(256, activation="relu"),
layers.Dropout(0.5),    # switch off 50% of the previous layer's outputs while training

Typical rates are 0.2 to 0.5. Dropout was one of the ideas behind AlexNet's breakthrough in 2012 (Module 8).

7.5L2 weight decay

You met this as Ridge regularisation in the Machine Learning track. L2 regularisation (also called weight decay) adds a penalty to the loss for large weights:

Big weights let a network make sharp, extreme decisions based on tiny details. Keeping weights small makes the network smoother and less likely to latch onto noise.

from keras import regularizers
layers.Dense(256, activation="relu",
             kernel_regularizer=regularizers.l2(0.01))

7.6Data augmentation

The best cure for overfitting is more data. If you can't collect more, you can often create it. Data augmentation makes new training examples by changing existing ones in ways that don't change the answer.

augment = keras.Sequential([
    layers.RandomTranslation(0.1, 0.1),   # shift up to 10%
    layers.RandomRotation(0.05),          # rotate up to about 18 degrees
    layers.RandomZoom(0.1),               # zoom in or out up to 10%
])
# Put these layers at the start of your model: they only act during training

Augmentation works for other data too: adding background noise to audio, or swapping words for synonyms in text.

7.7Batch normalisation

In the Machine Learning track, you scaled inputs so that every feature had a similar range. Batch normalisation does the same thing inside the network: it rescales the outputs of a layer for each mini-batch, so the next layer always receives numbers in a steady, sensible range.

layers.Dense(256),
layers.BatchNormalization(),
layers.Activation("relu"),

7.8Keeping training stable

In Modules 3 and 5 you saw that gradients can vanish or explode as they travel back through many layers. Here are the standard fixes:

Learning rate schedules

A big learning rate makes fast progress early on; a small one helps settle precisely into the valley later. A learning rate schedule lowers the rate as training goes on.

reduce_lr = keras.callbacks.ReduceLROnPlateau(
    monitor="val_loss", factor=0.5, patience=5)   # halve the rate when progress stalls

7.9Regularisation in Python

Here's the code behind the lab. It trains the same big network five ways on a small, noisy dataset (250 examples, with 10% of labels deliberately wrong, as real data often has).

import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import keras
from keras import layers, regularizers

# 1. A small, noisy dataset: only 250 training examples, 40 features,
#    and 10% of labels deliberately wrong. A recipe for overfitting.
X, y = make_classification(n_samples=2000, n_features=40, n_informative=8,
                           n_redundant=4, flip_y=0.1, random_state=3)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, train_size=250, random_state=0, stratify=y)

# 2. A big network (over 76,000 weights) that can easily memorise 250 rows
def make_model(dropout=0.0, l2=0.0):
    reg = regularizers.l2(l2) if l2 else None
    model = keras.Sequential([keras.Input(shape=(40,))])
    for _ in range(2):
        model.add(layers.Dense(256, activation="relu", kernel_regularizer=reg))
        if dropout:
            model.add(layers.Dropout(dropout))
    model.add(layers.Dense(1, activation="sigmoid"))
    model.compile(optimizer="adam", loss="binary_crossentropy",
                  metrics=["accuracy"])
    return model

early_stop = keras.callbacks.EarlyStopping(
    monitor="val_loss", patience=15, restore_best_weights=True)

# 3. Train the same network five ways
experiments = {
    "No regularisation": (dict(), []),
    "Dropout (0.5)":     (dict(dropout=0.5), []),
    "L2 (0.01)":         (dict(l2=0.01), []),
    "Early stopping":    (dict(), [early_stop]),
    "Dropout + L2 + ES": (dict(dropout=0.5, l2=0.01), [early_stop]),
}
for name, (settings, callbacks) in experiments.items():
    keras.utils.set_random_seed(0)
    model = make_model(**settings)
    hist = model.fit(X_train, y_train, epochs=150, batch_size=32, verbose=0,
                     validation_split=0.2, callbacks=callbacks)
    _, train_acc = model.evaluate(X_train, y_train, verbose=0)
    test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
    print(f"{name:18} epochs={len(hist.history['loss']):3}  train acc={train_acc:.3f}  "
          f"test acc={test_acc:.3f}  test loss={test_loss:.3f}")

No regularisation  epochs=150  train acc=0.964  test acc=0.815  test loss=1.084
Dropout (0.5)      epochs=150  train acc=0.968  test acc=0.822  test loss=1.158
L2 (0.01)          epochs=150  train acc=0.968  test acc=0.825  test loss=0.609
Early stopping     epochs= 21  train acc=0.948  test acc=0.822  test loss=0.424
Dropout + L2 + ES  epochs=150  train acc=0.972  test acc=0.834  test loss=0.624

What the output tells us

• Every version scores 95% or more on training data but only 81 to 83% on test data. With only 250 noisy examples, some overfitting is unavoidable.
• Test loss tells the bigger story. Without regularisation it's 1.08: the network is confidently wrong on many test examples. L2 cuts it to 0.61, and early stopping to 0.42, by stopping at epoch 21 before memorisation set in.
• Dropout alone didn't help here (test loss 1.16). With so little data it wasn't enough by itself. Regularisation isn't magic: each technique helps in different situations, which is why you test them.
• Combining dropout, L2 and early stopping gave the best test accuracy (83.4%).

SummaryKey takeaways

Check your understanding

Your neural networks roadmap

1. 01
   Introduction to Neural NetworksWhat neural networks are and how they work
2. 02
   The Perceptron and the Artificial NeuronBuild a single neuron from scratch
3. 03
   Activation FunctionsSigmoid, tanh, ReLU and softmax
4. 04
   Loss Functions and Gradient DescentHow networks measure and reduce error
5. 05
   BackpropagationHow every weight learns its share of the blame
6. 06
   Building Networks with KerasYour first deep learning library
7. 07
   Training Better NetworksYou are here
8. 08
   Convolutional Neural NetworksTeaching computers to see
9. 09
   Recurrent Neural Networks and LSTMsWorking with sequences
10. 10
    Transformers and AttentionThe architecture behind ChatGPT and Claude
11. 11
    Transfer Learning and Pretrained ModelsStanding on the shoulders of giants
12. 12
    Capstone ProjectBuild and present a deep learning project