Recurrent Neural Networks (RNN/LSTM)

9.1Sequences: when order matters

In a table of customers, the order of the rows doesn't matter. But in lots of data, order is everything:

Dense networks expect a fixed number of inputs and have no sense of order. CNNs (Module 8) look at local patterns but struggle to link things that are far apart. Sentences can also be any length. We need a network that reads one step at a time and remembers.

9.2How a recurrent neural network works

A recurrent neural network (RNN) processes a sequence one step at a time. At each step it takes two inputs: the new item (like the next word) and its own memory from the previous step. It combines them into an updated memory, called the hidden state, and passes that on to the next step.

It's the same neuron maths you already know, with one extra ingredient: the old memory. Two important points:

• The same weights are used at every step, just as a CNN reuses a filter at every position. So an RNN can handle sequences of any length.
• RNNs are trained with backpropagation through time: the unrolled network is treated as one very deep network, and the error flows back through every step.

9.3The memory problem

Backpropagation through a long sequence means multiplying by the same weights and slopes again and again. As you saw in Modules 3 and 5, that makes the signal vanish. In practice, a simple RNN forgets things from more than roughly 10 steps back. Try it:

This is a simplified picture, but it shows the real problem: with a simple RNN, early information fades at every step. The LSTM can choose to keep the important clue and carry it across the whole sentence.

9.4LSTMs: memory with gates

In 1997, Sepp Hochreiter and Jürgen Schmidhuber designed the Long Short-Term Memory (LSTM) network to solve the memory problem. An LSTM cell adds a separate long-term memory line, and three gates that control it.

GRUs: a lighter alternative

The Gated Recurrent Unit (GRU), from 2014, combines the gates into two and merges the two memories into one. It has fewer weights, trains faster, and often performs about as well as an LSTM. In Keras, swap layers.LSTM(32) for layers.GRU(32) and compare.

9.5Types of sequence task

In Keras, an LSTM layer gives just its final output by default (many to one). Add return_sequences=True to get an output at every step, which you need for many-to-many tasks or for stacking one LSTM layer on top of another.

9.6Forecasting with an LSTM in Python

A regional energy company wants to forecast tomorrow's electricity demand. We'll create three years of daily demand with realistic patterns (higher in winter, lower at weekends, a slow upward trend and random noise), train an LSTM on the first two years, and test it on the third.

The key preparation step for time series is windowing: turning one long series into many examples of "the last 28 days → the next day".

import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
import numpy as np
import keras
from keras import layers

keras.utils.set_random_seed(0)

# 1. Three years of daily electricity demand (in megawatts): a weekly
#    pattern, a yearly pattern (more in winter), a slow trend and noise
days = np.arange(3 * 365)
demand = (500 + 0.05 * days
          + 60 * np.cos(2 * np.pi * days / 365)          # winter peak
          + 40 * np.where(days % 7 >= 5, -1, 0.4)        # quieter weekends
          + np.random.default_rng(0).normal(0, 12, len(days)))

# 2. Turn the series into windows: 28 past days -> predict the next day
WINDOW = 28
X = np.array([demand[i:i + WINDOW] for i in range(len(demand) - WINDOW)])
y = demand[WINDOW:]
mean, std = demand[:730].mean(), demand[:730].std()     # scale using training years only
Xs, ys = (X - mean) / std, (y - mean) / std
split = 730 - WINDOW                                      # train on years 1-2, test on year 3
X_train, X_test = Xs[:split, :, None], Xs[split:, :, None]
y_train, y_test = ys[:split], ys[split:]

# 3. An LSTM reads the 28 days in order, one day at a time
model = keras.Sequential([
    keras.Input(shape=(WINDOW, 1)),      # 28 time steps, 1 value each
    layers.LSTM(32),                     # 32 memory cells
    layers.Dense(1),                     # tomorrow's demand
])
model.compile(optimizer="adam", loss="mse")
model.fit(X_train, y_train, epochs=100, batch_size=32, verbose=0,
          validation_split=0.1)

# 4. Compare with two simple baselines on the unseen third year
pred = model.predict(X_test, verbose=0).ravel() * std + mean
actual = y_test * std + mean
same_as_yesterday = X[split:, -1]
same_day_last_week = X[split:, -7]
mae = lambda p: np.mean(np.abs(actual - p))
print(f"Baseline 'same as yesterday':     MAE = {mae(same_as_yesterday):5.1f} MW")
print(f"Baseline 'same day last week':    MAE = {mae(same_day_last_week):5.1f} MW")
print(f"LSTM:                             MAE = {mae(pred):5.1f} MW")
print(f"LSTM parameters: {model.count_params():,}")

Baseline 'same as yesterday':     MAE =  25.1 MW
Baseline 'same day last week':    MAE =  13.1 MW
LSTM:                             MAE =  10.8 MW
LSTM parameters: 4,385

What the output tells us

• Always compare with a simple baseline. "Same as yesterday" is poor (25.1 MW error) because it misses the weekend pattern. "Same day last week" is much better (13.1 MW).
• The LSTM beats both, with an average error of 10.8 MW. It learned the weekly pattern, the seasonal pattern and the trend from the data by itself.
• The random noise in the data averages about 9.6 MW, so no model could do much better than about 10 MW. The LSTM is close to the best possible.
• We scaled the data using only the first two years, and tested on the future. With time series, never shuffle: the test set must come after the training data, or the model gets to peek at the future (data leakage, from the Machine Learning track).

9.7RNNs in the real world, and what came next

RNNs have two big limits. They must read one step at a time, so they can't make full use of GPUs, which prefer doing many things at once. And even LSTMs struggle with very long sequences, like whole documents. In 2017, a new design solved both problems by letting every word look directly at every other word: the Transformer, which is the next module.

RNNs and LSTMs are still useful for smaller time-series and sensor problems, where they're simple, compact and effective.

SummaryKey takeaways

Check your understanding

Your neural networks roadmap

1. 01
   Introduction to Neural NetworksWhat neural networks are and how they work
2. 02
   The Perceptron and the Artificial NeuronBuild a single neuron from scratch
3. 03
   Activation FunctionsSigmoid, tanh, ReLU and softmax
4. 04
   Loss Functions and Gradient DescentHow networks measure and reduce error
5. 05
   BackpropagationHow every weight learns its share of the blame
6. 06
   Building Networks with KerasYour first deep learning library
7. 07
   Training Better NetworksOverfitting, dropout, batch normalisation, early stopping
8. 08
   Convolutional Neural NetworksTeaching computers to see
9. 09
   Recurrent Neural Networks and LSTMsYou are here
10. 10
    Transformers and AttentionThe architecture behind ChatGPT and Claude
11. 11
    Transfer Learning and Pretrained ModelsStanding on the shoulders of giants
12. 12
    Capstone ProjectBuild and present a deep learning project