Transformers & Self-Attention

10.1Why a new design was needed

In Module 9, you saw that RNNs and LSTMs read a sentence one word at a time, carrying a memory forward. That caused two big problems:

In 2017, a team at Google published a paper with a famous title: "Attention Is All You Need". Their Transformer removed the step-by-step memory completely. Instead, every word looks directly at every other word, all at the same time.

10.2Attention: which words matter to each other?

Read this sentence: "The animal didn't cross the street because it was too tired." What does "it" mean? You instantly know it's the animal, because streets don't get tired. Now change one word: "…because it was too wide." Now "it" means the street.

To understand "it", you had to look at the other words and decide which ones were relevant. That's self-attention: for each word, the network works out how much attention to pay to every other word, then blends their information together to build a richer meaning.

10.3Queries, keys and values

How does the network decide how much attention to pay? Each word creates three vectors (lists of numbers) from its embedding, using three learned weight matrices:

The attention calculation

It looks scary, but it's just four steps you already know:

1. Score: compare each word's query with every word's key by multiplying them (a dot product). A good match gives a high score.
2. Scale: divide by √d (the square root of the vector length) to keep the numbers in a sensible range.
3. Softmax: turn the scores into percentages that add up to 100% (Module 3). These are the attention weights.
4. Blend: take a weighted mix of the values. The result is a new, context-aware version of each word.

10.4Multi-head attention and word order

Several heads look for different things

One attention calculation can only focus on one kind of relationship. So Transformers run several side by side, called attention heads. Each head has its own Q, K and V weights and learns to look for something different: one might link pronouns to nouns, another verbs to their subjects, another nearby words. Their results are combined. This is multi-head attention.

Adding back the order

Attention looks at all words at once, so on its own it has no idea of word order: "dog bites man" and "man bites dog" would look the same (Module 9). The fix is positional encoding: before attention, each word's embedding has information about its position added to it. In the Python example, this is the PositionEmbedding layer.

10.5Inside a Transformer

A Transformer is built from identical blocks stacked on top of each other:

Because nothing waits for the previous word, a whole sequence can be processed in parallel on GPUs. That's what made it possible to train on huge amounts of text, and to build models with billions of weights.

10.6From Transformers to ChatGPT and Claude

The original Transformer had two halves: an encoder that reads the input, and a decoder that writes the output. Later models often use just one half:

How a large language model writes

A large language model (LLM) is a very big decoder-only Transformer, trained on enormous amounts of text to do one simple task: predict the next token. Given "The cat sat on the", its final layer produces a score for every token it knows, and softmax turns them into probabilities, perhaps "mat" 41%, "floor" 22%, "sofa" 12% and so on (illustrative numbers). It picks one, adds it to the text, and repeats.

Trained at huge scale, this simple task produces models that can answer questions, write code and hold conversations. Assistants like ChatGPT and Claude then receive extra training, including learning from human feedback, to make them more helpful, honest and safe.

Transformers have also spread far beyond text: Vision Transformers for images (Module 8), speech recognition, protein structure prediction, and models that combine text, images and audio.

10.7Attention and a Transformer in Python

Part 1 builds the attention formula from scratch in NumPy, with four tiny made-up word embeddings. Part 2 builds a small Transformer in Keras for a task that needs long-range attention: in a sequence of 30 digits, is the first digit the same as the last?

import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
import numpy as np

# ---------- Part 1: attention from scratch ----------
def softmax(z):
    e = np.exp(z - z.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

def attention(Q, K, V):
    scores = Q @ K.T / np.sqrt(K.shape[-1])   # how well each query matches each key
    weights = softmax(scores)                  # turn scores into percentages
    return weights @ V, weights                # blend the values using the weights

# Four words, each described by 4 made-up numbers (an "embedding")
words = ["the", "cat", "sat", "it"]
E = np.array([[0.1, 0.0, 0.1, 0.0],    # the
              [0.9, 0.8, 0.0, 0.1],    # cat
              [0.0, 0.1, 0.9, 0.7],    # sat
              [0.8, 0.9, 0.1, 0.0]])   # it (similar to "cat")
out, w = attention(E, E, E)            # self-attention: the words look at each other
print("How much 'it' attends to each word:")
for word, weight in zip(words, w[3]):
    print(f"  {word:4} {weight:.2f}")

# ---------- Part 2: a tiny Transformer in Keras ----------
import keras
from keras import layers
keras.utils.set_random_seed(0)

# Task: in a sequence of 30 digits, is the FIRST digit the same as the LAST?
rng = np.random.default_rng(0)
def make_data(n, length=30):
    X = rng.integers(1, 10, (n, length))
    same = rng.random(n) < 0.5
    X[same, -1] = X[same, 0]
    return X, (X[:, 0] == X[:, -1]).astype(int)
X_train, y_train = make_data(8000)
X_test, y_test = make_data(2000)

class PositionEmbedding(layers.Layer):     # token meaning + position in the sequence
    def __init__(self, length, vocab, dim):
        super().__init__()
        self.tok = layers.Embedding(vocab, dim)
        self.pos = layers.Embedding(length, dim)
    def call(self, x):
        positions = keras.ops.arange(keras.ops.shape(x)[-1])
        return self.tok(x) + self.pos(positions)

inputs = keras.Input(shape=(30,))
x = PositionEmbedding(30, 10, 32)(inputs)
# One Transformer block: self-attention, then a small feed-forward network,
# each with a skip connection and layer normalisation
attn = layers.MultiHeadAttention(num_heads=4, key_dim=16)(x, x)
x = layers.LayerNormalization()(x + attn)
ff = layers.Dense(64, activation="relu")(x)
ff = layers.Dense(32)(ff)
x = layers.LayerNormalization()(x + ff)
x = layers.GlobalAveragePooling1D()(x)
outputs = layers.Dense(1, activation="sigmoid")(x)
model = keras.Model(inputs, outputs)
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(X_train, y_train, epochs=30, batch_size=64, verbose=0)
print(f"\nTransformer test accuracy: {model.evaluate(X_test, y_test, verbose=0)[1]:.3f}")
print(f"Parameters: {model.count_params():,}")

How much 'it' attends to each word:
  the  0.17
  cat  0.33
  sat  0.17
  it   0.33
Transformer test accuracy: 1.000
Parameters: 14,049

What the output tells us

• Attention from scratch: we made the embedding for "it" similar to "cat", so "it" pays the most attention to "cat" (0.33) and itself (0.33), and less to "the" and "sat" (0.17 each). The weights add up to 1, thanks to softmax.
• The Transformer scores 100% on the first-versus-last task. Position 1 and position 30 are 29 steps apart, but attention connects them directly, in a single step. A simple RNN would have to carry the first digit through 29 steps of fading memory.
• The model has only 14,049 parameters, from token and position embeddings, 4 attention heads and a small feed-forward network. Large language models use the same building blocks, but with billions of parameters and dozens of blocks.
• layers.MultiHeadAttention(...)(x, x) passes x twice because it's self-attention: the sequence provides both the queries and the keys and values.

SummaryKey takeaways

Check your understanding

Your neural networks roadmap

1. 01
   Introduction to Neural NetworksWhat neural networks are and how they work
2. 02
   The Perceptron and the Artificial NeuronBuild a single neuron from scratch
3. 03
   Activation FunctionsSigmoid, tanh, ReLU and softmax
4. 04
   Loss Functions and Gradient DescentHow networks measure and reduce error
5. 05
   BackpropagationHow every weight learns its share of the blame
6. 06
   Building Networks with KerasYour first deep learning library
7. 07
   Training Better NetworksOverfitting, dropout, batch normalisation, early stopping
8. 08
   Convolutional Neural NetworksTeaching computers to see
9. 09
   Recurrent Neural Networks and LSTMsWorking with sequences
10. 10
    Transformers and AttentionYou are here
11. 11
    Transfer Learning and Pretrained ModelsStanding on the shoulders of giants
12. 12
    Capstone ProjectBuild and present a deep learning project