Reviews, emails, support tickets and sensor readings all have something in common: they're sequences, where order matters. In this module you'll turn text into numbers with TextVectorization, give words meaning with Embedding layers, and use LSTMs to read sequences in order, then see why that matters with a real experiment.
- Level: Intermediate
- Time: about 45 minutes
- Needs: Modules 1 to 8
By the end of this module you will be able to
- Turn text into integer sequences with TextVectorization
- Explain vocabulary, [UNK], padding and truncation
- Use Embedding layers and count their parameters
- Build a bag-of-words model with GlobalAveragePooling1D
- Build recurrent models with LSTM, GRU and Bidirectional layers
- Use return_sequences and masking correctly
- Shape time-series data for LSTMs
- Show why word order matters with a real comparison
9.1From text to numbers
Neural networks need numbers, so text must be converted in three steps:
| Step | What happens | Example |
|---|---|---|
| 1. Tokenise | Split the text into words (called tokens), usually in lower case without punctuation | "The price was great!" → the, price, was, great |
| 2. Look up | Replace each token with its number in a vocabulary. Unknown words become 1, meaning [UNK] | → 3, 10, 2, 14 |
| 3. Pad or cut | Make every sequence the same length: add 0s to short ones, cut long ones | → 3, 10, 2, 14, 0, 0, 0 … |
Keras does all three with one layer, TextVectorization. Like the Normalization layer, you call adapt() on your training text so it can build the vocabulary:
vectorize = layers.TextVectorization(
max_tokens=10000, # keep the 10,000 most common words
output_sequence_length=25) # pad or cut every text to 25 tokens
vectorize.adapt(train_texts) # learn the vocabulary from TRAINING text only
ids = vectorize(["the price was not great"]) # -> [[3, 10, 2, 5, 14, 0, 0, ...]]
Index 0 is always reserved for padding and index 1 for unknown words. The rest are ordered from the most common word down.
Try it: text vectoriser
9.2Embeddings: giving words meaning
The number 14 for "great" is just a label; it doesn't mean great is bigger than 13. An Embedding layer turns each word id into a short list of learned numbers, called a word vector. During training, words used in similar ways end up with similar vectors, so "great" and "excellent" drift close together.
layers.Embedding(input_dim=10000, # vocabulary size
output_dim=16, # numbers per word
mask_zero=True) # ignore the padding 0s
# input shape (batch, 25) -> output shape (batch, 25, 16)
| Setting | Meaning |
|---|---|
input_dim | The vocabulary size: the number of different word ids |
output_dim | How many numbers describe each word. 16 to 300 is common |
mask_zero=True | Tells later layers, such as LSTMs, to skip the padding |
| Parameters | input_dim × output_dim: one learned vector per word |
9.3Two ways to read a sentence
Bag of words: fast, but ignores order
The simplest text model averages all the word vectors with GlobalAveragePooling1D, then classifies the average. It's fast and often surprisingly good, but it throws away word order. To it, "the price was great and the fit was not poor" and "the price was not great and the fit was poor" are identical: same words, same average.
Recurrent layers: read in order
A recurrent layer reads a sequence one step at a time, carrying a memory forward, so it knows that "not" came right before "great". Keras has three you'll use:
| Layer | What it is | When to use it |
|---|---|---|
LSTM(units) | Long Short-Term Memory: uses gates to decide what to remember and forget | The standard choice for sequences |
GRU(units) | A lighter, simpler version of the LSTM | Often just as good and faster |
Bidirectional(LSTM(units)) | Runs one LSTM forwards and another backwards, then combines them | Text classification, where the whole sentence is available at once |
For the full story of how LSTMs remember, see the RNN and LSTM module in our Neural Networks course.
9.4Sequence shapes and return_sequences
Recurrent layers expect a 3D input: (batch, time steps, features). For text, the Embedding layer produces exactly that: (batch, 25 words, 16 numbers per word). By default, an LSTM returns only its final output, which suits classifying a whole sentence. To stack LSTMs, the lower one must return an output for every step:
layers.LSTM(64, return_sequences=True), # (batch, 25, 64): one output per step
layers.LSTM(32), # (batch, 32): just the final output
layers.Dense(1, activation="sigmoid")
Time series use the same layers
Sequences aren't just text. For time series, such as daily sales or sensor readings, you don't need vectorisation or embeddings: shape your windows as (samples, time steps, features) and feed them straight to an LSTM.
# X has shape (samples, 30, 3): 30 days of 3 readings each; y is the next day's demand
model = keras.Sequential([
keras.Input(shape=(30, 3)),
layers.LSTM(32),
layers.Dense(1), # predict a number
])
model.compile(optimizer="adam", loss="mse", metrics=["mae"])
Remember from Module 2: for time series, split by time and never shuffle before splitting.
9.5Bag of words vs LSTM in Python
We'll create 8,000 short product reviews where word order matters: "not" flips the meaning of the word after it, and a review is positive if most of its points are positive. Then we'll compare a bag-of-words model with a bidirectional LSTM.
import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
import numpy as np
import keras
from keras import layers
keras.utils.set_random_seed(0)
rng = np.random.default_rng(0)
np.set_printoptions(linewidth=120)
# ---------- 1. Make product reviews where word ORDER matters ----------
things = ["delivery", "quality", "price", "battery", "screen", "service", "packaging", "fit"]
good = ["great", "excellent", "fast", "perfect", "lovely", "brilliant"]
bad = ["poor", "terrible", "slow", "awful", "disappointing", "cheap"]
fillers = ["honestly", "overall", "to be fair", "i think", "sadly", "luckily"]
def clause():
positive = rng.random() < 0.5
negated = rng.random() < 0.4 # "not great" flips the meaning
word = rng.choice(good if positive else bad)
text = f"the {rng.choice(things)} was {'not ' if negated else ''}{word}"
return text, (positive != negated)
def review():
parts, scores = zip(*[clause() for _ in range(rng.integers(1, 4))])
text = " and ".join(parts)
if rng.random() < 0.5:
text = f"{rng.choice(fillers)} {text}"
return text, int(np.mean(scores) > 0.5) # positive if most clauses are positive
data = [review() for _ in range(8000)]
texts = np.array([t for t, _ in data]); labels = np.array([l for _, l in data])
x_train, x_test, y_train, y_test = texts[:6000], texts[6000:], labels[:6000], labels[6000:]
print("Example:", x_train[0], "->", y_train[0])
print("Example:", x_train[1], "->", y_train[1])
# ---------- 2. TextVectorization: text -> padded integer sequences ----------
vectorize = layers.TextVectorization(max_tokens=100, output_sequence_length=25)
vectorize.adapt(x_train)
vocab = [str(w) for w in vectorize.get_vocabulary()]
print("\nVocabulary size:", len(vocab), "| first 8:", vocab[:8])
print("'the price was not great' ->", np.asarray(vectorize(["the price was not great"]))[0])
ids_train = vectorize(x_train) # shape (6000, 25): one row of word ids per review
ids_test = vectorize(x_test)
# ---------- 3. Two models: bag of words vs LSTM ----------
def bag_of_words():
return keras.Sequential([
keras.Input(shape=(25,)),
layers.Embedding(input_dim=len(vocab), output_dim=16),
layers.GlobalAveragePooling1D(), # averages the words: order is lost
layers.Dense(16, activation="relu"),
layers.Dense(1, activation="sigmoid")])
def lstm_model():
return keras.Sequential([
keras.Input(shape=(25,)),
layers.Embedding(input_dim=len(vocab), output_dim=16, mask_zero=True),
layers.Bidirectional(layers.LSTM(32)), # reads the words in order
layers.Dense(1, activation="sigmoid")])
tests = ["the service was great", "the service was not great",
"the price was awful and the quality was not bad",
"sadly the battery was not excellent and the fit was poor"]
for name, build in [("Bag of words", bag_of_words), ("Bidirectional LSTM", lstm_model)]:
keras.utils.set_random_seed(0)
model = build()
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(ids_train, y_train, epochs=10, batch_size=32, verbose=0, validation_split=0.1)
acc = model.evaluate(ids_test, y_test, verbose=0)[1]
probs = model.predict(vectorize(tests), verbose=0).ravel()
print(f"\n{name}: test accuracy = {acc:.3f} ({model.count_params():,} weights)")
for t, p in zip(tests, probs):
print(f" {p:5.1%} positive <- {t}")
Example: the delivery was not perfect and the screen was poor and the screen was awful -> 0 Example: honestly the delivery was perfect and the packaging was not poor and the battery was not slow -> 1 Vocabulary size: 35 | first 8: ['', '[UNK]', 'was', 'the', 'and', 'not', 'battery', 'screen'] 'the price was not great' -> [ 3 10 2 5 14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] Bag of words: test accuracy = 0.669 (849 weights) 54.4% positive <- the service was great 49.5% positive <- the service was not great 19.2% positive <- the price was awful and the quality was not bad 21.4% positive <- sadly the battery was not excellent and the fit was poor Bidirectional LSTM: test accuracy = 1.000 (13,169 weights) 99.9% positive <- the service was great 0.0% positive <- the service was not great 0.0% positive <- the price was awful and the quality was not bad 0.0% positive <- sadly the battery was not excellent and the fit was poor
What the output tells us
- TextVectorization built a vocabulary of 35 entries, with padding (
'') at 0 and[UNK]at 1, and turned "the price was not great" into[3, 10, 2, 5, 14]followed by padding zeros. - The bag-of-words model managed only 66.9%. It can't tell "the service was great" (54.4% positive) from "the service was not great" (49.5%): both are near 50-50, because the words are almost the same.
- The bidirectional LSTM scored 100% on 2,000 unseen reviews. Reading in order, it learned that "not" flips the next word: 99.9% positive for "great" and 0.0% for "not great".
- "bad" wasn't in the vocabulary, because the training reviews never used it, so it became
[UNK]. Real projects need plenty of varied training text, or a pretrained model (Module 10). - The LSTM needed 13,169 weights against 849 for bag of words. More power costs more computation, so try the simple model first, and switch when order matters.
Is 100% realistic?
Only because these reviews follow simple rules. Real reviews are messier, with sarcasm, typos and mixed opinions. The lesson that carries over is this: when word order changes the meaning, a model that reads in order beats one that doesn't. For the very best results on real text, today's go-to is a pretrained Transformer model, which you'll meet in Module 10.
SummaryKey takeaways
TextVectorizationtokenises text, looks words up in a vocabulary and pads or cuts to a fixed length. Calladapt()on training text.- Index 0 is padding and 1 is [UNK] (unknown words).
Embedding(input_dim, output_dim)learns a vector per word; parameters = vocabulary × dimensions.- Bag of words (
GlobalAveragePooling1D) is fast but ignores order. LSTM,GRUandBidirectionalread sequences in order; usemask_zero=Trueto skip padding.- Recurrent layers take (batch, time steps, features); use
return_sequences=Trueto stack them. - When order changes meaning, order-aware models win: 66.9% vs 100% in our experiment.
Check your understanding
Your Keras roadmap
- 01Introduction to KerasWhat Keras is, setup and your first model
- 02Tensors and Data for KerasShapes, scaling and splitting data
- 03The Sequential API and Dense LayersStacking layers into a model
- 04Compiling ModelsLosses, optimisers and metrics
- 05Training and Evaluatingfit(), validation and callbacks
- 06The Functional APIMultiple inputs, outputs and branches
- 07RegularisationDropout, batch normalisation and early stopping
- 08Image Classification with KerasCNNs and data augmentation
- 09Text and Sequences with KerasYou are here
- 10Pretrained Models, Saving and DeploymentFine-tuning and sharing your models
Next module
Pretrained Models, Saving and Deployment