CX Learn by CadetX
Exams My Dashboard
Keras · TensorFlow · Deep Learning
KR-101 · Module 9 Intermediate about 45 minutes 6 Lessons Prereq: Modules 1 to 8

Text & Sequential Models

Reviews, emails, support tickets and sensor readings all have something in common: they're sequences, where order matters. In this module you'll turn text into numbers with TextVectorization, give words meaning with Embedding layers, and use LSTMs to read sequences in order, then see why that matters with a real experiment.

  • Level: Intermediate
  • Time: about 45 minutes
  • Needs: Modules 1 to 8

By the end of this module you will be able to

  • Turn text into integer sequences with TextVectorization
  • Explain vocabulary, [UNK], padding and truncation
  • Use Embedding layers and count their parameters
  • Build a bag-of-words model with GlobalAveragePooling1D
  • Build recurrent models with LSTM, GRU and Bidirectional layers
  • Use return_sequences and masking correctly
  • Shape time-series data for LSTMs
  • Show why word order matters with a real comparison

9.1From text to numbers

Neural networks need numbers, so text must be converted in three steps:

StepWhat happensExample
1. TokeniseSplit the text into words (called tokens), usually in lower case without punctuation"The price was great!" → the, price, was, great
2. Look upReplace each token with its number in a vocabulary. Unknown words become 1, meaning [UNK]→ 3, 10, 2, 14
3. Pad or cutMake every sequence the same length: add 0s to short ones, cut long ones→ 3, 10, 2, 14, 0, 0, 0 …

Keras does all three with one layer, TextVectorization. Like the Normalization layer, you call adapt() on your training text so it can build the vocabulary:

text_vectorization.py
vectorize = layers.TextVectorization(
    max_tokens=10000,             # keep the 10,000 most common words
    output_sequence_length=25)    # pad or cut every text to 25 tokens
vectorize.adapt(train_texts)      # learn the vocabulary from TRAINING text only

ids = vectorize(["the price was not great"])   # -> [[3, 10, 2, 5, 14, 0, 0, ...]]
▶ Open Google ColabPaste the code into a new Colab notebook and press Run. Keras is already installed.

Index 0 is always reserved for padding and index 1 for unknown words. The rest are ordered from the most common word down.

Try it: text vectoriser

Tokens and their ids
What the model receives
Bag of words (order ignored)

9.2Embeddings: giving words meaning

The number 14 for "great" is just a label; it doesn't mean great is bigger than 13. An Embedding layer turns each word id into a short list of learned numbers, called a word vector. During training, words used in similar ways end up with similar vectors, so "great" and "excellent" drift close together.

embedding.py
layers.Embedding(input_dim=10000,    # vocabulary size
                 output_dim=16,      # numbers per word
                 mask_zero=True)     # ignore the padding 0s
# input shape (batch, 25)  ->  output shape (batch, 25, 16)
▶ Open Google ColabPaste the code into a new Colab notebook and press Run. Keras is already installed.
SettingMeaning
input_dimThe vocabulary size: the number of different word ids
output_dimHow many numbers describe each word. 16 to 300 is common
mask_zero=TrueTells later layers, such as LSTMs, to skip the padding
Parametersinput_dim × output_dim: one learned vector per word

9.3Two ways to read a sentence

Bag of words: fast, but ignores order

The simplest text model averages all the word vectors with GlobalAveragePooling1D, then classifies the average. It's fast and often surprisingly good, but it throws away word order. To it, "the price was great and the fit was not poor" and "the price was not great and the fit was poor" are identical: same words, same average.

Recurrent layers: read in order

A recurrent layer reads a sequence one step at a time, carrying a memory forward, so it knows that "not" came right before "great". Keras has three you'll use:

LayerWhat it isWhen to use it
LSTM(units)Long Short-Term Memory: uses gates to decide what to remember and forgetThe standard choice for sequences
GRU(units)A lighter, simpler version of the LSTMOften just as good and faster
Bidirectional(LSTM(units))Runs one LSTM forwards and another backwards, then combines themText classification, where the whole sentence is available at once

For the full story of how LSTMs remember, see the RNN and LSTM module in our Neural Networks course.

9.4Sequence shapes and return_sequences

Recurrent layers expect a 3D input: (batch, time steps, features). For text, the Embedding layer produces exactly that: (batch, 25 words, 16 numbers per word). By default, an LSTM returns only its final output, which suits classifying a whole sentence. To stack LSTMs, the lower one must return an output for every step:

stacked_lstm.py
layers.LSTM(64, return_sequences=True),   # (batch, 25, 64): one output per step
layers.LSTM(32),                          # (batch, 32): just the final output
layers.Dense(1, activation="sigmoid")
▶ Open Google ColabPaste the code into a new Colab notebook and press Run. Keras is already installed.

Time series use the same layers

Sequences aren't just text. For time series, such as daily sales or sensor readings, you don't need vectorisation or embeddings: shape your windows as (samples, time steps, features) and feed them straight to an LSTM.

time_series_lstm.py
# X has shape (samples, 30, 3): 30 days of 3 readings each; y is the next day's demand
model = keras.Sequential([
    keras.Input(shape=(30, 3)),
    layers.LSTM(32),
    layers.Dense(1),                      # predict a number
])
model.compile(optimizer="adam", loss="mse", metrics=["mae"])
▶ Open Google ColabPaste the code into a new Colab notebook and press Run. Keras is already installed.

Remember from Module 2: for time series, split by time and never shuffle before splitting.

9.5Bag of words vs LSTM in Python

We'll create 8,000 short product reviews where word order matters: "not" flips the meaning of the word after it, and a review is positive if most of its points are positive. Then we'll compare a bag-of-words model with a bidirectional LSTM.

review_sentiment.py
import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
import numpy as np
import keras
from keras import layers

keras.utils.set_random_seed(0)
rng = np.random.default_rng(0)
np.set_printoptions(linewidth=120)

# ---------- 1. Make product reviews where word ORDER matters ----------
things = ["delivery", "quality", "price", "battery", "screen", "service", "packaging", "fit"]
good = ["great", "excellent", "fast", "perfect", "lovely", "brilliant"]
bad = ["poor", "terrible", "slow", "awful", "disappointing", "cheap"]
fillers = ["honestly", "overall", "to be fair", "i think", "sadly", "luckily"]

def clause():
    positive = rng.random() < 0.5
    negated = rng.random() < 0.4                    # "not great" flips the meaning
    word = rng.choice(good if positive else bad)
    text = f"the {rng.choice(things)} was {'not ' if negated else ''}{word}"
    return text, (positive != negated)

def review():
    parts, scores = zip(*[clause() for _ in range(rng.integers(1, 4))])
    text = " and ".join(parts)
    if rng.random() < 0.5:
        text = f"{rng.choice(fillers)} {text}"
    return text, int(np.mean(scores) > 0.5)       # positive if most clauses are positive

data = [review() for _ in range(8000)]
texts = np.array([t for t, _ in data]); labels = np.array([l for _, l in data])
x_train, x_test, y_train, y_test = texts[:6000], texts[6000:], labels[:6000], labels[6000:]
print("Example:", x_train[0], "->", y_train[0])
print("Example:", x_train[1], "->", y_train[1])

# ---------- 2. TextVectorization: text -> padded integer sequences ----------
vectorize = layers.TextVectorization(max_tokens=100, output_sequence_length=25)
vectorize.adapt(x_train)
vocab = [str(w) for w in vectorize.get_vocabulary()]
print("\nVocabulary size:", len(vocab), "| first 8:", vocab[:8])
print("'the price was not great' ->", np.asarray(vectorize(["the price was not great"]))[0])

ids_train = vectorize(x_train)             # shape (6000, 25): one row of word ids per review
ids_test = vectorize(x_test)

# ---------- 3. Two models: bag of words vs LSTM ----------
def bag_of_words():
    return keras.Sequential([
        keras.Input(shape=(25,)),
        layers.Embedding(input_dim=len(vocab), output_dim=16),
        layers.GlobalAveragePooling1D(),                # averages the words: order is lost
        layers.Dense(16, activation="relu"),
        layers.Dense(1, activation="sigmoid")])

def lstm_model():
    return keras.Sequential([
        keras.Input(shape=(25,)),
        layers.Embedding(input_dim=len(vocab), output_dim=16, mask_zero=True),
        layers.Bidirectional(layers.LSTM(32)),          # reads the words in order
        layers.Dense(1, activation="sigmoid")])

tests = ["the service was great", "the service was not great",
         "the price was awful and the quality was not bad",
         "sadly the battery was not excellent and the fit was poor"]
for name, build in [("Bag of words", bag_of_words), ("Bidirectional LSTM", lstm_model)]:
    keras.utils.set_random_seed(0)
    model = build()
    model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
    model.fit(ids_train, y_train, epochs=10, batch_size=32, verbose=0, validation_split=0.1)
    acc = model.evaluate(ids_test, y_test, verbose=0)[1]
    probs = model.predict(vectorize(tests), verbose=0).ravel()
    print(f"\n{name}: test accuracy = {acc:.3f}  ({model.count_params():,} weights)")
    for t, p in zip(tests, probs):
        print(f"  {p:5.1%} positive  <- {t}")
Output
Example: the delivery was not perfect and the screen was poor and the screen was awful -> 0
Example: honestly the delivery was perfect and the packaging was not poor and the battery was not slow -> 1

Vocabulary size: 35 | first 8: ['', '[UNK]', 'was', 'the', 'and', 'not', 'battery', 'screen']
'the price was not great' -> [ 3 10  2  5 14  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0]

Bag of words: test accuracy = 0.669  (849 weights)
  54.4% positive  <- the service was great
  49.5% positive  <- the service was not great
  19.2% positive  <- the price was awful and the quality was not bad
  21.4% positive  <- sadly the battery was not excellent and the fit was poor

Bidirectional LSTM: test accuracy = 1.000  (13,169 weights)
  99.9% positive  <- the service was great
   0.0% positive  <- the service was not great
   0.0% positive  <- the price was awful and the quality was not bad
   0.0% positive  <- sadly the battery was not excellent and the fit was poor
▶ Open Google ColabPaste the code into a new Colab notebook and press Run. Keras is already installed.

What the output tells us

  • TextVectorization built a vocabulary of 35 entries, with padding ('') at 0 and [UNK] at 1, and turned "the price was not great" into [3, 10, 2, 5, 14] followed by padding zeros.
  • The bag-of-words model managed only 66.9%. It can't tell "the service was great" (54.4% positive) from "the service was not great" (49.5%): both are near 50-50, because the words are almost the same.
  • The bidirectional LSTM scored 100% on 2,000 unseen reviews. Reading in order, it learned that "not" flips the next word: 99.9% positive for "great" and 0.0% for "not great".
  • "bad" wasn't in the vocabulary, because the training reviews never used it, so it became [UNK]. Real projects need plenty of varied training text, or a pretrained model (Module 10).
  • The LSTM needed 13,169 weights against 849 for bag of words. More power costs more computation, so try the simple model first, and switch when order matters.

Is 100% realistic?

Only because these reviews follow simple rules. Real reviews are messier, with sarcasm, typos and mixed opinions. The lesson that carries over is this: when word order changes the meaning, a model that reads in order beats one that doesn't. For the very best results on real text, today's go-to is a pretrained Transformer model, which you'll meet in Module 10.

SummaryKey takeaways

  • TextVectorization tokenises text, looks words up in a vocabulary and pads or cuts to a fixed length. Call adapt() on training text.
  • Index 0 is padding and 1 is [UNK] (unknown words).
  • Embedding(input_dim, output_dim) learns a vector per word; parameters = vocabulary × dimensions.
  • Bag of words (GlobalAveragePooling1D) is fast but ignores order.
  • LSTM, GRU and Bidirectional read sequences in order; use mask_zero=True to skip padding.
  • Recurrent layers take (batch, time steps, features); use return_sequences=True to stack them.
  • When order changes meaning, order-aware models win: 66.9% vs 100% in our experiment.

Check your understanding

Your Keras roadmap

  1. 01
    Introduction to KerasWhat Keras is, setup and your first model
  2. 02
    Tensors and Data for KerasShapes, scaling and splitting data
  3. 03
    The Sequential API and Dense LayersStacking layers into a model
  4. 04
    Compiling ModelsLosses, optimisers and metrics
  5. 05
    Training and Evaluatingfit(), validation and callbacks
  6. 06
    The Functional APIMultiple inputs, outputs and branches
  7. 07
    RegularisationDropout, batch normalisation and early stopping
  8. 08
    Image Classification with KerasCNNs and data augmentation
  9. 09
    Text and Sequences with KerasYou are here
  10. 10
    Pretrained Models, Saving and DeploymentFine-tuning and sharing your models