CX Learn by CadetX
Exams My Dashboard
Keras · TensorFlow · Deep Learning
KR-101 · Module 4 Beginner about 35 minutes 7 Lessons Prereq: Modules 1 to 3

Compiling Models

Building a model describes its shape. Compiling it decides how it learns. In one line, you choose the loss (how mistakes are measured), the optimiser (how weights are improved) and the metrics (how progress is reported). Get these right and training goes smoothly; get them wrong and your model can fail silently.

  • Level: Beginner
  • Time: about 35 minutes
  • Needs: Modules 1 to 3

By the end of this module you will be able to

  • Explain what model.compile() does
  • Choose the right loss function for any task
  • Match the loss to your label format and output layer
  • Explain the from_logits option
  • Choose an optimiser and set the learning rate
  • Add metrics like precision, recall and AUC
  • Spot common compile mistakes before they waste your time

4.1What does compile() do?

Compiling prepares a model for training. You make three choices:

compile.py
model.compile(
    loss="binary_crossentropy",     # 1. How to measure mistakes
    optimizer="adam",               # 2. How to improve the weights
    metrics=["accuracy"],           # 3. What to report while training
)
▶ Open Google ColabPaste the code into a new Colab notebook and press Run. Keras is already installed.

Like a fitness coach

The loss is the measurement the coach tries to improve, like your time over 5 km. The optimiser is the training plan that decides how to improve it. The metrics are extra numbers the coach reports to you, like your heart rate, which you care about but which don't drive the plan.

You must compile before calling fit(). You can also compile again at any time to change the settings, for example to lower the learning rate.

4.2Choosing a loss function

The loss is the single number the model tries to make as small as possible during training. It must suit your task:

TaskOutput layerLoss
Predict a numberDense(1)"mse" (mean squared error), or "mae" if there are extreme outliers
Yes or noDense(1, activation="sigmoid")"binary_crossentropy"
One of several classes, integer labels (0, 1, 2…)Dense(n, activation="softmax")"sparse_categorical_crossentropy"
One of several classes, one-hot labelsDense(n, activation="softmax")"categorical_crossentropy"
Several labels at onceDense(n, activation="sigmoid")"binary_crossentropy"

Cross-entropy, used for classification, rewards being confidently right and punishes being confidently wrong very heavily. If the correct answer is "benign", predicting 95% benign gives a tiny loss, while predicting 5% benign gives a big one.

Strings or objects

You can write losses as strings like "binary_crossentropy", or as objects like keras.losses.BinaryCrossentropy(). Objects let you change settings, which you need for one important option.

The from_logits option

Some models leave out the final sigmoid or softmax and output raw scores, called logits. If you do that, you must tell the loss, or it will give wrong results without any error:

logits.py
# Output layer with NO activation: raw scores (logits)
layers.Dense(5)

model.compile(
    optimizer="adam",
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=["accuracy"],
)
▶ Open Google ColabPaste the code into a new Colab notebook and press Run. Keras is already installed.

This is slightly more numerically stable, and you'll see it in many tutorials and pretrained models. As a beginner, the simplest rule is: use sigmoid or softmax in the output layer, and leave from_logits at its default (False).

4.3Check your setup

The output layer, the loss and the label format must all agree. A mismatch often doesn't crash; the model just learns badly. Try some combinations:

Try it: compile checker

your_setup.py

4.4Optimisers and the learning rate

After each batch, the optimiser nudges every weight in the direction that reduces the loss. The size of each nudge is set by the learning rate, the most important setting in deep learning.

OptimiserHow it worksTypical learning rate
SGDPlain gradient descent: a fixed step downhill. Add momentum=0.9 to build up speed0.01
RMSpropGives each weight its own step size, smaller for jumpy ones0.001
AdamCombines momentum and per-weight step sizes. The best default0.001
AdamWAdam plus weight decay, which helps against overfitting; popular for large models0.001

Writing optimizer="adam" uses the default learning rate of 0.001. To choose your own, use an optimiser object:

learning_rate.py
model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.0005),
    loss="binary_crossentropy",
    metrics=["accuracy"],
)
▶ Open Google ColabPaste the code into a new Colab notebook and press Run. Keras is already installed.
Learning rateWhat you'll see
Too lowThe loss falls painfully slowly
About rightThe loss falls quickly, then levels off
Too highThe loss jumps around wildly, or becomes huge or nan

4.5Metrics

Metrics are extra scores Keras reports during training and evaluation. They don't affect learning; they help you understand it. Accuracy alone can mislead, especially when one class is rare, so add metrics that fit your problem:

MetricQuestion it answersIn Keras
AccuracyWhat share of predictions were right?"accuracy"
PrecisionWhen the model says "yes", how often is it right?keras.metrics.Precision()
RecallOf all the real "yes" cases, how many did it find?keras.metrics.Recall()
AUCHow well does it rank "yes" cases above "no" cases, from 0.5 (guessing) to 1 (perfect)?keras.metrics.AUC()
MAEFor numbers: how far off is it, on average?"mae"

Give each metric a name, like keras.metrics.Recall(name="recall"), to get tidy labels in your results.

4.6Compile choices in Python

Using the breast cancer dataset (569 tumours, 30 measurements), we'll compile a model with several metrics, compare three optimisers and four learning rates, and see what happens with a classic mistake.

compile_choices.py
import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
import keras
from keras import layers
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

# Breast cancer data: 569 tumours, 30 measurements, 1 = benign, 0 = malignant
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X.astype("float32"), y, test_size=0.25, random_state=42, stratify=y)
norm = layers.Normalization()
norm.adapt(X_train)

def build(output_layer):
    keras.utils.set_random_seed(0)
    return keras.Sequential([keras.Input(shape=(30,)), norm,
                             layers.Dense(16, activation="relu"), output_layer])

# ---------- 1. A well-compiled model with several metrics ----------
model = build(layers.Dense(1, activation="sigmoid"))
model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.001),
    loss="binary_crossentropy",
    metrics=["accuracy", keras.metrics.Precision(name="precision"),
             keras.metrics.Recall(name="recall"), keras.metrics.AUC(name="auc")])
model.fit(X_train, y_train, epochs=30, batch_size=32, verbose=0)
results = model.evaluate(X_test, y_test, verbose=0, return_dict=True)
print({k: round(v, 3) for k, v in results.items()})

# ---------- 2. Comparing optimisers ----------
print()
for name, opt in [("SGD", keras.optimizers.SGD(learning_rate=0.01)),
                  ("RMSprop", keras.optimizers.RMSprop(learning_rate=0.001)),
                  ("Adam", keras.optimizers.Adam(learning_rate=0.001))]:
    m = build(layers.Dense(1, activation="sigmoid"))
    m.compile(optimizer=opt, loss="binary_crossentropy", metrics=["accuracy"])
    h = m.fit(X_train, y_train, epochs=10, batch_size=32, verbose=0)
    print(f"{name:8} loss after 10 epochs = {h.history['loss'][-1]:.3f}")

# ---------- 3. Comparing learning rates (Adam) ----------
print()
for lr in [0.00001, 0.001, 0.01, 10.0]:
    m = build(layers.Dense(1, activation="sigmoid"))
    m.compile(optimizer=keras.optimizers.Adam(learning_rate=lr),
              loss="binary_crossentropy", metrics=["accuracy"])
    h = m.fit(X_train, y_train, epochs=10, batch_size=32, verbose=0)
    test_loss, test_acc = m.evaluate(X_test, y_test, verbose=0)
    print(f"learning_rate={lr:<7} test loss={test_loss:8.3f}  test accuracy={test_acc:.3f}")
print("Training loss by epoch at 10.0:", [round(v) for v in h.history["loss"]])

# ---------- 4. A classic mistake: softmax with ONE output neuron ----------
wrong = build(layers.Dense(1, activation="softmax"))
wrong.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
wrong.fit(X_train, y_train, epochs=10, batch_size=32, verbose=0)
print("\nSoftmax with 1 neuron always predicts:",
      wrong.predict(X_test[:5], verbose=0).ravel(),
      f"| test accuracy = {wrong.evaluate(X_test, y_test, verbose=0)[1]:.3f}")
Output
{'accuracy': 0.944, 'auc': 0.995, 'loss': 0.113, 'precision': 0.956, 'recall': 0.956}

SGD      loss after 10 epochs = 0.307
RMSprop  loss after 10 epochs = 0.169
Adam     loss after 10 epochs = 0.220

learning_rate=1e-05   test loss=   1.107  test accuracy=0.308
learning_rate=0.001   test loss=   0.261  test accuracy=0.895
learning_rate=0.01    test loss=   0.085  test accuracy=0.965
learning_rate=10.0    test loss= 490.764  test accuracy=0.965
Training loss by epoch at 10.0: [553, 574, 397, 117, 25, 6, 4, 126, 393, 67]

Softmax with 1 neuron always predicts: [1. 1. 1. 1. 1.] | test accuracy = 0.629
▶ Open Google ColabPaste the code into a new Colab notebook and press Run. Keras is already installed.

What the output tells us

  • Several metrics at once: the model reached 94.4% accuracy with precision and recall of 95.6% each, and an AUC of 0.995, showing it ranks benign tumours above malignant ones almost perfectly.
  • Optimisers learn at different speeds. After 10 epochs, RMSprop and Adam had lower loss than plain SGD. With more epochs, all three would do well here; Adam is simply a reliable default.
  • A learning rate of 0.00001 is far too low: after 10 epochs the model has barely moved from its random start and scores just 30.8%.
  • 0.01 was the best here, with the lowest test loss. Adam's default of 0.001 was still improving.
  • A learning rate of 10 is far too high. The training loss leaps around between 4 and 574, and the test loss is 490. Accuracy still looks fine because this dataset is easy, but a test loss that high means the model is wildly overconfident whenever it's wrong. Always watch the loss, not just accuracy.
  • Softmax with one neuron always outputs exactly 1, so the model can only ever say "benign". Its 62.9% accuracy is simply the share of benign tumours, and there was no error message to warn you. The compile checker above catches this.

SummaryKey takeaways

  • compile() sets the loss, the optimiser and the metrics. Always compile before fit().
  • Match the loss to the task: mse for numbers, binary_crossentropy for yes/no, sparse_categorical_crossentropy for classes with integer labels.
  • If your output layer has no activation, use from_logits=True in the loss.
  • Adam with a learning rate around 0.001 is a great default; use an optimiser object to change it.
  • Too low a learning rate is slow; too high makes the loss jump around or explode.
  • Add precision, recall and AUC when accuracy isn't enough.
  • Mismatched outputs, losses and labels often fail silently, so check them.

Check your understanding

Your Keras roadmap

  1. 01
    Introduction to KerasWhat Keras is, setup and your first model
  2. 02
    Tensors and Data for KerasShapes, scaling and splitting data
  3. 03
    The Sequential API and Dense LayersStacking layers into a model
  4. 04
    Compiling ModelsYou are here
  5. 05
    Training and Evaluatingfit(), validation and callbacks
  6. 06
    The Functional APIMultiple inputs, outputs and branches
  7. 07
    RegularisationDropout, batch normalisation and early stopping
  8. 08
    Image Classification with KerasCNNs and data augmentation
  9. 09
    Text and Sequences with KerasEmbeddings, LSTMs and text classification
  10. 10
    Pretrained Models, Saving and DeploymentFine-tuning and sharing your models