Model Evaluation

5.1Why evaluation matters

Imagine a driving instructor who says "you're ready for your test" without ever watching you drive. You wouldn't trust them. A model is the same: until you've measured it properly, you don't know if it works.

You've already met some measures in this track:

In Module 4 you also met the accuracy trap: when one answer is rare, a useless model can still score very high accuracy. This module gives you the tools to see through that, and to check whether your model will really work on new data.

5.2The confusion matrix, properly

Throughout this module, we'll use one example. A bank has built a model to catch fraudulent card payments. It's tested on 1,000 payments, and 50 of them are really fraud.

The model's accuracy is (920 + 40) ÷ 1,000 = 96%. Sounds great. But it missed 10 frauds and blocked 30 genuine customers. To understand if that's good enough, we need better measures.

5.3Precision and recall

These are the two most important classification measures. They answer two different questions.

Which one matters more?

As you saw with the threshold in Module 4, there's usually a trade-off: raising one tends to lower the other.

5.4The F1 score

Sometimes you want one number that balances precision and recall. That's the F1 score.

For the bank: 2 × (0.57 × 0.80) ÷ (0.57 + 0.80) = 0.67.

F1 is a special kind of average (called a harmonic mean) that is pulled down hard by whichever score is lower. A model can't hide a terrible recall behind a great precision:

The last row is a model that almost never says "fraud", so it's never wrong when it does, but it misses 90% of fraud. The normal average (0.55) makes it look okay. F1 (0.18) tells the truth.

5.5ROC curves and AUC

Precision, recall and F1 all depend on the threshold you choose. An ROC curve shows how the model performs at every threshold at once. It plots:

• True positive rate (recall): the share of frauds caught, going up the side.
• False positive rate: the share of genuine payments wrongly flagged, going along the bottom.

Try it with a fraud model scoring 200 payments (40 are fraud). Move the threshold and watch the dot travel along the curve.

What is AUC?

AUC stands for "Area Under the Curve". It squeezes the whole ROC curve into one number between 0 and 1.

5.6Overfitting and underfitting

This is the most important idea in this module. A model can fail in two opposite ways.

The tell-tale sign of overfitting is a big gap between the training score and the test score. As a model gets more complex, training error keeps dropping, but test error eventually starts rising again:

5.7How to fix overfitting and underfitting

Regularisation: Ridge and Lasso

In Module 3, linear regression chose coefficients to make the error as small as possible. Regularisation changes the goal to: make the error small, but keep the coefficients small too. Big coefficients mean a model is leaning very heavily on certain features, which is often a sign of overfitting.

The setting alpha controls how strong the penalty is: higher alpha means a simpler model. Logistic regression is regularised by default in scikit-learn, controlled by a setting called C (where a smaller C means stronger regularisation).

5.8Cross-validation

A single train/test split has a weakness: you might get lucky (or unlucky) with which rows end up in the test set. Your score could change a lot just by splitting differently.

K-fold cross-validation fixes this. The training data is cut into k equal parts (usually 5 or 10), called folds. The model is trained and tested k times, each time using a different fold as the test set. Then we average the scores.

5.9Evaluating a model in Python

Now for a real dataset. scikit-learn includes a famous medical dataset of 569 breast tumour scans, each with 30 measurements and a label saying whether it was benign (harmless) or malignant (cancer). Missing a malignant tumour is very serious, so recall for "malignant" is the measure that matters most.

We also use a Pipeline, which bundles scaling and the model together. As promised in Module 2, this stops data leakage automatically, even inside cross-validation.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, PolynomialFeatures
from sklearn.linear_model import LogisticRegression, LinearRegression
from sklearn.metrics import (confusion_matrix, classification_report,
                             roc_auc_score, mean_squared_error)

# 1. Load a real medical dataset: 569 tumour scans, 30 measurements each
X, y = load_breast_cancer(return_X_y=True)
y = 1 - y            # make 1 = malignant (the case we want to catch)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

# 2. A Pipeline scales and trains in one step (no leakage)
model = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000)),
])
model.fit(X_train, y_train)

# 3. Confusion matrix and the full report
pred = model.predict(X_test)
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred,
      target_names=["benign", "malignant"], digits=2))

# 4. ROC AUC uses probabilities, not just yes/no
prob = model.predict_proba(X_test)[:, 1]
print("ROC AUC:", round(roc_auc_score(y_test, prob), 3))

# 5. 5-fold cross-validation on the training data
scores = cross_val_score(model, X_train, y_train, cv=5, scoring="recall")
print("Recall per fold:", scores.round(2))
print("Average recall:", round(scores.mean(), 3))

# 6. Spot overfitting: 40 noisy points, three models of rising complexity
rng = np.random.default_rng(0)
x = rng.uniform(0, 10, 40).reshape(-1, 1)
target = 3 * np.sin(x).ravel() + x.ravel() + rng.normal(0, 1, 40)
x_tr, x_te, t_tr, t_te = train_test_split(x, target, test_size=0.5, random_state=0)

for degree in [1, 4, 15]:
    curve = Pipeline([("poly", PolynomialFeatures(degree)),
                      ("scale", StandardScaler()),
                      ("reg", LinearRegression())])
    curve.fit(x_tr, t_tr)
    train_err = mean_squared_error(t_tr, curve.predict(x_tr)) ** 0.5
    test_err = mean_squared_error(t_te, curve.predict(x_te)) ** 0.5
    print(f"Degree {degree:2}: train RMSE={train_err:5.2f}  test RMSE={test_err:5.2f}")

[[89  1]
 [ 4 49]]
              precision    recall  f1-score   support

      benign       0.96      0.99      0.97        90
   malignant       0.98      0.92      0.95        53

    accuracy                           0.97       143
   macro avg       0.97      0.96      0.96       143
weighted avg       0.97      0.97      0.96       143

ROC AUC: 0.996
Recall per fold: [0.94 1.   0.94 0.97 0.94]
Average recall: 0.956
Degree  1: train RMSE= 1.80  test RMSE= 2.11
Degree  4: train RMSE= 1.20  test RMSE= 1.26
Degree 15: train RMSE= 0.60  test RMSE=28.47

What the output tells us

• Confusion matrix: of 53 malignant tumours in the test set, the model caught 49 and missed 4. Only 1 benign tumour was wrongly flagged.
• Classification report: for malignant, precision is 0.98 (when it says cancer, it's nearly always right) and recall is 0.92 (it finds 92% of cancers). The 4 missed cases are the ones a doctor would worry about, so in practice we might lower the threshold to push recall higher.
• ROC AUC of 0.996 means the model is excellent at ranking malignant tumours above benign ones.
• Cross-validation recall is consistent across all 5 folds (0.94 to 1.00), so this isn't a lucky split.
• The overfitting check shows all three patterns. Degree 1 is poor on both (underfitting). Degree 4 is good on both, with a tiny gap (just right). Degree 15 has the best training score but a terrible test score (overfitting).

5.10Choosing the right measure

There's no single "best" measure. The right one depends on the cost of each kind of mistake in your business. Here's a cheat sheet.

SummaryKey takeaways

Check your understanding

Your machine learning roadmap

1. 01
   Introduction to Machine LearningWhat ML is and how it works
2. 02
   Preparing Data for Machine LearningFeatures, encoding, scaling, train/test split
3. 03
   Linear RegressionPredicting numbers
4. 04
   Logistic RegressionPredicting yes or no
5. 05
   Evaluating ModelsYou are here
6. 06
   K-Nearest NeighboursLearning from similar examples
7. 07
   Decision TreesFlowcharts that learn
8. 08
   Random Forest and Gradient BoostingMany models working together
9. 09
   Support Vector MachinesFinding the best boundary
10. 10
    Naive BayesProbability-based classification
11. 11
    Clustering with K-MeansFinding groups without labels
12. 12
    Dimensionality Reduction with PCASimplifying big datasets
13. 13
    Capstone ProjectBuild and present a full ML project