Random Forest & Gradient Boosting

8.1Ensemble learning: the wisdom of the crowd

In 1906, the scientist Francis Galton visited a country fair where around 800 people paid to guess the weight of an ox. Individual guesses were all over the place. But when Galton averaged them, the crowd's answer was within about 1% of the true weight, closer than almost any single expert.

This is the idea behind ensemble learning: combine many models, and the group is usually more accurate than any one of them.

There's one important condition: the models must make different mistakes. If every member of the crowd copied the same person's guess, averaging them would change nothing. The two methods in this module make models different in two different ways:

8.2Why combine trees?

In Module 7, you saw two problems with a single decision tree:

• It overfits easily. Deep trees memorise the training data.
• It's unstable. Change a few rows of data and you can get a completely different tree.

In Module 5 terms, a deep tree has high variance: its predictions jump around depending on exactly which data it saw. The good news is that high variance is exactly what averaging fixes. One wobbly tree is unreliable; the average of 300 wobbly trees is remarkably steady.

8.3Bagging and bootstrap samples

How do you train many different trees from one dataset? Bagging (short for bootstrap aggregating) gives each tree its own random version of the data.

Each version is a bootstrap sample: a random sample of the same size as the original, picked with replacement. That means after picking a row, you "put it back", so it can be picked again.

On average, each bootstrap sample contains about 63% of the original rows, with the rest being repeats. The rows a tree didn't see are called its out-of-bag (OOB) rows. They make a handy free test set: each row can be scored using only the trees that never saw it. This gives the OOB score, a built-in estimate of how well the forest will do on new data.

8.4Random forest

A random forest is bagging with decision trees, plus one extra twist. At every split, each tree is only allowed to look at a random handful of features, not all of them.

Why? Imagine one feature, like "months with us", is very strong. Without the twist, almost every tree would ask about it first, and the trees would end up looking similar and making similar mistakes. Forcing each split to choose from a random subset makes the trees more different from each other, so their mistakes cancel out better.

Watch this happen with a real forest trained on 4,000 broadband customers.

Notice two things. First, the score climbs quickly for the first 20 or so trees, then levels off. Adding more trees never makes a random forest overfit; it just stops helping and makes it slower. Second, with only a few trees, the vote on a single customer swings wildly. With hundreds, it settles on a stable answer. (This lab uses fully grown trees. In lesson 8.9, limiting each leaf to at least 10 customers pushes the score even higher, to 0.857.)

8.5Gradient boosting: learning from mistakes

Boosting takes the opposite approach. Instead of growing many big trees independently, it grows many small trees one after another. Each new tree focuses on what the model so far is still getting wrong.

How gradient boosting works

Let's use the ten Newcastle house prices from Module 3.

1. Start simple: predict the average price for every house: £201.1k.
2. Measure the mistakes: work out each house's residual (actual − predicted), just like in Module 3.
3. Train a small tree to predict those mistakes, not the prices.
4. Add a small part of that tree's answer to the current prediction.
5. Repeat from step 2, with a new tree each time.

Each round moves every prediction a little closer to the truth. The word "gradient" comes from the same idea as gradient descent in Module 3: each step heads "downhill" towards a smaller error.

The learning rate

In step 4, we only add a small part of each new tree's answer. That fraction is the learning rate (0.3 in Figure 2). A small learning rate means small, careful steps: you need more trees, but the final model usually generalises better. It's like turning a steering wheel gently rather than yanking it.

8.6Bagging vs boosting

8.7The key hyperparameters

Random forest

Gradient boosting

Remember: learning_rate and n_estimators work together. Halve the learning rate and you'll roughly need to double the number of trees.

8.8XGBoost, LightGBM and CatBoost

scikit-learn has its own boosting models, but in industry you'll often hear three names. They're all gradient boosting, with clever engineering that makes them faster and more accurate on big datasets.

They all follow the same .fit() and .predict() pattern you already know, so switching between them is easy. For data in rows and columns (like spreadsheets and database tables), a well-tuned gradient boosting model is very often the strongest option, frequently beating neural networks.

8.9Comparing models in Python

In Module 4, our churn data was made from logistic regression's own formula, which gave it an unfair advantage. Real customer behaviour is messier: it has thresholds ("more than 3 calls"), special cases ("new customers are risky") and combinations ("an expensive bill and no contract"). So here we use 5,000 customers with more realistic, tangled patterns, and let four models compete.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import accuracy_score, roc_auc_score

# 1. 5,000 customers with more realistic, tangled patterns
rng = np.random.default_rng(8)
n = 5000
df = pd.DataFrame({
    "support_calls":    rng.poisson(3, n),
    "months":           rng.integers(1, 72, n),
    "monthly_bill":     rng.integers(20, 90, n),
    "monthly_contract": rng.integers(0, 2, n),
    "age":              rng.integers(18, 80, n),
})
z = (-2.2
     + 1.4 * (df["support_calls"] >= 4) + 1.0 * (df["support_calls"] >= 7)
     + 1.8 * (df["months"] < 6) - 1.2 * (df["months"] > 36)
     + 2.0 * df["monthly_contract"] * (df["monthly_bill"] > 60)
     + 0.9 * ((df["age"] < 25) | (df["age"] > 70))
     - 0.6 * df["monthly_contract"] * (df["months"] > 24))
df["left"] = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
print(f"Customers who left: {df['left'].mean():.0%}\n")

X = df.drop(columns="left")
y = df["left"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

# 2. Four models to compare
models = {
    "Logistic regression": Pipeline([("scale", StandardScaler()),
                                     ("clf", LogisticRegression())]),
    "Single tree":         DecisionTreeClassifier(max_depth=4, random_state=42),
    "Random forest":       RandomForestClassifier(
                               n_estimators=300, min_samples_leaf=10,
                               oob_score=True, random_state=42),
    "Gradient boosting":   GradientBoostingClassifier(
                               n_estimators=150, learning_rate=0.05,
                               max_depth=2, random_state=42),
}

# 3. Train each one and score it on the test set
for name, model in models.items():
    model.fit(X_train, y_train)
    acc = accuracy_score(y_test, model.predict(X_test))
    auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
    print(f"{name:20} accuracy={acc:.3f}  ROC AUC={auc:.3f}")

# 4. Random forest extras
rf = models["Random forest"]
print("\nOut-of-bag score:", round(rf.oob_score_, 3))
for col, imp in sorted(zip(X.columns, rf.feature_importances_), key=lambda t: -t[1]):
    print(f"{col:17} {imp:.2f}")

Customers who left: 23%

Logistic regression  accuracy=0.816  ROC AUC=0.811
Single tree          accuracy=0.816  ROC AUC=0.817
Random forest        accuracy=0.829  ROC AUC=0.857
Gradient boosting    accuracy=0.836  ROC AUC=0.860

Out-of-bag score: 0.814
months            0.38
monthly_bill      0.20
support_calls     0.18
age               0.14
monthly_contract  0.09

What the output tells us

• Accuracy barely separates them (81.6% to 83.6%). Remember the accuracy trap from Modules 4 and 5: 77% of customers stay, so "always stay" would already score 77%.
• ROC AUC tells the real story. Logistic regression (0.811) and the single tree (0.817) are close. The random forest (0.857) and gradient boosting (0.860) are clearly better at ranking which customers are most likely to leave.
• Why do the ensembles win? Logistic regression can only draw one straight boundary, so it can't capture "bill over £60 and no contract". A single tree can, but it's unstable. Hundreds of trees capture the tangled patterns and stay stable.
• The OOB score (81.4%) is close to the real test accuracy (82.9%), which shows it's a useful free estimate when you don't want to hold back a separate test set.
• Feature importance now ranks months with us first. That makes sense: the data has a sharp "new customer" effect and a "loyal customer" effect, which trees capture well.

8.10Strengths, weaknesses and real-world uses

SummaryKey takeaways

Check your understanding

Your machine learning roadmap

1. 01
   Introduction to Machine LearningWhat ML is and how it works
2. 02
   Preparing Data for Machine LearningFeatures, encoding, scaling, train/test split
3. 03
   Linear RegressionPredicting numbers
4. 04
   Logistic RegressionPredicting yes or no
5. 05
   Evaluating ModelsAccuracy, precision, recall, overfitting, cross-validation
6. 06
   K-Nearest NeighboursLearning from similar examples
7. 07
   Decision TreesFlowcharts that learn
8. 08
   Random Forest and Gradient BoostingYou are here
9. 09
   Support Vector MachinesFinding the best boundary
10. 10
    Naive BayesProbability-based classification
11. 11
    Clustering with K-MeansFinding groups without labels
12. 12
    Dimensionality Reduction with PCASimplifying big datasets
13. 13
    Capstone ProjectBuild and present a full ML project