Logistic Regression

4.1What is logistic regression?

In Module 3, linear regression predicted a number, such as a house price. But many business questions have only two possible answers:

This is called binary classification (binary means "two"). Logistic regression is the classic algorithm for it. Instead of saying just "yes" or "no", it gives a probability: "there's a 78% chance this customer will leave".

We'll continue with the broadband company from Module 2. They want to predict which customers will leave. One strong clue is how many times a customer has called customer support in the last three months. Unhappy customers call more.

4.2Why not just use linear regression?

We can write "left" as 1 and "stayed" as 0. So why not draw a straight line through the data, like in Module 3? Let's try it with 26 customers.

There are two problems:

1. Impossible answers. A probability must be between 0% and 100%. The straight line happily predicts 120% for a customer with 10 calls, and −13% for one with none.
2. Wrong shape. In real life, the chance of leaving doesn't rise steadily forever. It rises quickly in the middle, then levels off near 100%, because you can't be "more certain than certain".

We need a line that bends into an S-shape and stays between 0 and 1. That's exactly what logistic regression does.

4.3The sigmoid: the S-curve

Logistic regression works in two steps.

Step 1: calculate a score, exactly like linear regression. We'll call it z.

Step 2: squash the score into a probability using a special function called the sigmoid (or logistic function).

Don't worry about the maths. e is just a special number (about 2.718). All you need to know is what the sigmoid does: it takes any number, however big or small, and turns it into a value between 0 and 1.

For our 26 customers, the model learned m = 0.79 and c = −3.69. So for a customer with 6 support calls:

That customer has about a 74% chance of leaving.

4.4From probability to decision: the threshold

A probability is useful, but a business usually needs a decision: do we call this customer with a special offer or not? We use a threshold. If the probability is above the threshold, the answer is "yes".

The default threshold is 0.5. But you can move it, and that changes the results. Try it below.

The threshold trade-off

4.5How it learns: log loss

In Module 3, linear regression learned by making the squared errors as small as possible. Logistic regression uses a different measure of "wrongness" called log loss (also called cross-entropy).

The idea is simple: punish confident wrong answers very hard. Imagine a customer who really did leave:

4.6Reading the coefficients

Just like linear regression, logistic regression gives each feature a coefficient. The sign tells you the direction:

Odds: a handy way to explain the size

The size is a little harder to read than in linear regression, because the S-curve bends. The easiest way is to talk about odds. If you calculate e^coefficient, you get how much the odds are multiplied for each 1-unit increase.

Our coefficient for support calls is 0.79, and e^0.79 ≈ 2.2. So in plain English: each extra support call roughly doubles the odds that a customer leaves. That's a sentence a manager can act on.

4.7The decision boundary

With one feature, the threshold of 0.5 is just a single point: customers with more than about 4.7 calls are predicted to leave. With two features, it becomes a line that splits the chart in two. This is called the decision boundary.

Notice some dots are on the "wrong" side. That's normal. Real life is messy, and no model is perfect. Also notice the boundary is a straight line. That's the main limitation of logistic regression: if the groups can only be separated by a curvy line, you'll need a different algorithm, such as the decision trees or SVMs you'll meet later in this track.

4.8Measuring a classifier

For regression we used MAE and RMSE. For classification, the simplest measure is accuracy: the percentage of predictions that were right.

A better picture comes from a confusion matrix, which shows exactly which kinds of mistakes the model makes. You already saw one in the threshold lab:

4.9Logistic regression in Python

Now let's build a churn model for 1,000 customers, using four features: support calls, months as a customer, monthly bill, and whether they're on a monthly (no fixed) contract. We scale the features first, as you learned in Module 2, because logistic regression trains best that way.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix

# 1. Create 1,000 broadband customers
rng = np.random.default_rng(7)
n = 1000
df = pd.DataFrame({
    "support_calls":    rng.poisson(3, n),
    "months":           rng.integers(1, 60, n),
    "monthly_bill":     rng.integers(20, 70, n),
    "monthly_contract": rng.integers(0, 2, n),   # 1 = no fixed contract
})
z = (-4.5 + 1.0 * df["support_calls"] - 0.08 * df["months"]
     + 0.04 * df["monthly_bill"] + 1.5 * df["monthly_contract"])
df["left"] = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
print("Customers who left:", f"{df['left'].mean():.0%}")

# 2. Features (X), target (y) and split
X = df.drop(columns="left")
y = df["left"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

# 3. Scale (logistic regression works best with scaled features)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)

# 4. Train the model
model = LogisticRegression()
model.fit(X_train_s, y_train)

# 5. What did it learn? (positive = more likely to leave)
for name, coef in zip(X.columns, model.coef_[0]):
    print(f"{name:17} {coef:+.2f}")

# 6. Test it
pred = model.predict(X_test_s)
print("Accuracy:", round(accuracy_score(y_test, pred), 3))
print("Confusion matrix:")
print(confusion_matrix(y_test, pred))

# 7. Probability for a new customer: 4 calls, 10 months, £45 bill, no contract
new = pd.DataFrame({"support_calls": [4], "months": [10],
                    "monthly_bill": [45], "monthly_contract": [1]})
prob = model.predict_proba(scaler.transform(new))[0, 1]
print(f"Chance this customer leaves: {prob:.0%}")

Customers who left: 33%
support_calls     +1.59
months            -1.19
monthly_bill      +0.59
monthly_contract  +0.83
Accuracy: 0.815
Confusion matrix:
[[113  21]
 [ 16  50]]
Chance this customer leaves: 87%

What the output tells us

• The coefficients make sense. Support calls (+1.59) push customers towards leaving the most. Months with us (−1.19) keeps them loyal. A higher bill and no fixed contract also increase the risk. Because we scaled the features, we can fairly compare these sizes.
• Accuracy is 81.5%. Remember, 33% of customers left, so a lazy model that always says "stays" would score 67%. Ours is clearly better.
• The confusion matrix reads: 113 stayers correctly left alone, 21 false alarms, 16 leavers missed and 50 leavers caught. The model found 50 of the 66 customers who left.
• predict_proba() gives the probability, not just yes/no. This new customer has an 87% chance of leaving, so the retention team should call them first.

4.10Logistic regression in the real world

Banks and hospitals especially like logistic regression because it's easy to explain. When a loan is refused, regulators may require the bank to explain why, and "the coefficient for existing debt is strongly positive" is a clear answer.

SummaryKey takeaways

Check your understanding

Your machine learning roadmap

1. 01
   Introduction to Machine LearningWhat ML is and how it works
2. 02
   Preparing Data for Machine LearningFeatures, encoding, scaling, train/test split
3. 03
   Linear RegressionPredicting numbers
4. 04
   Logistic RegressionYou are here
5. 05
   Evaluating ModelsAccuracy, precision, recall, overfitting, cross-validation
6. 06
   K-Nearest NeighboursLearning from similar examples
7. 07
   Decision TreesFlowcharts that learn
8. 08
   Random Forest and Gradient BoostingMany models working together
9. 09
   Support Vector MachinesFinding the best boundary
10. 10
    Naive BayesProbability-based classification
11. 11
    Clustering with K-MeansFinding groups without labels
12. 12
    Dimensionality Reduction with PCASimplifying big datasets
13. 13
    Capstone ProjectBuild and present a full ML project