Naive Bayes

10.1What is Naive Bayes?

Think about how you decide if an email is spam. You notice clues: it says "WINNER", it's asking you to "click now", and it offers something "free". Each clue on its own isn't proof, but each one makes spam more likely. Put enough clues together and you're pretty sure.

Naive Bayes does exactly this, using probability. It starts with a basic belief (for example, "30% of emails are spam"), then updates that belief with each clue it finds.

It's named after Thomas Bayes, an English minister and mathematician from the 1700s whose theorem sits at its heart. The "naive" part is explained in lesson 10.4.

10.2A quick guide to probability

You only need two ideas for this module.

10.3Bayes' theorem

Here's what we know about a company's inbox:

• 30% of emails are spam.
• 60% of spam emails contain the word "free".
• Only 5% of normal emails contain the word "free".

A new email arrives containing "free". How likely is it to be spam? The easiest way to see the answer is to imagine 1,000 emails:

That's Bayes' theorem in action. Written as a formula, it looks like this:

Why the prior matters so much

Here's a famous example that surprises almost everyone. A rare illness affects 1 in 100 people. A test for it catches 90% of people who are ill, but also wrongly flags 9% of healthy people. You test positive. What's the chance you're actually ill?

Most people guess 90%. The real answer is about 9%, because the illness is so rare (a low prior) that false alarms from the huge healthy group outnumber the true cases. This is why doctors usually repeat a test before making a diagnosis, and why Naive Bayes always starts from the prior.

10.4Why "naive"? Combining many clues

Real emails contain many words, not just one. To combine them, Naive Bayes makes a big simplifying assumption: every clue is independent of the others. It assumes that seeing "free" tells you nothing about whether you'll also see "winner", once you know whether the email is spam.

That's clearly not true in real life ("click" and "here" often appear together). That's why it's called naive. But the assumption makes the maths very simple: you can just multiply the evidence from each word together.

Do the same for "not spam", compare the two scores, and pick the bigger one. That's the whole algorithm.

Try it yourself with a spam filter that has learned how often eight words appear in spam and normal emails.

Notice how a word like "meeting", which is much more common in normal email, can pull a spammy-looking message back down. Each word multiplies the odds by how much more common it is in spam than in normal email.

10.5The zero-frequency problem

Imagine the word "voucher" never appeared in any normal email in the training data. Then P("voucher" | normal) = 0. Because Naive Bayes multiplies everything, a single zero makes the whole score zero. One word would make an email "impossible" to be normal, however many other normal words it contains.

The fix is Laplace smoothing (also called add-one smoothing). We pretend we've seen every word at least once in every class, by adding 1 to every count.

In scikit-learn, this is the alpha setting: alpha=1.0 is classic Laplace smoothing (the default). Smaller values smooth less. It's another hyperparameter you can tune with cross-validation.

10.6Three types of Naive Bayes

The idea is always the same; what changes is how the likelihood of each feature is worked out.

How Gaussian Naive Bayes handles numbers

You can't count how often "a radius of 15.53" appears. Instead, Gaussian Naive Bayes assumes each feature follows a bell curve (a normal distribution) within each class. It learns the average and spread of each curve, then checks how high each curve is at the new value.

10.7Naive Bayes in Python

This example has two parts. First, a tiny spam filter trained on 14 text messages. Second, Gaussian Naive Bayes on the real breast cancer dataset from Module 5, compared with logistic regression.

For text, we use CountVectorizer, which turns each message into word counts: one column for every word it has seen. This is called a bag of words, because it keeps the words but throws away their order, like tipping them into a bag.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB, GaussianNB
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# ---------- Part 1: a text spam filter ----------
messages = [
    "WINNER! Claim your free prize now, click here",
    "Free entry to win a luxury holiday, reply WIN",
    "URGENT: your account is locked, click to verify",
    "You have won a £500 voucher, claim it free today",
    "Cheap loans approved instantly, click now",
    "Congratulations, you are our lucky winner, call now",
    "Are we still on for lunch tomorrow?",
    "Please find the invoice attached for last month",
    "Can you send me the slides before the meeting?",
    "Running 10 minutes late, see you at the station",
    "Happy birthday! Hope you have a lovely day",
    "The meeting has moved to 3pm in room 2",
    "Thanks for your help with the report today",
    "Don't forget to bring the charger tonight",
]
labels = ["spam"] * 6 + ["ham"] * 8     # "ham" = a normal message

spam_filter = Pipeline([
    ("count", CountVectorizer()),          # turn text into word counts
    ("nb", MultinomialNB(alpha=1.0)),      # alpha=1 is Laplace smoothing
])
spam_filter.fit(messages, labels)

new = ["Click now to claim your free holiday",
       "Can we move lunch to 1pm?",
       "Urgent: invoice for the meeting room"]
for text, probs in zip(new, spam_filter.predict_proba(new)):
    spam_prob = probs[list(spam_filter.classes_).index("spam")]
    print(f"{spam_prob:6.1%} spam | {text}")

# ---------- Part 2: Gaussian NB on number features ----------
print()
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

for name, model in [("Gaussian NB", GaussianNB()),
                    ("Logistic regression", Pipeline([
                        ("scale", StandardScaler()),
                        ("lr", LogisticRegression(max_iter=1000))]))]:
    model.fit(X_train, y_train)
    print(f"{name:20} accuracy={model.score(X_test, y_test):.3f}")

 99.9% spam | Click now to claim your free holiday
 13.1% spam | Can we move lunch to 1pm?
  0.8% spam | Urgent: invoice for the meeting room

Gaussian NB          accuracy=0.937
Logistic regression  accuracy=0.986

What the output tells us

• "Click now to claim your free holiday" is 99.9% spam. Every one of its key words appeared in the spam examples. Notice how confident it is: that's the overconfidence mentioned earlier.
• "Can we move lunch to 1pm?" is only 13.1% spam. "Lunch" appeared in a normal message.
• "Urgent: invoice for the meeting room" is just 0.8% spam. "Urgent" is a spam word, but "invoice", "meeting" and "room" all point strongly the other way, and three clues beat one.
• With only 14 training messages, this filter is a toy. Real spam filters learn from millions of emails, but the code is exactly the same.
• On the cancer data, Gaussian Naive Bayes reaches 93.7%, while logistic regression reaches 98.6%. Naive Bayes is a quick, decent baseline, but here its "independent features" assumption hurts, because many tumour measurements (radius, perimeter, area) are closely linked.

10.8Strengths, weaknesses and real-world uses

SummaryKey takeaways

Check your understanding

Your machine learning roadmap

1. 01
   Introduction to Machine LearningWhat ML is and how it works
2. 02
   Preparing Data for Machine LearningFeatures, encoding, scaling, train/test split
3. 03
   Linear RegressionPredicting numbers
4. 04
   Logistic RegressionPredicting yes or no
5. 05
   Evaluating ModelsAccuracy, precision, recall, overfitting, cross-validation
6. 06
   K-Nearest NeighboursLearning from similar examples
7. 07
   Decision TreesFlowcharts that learn
8. 08
   Random Forest and Gradient BoostingMany models working together
9. 09
   Support Vector MachinesFinding the best boundary
10. 10
    Naive BayesYou are here
11. 11
    Clustering with K-MeansFinding groups without labels
12. 12
    Dimensionality Reduction with PCASimplifying big datasets
13. 13
    Capstone ProjectBuild and present a full ML project