K-Nearest Neighbours (KNN)

6.1What is K-Nearest Neighbours?

Imagine you move to a new area and want to guess which football team your new neighbour supports. You don't know them yet, but you know the five houses closest to theirs: four support Newcastle and one supports Sunderland. A sensible guess? Newcastle.

That's KNN. To predict something about a new data point, it:

1. finds the k most similar data points it already knows (its "nearest neighbours"), then
2. lets them vote on the answer.

In this module we'll help a UK streaming service. It offers a Basic and a Premium plan, and wants to predict which plan a new user will choose, based on their age and how many hours they watch per week.

6.2How KNN works, step by step

KNN can also give a probability, just like logistic regression: here, 3 out of 5 votes means a 60% chance of Premium.

6.3Measuring distance

"Nearest" means the smallest distance. But how do we measure distance between two data points? There are two common ways.

With more than two features, the idea is exactly the same. Euclidean distance just adds more squared differences under the square root: one for every feature. The computer handles this easily, even with hundreds of features.

6.4Choosing k

The number of neighbours, k, is a hyperparameter (you met these in Module 5): you choose it before making predictions. It has a big effect on the results. Try it yourself.

Small k vs large k

Try clicking near a blue point that sits among amber ones, then switch between k = 1 and k = 15. With k = 1, the prediction follows that single odd point. With k = 15, the crowd outvotes it. You can see the same effect across the whole chart:

6.5Why KNN needs scaled features

In Module 2 you learned that some algorithms need scaled features. KNN is the most important example, because everything depends on distance. Look at three customers:

Who is more similar to A? Most people would say C: almost the same age. Now look at what the raw distances say:

Without scaling, a £1,000 difference in income completely drowns out a 35-year age gap, simply because income numbers are bigger. After scaling (age range 18 to 70, income range £20k to £80k), both features get a fair say, and KNN finds the right neighbour.

6.6KNN for regression

KNN can predict numbers too. Instead of a vote, it takes the average of the neighbours' values.

Back to the Newcastle estate agent from Module 3. A new 95 m², 3-bedroom house comes on the market. KNN with k = 3 finds the three most similar houses already sold:

This is exactly how estate agents value a house in real life: they look at "comparables", similar houses that sold nearby. In scikit-learn, use KNeighborsRegressor instead of KNeighborsClassifier.

Weighted KNN: closer neighbours count more

In normal KNN, every neighbour gets an equal vote, even if one is very close and another is much further away. With distance weighting, closer neighbours get a bigger say. In scikit-learn, set weights="distance". It's worth trying both options when you tune your model.

6.7Strengths and weaknesses

The curse of dimensionality

With lots of features (say, 500), something strange happens: all points start to look roughly the same distance from each other. If everyone is "far away", the idea of a "nearest" neighbour stops meaning much. This is called the curse of dimensionality.

6.8KNN in Python

We'll use another real dataset built into scikit-learn: 178 Italian wines, each with 13 chemical measurements (such as alcohol and colour intensity), from three different grape types. Can KNN tell which grape a wine was made from?

We'll also use GridSearchCV. It automatically tries every combination of settings you give it, scores each one with cross-validation, and keeps the best. This is the proper way to choose k.

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

# 1. Load a real dataset: 178 wines, 13 chemical measurements, 3 grape types
X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

# 2. KNN WITHOUT scaling
raw = KNeighborsClassifier(n_neighbors=5)
raw.fit(X_train, y_train)
print("Accuracy without scaling:", round(raw.score(X_test, y_test), 3))

# 3. KNN WITH scaling (inside a Pipeline)
knn = Pipeline([
    ("scale", StandardScaler()),
    ("knn", KNeighborsClassifier(n_neighbors=5)),
])
knn.fit(X_train, y_train)
print("Accuracy with scaling:   ", round(knn.score(X_test, y_test), 3))

# 4. Find the best k with 5-fold cross-validation
grid = GridSearchCV(
    knn,
    param_grid={"knn__n_neighbors": [1, 3, 5, 7, 9, 11, 15, 21],
                "knn__weights": ["uniform", "distance"]},
    cv=5, scoring="accuracy")
grid.fit(X_train, y_train)
print("Best settings:", grid.best_params_)
print("Best cross-validation accuracy:", round(grid.best_score_, 3))
print("Final test accuracy:", round(grid.score(X_test, y_test), 3))

# 5. Predict a new wine and look at its neighbours
new_wine = X_test[:1]
print("Predicted grape type:", grid.predict(new_wine)[0])
print("Vote shares:", grid.predict_proba(new_wine).round(2)[0])

Accuracy without scaling: 0.778
Accuracy with scaling:    0.933
Best settings: {'knn__n_neighbors': 15, 'knn__weights': 'uniform'}
Best cross-validation accuracy: 0.977
Final test accuracy: 1.0
Predicted grape type: 0
Vote shares: [0.93 0.07 0.  ]

What the output tells us

• Scaling is essential. Accuracy jumps from 77.8% to 93.3% just by scaling. Some wine measurements (like "proline") are in the hundreds, while others are below 1, so without scaling they completely dominate the distances.
• GridSearchCV tried 16 combinations (8 values of k × 2 weighting options) and found k = 15 with equal votes worked best, with 97.7% average cross-validation accuracy.
• The final test accuracy is 100%. That's excellent, but remember the test set only has 45 wines, so one or two mistakes would change it a lot. The cross-validation score (97.7%) is the more reliable estimate.
• Vote shares show that 93% of the 15 neighbours (14 of them) were grape type 0, so the model is very confident about this wine.

6.9KNN in the real world

KNN is rarely the most accurate model on large, complex datasets, but it's an excellent baseline, and its "similar examples" approach is easy for anyone in a business to understand and trust.

SummaryKey takeaways

Check your understanding

Your machine learning roadmap

1. 01
   Introduction to Machine LearningWhat ML is and how it works
2. 02
   Preparing Data for Machine LearningFeatures, encoding, scaling, train/test split
3. 03
   Linear RegressionPredicting numbers
4. 04
   Logistic RegressionPredicting yes or no
5. 05
   Evaluating ModelsAccuracy, precision, recall, overfitting, cross-validation
6. 06
   K-Nearest NeighboursYou are here
7. 07
   Decision TreesFlowcharts that learn
8. 08
   Random Forest and Gradient BoostingMany models working together
9. 09
   Support Vector MachinesFinding the best boundary
10. 10
    Naive BayesProbability-based classification
11. 11
    Clustering with K-MeansFinding groups without labels
12. 12
    Dimensionality Reduction with PCASimplifying big datasets
13. 13
    Capstone ProjectBuild and present a full ML project