Support Vector Machines (SVM)

9.1What is a Support Vector Machine?

Imagine two villages on either side of a field, and the council wants to build a road between them. They could build a narrow lane squeezed right up against one village's houses. Or they could build the widest possible road, running right down the middle, with as much space as possible on both sides. The second option is safer: there's plenty of room for error.

An SVM does exactly that with data. For a classification problem, it looks for the boundary that separates the classes with the biggest gap between them.

SVMs are mainly used for classification, though there's also a version for regression (lesson 9.7). They work especially well when there are lots of features, such as text or images.

9.2Which line is best?

When two groups are cleanly separated, there are endless lines that could divide them. Logistic regression (Module 4) picks one based on probabilities. An SVM asks a different question: which line gives the most breathing room?

The idea is simple: if the boundary is far from all the training points, then a new point that's slightly different from what we've seen is still likely to land on the correct side. A wide gap means better generalisation, the thing we care about most (Module 5).

9.3The margin and support vectors

Here's the surprising part: only the support vectors matter. You could delete every other point, retrain, and get exactly the same boundary. They "support" the margin like tent poles hold up a tent, which is where the algorithm gets its name.

9.4The soft margin and the C setting

Real data is rarely perfectly separable. There's usually some overlap: a few customers who "should" have stayed but left anyway. A strict SVM that demands every point is on the correct side would fail, or draw a terrible boundary to fit a few odd points.

The solution is a soft margin: the SVM is allowed to let some points sit inside the margin, or even on the wrong side, but it pays a penalty for each one. The setting C controls how big that penalty is.

C is a hyperparameter, just like k in KNN or max_depth in trees. As always, choose it with cross-validation. Good values to try are 0.1, 1, 10 and 100.

9.5The kernel trick

So far, SVMs draw straight lines. But what if the data looks like this: one group in the middle, the other on both sides?

That's the key idea: data that can't be separated by a straight line might become separable if you look at it in more dimensions. A straight line in the new, bigger space becomes a curve when you bring it back to the original space.

Creating all those extra features could be slow. The kernel trick is a mathematical shortcut that lets the SVM work as if it had added the extra dimensions, without ever actually creating them. You don't need to know the maths; you just choose a kernel.

9.6Choosing a kernel

Gamma: how curvy can the RBF boundary get?

The RBF kernel has a second setting, gamma. It controls how far each training point's influence reaches.

• Small gamma: each point influences a wide area, so the boundary is smooth and simple.
• Large gamma: each point only influences a tiny area around itself, so the boundary can wrap tightly around individual points.

C and gamma work together, so it's normal to tune them at the same time with GridSearchCV, just as you'll see in lesson 9.8.

9.7SVMs for regression

SVMs can predict numbers too, using Support Vector Regression (SVR). It flips the idea around. Instead of the widest gap between groups, it fits a "tube" around a line or curve and tries to fit as many points as possible inside the tube.

• Points inside the tube count as "close enough": they cost nothing.
• Only points outside the tube count as errors, and they become the support vectors.
• The width of the tube is set with a setting called epsilon.

In scikit-learn, use SVR in place of SVC. It supports the same kernels. In practice, for most number-prediction problems on table-shaped data, the gradient boosting models from Module 8 are usually a stronger choice.

9.8Reading handwriting with an SVM in Python

Before deep learning took over, SVMs were famous for recognising handwritten digits. scikit-learn includes a dataset of 1,797 small images of handwritten digits. Each image is an 8 × 8 grid of pixels, so each digit has 64 features: one for how dark each pixel is.

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import MinMaxScaler
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score

# 1. 1,797 handwritten digits, each an 8x8 image = 64 pixel features
X, y = load_digits(return_X_y=True)
print("Images:", X.shape[0], "| Features per image:", X.shape[1])
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

# 2. Compare a linear SVM with an RBF (curved) SVM
#    Pixels run from 0 to 16, so min-max scaling puts them all on 0 to 1
for kernel in ["linear", "rbf"]:
    svm = Pipeline([("scale", MinMaxScaler()),
                    ("svm", SVC(kernel=kernel))])
    svm.fit(X_train, y_train)
    print(f"{kernel:6} kernel accuracy: {svm.score(X_test, y_test):.3f}")

# 3. Tune C and gamma for the RBF kernel with cross-validation
grid = GridSearchCV(
    Pipeline([("scale", MinMaxScaler()), ("svm", SVC(kernel="rbf"))]),
    param_grid={"svm__C": [0.1, 1, 10, 100],
                "svm__gamma": ["scale", 0.01, 0.1, 1]},
    cv=5)
grid.fit(X_train, y_train)
print("Best settings:", grid.best_params_)
print("Cross-validation accuracy:", round(grid.best_score_, 3))

# 4. Final test
pred = grid.predict(X_test)
print("Test accuracy:", round(accuracy_score(y_test, pred), 3))
print("Mistakes:", (pred != y_test).sum(), "out of", len(y_test))
print("Support vectors used:", grid.best_estimator_["svm"].n_support_.sum())

Images: 1797 | Features per image: 64
linear kernel accuracy: 0.984
rbf    kernel accuracy: 0.991
Best settings: {'svm__C': 10, 'svm__gamma': 'scale'}
Cross-validation accuracy: 0.989
Test accuracy: 0.991
Mistakes: 4 out of 450
Support vectors used: 561

What the output tells us

• Even a straight-line SVM does brilliantly (98.4%), because with 64 features there's lots of room to separate the digits.
• The RBF kernel does even better (99.1%), because it can draw curved boundaries around the different ways people write each digit.
• GridSearchCV tried 16 combinations of C and gamma and picked C = 10 with gamma = "scale" (scikit-learn's sensible automatic value).
• Only 4 mistakes in 450 test images. That's a 99.1% test accuracy, very close to the 98.9% cross-validation score, so it isn't a lucky split.
• 561 support vectors out of 1,347 training images. These are the "tricky" digits, like a 1 that looks a bit like a 7, which define the boundaries.

9.9Strengths, weaknesses and real-world uses

SummaryKey takeaways

Check your understanding

Your machine learning roadmap

1. 01
   Introduction to Machine LearningWhat ML is and how it works
2. 02
   Preparing Data for Machine LearningFeatures, encoding, scaling, train/test split
3. 03
   Linear RegressionPredicting numbers
4. 04
   Logistic RegressionPredicting yes or no
5. 05
   Evaluating ModelsAccuracy, precision, recall, overfitting, cross-validation
6. 06
   K-Nearest NeighboursLearning from similar examples
7. 07
   Decision TreesFlowcharts that learn
8. 08
   Random Forest and Gradient BoostingMany models working together
9. 09
   Support Vector MachinesYou are here
10. 10
    Naive BayesProbability-based classification
11. 11
    Clustering with K-MeansFinding groups without labels
12. 12
    Dimensionality Reduction with PCASimplifying big datasets
13. 13
    Capstone ProjectBuild and present a full ML project