Principal Component Analysis (PCA)

12.1Why reduce the number of features?

Each feature in a dataset is called a dimension. A table with 3 columns is 3-dimensional; the handwritten digits from Module 9 are 64-dimensional (one per pixel). Some real datasets have many thousands.

Lots of dimensions cause four problems:

The good news is that many features are redundant. A house's size in square metres and its number of rooms rise together. A person's height and arm span are almost the same measurement. In an image, neighbouring pixels are usually similar. If features repeat each other, we can combine them without losing much.

12.2The big idea behind PCA

PCA looks for the direction in which the data is most spread out. That direction becomes the first new feature, called the first principal component (PC1). Then it finds the next most spread-out direction, at right angles to the first (PC2), and so on.

Why does "most spread out" matter? Because variation is information. If every house in a dataset had exactly 3 bedrooms, the "bedrooms" column would tell a model nothing. The directions where the data varies most are the ones that separate one example from another.

Each principal component is a mix of the original features. For example, PC1 above might be roughly "0.8 × height + 0.6 × arm span": a combined "body size" feature.

12.3Finding the first principal component

Let's find PC1 by hand. Rotate the line below. Each point is "projected" (squashed) onto the line, like a shadow. PCA wants the line where those shadows are as spread out as possible, so they keep as much of the original variation as they can.

When the line runs along the long diagonal of the cloud, the shadows spread out the most, keeping about 93% of the variation. That's PC1. Rotate it 90 degrees and you get PC2, which keeps the remaining 7%.

12.4How many components should you keep?

PCA creates as many components as there were original features, but ordered from most to least useful. The first few usually carry most of the information. The share each one keeps is called its explained variance.

In scikit-learn, you can even ask for a share directly: PCA(n_components=0.95) keeps as many components as needed for 95% of the variation.

12.5Using PCA, step by step

As always, the easiest and safest way is to put the scaler, PCA and the model together in a Pipeline, which handles steps 1, 2 and 4 automatically, even during cross-validation.

12.6Seeing data with many features

One of the most popular uses of PCA is to draw high-dimensional data. We can't picture 64 dimensions, but we can squeeze the digits down to 2 components and plot them.

This kind of chart is a great first step with any new dataset. It shows whether the classes naturally separate, which groups are likely to get confused (here, digits such as 1, 5, 8 and 9, which overlap in the middle), and whether there are any strange outliers.

12.7PCA as compression: what gets lost?

PCA can also go backwards: take the few components and rebuild an approximate version of the original data. The rebuilt version shows exactly what was kept, and what was thrown away.

What's thrown away first is the fine detail and noise: tiny variations that are different in every image. What's kept is the main shape. That's why PCA can sometimes even improve a model: it removes noise that the model might otherwise overfit.

12.8PCA in Python

Let's return to the digits and the SVM from Module 9. How much information do the components keep, and what happens to accuracy when we shrink 64 features down to 30, 10 or even 2?

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import MinMaxScaler
from sklearn.svm import SVC

# 1. The handwritten digits from Module 9: 64 pixel features each
X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

# 2. How much information does each component keep?
pca = PCA().fit(MinMaxScaler().fit_transform(X_train))
cumulative = np.cumsum(pca.explained_variance_ratio_)
for n in [1, 2, 5, 10, 20, 30]:
    words = "components keep" if n > 1 else "component keeps"
    print(f"{n:2} {words} {cumulative[n-1]:.0%} of the variation")
print("Components needed for 95%:", np.argmax(cumulative >= 0.95) + 1)

# 3. Does shrinking 64 features hurt accuracy?
print()
for n in [None, 30, 10, 2]:
    steps = [("scale", MinMaxScaler())]
    if n:
        steps.append(("pca", PCA(n_components=n, random_state=42)))
    steps.append(("svm", SVC(C=10)))
    model = Pipeline(steps)
    model.fit(X_train, y_train)
    label = f"{n} components" if n else "All 64 features"
    print(f"{label:16} accuracy={model.score(X_test, y_test):.3f}")

 1 component keeps 15% of the variation
 2 components keep 28% of the variation
 5 components keep 54% of the variation
10 components keep 73% of the variation
20 components keep 89% of the variation
30 components keep 95% of the variation
Components needed for 95%: 30

All 64 features  accuracy=0.991
30 components    accuracy=0.991
10 components    accuracy=0.982
2 components     accuracy=0.653

What the output tells us

• The information is concentrated. The first component alone keeps 15%. Ten keep 73%, and 30 keep 95%, so more than half of the 64 original features are mostly redundant.
• 30 components give exactly the same accuracy (99.1%) as all 64 features, with less than half the data. On big datasets, that means much faster training.
• 10 components lose very little (98.2%), with just a sixth of the features.
• 2 components drop to 65.3%. Good enough for a chart (Figure 3), but far too little information for a reliable model. It's still much better than guessing, which would be 10%.

12.9Strengths, weaknesses and real-world uses

SummaryKey takeaways

Check your understanding

Your machine learning roadmap

1. 01
   Introduction to Machine LearningWhat ML is and how it works
2. 02
   Preparing Data for Machine LearningFeatures, encoding, scaling, train/test split
3. 03
   Linear RegressionPredicting numbers
4. 04
   Logistic RegressionPredicting yes or no
5. 05
   Evaluating ModelsAccuracy, precision, recall, overfitting, cross-validation
6. 06
   K-Nearest NeighboursLearning from similar examples
7. 07
   Decision TreesFlowcharts that learn
8. 08
   Random Forest and Gradient BoostingMany models working together
9. 09
   Support Vector MachinesFinding the best boundary
10. 10
    Naive BayesProbability-based classification
11. 11
    Clustering with K-MeansFinding groups without labels
12. 12
    Dimensionality Reduction with PCAYou are here
13. 13
    Capstone ProjectBuild and present a full ML project