Capstone Project: Defect Detection

12.1The brief

You've joined Tyneside Precision Components, a (fictional) factory that makes metal washers for the car industry. A camera photographs every washer on the production line. On your first day, this email arrives:

12.2Your project plan

12.3Step 1: Define the problem

12.4Step 2: Get and explore the data

Run this script to create the dataset. It draws 4,000 realistic 48 × 48 greyscale washer photos, with lighting changes and camera noise, and saves them in one file. Everyone gets exactly the same images, so you can compare results with the reference solution.

# Creates the capstone dataset: 4,000 greyscale images (48x48) of metal washers
import numpy as np

def washer(rng, kind, size=48):
    yy, xx = np.mgrid[0:size, 0:size]
    cx, cy = size / 2 + rng.normal(0, 1.5, 2)
    r = np.hypot(xx - cx, yy - cy)
    outer, inner = rng.uniform(18, 21), rng.uniform(7, 9)
    img = np.full((size, size), 0.12)                                  # dark background
    metal = (r < outer) & (r > inner)
    img[metal] = rng.uniform(0.55, 0.7) + 0.08 * np.cos(np.arctan2(yy - cy, xx - cx))[metal]
    if kind == "scratch":                                              # thin dark line
        a = rng.uniform(0, np.pi)
        off = rng.uniform(-10, 10)
        dist = np.abs((xx - cx) * np.sin(a) - (yy - cy) * np.cos(a) - off)
        length_ok = np.abs((xx - cx) * np.cos(a) + (yy - cy) * np.sin(a)) < rng.uniform(6, 14)
        img[metal & (dist < rng.uniform(0.6, 1.2)) & length_ok] -= rng.uniform(0.2, 0.35)
    if kind == "dent":                                                 # small dark blob
        ang, rad = rng.uniform(0, 2 * np.pi), rng.uniform(inner + 3, outer - 3)
        bx, by = cx + rad * np.cos(ang), cy + rad * np.sin(ang)
        blob = np.exp(-((xx - bx) ** 2 + (yy - by) ** 2) / (2 * rng.uniform(1.2, 2.2) ** 2))
        img -= metal * blob * rng.uniform(0.2, 0.35)
    img += rng.normal(0, 0.06, img.shape)                              # camera noise
    return np.clip(img, 0, 1)

rng = np.random.default_rng(2026)
kinds = rng.choice(["ok", "scratch", "dent"], 4000, p=[0.7, 0.15, 0.15])
images = np.array([washer(rng, k) for k in kinds]).astype("float32")
np.savez_compressed("washers.npz", images=images, labels=kinds)
print(images.shape, {k: int((kinds == k).sum()) for k in ["ok", "scratch", "dent"]})

(4000, 48, 48) {'ok': 2803, 'scratch': 604, 'dent': 593}

Your tasks

1. Load the file with np.load("washers.npz") and check the shape of the images and labels.
2. Count each class. Is the data balanced?
3. Plot 10 images of each class with Matplotlib (plt.imshow(img, cmap="gray")). What makes scratches and dents hard to see?
4. Check the pixel range: are the values already between 0 and 1?

12.5Step 3: Build a baseline

Before building anything clever, train a simple dense network: flatten the 48 × 48 image into 2,304 numbers and pass them through one hidden layer. Split the data 80/20 into training and test sets with stratify=y, and keep the test set locked away.

The reference solution's dense baseline scored 70.0%, exactly the share of good washers. In other words, it learned nothing except "always say ok". Despite having 295,427 weights, it couldn't find tiny defects that move around the image. This is a perfect example of why we need a CNN, and of why accuracy on its own can hide a useless model.

12.6Step 4: Build a CNN

Now design a CNN in Keras. Here are the design decisions in the reference solution, and why each one was made:

The reference CNN reached 91.6% test accuracy with 69,347 weights: far better than the baseline, with less than a quarter of the weights. Training took 42 epochs before early stopping kicked in.

12.7Step 5: Evaluate properly

Accuracy is only the start. The confusion matrix shows which mistakes the model makes:

Error analysis: your task

Find the test images where the true class was "scratch" but the model said "ok", and plot 20 of them. What do they have in common? This is one of the most valuable habits in machine learning: look at your mistakes.

12.8Step 6: Choose when to reject

The model gives each washer a probability of being faulty: P(defect) = 1 − P(ok). The factory rejects any washer above a threshold. Using Priya's costs, find the threshold that costs the least. These are the real predictions for the 800 test washers.

The model cuts the cost of faults by over 80% compared with no inspection. Tuning the threshold saves a little more. But notice that even the best threshold still misses around 35 defects, mostly faint scratches, because the model gives them a very low P(defect). A lower threshold can't catch them without rejecting lots of good washers. To fix that, the model itself must get better at scratches, which is a key message for your presentation.

12.9The reference solution

Here's the complete solution for steps 3 to 6. Your code doesn't have to match; there are many good designs.

import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, classification_report
import keras
from keras import layers

keras.utils.set_random_seed(42)
CLASSES = ["ok", "scratch", "dent"]

# ---------- 1. Load and split ----------
data = np.load("washers.npz")
X = data["images"][..., None]                        # add the channel dimension
y = np.array([CLASSES.index(k) for k in data["labels"]])
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)
print("Train:", X_train.shape, "| Test:", X_test.shape)

# ---------- 2. Baseline: a dense network ----------
dense = keras.Sequential([
    keras.Input(shape=(48, 48, 1)),
    layers.Flatten(),
    layers.Dense(128, activation="relu"),
    layers.Dense(3, activation="softmax"),
])

# ---------- 3. A CNN with augmentation ----------
cnn = keras.Sequential([
    keras.Input(shape=(48, 48, 1)),
    layers.RandomFlip("horizontal_and_vertical"),    # a washer can face any way
    layers.Conv2D(32, 3, padding="same", activation="relu"),
    layers.Conv2D(32, 3, padding="same", activation="relu"),
    layers.MaxPooling2D(2),
    layers.Conv2D(64, 3, padding="same", activation="relu"),
    layers.MaxPooling2D(2),
    layers.Conv2D(64, 3, padding="same", activation="relu"),
    layers.GlobalMaxPooling2D(),                     # "is there a defect anywhere?"
    layers.Dense(64, activation="relu"),
    layers.Dense(3, activation="softmax"),
])

for name, model in [("Dense baseline", dense), ("CNN", cnn)]:
    early_stop = keras.callbacks.EarlyStopping(monitor="val_loss", patience=8,
                                               restore_best_weights=True)
    model.compile(optimizer="adam", loss="sparse_categorical_crossentropy",
                  metrics=["accuracy"])
    hist = model.fit(X_train, y_train, epochs=60, batch_size=64, verbose=0,
                     validation_split=0.15, callbacks=[early_stop])
    acc = model.evaluate(X_test, y_test, verbose=0)[1]
    print(f"{name:15} epochs={len(hist.history['loss']):2}  "
          f"params={model.count_params():7,}  test accuracy={acc:.3f}")

# ---------- 4. Evaluate the CNN properly ----------
probs = cnn.predict(X_test, verbose=0)
pred = probs.argmax(axis=1)
print("\nConfusion matrix (rows = actual, columns = predicted: ok, scratch, dent)")
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, target_names=CLASSES, digits=3))

# ---------- 5. Business decision: reject if P(defect) is above a threshold ----------
p_defect = 1 - probs[:, 0]
is_defect = y_test != 0
MISSED_DEFECT, FALSE_REJECT = 120, 15              # cost in pounds
for t in [0.5, 0.39, 0.2]:
    rejected = p_defect >= t
    missed = (~rejected & is_defect).sum()
    wasted = (rejected & ~is_defect).sum()
    cost = missed * MISSED_DEFECT + wasted * FALSE_REJECT
    print(f"Threshold {t:<4}: missed defects={missed:2}  good parts rejected={wasted:3}  "
          f"cost=£{cost:,}")

np.save("test_probs.npy", np.c_[p_defect, is_defect])   # for your profit chart

Train: (3200, 48, 48, 1) | Test: (800, 48, 48, 1)
Dense baseline  epochs=13  params=295,427  test accuracy=0.700
CNN             epochs=42  params= 69,347  test accuracy=0.916
Confusion matrix (rows = actual, columns = predicted: ok, scratch, dent)
[[546  12   2]
 [ 37  84   0]
 [  2  14 103]]
              precision    recall  f1-score   support
          ok      0.933     0.975     0.954       560
     scratch      0.764     0.694     0.727       121
        dent      0.981     0.866     0.920       119
    accuracy                          0.916       800
   macro avg      0.893     0.845     0.867       800
weighted avg      0.915     0.916     0.914       800
Threshold 0.5 : missed defects=39  good parts rejected= 15  cost=£4,905
Threshold 0.39: missed defects=35  good parts rejected= 31  cost=£4,665
Threshold 0.2 : missed defects=33  good parts rejected= 71  cost=£5,025

Stretch goals

• Fix the scratch problem: try class_weight in model.fit() to make scratches count more, add more filters, or use larger images.
• Transfer learning: resize the images to 96 × 96, copy them into 3 channels, and try a frozen MobileNetV2 (Module 11). Does it beat your CNN?
• More augmentation: add small rotations, brightness changes and noise (Module 7). Which help?
• Explainability: research Grad-CAM, which creates a heatmap of where the CNN "looked". Does it look at the defect?
• Sensitivity: what if a missed defect costs £500? How does the best threshold change?

12.10Step 7: Present your findings

The operations director wants decisions, not code. Aim for about eight slides:

12.11Submission checklist and marking guide

• A notebook that runs from top to bottom without errors
• Exploration: class counts and example images for each class
• A baseline model and a CNN, compared on the same test set
• Training history charts (loss and validation loss)
• A confusion matrix and recall for each class
• An error analysis with example images of mistakes
• A recommended threshold backed by a cost calculation
• A short presentation (around 8 slides) or one-page summary for Priya

SummaryWhat you've achieved

Final check: think like a deep learning engineer

Your neural networks roadmap

1. 01
   Introduction to Neural NetworksWhat neural networks are and how they work
2. 02
   The Perceptron and the Artificial NeuronBuild a single neuron from scratch
3. 03
   Activation FunctionsSigmoid, tanh, ReLU and softmax
4. 04
   Loss Functions and Gradient DescentHow networks measure and reduce error
5. 05
   BackpropagationHow every weight learns its share of the blame
6. 06
   Building Networks with KerasYour first deep learning library
7. 07
   Training Better NetworksOverfitting, dropout, batch normalisation, early stopping
8. 08
   Convolutional Neural NetworksTeaching computers to see
9. 09
   Recurrent Neural Networks and LSTMsWorking with sequences
10. 10
    Transformers and AttentionThe architecture behind ChatGPT and Claude
11. 11
    Transfer Learning and Pretrained ModelsStanding on the shoulders of giants
12. 12
    Capstone ProjectYou are here