NN-101 · Module 3 Beginner about 30 minutes 9 Lessons Prereq: Modules 1 and 2

Activation Functions

The activation function is a small piece of maths inside every neuron, but it decides whether a network can learn anything interesting at all. In this module you'll see why networks need them, meet the ones used every day (sigmoid, tanh, ReLU and softmax), and learn which to choose.

Level: Beginner
Time: about 30 minutes
Needs: Modules 1 and 2

By the end of this module you will be able to

Explain why a network without activation functions can only draw straight lines
Describe sigmoid, tanh, ReLU and Leaky ReLU, and their output ranges
Explain what a gradient (slope) is and why it matters for learning
Explain the vanishing gradient problem and why ReLU helps
Use softmax to turn scores into probabilities
Choose the right activation for hidden and output layers
Write activation functions in NumPy and compare them in a real network

3.1Why do we need activation functions?

Remember the neuron: it multiplies inputs by weights, adds a bias, then applies an activation function. What if we skipped that last step?

Without it, each neuron just does "multiply and add". The trouble is that stacking layers of multiply-and-add is still just multiply-and-add. Ten layers collapse into the same thing as one layer, so the network can only ever draw a straight line, however deep it is.

An everyday way to see it

"Double it, then triple it" is the same as "multiply by 6". "Add 5, then add 3" is the same as "add 8". Chaining simple steps like these never creates anything new. To build curves, a network needs something that bends the numbers between layers. That's the activation function.

Figure 1. The same network (two hidden layers of 16 neurons) trained on ring-shaped data. Without activation functions (left), 32 neurons still behave like one straight line. With ReLU (right), the network bends its boundary into a circle. Real results from the Python code in lesson 3.8.

An activation function that bends the numbers is called non-linear. This non-linearity is what lets neural networks learn curves, circles, faces and language.

3.2Slopes, and what makes a good activation

In Module 2 you saw that the perceptron's step function is flat almost everywhere, which makes training hidden layers impossible. Networks learn by asking, for every weight: "if I nudge this weight a little, does the error go up or down, and by how much?" The answer depends on the slope (also called the gradient, or derivative) of each activation function.

A steep slope means a small change has a big effect: a strong learning signal.
A flat slope means a change does almost nothing: a weak signal, so learning is slow.

A good activation function is:

Quality	Why it matters
Non-linear	So layers can build curves (lesson 3.1)
Has a useful slope	So there's a learning signal for gradient descent (Module 4)
Doesn't flatten out too much	So the signal survives through many layers (lesson 3.5)
Fast to calculate	Big networks apply it billions of times

3.3The main activation functions

Figure 2. Four common activation functions (green) and their slopes (red dashed). Notice where each slope is high, where it's flat, and where it's zero.

Sigmoid

sigmoid(z) = 1 ÷ (1 + e^−z)

You know this one from logistic regression. It squashes any number into the range 0 to 1, which makes it perfect for a probability. But its slope is never more than 0.25, and it's almost flat for big positive or negative numbers. When a neuron's output is stuck near 0 or 1 like this, we say it has saturated, and it barely learns.

Tanh (hyperbolic tangent)

tanh(z) = (e^z − e^−z) ÷ (e^z + e^−z)

A stretched sigmoid that runs from −1 to 1, centred on 0. Its slope reaches 1 in the middle, so it learns faster than sigmoid, and outputs centred on zero make life easier for the next layer. But it still saturates at the edges.

ReLU (Rectified Linear Unit)

ReLU(z) = max(0, z)

The simplest of all: if the number is negative, output 0; otherwise, pass it straight through. That's it. It became the default choice for hidden layers after 2012 because:

it's extremely fast to calculate (just a comparison),
for positive inputs its slope is always 1, so it never saturates on that side,
networks using it train much faster than with sigmoid or tanh.

Its weakness: for negative inputs the slope is 0. If a neuron's total is negative for every example, it outputs 0 forever and stops learning. This is called a dying ReLU.

Leaky ReLU and friends

Leaky ReLU fixes dying ReLUs by letting a small amount through for negative inputs (for example, 0.01 × z), so the slope is never exactly zero. Modern networks also use smooth relatives of ReLU:

Function	Idea	Where you'll see it
Leaky ReLU	ReLU with a small slope for negatives	When many neurons are "dying"
ELU	Smoothly curves towards −1 for negatives	Some image networks
GELU	A smooth, slightly curved version of ReLU	Transformers, including GPT and BERT-style models
Swish / SiLU	z × sigmoid(z): smooth and slightly dips below zero	Many modern image and language models

3.4Explore the functions yourself

Pick a function and move the input. Watch the output, the slope, and how much learning signal would survive after passing back through 10 layers with that same slope.

Try it: activation function explorer

Input to the neuron (z):

Output

Slope here

Signal left after 10 layers

Output range

3.5The vanishing gradient problem

When a network learns, the error signal travels backwards from the output through every layer (you'll see exactly how in Module 5). At each layer, the signal is multiplied by the slope of that layer's activation function.

Sigmoid's slope is at most 0.25. Multiply by 0.25 again and again, and the signal shrinks fast:

Figure 3. Even in the best case, a sigmoid network loses three quarters of its learning signal at every layer. After 10 layers, less than one millionth is left, so the early layers barely learn at all.

This is the vanishing gradient problem, and it's a big reason why deep networks were so hard to train before around 2010. The early layers, which should learn the basic building blocks like edges in an image, received almost no learning signal.

How ReLU helped

For active neurons (positive inputs), ReLU's slope is exactly 1. Multiply by 1 ten times and the signal is unchanged. Switching from sigmoid to ReLU was one of the key changes that made deep learning work.

The opposite can happen too: if slopes or weights are bigger than 1, the signal can grow out of control. That's called exploding gradients. You'll learn how to prevent both in Module 7.

3.6Softmax: choosing between many classes

Sigmoid is great when the output is one yes/no probability. But what if a network must choose between several classes, like the 10 digits, or cat, dog and rabbit? Then we need probabilities that add up to 1. That's what softmax does.

The output layer gives each class a raw score (called a logit). Softmax then:

raises e to the power of each score, which makes every value positive and makes big scores stand out more,
divides each by the total, so they add up to 1.

Class	Raw score	e^score	Softmax probability
Cat	2.0	7.39	7.39 ÷ 11.21 = 65.9%
Dog	1.0	2.72	2.72 ÷ 11.21 = 24.2%
Rabbit	0.1	1.11	1.11 ÷ 11.21 = 9.9%
Total		11.21	100%

The network is 65.9% sure it's a cat. Notice that softmax exaggerates the gap: a score of 2 versus 1 becomes 66% versus 24%. That's where the name comes from: it's a "soft" version of simply picking the maximum.

3.7Which activation should I use?

The choice is different for the hidden layers and the output layer.

Hidden layers

Choice	When
ReLU	The default. Start here for almost every network
Leaky ReLU	If many neurons are dying (always outputting 0)
GELU	In Transformer-style models (Module 10)
Tanh	Inside some recurrent networks (Module 9)
Sigmoid	Rarely in hidden layers today, because of vanishing gradients

Output layer: match it to your task

Task	Example	Output activation	Output neurons
Regression	Predict a house price	None (linear)	1
Binary classification	Spam or not spam	Sigmoid	1
Multi-class classification	Which digit, 0 to 9?	Softmax	One per class
Multi-label classification	Which tags fit this photo: beach, dog, sunset?	Sigmoid on each	One per label

A simple rule to remember

ReLU inside, and let the task choose the output. A number needs no activation, a yes/no needs sigmoid, and "pick one of many" needs softmax.

3.8Activation functions in Python

We'll write the main functions in NumPy (each is just one line), check the softmax example, then train the same network four times on ring-shaped data, changing only the activation function.

activations.py

import numpy as np
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

# 1. The most common activation functions, in NumPy
def sigmoid(z):    return 1 / (1 + np.exp(-z))
def tanh(z):       return np.tanh(z)
def relu(z):       return np.maximum(0, z)
def leaky_relu(z): return np.where(z > 0, z, 0.01 * z)
def softmax(z):
    e = np.exp(z - z.max())            # subtract the max for numerical safety
    return e / e.sum()

z = np.array([-2.0, 0.0, 3.0])
for f in [sigmoid, tanh, relu, leaky_relu]:
    print(f"{f.__name__:10} {f(z).round(3)}")

# 2. Softmax turns scores into probabilities that add up to 1
scores = np.array([2.0, 1.0, 0.1])     # e.g. cat, dog, rabbit
probs = softmax(scores)
print("\nSoftmax:", probs.round(3), "| total =", probs.sum().round(3))

# 3. Does the activation choice matter? Ring-shaped data (Module 9 of ML)
X, y = make_circles(n_samples=1000, noise=0.1, factor=0.5, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

print()
for act in ["identity", "logistic", "tanh", "relu"]:
    net = MLPClassifier(hidden_layer_sizes=(16, 16), activation=act,
                        max_iter=2000, random_state=0)
    net.fit(X_train, y_train)
    print(f"{act:9} hidden layers: test accuracy = {net.score(X_test, y_test):.3f}"
          f"  ({net.n_iter_} epochs)")

Output

sigmoid    [0.119 0.5   0.953]
tanh       [-0.964  0.     0.995]
relu       [0. 0. 3.]
leaky_relu [-0.02  0.    3.  ]

Softmax: [0.659 0.242 0.099] | total = 1.0

identity  hidden layers: test accuracy = 0.364  (21 epochs)
logistic  hidden layers: test accuracy = 0.468  (24 epochs)
tanh      hidden layers: test accuracy = 1.000  (523 epochs)
relu      hidden layers: test accuracy = 1.000  (372 epochs)

What the output tells us

Each function behaves as expected: sigmoid keeps everything between 0 and 1, tanh between −1 and 1, ReLU zeroes the negative input, and Leaky ReLU lets a little through (−0.02).
Softmax gives 65.9%, 24.2% and 9.9%, adding up to exactly 1, matching our table.
"identity" means no activation. With 32 hidden neurons, it scores just 36.4%: no better than guessing, because the network can only draw a straight line through rings.
Sigmoid ("logistic") stalled at 46.8%. Its small slopes gave such a weak learning signal that progress stayed tiny, so training stopped after 24 epochs, thinking it had finished. This is the vanishing gradient problem in action.
Tanh and ReLU both reach 100%, and ReLU got there in fewer epochs (372 vs 523).

Setting activations in the tools you'll use

In scikit-learn, use MLPClassifier(activation="relu"). In Keras (Module 6), you'll write Dense(64, activation="relu") for a hidden layer and Dense(10, activation="softmax") for a 10-class output layer.

SummaryKey takeaways

Without activation functions, any number of layers collapses into one straight line. Activations add non-linearity.
Networks learn from slopes (gradients): flat regions give a weak learning signal.
Sigmoid: 0 to 1, great for probabilities, but slope ≤ 0.25 and it saturates.
Tanh: −1 to 1, centred on zero, stronger slope, but still saturates.
ReLU: max(0, z). Fast, slope 1 for positives, the default for hidden layers. Watch for dying ReLUs; Leaky ReLU helps.
Multiplying small slopes through many layers causes vanishing gradients.
Softmax turns scores into probabilities that add up to 1, for multi-class outputs.
Output layer: none for numbers, sigmoid for yes/no, softmax for one-of-many.

Check your understanding

Your neural networks roadmap

01
Introduction to Neural NetworksWhat neural networks are and how they work
02
The Perceptron and the Artificial NeuronBuild a single neuron from scratch
03
Activation FunctionsYou are here
04
Loss Functions and Gradient DescentHow networks measure and reduce error
05
BackpropagationHow every weight learns its share of the blame
06
Building Networks with KerasYour first deep learning library
07
Training Better NetworksOverfitting, dropout, batch normalisation, early stopping
08
Convolutional Neural NetworksTeaching computers to see
09
Recurrent Neural Networks and LSTMsWorking with sequences
10
Transformers and AttentionThe architecture behind ChatGPT and Claude
11
Transfer Learning and Pretrained ModelsStanding on the shoulders of giants
12
Capstone ProjectBuild and present a deep learning project

PreviousThe Perceptron and the Artificial Neuron

Next module

Loss Functions and Gradient Descent

Start Module 4

← Previous: Module 2 Next: Module 4 →