Activation Functions

sgn · sigmoid · tanh · ReLU · linear — the nonlinear soul of neural networks

Machine Learning · Activation Functions

Mathematical Formulas

An activation function defines how a neuron maps its input to its output, injecting nonlinearity into the network so that it can learn and model complex patterns. Choosing a suitable activation function is critical to model performance. Below are some of the most classic activation functions together with their mathematical expressions.

$$a = f(z) = \sigma(z) = \frac{1}{1+e^{-z}}$$
$$z \text{ is the input and } a \text{ is the output}$$

Parameter Notes

z (input) Usually the weighted sum of the previous layer's outputs (the net input); it is the independent variable of the activation function.
a (output) The output of the activation function, which is passed to the next layer of neurons or used as the final prediction.
f (activation function) The function that determines the mapping; it can be sgn, sigmoid, tanh, ReLU, linear, and so on.
Nonlinearity Activation functions introduce nonlinearity, letting neural networks approximate arbitrarily complex functions and greatly improving their expressive power.

Interactive Simulation

Pick an activation function and drag the slider to change the input z, then watch the output and your position on the curve update in real time.

Parameter Control

Output a = f(z)
0.50
sigmoid function

Activation Function Curve

Standard Overview

An activation function is a crucial component of an artificial neural network: it defines the rule that maps a neuron's input to its output. Formally, a neuron first computes the weighted sum of its inputs z = Σwᵢxᵢ + b, then applies a nonlinear transformation f to produce the final output a = f(z). Without an activation function (or with a linear one), no matter how many layers the network has, the whole model still reduces to a linear model and cannot solve complex nonlinear problems.

Common activation functions include sgn (the sign function), sigmoid, tanh (hyperbolic tangent), ReLU (rectified linear unit) and the linear function. They differ in output range, gradient behavior and saturation, and each suits different network architectures and tasks. For example, sigmoid squashes the output into the interval (0, 1) and is often used in the output layer of binary classification models; ReLU keeps its gradient constant on the positive side, which mitigates the vanishing-gradient problem in deep networks and has made it one of the default choices in modern deep learning.

Plain-Language Introduction

If a neuron is a decision maker, then the activation function is its personality. Faced with the same input z, different activation functions react in different ways:

sgn is a strict yes/no judge — it only outputs -1 or 1 (or 0), like a light switch with no in-between state.
sigmoid is a probabilist — it squeezes the result into the range 0 to 1 and tells you how likely something is.
tanh works like sigmoid, but stretches the range to -1 to 1 — essentially sigmoid scaled up and centered at 0.
ReLU is a pragmatic efficiency expert — it passes positive inputs through unchanged and outputs 0 when the input is negative; it is cheap to compute and performs extremely well in deep networks.
linear is a straight shooter — it outputs exactly what it receives, adds no nonlinearity, and fits the output layer of regression problems.

It is precisely these different personalities that let neural networks learn patterns ranging from the very simple to the extremely complex.

What Problem Does It Solve

The core problem activation functions solve is giving neural networks the ability to model nonlinearity. Before activation functions were introduced, early neural networks (such as the perceptron) could only handle linearly separable problems. In 1969, Minsky and Papert pointed out in their book Perceptrons the limitations of single-layer perceptrons — most notably their inability to solve nonlinear problems such as XOR. That finding triggered the first winter of neural network research.

Activation functions allow multilayer networks to approximate arbitrarily complex functions by stacking nonlinear transformations. In 1989, Cybenko and others proved the Universal Approximation Theorem: as long as a network has at least one hidden layer and uses a nonlinear activation function (such as sigmoid), it can approximate any continuous function to arbitrary accuracy. This theorem laid a solid mathematical foundation for the rise of deep learning.

Beyond that, different activation functions also address gradient problems during training. For example, ReLU has a constant gradient of 1 on the positive side, which alleviates vanishing gradients in deep networks and makes it possible to train networks with hundreds of layers.

Why It Matters

The activation function is the indispensable nonlinear engine of a neural network. Without them, no matter how deep the network is, it is essentially just a linear regression model, unable to learn complex data patterns. The choice of activation function directly affects a network's expressive power, training speed and convergence.

Throughout the history of deep learning, innovations in activation functions have repeatedly pushed the field forward: sigmoid and tanh opened the era of multilayer perceptrons; the arrival of ReLU made it possible to train deep convolutional networks (such as AlexNet, VGG and ResNet), directly igniting the deep learning revolution; later improved variants such as Leaky ReLU, ELU and GELU further improved training stability and performance.

One could say the activation function is the bridge that connects linear algebra to the nonlinear world — the key that turned neural networks from "linear classifiers" into "universal approximators".

Application Scenarios

Image Recognition Speech Recognition Natural Language Processing Recommender Systems Reinforcement Learning Generative Models

Different activation functions suit different scenarios:

sigmoid: commonly used in the output layer of binary classification tasks, where the output is interpreted as a probability.
tanh: often used in the hidden layers of recurrent neural networks (RNNs) and generative models, because its zero-centered output helps gradients flow.
ReLU: the first choice for modern convolutional neural networks (CNNs) and deep feedforward networks — computationally efficient and resistant to vanishing gradients.
linear: the output layer for regression problems, producing continuous values directly.
sgn: used in early perceptrons and certain binary neural networks, and also as a hard decision for binary classification.

Limitations and Drawbacks

sgn: its gradient is zero almost everywhere, so it cannot be used with gradient-based learning; the output takes only discrete values, which loses a great deal of information.

sigmoid: its output is not zero-centered, which makes the optimization path zigzag; in the saturation regions (very large or very small inputs) the gradient approaches zero, easily causing vanishing gradients; the exponential operation is also relatively expensive.

tanh: although zero-centered, it suffers from the same vanishing-gradient problem in the saturation regions, and the exponential operation is costly.

ReLU: when the input is negative the gradient is zero, which can create "dead neurons" (neurons that can never be activated again); the output is not zero-centered; and there is no gradient on the negative side, which limits learning capacity.

linear: it introduces no nonlinearity, so stacking multiple layers is still equivalent to a single linear transformation, and the network cannot learn complex patterns.

In practice, one usually picks an activation function based on the task, network depth and computational budget, and may combine techniques such as batch normalization and residual connections to alleviate gradient problems.