Activation Functions
sgn · sigmoid · tanh · ReLU · linear — the nonlinear soul of neural networks
Machine Learning · Activation FunctionsMathematical Formulas
An activation function defines how a neuron maps its input to its output, injecting nonlinearity into the network so that it can learn and model complex patterns. Choosing a suitable activation function is critical to model performance. Below are some of the most classic activation functions together with their mathematical expressions.
Parameter Notes
Interactive Simulation
Pick an activation function and drag the slider to change the input z, then watch the output and your position on the curve update in real time.
Parameter Control
Activation Function Curve
Standard Overview
An activation function is a crucial component of an artificial neural network: it defines the rule that maps a neuron's input to its output. Formally, a neuron first computes the weighted sum of its inputs z = Σwᵢxᵢ + b, then applies a nonlinear transformation f to produce the final output a = f(z). Without an activation function (or with a linear one), no matter how many layers the network has, the whole model still reduces to a linear model and cannot solve complex nonlinear problems.
Common activation functions include sgn (the sign function), sigmoid, tanh (hyperbolic tangent), ReLU (rectified linear unit) and the linear function. They differ in output range, gradient behavior and saturation, and each suits different network architectures and tasks. For example, sigmoid squashes the output into the interval (0, 1) and is often used in the output layer of binary classification models; ReLU keeps its gradient constant on the positive side, which mitigates the vanishing-gradient problem in deep networks and has made it one of the default choices in modern deep learning.
Plain-Language Introduction
If a neuron is a decision maker, then the activation function is its personality. Faced with the same input z, different activation functions react in different ways:
sgn is a strict yes/no judge — it only outputs -1 or 1 (or 0), like a light
switch with no in-between state.
sigmoid is a probabilist — it squeezes the result into the range 0 to 1 and
tells you how likely something is.
tanh works like sigmoid, but stretches the range to -1 to 1 — essentially
sigmoid scaled up and centered at 0.
ReLU is a pragmatic efficiency expert — it passes positive inputs through
unchanged and outputs 0 when the input is negative; it is cheap to compute and performs
extremely well in deep networks.
linear is a straight shooter — it outputs exactly what it receives, adds no
nonlinearity, and fits the output layer of regression problems.
It is precisely these different personalities that let neural networks learn patterns ranging from the very simple to the extremely complex.
What Problem Does It Solve
The core problem activation functions solve is giving neural networks the ability to model nonlinearity. Before activation functions were introduced, early neural networks (such as the perceptron) could only handle linearly separable problems. In 1969, Minsky and Papert pointed out in their book Perceptrons the limitations of single-layer perceptrons — most notably their inability to solve nonlinear problems such as XOR. That finding triggered the first winter of neural network research.
Activation functions allow multilayer networks to approximate arbitrarily complex functions by stacking nonlinear transformations. In 1989, Cybenko and others proved the Universal Approximation Theorem: as long as a network has at least one hidden layer and uses a nonlinear activation function (such as sigmoid), it can approximate any continuous function to arbitrary accuracy. This theorem laid a solid mathematical foundation for the rise of deep learning.
Beyond that, different activation functions also address gradient problems during training. For example, ReLU has a constant gradient of 1 on the positive side, which alleviates vanishing gradients in deep networks and makes it possible to train networks with hundreds of layers.
Why It Matters
The activation function is the indispensable nonlinear engine of a neural network. Without them, no matter how deep the network is, it is essentially just a linear regression model, unable to learn complex data patterns. The choice of activation function directly affects a network's expressive power, training speed and convergence.
Throughout the history of deep learning, innovations in activation functions have repeatedly pushed the field forward: sigmoid and tanh opened the era of multilayer perceptrons; the arrival of ReLU made it possible to train deep convolutional networks (such as AlexNet, VGG and ResNet), directly igniting the deep learning revolution; later improved variants such as Leaky ReLU, ELU and GELU further improved training stability and performance.
One could say the activation function is the bridge that connects linear algebra to the nonlinear world — the key that turned neural networks from "linear classifiers" into "universal approximators".
Application Scenarios
Image Recognition Speech Recognition Natural Language Processing Recommender Systems Reinforcement Learning Generative Models
Different activation functions suit different scenarios:
sigmoid: commonly used in the output layer of binary classification tasks,
where the output is interpreted as a probability.
tanh: often used in the hidden layers of recurrent neural networks (RNNs) and
generative models, because its zero-centered output helps gradients flow.
ReLU: the first choice for modern convolutional neural networks (CNNs) and
deep feedforward networks — computationally efficient and resistant to vanishing
gradients.
linear: the output layer for regression problems, producing continuous values
directly.
sgn: used in early perceptrons and certain binary neural networks, and also as
a hard decision for binary classification.
Limitations and Drawbacks
sgn: its gradient is zero almost everywhere, so it cannot be used with
gradient-based learning; the output takes only discrete values, which loses a great deal of
information.
sigmoid: its output is not zero-centered, which makes the optimization path
zigzag; in the saturation regions (very large or very small inputs) the gradient approaches
zero, easily causing vanishing gradients; the exponential operation is also relatively
expensive.
tanh: although zero-centered, it suffers from the same vanishing-gradient
problem in the saturation regions, and the exponential operation is costly.
ReLU: when the input is negative the gradient is zero, which can create "dead
neurons" (neurons that can never be activated again); the output is not zero-centered; and
there is no gradient on the negative side, which limits learning capacity.
linear: it introduces no nonlinearity, so stacking multiple layers is still
equivalent to a single linear transformation, and the network cannot learn complex
patterns.
In practice, one usually picks an activation function based on the task, network depth and
computational budget, and may combine techniques such as batch normalization and residual
connections to alleviate gradient problems.