Error Backpropagation

Backpropagation — the core engine behind neural network learning

Deep Learning · Gradient Descent

Core Formulas

The error backpropagation (BP) algorithm was popularized by Rumelhart, Hinton and Williams in 1986 and is still the most widely used algorithm for training multilayer neural networks. Using the chain rule, it propagates the output-layer error back to the hidden layers, computes the gradient of the loss function with respect to every weight, and then updates the parameters by gradient descent. The formulas below describe a typical 2-2-1 network (2 input neurons, 2 hidden neurons, 1 output neuron).

Forward propagation: $$h_j = \sigma\left(\sum_{i=1}^{2} w_{ij} x_i + b_j\right), \quad j=1,2$$ $$y = \sigma\left(\sum_{j=1}^{2} v_j h_j + c\right)$$

Loss function (mean squared error): $$E = \frac{1}{2}(y - t)^2$$

Output layer error: $$\delta_y = \frac{\partial E}{\partial y} \cdot \sigma'(z_y) = (y - t) \cdot y(1-y)$$

Hidden layer error: $$\delta_j = v_j \cdot \delta_y \cdot h_j(1-h_j), \quad j=1,2$$

Weight update (gradient descent): $$v_j \leftarrow v_j - \eta \frac{\partial E}{\partial v_j} = v_j - \eta \delta_y h_j$$ $$w_{ij} \leftarrow w_{ij} - \eta \frac{\partial E}{\partial w_{ij}} = w_{ij} - \eta \delta_j x_i$$ $$c \leftarrow c - \eta \delta_y, \quad b_j \leftarrow b_j - \eta \delta_j$$

Parameter Notes

x₁, x₂ (inputs) The two input features — the raw data fed into the network.
h₁, h₂ (hidden layer) The outputs of the hidden neurons, computed through the sigmoid activation function.
y (output) The prediction of the output neuron, mapped into (0, 1) by the sigmoid function.
t (target) The desired output — the true label of the training sample.
wᵢⱼ (input-to-hidden weights) The weight connecting the i-th input neuron to the j-th hidden neuron.
vⱼ (hidden-to-output weights) The weight connecting the j-th hidden neuron to the output neuron.
bⱼ, c (biases) Bias terms of the hidden and output layers, letting a neuron activate even when its input is zero.
σ (sigmoid function) The activation function, which maps the net input into (0, 1): σ(z) = 1/(1+e⁻ᶻ).
η (learning rate) Controls the step size of the weight update: too large causes oscillation, too small slows convergence.
E (loss) Mean squared error, measuring the gap between the prediction and the target.

Interactive Simulation

This is a 2-2-1 neural network. Adjust the inputs, the target value and the learning rate, set the initial weights, then click "Run one training step" to watch forward propagation, error computation and backpropagation update the weights.

Parameter Control

Initial weights and biases (training resets automatically when they change):

Current output y
0.50
Loss E
0.13
Training steps: 0

Network Architecture (2-2-1)

Loss Curve

Standard Overview

The error backpropagation (BP) algorithm is one of the core algorithms for training artificial neural networks; it was systematically described and popularized by Rumelhart, Hinton and Williams in their 1986 paper. Built on gradient descent, BP uses the chain rule to efficiently compute the partial derivative of the loss function with respect to every weight in the network, and uses those gradients to update the weights. Its core idea is: first compute the network output by forward propagation, then measure the error between that output and the desired target, then propagate the error layer by layer from the output back to the input, computing an error signal for every neuron along the way, and finally update the connection weights and biases accordingly.

BP made it possible to train multilayer neural networks and overcame the linear limitation of the single-layer perceptron. It laid the foundation of deep learning and is still used to train virtually every modern architecture (CNN, RNN, Transformer and others). Although modern deep learning frameworks compute gradients automatically, understanding BP remains essential for truly mastering how neural networks work.

Plain-Language Introduction

Imagine teaching a child to shoot a basketball. You tell them which way to throw; they throw and the ball lands to the left. You say "a bit more to the right." They adjust and throw again — this time it goes too far right. You say "back to the left a little." After many adjustments they get more and more accurate. Throughout this process you are really using the error (how far the ball is from the hoop) to continuously correct their motion.

BP does something similar. The neural network is the shooter: it takes a first shot (forward propagation) and sees how far it is from the target (computing the loss). Then it asks "which part of the motion caused the deviation?" If a weight (the force, the angle) has a large influence on the error it is adjusted more; if its influence is small it is adjusted less. That step of "analyzing which motion caused the deviation" is backpropagation — starting from the final error, the chain rule traces how much each parameter contributed, and gradient descent corrects them step by step. After a great deal of training the network's shots get steadily more accurate.

What Problem Does It Solve

BP solves the problem of training the hidden-layer weights of a multilayer neural network. Before BP, a perceptron had only an input layer and an output layer, so its weights could be adjusted directly from the error. In a multilayer network, however, the hidden layers have no desired output of their own, so how to adjust hidden-layer weights was an open problem.

BP solves this elegantly through error backpropagation: it distributes the output-layer error back to the hidden layers in proportion to the connection weights, giving each hidden neuron a "virtual error" so that its weights can be updated just like those of the output layer. This innovation made it possible to train feedforward networks of any depth and directly triggered the revival of neural network research.

In addition, combining BP with gradient descent provides a general parameter optimization framework that can train any differentiable model — CNNs, RNNs, Transformers and beyond. One could say that the training of almost every modern deep learning model relies on the idea behind BP.

Why It Matters

BP is the technical foundation of deep learning. Without it, modern deep learning would not exist. It turned the training of multilayer networks from a theoretical possibility into an engineering reality, and directly sparked the neural network boom of the late 1980s and early 1990s. Deep learning later went through a long winter, but the core idea of BP survived and, after the deep learning revival of 2006, became an indispensable tool.

Mathematically, BP is essentially an efficient implementation of the chain rule. Exploiting the dynamic-programming structure of the computational graph, it avoids recomputing gradients and reduces the cost from exponential to linear, which is what makes training large-scale networks feasible. The automatic differentiation built into modern frameworks such as TensorFlow and PyTorch is still BP at its core.

BP also inspired many later optimization techniques — momentum, Adam, RMSprop and others all improve the update rule on top of the gradients BP computes, in order to train faster and more stably. Understanding BP is a necessary step on the path to deep learning research.

Application Scenarios

Image Recognition Speech Recognition Natural Language Processing Recommender Systems Medical Diagnosis Financial Forecasting Autonomous Driving Game AI

BP runs through essentially every deep learning application:

Computer vision: training convolutional neural networks (CNNs) for image classification, object detection and semantic segmentation;
Natural language processing: training recurrent neural networks (RNNs), LSTMs and Transformers for machine translation, text generation and more;
Speech recognition: training deep networks that map acoustic features to phonemes or characters;
Recommender systems: training deep collaborative filtering models to learn user and item embeddings;
Reinforcement learning: training policy and value networks, such as the deep networks behind AlphaGo.

Any task that needs to learn a complex mapping from data depends on BP. It is the bridge between data and models, and essential knowledge for every machine learning engineer.

Limitations and Drawbacks

1. Vanishing and exploding gradients: in deep networks that use saturating activations such as sigmoid, gradients shrink (vanish) or grow (explode) exponentially as they propagate backward, making training difficult.

2. Prone to local optima: the loss function is usually non-convex, so gradient descent may converge to a local minimum instead of the global one. Results are often good enough in practice, but global optimality cannot be guaranteed in theory.

3. Sensitive to weight initialization: poor initialization can slow convergence or trap the model in a bad local minimum, so schemes such as Xavier or He initialization are needed.

4. Heavy computational cost: training a large network means repeating forward and backward passes, which is expensive — especially as data volume and model size grow.

5. Risk of overfitting: models trained with BP have many parameters and easily overfit the training data, so regularization, dropout and early stopping are needed to mitigate this.

6. Difficulty choosing the learning rate: too large a learning rate causes oscillation or even divergence; too small one converges slowly. Manual tuning or adaptive learning-rate algorithms are usually required.

Despite these limitations, BP is still one of the most effective ways to train neural networks. A large body of improvements — ReLU activations, batch normalization, residual connections and more — has alleviated the problems above and made deep networks trainable in practice.