Core Formulas of Deep Belief Networks (DBN)
A deep belief network is stacked from multiple restricted Boltzmann machines (RBMs). The following formulas fully describe the RBM energy model, the probability distribution, the sampling process and the contrastive-divergence learning algorithm.
Meaning: The energy function defines the "energy" of a joint configuration of the visible states v and hidden states h in an RBM. The lower the energy, the higher the probability of that configuration. This is the foundation of the RBM model; all probability derivations start here.
•
E(v,h;θ): energy of the joint configuration of the visible and hidden layers•
v_i: state (0 or 1) of the i-th visible unit•
h_j: state (0 or 1) of the j-th hidden unit•
a_i: bias of visible unit i•
b_j: bias of hidden unit j•
W_ij: weight connecting visible unit i and hidden unit j•
n_v, n_h: number of units in the visible and hidden layers
Meaning: The partition function is the sum of the exponentiated energies over all possible combinations of visible and hidden layers, serving as the normalization constant of the probability distribution so that all joint probabilities sum to 1.
•
Z(θ): the partition function, depending only on the model parameters θ• the double sum over
v and h iterates over all 2^(n_v+n_h) possible binary configurations• in practice, due to combinatorial explosion, sampling methods are usually used for approximation
Meaning: The joint probability distribution is given by the Boltzmann distribution, the exponential of the energy function divided by the partition function. Lower-energy configurations have higher probability — this is the core assumption of the RBM model.
•
P(v,h;θ): joint probability of the visible and hidden layers•
e^(-E): the Boltzmann factor; lower energy gives higher probability•
Z(θ): the partition function from Formula 2, ensuring probability normalization•
θ = {W, a, b}: the set of all model parameters
Meaning: The marginal probability describes the probability of observing only the visible data v (summed over all possible hidden states). The goal of training an RBM is to maximize the marginal probability of the training data.
•
P(v;θ): the marginal probability of the visible configuration v• the second equality uses the conditional independence of the hidden units to turn the sum into a product
•
a^T v: the linear term of the visible-layer bias• product term: each hidden unit's contribution to the probability
Meaning: Given the visible state v, the probability that the j-th hidden unit is active (takes value 1). Thanks to the bipartite structure of an RBM, the hidden units are conditionally independent given the visible layer, which makes sampling very efficient.
•
P(h_j=1|v): conditional probability that hidden unit j is active•
σ(·): the sigmoid activation function (see Formula 7)•
b_j + Σ v_i W_ij: the "net input" of hidden unit j• conditional independence:
P(h|v) = Π_j P(h_j|v)
Meaning: Given the hidden state h, the probability that the i-th visible unit is active. This is the "reconstruction" process of the RBM, used to recover the visible data from the hidden representation.
•
P(v_i=1|h): conditional probability that visible unit i is active•
a_i + Σ W_ij h_j: the "net input" of visible unit i• the weight matrix
W is shared in both directions (symmetric connections)• also satisfies conditional independence:
P(v|h) = Π_i P(v_i|h)
Meaning: The sigmoid function maps any real number into the interval (0, 1); in an RBM it converts the "net input" into an activation probability. It is the core non-linear transform in the RBM conditional-probability computation.
•
x: the input value (net input), which can be any real number•
σ(x): the output value, in the range (0, 1)• when
x=0, σ(0)=0.5• symmetry:
σ(-x) = 1 - σ(x)
Meaning: The contrastive-divergence algorithm updates the weights by contrasting the expectation under the data distribution with the expectation under the reconstruction distribution. This is the core learning rule of RBM training, effectively avoiding the difficulty of computing the partition function directly.
•
ΔW_ij: the update amount of weight W_ij•
ε: the learning rate, controlling the update step size•
⟨v_i h_j⟩_data: the expectation of v_i and h_j being simultaneously active under the data•
⟨v_i h_j⟩_recon: the expectation under the model reconstruction (after k Gibbs steps)
Meaning: The bias parameters are also updated based on the difference between the data expectation and the model reconstruction expectation. The visible-layer bias a and the hidden-layer bias b are adjusted according to the activation differences of their respective units.
•
Δa_i: the bias update of visible unit i•
Δb_j: the bias update of hidden unit j•
⟨v_i⟩_data: the average activation rate of visible unit i in the data•
⟨h_j⟩_data: the average activation rate of hidden unit j under the data
Meaning: The DBN maps the input data h^(0) = v to increasingly abstract feature representations through layer-wise forward propagation. Each layer is a non-linear transform of the previous layer's representation.
•
h^(l): the activation vector of layer l (h^(0)=v is the input layer)•
W^(l): the weight matrix connecting layer l-1 to layer l•
b^(l): the bias vector of layer l•
L: total number of layers (1 visible layer + L-1 hidden layers)
Meaning: The log-likelihood function measures how well the model fits the whole training dataset. The training objective is to maximize the log-likelihood by adjusting the parameters θ — that is, to make the model assign as high a probability as possible to the training data.
•
L(θ): the average log-likelihood on the training set•
N: total number of training samples•
v^(n): the n-th training sample•
P(v^(n);θ): the model's marginal probability of sample v^(n) (see Formula 4)
Meaning: Gibbs sampling is a Markov-chain method for sampling from the joint distribution in an RBM. By alternately sampling the hidden and visible layers, it eventually converges to the model's equilibrium distribution. The CD-k algorithm uses k steps of Gibbs sampling as an approximation.
•
h^(0): the initial hidden state sampled starting from the data v•
v^(1): the visible state reconstructed from h^(0)•
h^(1): the hidden state sampled again from v^(1)• in CD-k, only k steps of sampling (usually k=1) are used to approximate the gradient
DBN Training: Interactive Simulation
Adjust the parameters on the left and click "Start training" to watch the RBM learn layer by layer. The charts show the network structure, the weight matrix and the training loss changing in real time.
Training parameter controls
A Deep Dive into Deep Belief Networks
From the standard definition to a plain-language explanation — understand the value and limits of DBNs.
Standard Introduction
A deep belief network (DBN) is a generative probabilistic model proposed by Geoffrey Hinton and others in 2006. A DBN is stacked from multiple restricted Boltzmann machines (RBMs) and trained with a layer-wise greedy strategy. Each RBM learns the statistical structure of the previous layer's representation, extracting features layer by layer in an unsupervised way. The bottom layer receives the raw input data, and each hidden layer transforms the previous layer's representation into more abstract, more discriminative features. After training, a DBN can be used to initialize features for supervised learning, or as a generative model that produces new samples. The introduction of the DBN marked the beginning of the deep-learning revival and laid an important foundation for the subsequent development of deep neural networks.
Plain-Language Introduction
Think of a DBN as a multi-layer "data translator". The first layer (an RBM) finds the most basic regularities in the raw data, such as edges and textures in an image; the second layer discovers more complex patterns on top of the first, such as shapes and parts; the third layer further combines these parts to recognize complete objects. Each layer does not learn everything at once, but "refines" the information layer by layer, gradually. It is like learning to draw: you first learn to draw lines, then shapes, and only then can you draw a complete picture. In this way the DBN "unfolds" the hidden structure in complex data layer by layer, letting the computer understand the patterns behind the data.
What Problem Does It Solve
DBN solves two core problems in deep learning:
- Difficulty of training deep networks: before DBNs, deep neural networks were hard to train effectively because of vanishing gradients and local optima. The DBN's layer-wise greedy pre-training provides a good parameter initialization for the deep network, allowing the subsequent supervised fine-tuning to converge successfully.
- Unsupervised feature learning: a DBN can automatically learn useful feature representations from unlabeled data, reducing the dependence on large-scale labeled data. This unsupervised pre-training approach greatly advanced the application of deep learning in scenarios where labeled data is scarce.
Moreover, as a generative model, the DBN also solves the problem of sampling new samples from the data distribution, which is important for tasks such as image generation and data augmentation.
Why It Matters
DBN holds a milestone position in the history of machine learning. In 2006, Hinton and others achieved a breakthrough on MNIST handwritten-digit recognition using a DBN, significantly surpassing the best shallow models of the time. This work proved the feasibility of deep architectures and directly triggered the research boom in deep learning. The DBN's layer-wise pre-training idea inspired the design of later models such as deep autoencoders and deep convolutional networks. Even today, the core idea of the DBN — initializing deep networks through unsupervised learning — still has a far-reaching impact on fields such as self-supervised learning and representation learning. Understanding the DBN is a key part of understanding the development of deep learning.
Application Scenarios
DBN and its variants are widely used in many fields:
- Image recognition and generation: in the early deep-learning era, DBNs were used for tasks such as handwritten-digit recognition, face recognition and image denoising.
- Speech recognition: DBNs were used to pre-train acoustic models, significantly improving speech-recognition accuracy at the time.
- Fault diagnosis: in industry, DBNs learn feature representations from sensor data to help classify and predict equipment faults.
- Bioinformatics: in tasks such as gene-expression-data analysis and protein-structure prediction, DBNs are used to discover hidden patterns in the data.
- Recommender systems: DBNs can learn latent-vector representations of users and items, improving the performance of collaborative filtering.
- Financial risk control: extract anomalous patterns from transaction data to support anti-fraud and credit assessment.
Limitations and Drawbacks
Despite its important historical significance, the DBN also has some clear limitations:
- High computational cost: training an RBM requires Gibbs sampling, and even with the CD-k approximation the computation is still heavy. For large datasets, training can take a very long time.
- Unstable training: RBM training is sensitive to hyperparameters (learning rate, initialization, CD steps, etc.); unsuitable settings can prevent the model from converging or lead it to learn degenerate representations.
- Superseded by more advanced models: with the development of techniques such as the ReLU activation, batch normalization and residual connections, deep networks can now be trained effectively by supervised learning directly, so the DBN's pre-training strategy is no longer necessary. Variational autoencoders (VAEs) and generative adversarial networks (GANs) have also shown stronger capability in generative modeling.
- Incomplete theoretical understanding: although the DBN's layer-wise greedy training strategy works well in practice, its theoretical guarantees are still insufficient. Why layer-wise pre-training helps subsequent supervised learning still lacks a fully rigorous mathematical explanation.
- Hard to scale to high-dimensional data: for high-dimensional image or video data, the fully-connected structure of an RBM causes the number of parameters to grow sharply, making training extremely difficult. Introducing a convolutional structure partly alleviates this, but in essence a DBN is better suited to data of moderate dimensionality.