Support Vector Machine — Core Formulas

The following formulas form the theoretical foundation of SVM. Each formula is accompanied by detailed parameter and meaning explanations.

① Linear SVM Decision Function

Parameters:
w — weight vector, determining the normal direction of the decision boundary
x — feature vector of the input sample
b — bias term (intercept), determining the position offset of the decision boundary
Meaning: The decision function outputs a real value whose sign determines the classification result. When f(x) > 0, it predicts the positive class (+1); when f(x) < 0, it predicts the negative class (-1). f(x) = 0 defines the decision boundary hyperplane.

② Hard Margin SVM Primal Optimization Problem

Parameters:
‖w‖² — squared L2 norm of the weight vector, inversely proportional to the margin width
yᵢ — label of the i-th sample, taking values +1 or -1
xᵢ — the i-th training sample
Meaning: Under the premise of ensuring that all samples are correctly classified (constraint conditions), minimize the weight norm to maximize the classification margin. The margin width is 2/‖w‖, and maximizing the margin is equivalent to minimizing ‖w‖².

③ Soft Margin SVM Optimization Problem

Parameters:
C — penalty coefficient, balancing margin maximization and tolerance of misclassification
ξᵢ — slack variable, allowing the i-th sample to violate the margin constraint to some extent
n — total number of training samples
Meaning: Soft margin SVM allows some samples to be misclassified or fall inside the margin by introducing slack variables ξᵢ. The parameter C controls the trade-off: the larger C is, the heavier the penalty for misclassification and the narrower the margin; the smaller C is, the more misclassification is allowed and the wider the margin.

④ Lagrangian Function

Parameters:
αᵢ — Lagrange multiplier, corresponding to the constraint of each training sample
αᵢ ≥ 0 — non-negativity constraint ensuring the validity of the optimal solution
Meaning: Transform the constrained primal optimization problem into an unconstrained Lagrangian function. By taking partial derivatives with respect to w and b and setting them to zero, the dual problem can be obtained. The samples corresponding to the non-zero terms in the Lagrange multipliers αᵢ are the support vectors.

⑤ Dual Optimization Problem

Parameters:
xᵢᵀxⱼ — inner product between samples, which is the basis of the kernel function
αᵢ — dual variable, with upper bound C (consistent with the soft margin penalty coefficient)
Meaning: The dual problem depends only on the inner products between samples, which provides the basis for introducing the kernel trick. The αᵢ obtained by solving the dual problem determines the decision function. When αᵢ=0, the sample does not affect the decision; when 0<αᵢ<C, the sample lies on the margin boundary; when αᵢ=C, the sample is misclassified or inside the margin.

⑥ Kernel Function Definition

Parameters:
K(xᵢ, xⱼ) — kernel function, computing the inner product of two samples in feature space
φ(·) — mapping function from the original space to a high-dimensional feature space
Meaning: The kernel function allows us to directly compute the inner product in a high-dimensional feature space without explicitly computing φ(x). This enables SVM to handle nonlinear classification problems by mapping data into a high-dimensional space, while avoiding the "curse of dimensionality."

⑦ Gaussian Radial Basis (RBF) Kernel Function

Parameters:
γ (gamma) — kernel width parameter, controlling the influence range of a single sample
‖xᵢ - xⱼ‖² — squared Euclidean distance between two samples
Meaning: The RBF kernel is one of the most commonly used kernel functions. The larger γ is, the faster the kernel function decays and the more complex the decision boundary (possible overfitting); the smaller γ is, the smoother the decision boundary (possible underfitting). The RBF kernel corresponds to an infinite-dimensional feature space.

⑧ Dual-Form Decision Function

Parameters:
αᵢ — dual variable (Lagrange multiplier), with non-zero terms corresponding to support vectors
yᵢ — label of the training sample
K(xᵢ, x) — kernel function, computing the similarity between a training sample and a test sample
Meaning: The dual-form decision function expresses the decision as a weighted sum of kernel functions over the support vectors. Only support vectors (αᵢ>0) contribute to the decision, which gives SVM its sparsity. b is solved through the KKT conditions satisfied by the support vectors.

Interactive Simulation

Adjust the parameters on the left and click "Train SVM" to observe changes in the decision boundary, margin region, and support vectors.

Model Parameters

Model status: Waiting for training
Training accuracy: --
Number of support vectors: --
Margin width: --
Final loss: --

Decision Boundary Visualization

About Support Vector Machine (SVM)

Standard Introduction

Support Vector Machine (SVM) is a supervised learning model based on statistical learning theory, formally proposed by Vapnik et al. in 1995. The core idea of SVM is to find an optimal hyperplane in feature space that separates samples of different classes, such that the distance from this hyperplane to the nearest training samples (i.e., support vectors)—called the margin—is maximized. For linearly inseparable data, SVM introduces kernel functions (such as the Gaussian RBF kernel and polynomial kernel) to implicitly map the data into a high-dimensional feature space, constructing a linear decision boundary in that high-dimensional space, thereby solving nonlinear problems. SVM has a solid mathematical foundation, based on VC dimension theory and the structural risk minimization principle, and possesses strong generalization ability, making it especially suitable for classification problems with small samples and high-dimensional data.

The SVM optimization problem can be stated as: under the constraints yᵢ(wᵀxᵢ + b) ≥ 1 - ξᵢ (ξᵢ ≥ 0), minimize the objective function (1/2)‖w‖² + C·Σξᵢ. Here C is the penalty coefficient, controlling the model's tolerance for misclassification. Through dual transformation, the problem becomes a quadratic programming problem that depends only on the inner products between samples (or kernel function values), and can be solved efficiently using algorithms such as SMO (Sequential Minimal Optimization). The final decision function is f(x) = ΣαᵢyᵢK(xᵢ, x) + b, in which only support vectors (samples with αᵢ > 0) participate in the decision, reflecting the sparsity advantage of SVM.

Plain-Language Introduction

Imagine you have two piles of balls of different colors (red and blue) on a plane, and you need to draw a line to separate them. What Support Vector Machine does is find the best line—not just any line that can separate them, but the one that makes the distance from the nearest balls to the line as large as possible. These balls closest to the line are the "support vectors," and like "support pillars," they determine the position of the line. If the balls are not linearly distributed (for example, red balls form a circle with blue balls inside), then you need to "project" the balls into a higher dimension (like bouncing points on a plane into the air), and then find a "hyperplane" in the high-dimensional space to separate them. The mathematical tool for this "projection" is the kernel function. The advantage of SVM is that it only cares about the points "on the boundary" (support vectors) rather than all data points, so it is computationally efficient and not prone to overfitting.

What Problem Does It Solve

  • Binary classification problems: SVM was originally designed to solve binary classification problems by finding the optimal classification hyperplane to separate samples of different classes.
  • Nonlinear classification: Through the kernel trick, SVM can handle linearly inseparable data and achieve nonlinear decision boundaries without explicitly computing high-dimensional mappings.
  • Small-sample learning: Compared with traditional statistical methods (such as neural networks), SVM can still maintain good generalization performance in small-sample situations and is not prone to overfitting.
  • High-dimensional data classification: SVM performs excellently when handling high-dimensional sparse data (such as word frequency vectors in text classification), because its complexity mainly depends on the number of support vectors rather than the feature dimension.
  • Multi-class problems: Through strategies such as one-vs-one or one-vs-rest, SVM can be extended to multi-class tasks.

Why It Matters

  • Solid theoretical foundation: SVM is based on statistical learning theory and the structural risk minimization principle, and its generalization error upper bound has rigorous mathematical guarantees, which gives SVM theoretical superiority.
  • Global optimal solution: Unlike algorithms such as neural networks that easily fall into local optima, the SVM optimization problem is a convex quadratic programming problem, guaranteeing that a global optimal solution is found.
  • Sparsity: The final model is determined only by a small number of support vectors, and the model complexity does not grow linearly with the number of training samples, resulting in high computational efficiency at prediction time.
  • Generality of the kernel trick: The idea of kernel methods is not limited to SVM; it has also inspired many kernel methods such as kernel PCA and kernel regression, becoming an important branch of Machine Learning.
  • Historical status: Before the rise of deep learning, SVM was one of the most mainstream classification algorithms in the Machine Learning field, and it remains competitive in many areas (such as bioinformatics and text classification) to this day.

Application Scenarios

  • Text classification and sentiment analysis: SVM performs excellently in tasks such as spam filtering, news classification, and sentiment polarity judgment, and is especially suitable for high-dimensional sparse text features.
  • Image recognition: For handwritten digit recognition (such as MNIST), face detection, and object classification, SVM is often used in combination with feature extraction methods such as HOG and SIFT.
  • Bioinformatics: For gene expression data classification, protein structure prediction, and disease diagnosis, SVM can handle high-dimensional small-sample biological data.
  • Financial risk control: For credit scoring, fraud detection, and stock trend prediction, the robustness of SVM has led to its wide application in the financial field.
  • Medical diagnosis: For auxiliary disease diagnosis based on medical images or test data, such as cancer classification and pathological image analysis.
  • Industrial inspection: For classification problems in industrial scenarios such as product quality inspection, fault diagnosis, and anomaly detection.

Limitations and Drawbacks

  • Difficult training on large-scale data: The training time complexity of SVM ranges between O(n²) and O(n³) (where n is the number of samples). When the sample size exceeds hundreds of thousands, training efficiency drops significantly.
  • Sensitive to parameter selection: The penalty coefficient C and kernel function parameters (such as γ of the RBF kernel) greatly affect model performance, and the optimal parameter combination usually needs to be determined through grid search or cross-validation.
  • Kernel function selection relies on experience: Different problems may require different kernel functions, and there is a lack of unified theoretical guidance on how to choose a suitable kernel function, often requiring domain knowledge and experimental validation.
  • Probability output is not direct: Standard SVM outputs the value of the decision function rather than a probability, and probability estimates require post-processing methods such as Platt scaling.
  • Sensitive to class imbalance: When the ratio of positive to negative samples is severely imbalanced, the SVM decision boundary may be biased toward the minority class, requiring class weights or sampling strategies to mitigate.
  • Relatively limited interpretability: Although support vectors provide an intuitive understanding of the decision boundary, when nonlinear kernel functions are used, the model's interpretability is less intuitive than methods such as decision trees.
  • Not suitable for large-scale multi-class classification: Extending SVM to multi-class problems requires constructing multiple binary classifiers, which incurs high computational overhead in large-scale multi-class scenarios.