Linear Discriminant Analysis (LDA)
Linear Discriminant Analysis — Finding the Optimal Projection Direction to Maximize Between-Class Separation
Core Formulas and Parameters
Measures the dispersion of data points within the same class around their mean. The smaller SW is, the more compact the class is.
Measures the degree of difference between the means of different classes. The larger SB is, the better the separation between classes.
Goal: find the projection direction w that maximizes the ratio of between-class scatter to within-class scatter.
By solving the generalized eigenvalue problem, the optimal direction is SW-1(μ₁ - μ₂) (for binary classification).
Parameters
- μ₁x, μ₁y — the center coordinates of class 1 in two-dimensional space
- μ₂x, μ₂y — the center coordinates of class 2 in two-dimensional space
- σ₁ — the data standard deviation of class 1 (controlling dispersion)
- σ₂ — the data standard deviation of class 2
- ρ — the correlation coefficient between the two dimensions (controlling the tilt of the ellipse)
- n — the number of samples per class
Parameter Adjustment
Real-Time Metrics
Original Data Distribution and LDA Projection Direction 2D Scatter Plot
One-Dimensional Distribution After Projection LDA Dimensionality Reduction Effect
Standard Introduction
Linear Discriminant Analysis (LDA) is a classic supervised learning method proposed by the statistician Ronald Fisher in 1936, and is therefore also known as Fisher Discriminant Analysis. The core idea of LDA is: find one (or more) projection directions that project high-dimensional data into a low-dimensional space, so that after projection the data of different classes are separated as much as possible, while the data within the same class are as compact as possible.
Mathematically, LDA solves for the optimal projection direction by maximizing the ratio of the between-class scatter matrix to the within-class scatter matrix (the Fisher criterion). For binary classification problems, the optimal projection direction can be expressed analytically as w* = SW-1(μ₁ - μ₂), where SW is the within-class scatter matrix and μ₁ and μ₂ are the mean vectors of the two classes, respectively.
The main difference between LDA and PCA (Principal Component Analysis) is: PCA is an unsupervised method that looks for the direction of maximum data variance; LDA is a supervised method that uses class label information to find the direction most favorable for classification.
Plain-Language Introduction
Imagine you have two kinds of fruit, say apples and oranges, and they differ along two dimensions: size and color. Your task is to find the best angle from which to observe them, so that from this angle the apples and oranges are most clearly distinguished.
This is exactly what LDA does: it looks for an "observation line" (projection direction), and when you project all the data points onto this line, the overlap between different classes is minimized and they are separated the most. It is like finding a dividing line in a crowd so that two teams stand clearly apart.
LDA not only finds this line, but also tells you how to draw a "dividing line" after projection to distinguish the two classes. If the overlapping region of the two distributions after projection is very small, it means this projection direction is well chosen, and the classification accuracy will be high.
What Problem Does It Solve
LDA solves the problem of classification and dimensionality reduction of high-dimensional data. In many practical scenarios, the data dimension is very high (for example, the pixels of a face image may number in the tens of thousands), and performing classification directly in high-dimensional space is very difficult (the "curse of dimensionality").
- Dimensionality reduction problem: projecting high-dimensional data into a low-dimensional space (such as one or two dimensions) while preserving class-discriminative information
- Classification problem: establishing a decision rule in the projected space to predict the class of new samples
- Feature extraction: finding the combination of features that is most discriminative for classification
- Data visualization: reducing high-dimensional data to two or three dimensions for intuitive understanding of the data structure
Why It Matters
LDA is one of the most fundamental and classic algorithms in the fields of Machine Learning and pattern recognition. Its importance is reflected in the following aspects:
- Deep theoretical foundation: LDA is based on rigorous statistical inference theory, assuming that the data of each class follow a Gaussian distribution with the same covariance; under these assumptions, LDA is the optimal classifier (Bayes optimal)
- Computationally efficient: LDA has an analytical solution and does not require iterative optimization; it is fast to compute and suitable for large-scale data
- Strong interpretability: the projection direction w has a clear geometric meaning, allowing intuitive understanding of which features are more important for classification
- Widely applicable: from medical diagnosis to financial risk control, from face recognition to text classification, LDA has successful applications in various fields
- Algorithmic cornerstone: many modern dimensionality reduction methods (such as regularized LDA, kernel LDA, and multidimensional LDA) are extended from classical LDA
Application Scenarios
- Face recognition: the Fisherface method uses LDA to extract discriminative face features and is widely used in security and attendance systems
- Medical diagnosis: disease classification and risk prediction based on multiple patient indicators (blood pressure, blood sugar, cholesterol, etc.)
- Financial risk control: distinguishing normal transactions from fraudulent transactions, and determining the credit rating of loan applicants
- Speech recognition: extracting discriminative acoustic features from speech signals
- Text classification: topic classification of documents, spam filtering, and sentiment analysis
- Bioinformatics: cancer subtype classification of gene expression data
- Quality control: classification problems in industrial product defect detection
- Marketing: customer segmentation to distinguish user groups with different consumption behaviors
Limitations and Drawbacks
- Gaussian distribution assumption: LDA assumes that the data of each class follow a multivariate Gaussian distribution. When the data distribution deviates significantly from Gaussian (such as clearly multimodal or heavy-tailed distributions), LDA performance degrades
- Equal covariance assumption: classical LDA assumes that the covariance matrices of all classes are the same. If the actual covariance matrices differ greatly, QDA (Quadratic Discriminant Analysis) should be used
- Linear boundary limitation: LDA can only find linear decision boundaries. For non-linearly separable datasets, LDA performs poorly and requires kernel methods or other nonlinear algorithms
- Small-sample problem: when the number of samples is much smaller than the feature dimension (such as gene expression data), SW may be singular (non-invertible), requiring regularization
- Sensitive to outliers: the estimation of the mean and covariance matrix is very sensitive to outliers, and a few outliers may severely affect the projection direction
- Limited projection dimension: for a K-class problem, LDA can project at most into a K-1 dimensional space. When the number of classes is small, the dimensionality reduction capability is limited
- Not suitable for complex structures: when the within-class structure is complex (such as data distributed in a ring or spiral shape), LDA cannot find an effective discriminant direction