Log-Odds Regression (Logistic Regression)

A classic linear model for binary classification problems — mapping the output of linear regression to a probability through the Sigmoid function, optimized using cross-entropy loss and gradient descent.

Core Formulas Formula Breakdown

Linear combination
$$z = \mathbf{w}^\top \mathbf{x} + b = w_1 x_1 + w_2 x_2 + b$$
Sigmoid activation function
$$\sigma(z) = \frac{1}{1 + e^{-z}}$$
Predicted probability
$$P(y=1 \mid \mathbf{x}) = \sigma(\mathbf{w}^\top \mathbf{x} + b)$$
Cross-entropy loss function
$$J(\mathbf{w}, b) = -\frac{1}{n}\sum_{i=1}^{n}\left[y_i \log(\hat{y}_i) + (1-y_i)\log(1-\hat{y}_i)\right]$$
Formula Introduction: Log-odds regression performs a weighted sum of the input features through the linear function \(z = \mathbf{w}^\top \mathbf{x} + b\), and then uses the Sigmoid function to map the result to the (0,1) interval, interpreted as the probability that the sample belongs to the positive class. The log-odds of the model is linearly related to the input: \(\log\frac{P(y=1|\mathbf{x})}{P(y=0|\mathbf{x})} = \mathbf{w}^\top \mathbf{x} + b\).
w₁, w₂ — feature weights, controlling the slope of the decision boundary
b — bias term, controlling the position of the decision boundary
σ(z) — Sigmoid function, mapping any real number to (0,1)
η — learning rate, controlling the parameter update step size
iterations — number of gradient descent iterations

Interactive Simulation Real-Time Visualization

Ready — adjust parameters or click Train
Tip: Left-click on the main chart to add a positive class point (blue), right-click to add a negative class point (orange)

Decision Boundary and Probability Heatmap

Loss Function Curve

Sigmoid Activation Function

In-Depth Understanding From Principle to Application

Standard Introduction

Logistic Regression is a linear discriminant model used for binary classification problems. Although the name contains the word "regression," it is actually a classification algorithm. The model feeds a linear combination of the features \(z = \mathbf{w}^\top \mathbf{x} + b\) into the Sigmoid function \(\sigma(z) = 1/(1+e^{-z})\), compressing the output into the (0,1) interval, which represents the probability that the sample belongs to the positive class. The decision boundary is defined by \(\mathbf{w}^\top \mathbf{x} + b = 0\), appearing as a straight line in two-dimensional space and dividing the feature space into two class regions. The model parameters are usually solved by maximizing the likelihood function (equivalent to minimizing the cross-entropy loss) using gradient descent.

Plain-Language Introduction

Imagine you need to determine whether an email is "spam" or "normal." You score it based on the keywords that appear in the email (such as "winner," "free," etc.). The higher the score, the more it looks like spam. But the score itself is a number ranging from negative infinity to positive infinity, and you need to convert it into a "probability." Logistic regression is like a "translator": it first computes a weighted sum of all features to obtain a score, and then uses the Sigmoid function to "compress" this score into the range between 0 and 1—for example, 0.85 means "this email has an 85% probability of being spam." A score of 0 corresponds exactly to a 50% probability, which is the position of the decision boundary.

What Problem Does It Solve

  • Binary classification problems: assigning input samples to one of two mutually exclusive classes, with an interpretable probability output.
  • Probability estimation: unlike hard classification, it provides the confidence that each sample belongs to the positive class, making it easy to set different classification thresholds.
  • Discrimination of linearly separable data: finding the optimal hyperplane (a straight line in two dimensions) in the feature space to separate the two classes of samples.
  • Feature importance analysis: the absolute value of the weight vector w reflects the degree and direction of each feature's influence on the classification result.

Why It Matters

Logistic Regression is one of the cornerstone algorithms in the Machine Learning field and holds an irreplaceable position. First, it is the foundation for understanding neural networks—a single neuron (an upgraded version of the Perceptron) is essentially a logistic regression model. Second, it has good mathematical properties: the loss function is convex, guaranteeing that gradient descent can find the global optimum; the output has a probabilistic interpretation, making it easy to combine with Bayesian methods. In addition, it is computationally efficient and highly interpretable, and is widely used in practical industrial applications (such as credit scoring, medical diagnosis, and click-through rate prediction), serving as the performance benchmark for many complex models.

Application Scenarios

  • Medical diagnosis: predicting disease probability based on patient indicators (such as diabetes risk prediction).
  • Financial risk control: credit scoring, fraud detection, and loan default prediction.
  • Internet: click-through rate prediction, user conversion prediction, and spam filtering.
  • Natural language processing: sentiment analysis (positive/negative reviews) and text classification.
  • Marketing: customer churn prediction and advertising response prediction.

Limitations and Drawbacks

  • Linear decision boundary: cannot handle non-linearly separable problems, requiring feature engineering or kernel method extensions.
  • Sensitive to outliers: extreme outliers can significantly affect the position of the decision boundary.
  • Multicollinearity: when features are highly correlated, parameter estimation becomes unstable and requires regularization.
  • Class imbalance: when the ratio of positive to negative samples is skewed, the model tends to favor the majority class, requiring threshold adjustment or sampling strategies.
  • Limited expressive power: for complex high-dimensional nonlinear patterns, more powerful models (such as decision trees and neural networks) are needed.