Core Decision Tree Formulas

A decision tree chooses the best split by measuring the impurity of the dataset. The three most fundamental measures are given below.

β‘  Information Entropy

Entropy measures the degree of disorder in a dataset. The larger the entropy, the more mixed the data; when entropy is 0 the data is completely pure (all samples belong to the same class).

Parameters:
H(D) β€” information entropy of dataset D
K β€” total number of classes (K=2 for binary classification)
pk β€” proportion of class-k samples in D
logβ‚‚ β€” base-2 logarithm (the information-theory convention)

β‘‘ Information Gain

Information gain measures how much the uncertainty drops after splitting on feature A. The larger the gain, the more effective the split. At each step the tree splits on the feature with the largest information gain.

Parameters:
IG(D, A) β€” information gain from splitting dataset D on feature A
V β€” number of distinct values of feature A (V=2 for a binary split)
|Dv| β€” number of samples in subset Dv
|D| β€” total number of samples

β‘’ Gini Index

The Gini index is another impurity measure: it is the probability that two samples drawn at random from the dataset belong to different classes. The smaller the Gini index, the purer the data.

Parameters:
Gini(D) β€” Gini index of dataset D
pk β€” proportion of class-k samples
Range: [0, 0.5] (binary classification); 0 means completely pure

Decision Tree Split Simulation

Adjust the parameters on the left and watch how the data distribution and the tree structure on the right change in real time. The system automatically computes the best split for reference.

Parameters

Current: 100 samples
Larger value β†’ the two classes separate more easily; smaller value β†’ more noise
Threshold: 5.00
Data distribution and split boundary
Root entropy H(D)
--
Entropy after split (weighted)
--
Information gain IG
--
Gini index
--
Computing...
Decision tree structure

Standard Introduction

A decision tree is a supervised learning algorithm used for classification and regression. It builds a tree structure by recursively splitting the dataset into smaller subsets. At every internal node the algorithm picks a feature and a threshold that divide the data into two (or more) subsets, aiming to make each subset as pure as possible β€” that is, to maximize the share of samples belonging to a single class.

The quality of a split is measured with information entropy, information gain or the Gini index. Building a decision tree is a greedy algorithm: at each step it picks the split that is best right now, without regard to global optimality. The resulting tree is then used to classify new samples β€” starting at the root node, you walk down the tree according to the feature values until you reach a leaf.

Plain-Language Introduction

A decision tree is like an elaborate game of Twenty Questions. Suppose you have to guess a fruit: you might first ask β€œIs it round?” β€” if so, then ask β€œIs its skin yellow?” β€” if so, it is probably a banana. Every question is a split that halves the range of possibilities.

In machine learning, a decision tree β€œlearns” which questions to ask from the data. It automatically finds the feature and the cut-off value that best separate the classes β€” a question such as β€œIs the petal length ≀ 3.5 cm?”. Good questions make the classification clear very quickly, and that is the core idea of information gain: the more the disorder drops after the question is asked, the better.

What Problem Does It Solve

Decision trees mainly solve classification and regression problems, and are especially well suited to the following situations:

  • High demand for interpretability: the decision logic has to be explained to non-technical users
  • Complex feature relationships: non-linear relations and interactions exist among the features
  • Mixed data types: numeric and categorical features handled at the same time
  • Limited preprocessing: little need for scaling or normalization
  • Rule extraction: the model needs to be turned into explicit business rules

Why It Matters

Decision trees are a cornerstone algorithm of machine learning. Their importance shows up in several ways:

  • Strong interpretability: it is a β€œwhite-box” model whose decision path is clearly visible, satisfying the compliance requirements of finance, healthcare and similar fields
  • Basis of ensemble learning: powerful algorithms such as random forests, gradient boosted trees (GBDT) and XGBoost all use decision trees as their basic unit
  • No complex preprocessing: insensitive to feature scaling, handles missing values naturally
  • Pedagogical value: the best entry model for grasping core machine learning ideas such as impurity and recursive partitioning

Application Scenarios

  • Medical diagnosis: predicting the type of disease from symptoms and test indicators
  • Credit risk assessment: banks estimate default probability when approving a loan
  • Customer segmentation: e-commerce platforms run targeted marketing based on user behaviour
  • Fault detection: analysing industrial sensor data to predict equipment failure
  • Recommender systems: matching recommendations to user profiles and item features
  • Bioinformatics: gene sequence classification, protein structure prediction

Limitations and Drawbacks

  • Prone to overfitting: the tree can grow without limit, fitting the training data perfectly but generalizing poorly (pruning or depth limits are needed)
  • Sensitive to noise: a few outliers can lead to a completely different split
  • Instability: small changes in the data can produce a completely different tree
  • Bias toward many-valued features: information gain tends to prefer features with more distinct values (the gain ratio alleviates this)
  • Limited expressive power: decision boundaries are axis-aligned, so complex non-linear relations are hard to represent
  • Local optimum: the greedy strategy offers no guarantee of finding the globally optimal tree