Empirical Error and Overfitting
Polynomial Regression Demo — drag the slider to adjust model complexity and observe the fitted curve and error changes
Standard Introduction: Empirical Error and Overfitting
In Machine Learning, the empirical error (Empirical Error) is the error of the model on the training set, reflecting how well the model fits the known data. The generalization error (Generalization Error) is the expected error of the model on new samples, representing its true predictive ability.
As discussed in Section 2.1 of Zhou Zhihua's "Machine Learning" (the "Watermelon Book"), when model complexity is too high, overfitting (Overfitting) occurs: the model "memorizes" the noise in the training data as well, causing the empirical error to be extremely low while the generalization error rises significantly. Conversely, if the model is too simple, it leads to underfitting (Underfitting), where both the empirical and generalization errors remain at a high level.
The core goal of model selection is to strike a balance between empirical error and generalization error—to find the model complexity that minimizes the generalization error, rather than blindly pursuing a perfect fit on the training set. This reflects Occam's razor principle: given the ability to explain the data, the simpler model should be chosen.
Key term mapping: Empirical error ≈ training error | Generalization error ≈ test error (approx.) | Overfitting: low training error + high test error | Underfitting: both high
Plain-Language Introduction: Understanding Overfitting Like "Cramming"
Imagine you are preparing for a math exam. If you rote-learn all the practice problems (including the typos and occasional slips in the questions), you can score full marks on the practice sheets—this is extremely low empirical error. But once a slightly different new question appears on the exam, you make mistakes—this is high generalization error, i.e., overfitting.
On the other hand, if you only master the most basic addition and subtraction before taking the exam, you will get neither the practice nor the exam questions right—this is underfitting, where both empirical and generalization errors are large.
- Underfitting: cannot even get the training questions right → the model is too simple and has not learned the basic pattern of the data.
- Overfitting: gets all training questions right but fails the exam → the model is too complex and treats noise as a pattern.
- Ideal state: does well on the training questions and performs steadily on the exam → the model complexity is just right.
In the demo above, drag the slider to change the polynomial degree (i.e., model complexity) and observe: At low degree the curve is smooth but the error is large (underfitting), At high degree the curve fluctuates violently, trying to pass through every training point (overfitting), while some intermediate degree (usually d=3~4) achieves the best balance—this is what model selection is all about.