Classification Performance Measures

Adjust the decision threshold and observe how precision, recall, F1, and the P-R curve and ROC curve change

Standard Introduction: Error Rate, Accuracy, Precision, Recall, and F1

For binary classification problems, the confusion matrix can be obtained from the combinations of true classes and predicted classes:

Error rate and accuracy are the most commonly used performance measures:

Precision and Recall measure, respectively, how many of the samples predicted as positive are truly positive, and how many of all positive samples are correctly identified:

The F1 measure is the harmonic mean of precision and recall, placing more emphasis on the smaller value:

In practical applications, precision and recall often constrain each other, and a P-R curve can be plotted to observe the overall performance of the model.

Plain-Language Introduction: Imagine a security screening system: precision answers "given that the alarm went off, what is the probability that it really is a dangerous item?"; recall answers "of all dangerous items, how many were detected by the alarm?". If the screener is too cautious (lowering the threshold), almost every item will trigger the alarm, giving high recall but low precision (many false alarms); conversely, if the screener is too lenient (raising the threshold), precision becomes higher but real dangerous items will be missed. F1 is a balanced score of the two. Dragging the slider above lets you intuitively feel this trade-off.

P-R Curve and ROC/AUC

The P-R curve uses recall as the horizontal axis and precision as the vertical axis, and is drawn by varying the decision threshold. The area under the curve (AP) can measure model performance.

The ROC curve uses the false positive rate (FPR) as the horizontal axis and the true positive rate (TPR, i.e., recall) as the vertical axis. AUC is the area under the ROC curve, and the larger the value, the better the model performance:

Plain-Language Introduction: The ROC curve is like a trade-off chart between "better to kill a thousand by mistake than let one slip through" and "precision strike." The diagonal represents random guessing; a good model's curve bulges toward the upper-left corner. AUC can be understood as: given one positive example and one negative example at random, the probability that the model ranks the positive example ahead of the negative example.