McNemar's Test — Formula and Principle

McNemar's test is used to compare whether the performance difference between two classifiers on the same test set is statistically significant. It is based on the counts of samples on which the two models disagree (b and c), and tests whether they are symmetric.

Core test statistic (continuity correction)

\( \chi^2 = \frac{(|b - c| - 1)^2}{b + c} \)
Contingency table
\( \begin{array}{c|cc} & B\text{ correct} & B\text{ wrong} \\ \hline A\text{ correct} & a & b \\ A\text{ wrong} & c & d \end{array} \)
Disagreement counts
\( b = \#(A\text{ correct}, B\text{ wrong}) \)
\( c = \#(A\text{ wrong}, B\text{ correct}) \)
Null hypothesis
\( H_0: p_b = p_c \)
\( H_1: p_b \neq p_c \)
Degrees of freedom
\( df = 1 \)

Formula Parameters

a Number of samples correctly classified by both models
b Number of samples where model A is correct and model B is wrong (disagreement)
c Number of samples where model A is wrong and model B is correct (disagreement)
d Number of samples misclassified by both models
χ² Chi-square statistic, which follows a chi-square distribution with 1 degree of freedom (asymptotically)
α Significance level, commonly 0.01, 0.05, or 0.10
p-value The probability of observing the current or a more extreme result when the null hypothesis holds
Continuity correction When b+c is small, using the corrected formula can reduce Type I errors

Interactive Simulation Lab

Adjust the disagreement counts b and c, or set the total sample size N, and the system will compute the McNemar statistic and p-value in real time and visualize the test result.

Significance level α
Sample size settings
samples

N = a + b + c + d, and it must satisfy b + c ≤ N.

Disagreement counts (model A correct & B wrong vs A wrong & B correct)
Quick presets
Contingency Table and Disagreement Counts
Chi-Square Distribution (df=1) and Observed Statistic
Test Result Summary

In-Depth Understanding of McNemar's Test

Understand the value and limitations of this classic statistical testing method from multiple dimensions.

Standard Introduction ▼

McNemar's test was proposed by the statistician Quinn McNemar in 1947. It is a non-parametric test for paired nominal data, especially suitable for comparing the performance of two classifiers on the same test set. Its core is to construct a 2×2 contingency table recording the number of samples where both models are correct (a), both are wrong (d), and where they disagree (b and c). The test statistic is based on the difference between the disagreement counts b and c. Under the null hypothesis (the two models have the same error rate), b and c should follow a symmetric binomial distribution, and the test statistic approximately follows a chi-square distribution with 1 degree of freedom. When b+c is small, a continuity correction is usually applied, that is, using (|b-c|-1)² in the formula.

Plain-Language Introduction ▼

Imagine two doctors diagnosing the same group of patients, and we care about the differences in their diagnoses. Record the cases where both are correct, both are wrong, and where one is correct and the other is wrong. If the two doctors have similar skill levels, then the number of cases where "A is right and B is wrong" and "A is wrong and B is right" should be about the same. McNemar's test checks whether the difference between these two "disagreement" counts is too large to be due to chance. If the difference is very large, we say the two doctors have significantly different diagnostic abilities. In Machine Learning, this means replacing the doctors with two models and the patients with test samples.

What Problem Does It Solve ▼

When comparing two classifiers, merely comparing the difference in accuracy can be misleading, because:

  • Sample overlap: the two models are evaluated on the same test set, so their errors are correlated, and independent-sample tests are not applicable;
  • Focus only on disagreements: samples where both models are correct or both are wrong do not help distinguish which model is better, and McNemar's test focuses on disagreement samples, making it more efficient;
  • Robustness in small samples: compared with other tests, McNemar's test can still provide reasonable inference when the sample size is small (combined with continuity correction or exact binomial test);
  • Paired design: it is suitable for any paired comparison scenario, such as the performance of two algorithms on the same batch of cross-validation folds.
Why It Matters ▼

In Machine Learning model evaluation, the importance of McNemar's test is reflected in:

  • Simple and efficient: it can be computed using only the prediction results of the two classifiers on the same test set, without retraining or additional data;
  • Widely applicable: it is suitable for performance comparison in binary and multi-class problems (through the definition of agreement/disagreement);
  • Academic standard: many papers and competitions (such as Kaggle) use McNemar's test to verify whether a model improvement is significant;
  • Combined with cross-validation: it can be applied to the paired results of each fold, or aggregated into multiple 2×2 tables for combined testing.
Application Scenarios ▼

McNemar's test is often used in the following scenarios:

  • Model comparison: comparing whether there is a significant difference in the error rates of two classifiers on the same test set;
  • Feature selection: testing whether adding or removing a certain feature significantly increases the number of samples on which the model predictions disagree;
  • Medical diagnosis: comparing the agreement of positive/negative judgments of two diagnostic methods on the same group of patients;
  • A/B testing: comparing the performance difference of two strategies on the same user group in a recommendation system;
  • Text classification: evaluating the impact of two preprocessing methods on classification results.
Limitations and Drawbacks ▼

Although McNemar's test is simple and practical, it also has some limitations:

  • Focuses only on disagreement samples: it ignores samples where both models are correct or both are wrong, which may lose some information;
  • Not applicable to independent samples: it requires the two models to be evaluated on exactly the same samples;
  • Asymptotic approximation bias: when b+c is small (usually <25), the chi-square approximation is not accurate enough, and an exact binomial test is needed;
  • Cannot assess effect size: it only answers "whether there is a difference," but does not provide a confidence interval for the size of the difference;
  • Multiple comparison problem: when comparing multiple models simultaneously, a correction is needed (such as Bonferroni);
  • Sensitive to imbalanced data: if one class accounts for the vast majority in the test set, the number of disagreement samples may be very small, reducing the power of the test.