Friedman Test & Nemenyi Post-hoc Test — Formula and Principle
Used to compare the performance of more than two algorithms across multiple datasets. The Friedman test judges whether an overall difference exists among the algorithms; if significant, the Nemenyi post-hoc test identifies exactly which algorithm pairs differ significantly.
Friedman Statistic
\(\chi_F^2 = \frac{12N}{k(k+1)} \left[ \sum_{j=1}^{k} R_j^2 - \frac{k(k+1)^2}{4} \right]\)Friedman's F Statistic (More Accurate)
\(F_F = \frac{(N-1)\chi_F^2}{N(k-1) - \chi_F^2}\)Formula Parameters
Interactive Simulation Lab
Set the number of algorithms, number of datasets, significance level, and the degree of performance difference between algorithms, then click "Generate Data" to simulate the rankings of multiple algorithms across multiple datasets and run the statistical test.
Understanding the Friedman Test & Nemenyi Post-hoc Test
Standard Introduction
The Friedman test is a non-parametric statistical test for comparing multiple related samples. In machine learning, when we need to compare several algorithms (k>2) across multiple datasets (N datasets), each dataset provides a performance measure for every algorithm. For each dataset we rank the algorithms by performance, assigning ranks 1 to k (best = 1, worst = k; ties take the average rank). The Friedman test uses these ranks to judge whether all algorithms perform equally. The null hypothesis is "all algorithms perform equally" and the alternative is "at least one algorithm differs from the others". The test statistic is based on the deviation of each algorithm's average rank from the expected rank, and is approximately chi-square distributed (or, more precisely, F distributed). If the null hypothesis is rejected, a post-hoc test (e.g. the Nemenyi test) is needed to determine which algorithm pairs differ significantly. The Nemenyi test computes a critical difference (CD); when the difference between two algorithms' average ranks exceeds the CD, their difference is deemed statistically significant.
Plain-Language Introduction
Imagine a cooking contest where several chefs (algorithms) prepare dishes and multiple judges (datasets) score them. Each judge ranks the dishes (1 is best, 2 next, and so on). After collecting all judges' rankings, you want to know whether the chefs truly differ in skill or are all about the same. The Friedman test is like an impartial statistician who analyzes all the rankings and decides whether the differences among chefs are too large to be due to chance. If he says "there is a difference", you still need to see which chefs differ significantly. The Nemenyi test is that follow-up referee: he hands you a ruler for the "minimum significant difference" (CD), and if two chefs' rank gap exceeds this ruler, they are considered genuinely different in skill.
What Problem Does It Solve
When comparing two algorithms we can use a t-test, the Wilcoxon signed-rank test, or the McNemar test. But with more than two algorithms, performing all pairwise comparisons directly creates a multiple-comparison problem that inflates the type I error (false positives). The Friedman test is an omnibus test: it judges at once whether any difference exists among all algorithms, controlling the overall type I error. If the omnibus test is significant, a post-hoc test (e.g. Nemenyi) then performs pairwise comparisons with critical values adjusted for multiple comparisons. This both increases power and guarantees reliability. Moreover, because the Friedman test uses ranks rather than raw performance values, it makes fewer distributional assumptions, is insensitive to outliers, and suits cases where the performance scale differs across datasets (e.g. one dataset has accuracy around 0.7 and another around 0.9, where comparing raw values directly may be inappropriate, but ranks are unified).
Why It Matters
In machine learning research we often need to compare the performance of multiple algorithms or models. Simply ranking by average accuracy can be misleading, because performance differences may arise merely from random variation across datasets. The Friedman test provides a rigorous statistical framework for assessing whether such differences are significant, and has become a widely accepted standard in the machine learning community. Many top-tier conferences and journals require the Friedman test and Nemenyi post-hoc test when reporting comparisons of multiple algorithms. This method applies not only to comparing classifiers but also to regression, clustering, and other tasks, as long as a sensible rank can be assigned to each algorithm on every dataset.
Application Scenarios
- Algorithm Benchmarking:Comparing multiple classifiers (e.g. SVM, random forest, neural networks) on several benchmark datasets;
- Model Selection and Hyperparameter Tuning:Comparing the performance of different hyperparameter configurations or model architectures;
- Feature Selection Method Evaluation:Comparing the effect of different feature subsets on classifier performance;
- Data Preprocessing Strategy Comparison:Comparing different data cleaning, normalization, and augmentation methods;
- Domain Application Research:Comparing the effectiveness of various methods in medical image analysis, natural language processing, recommender systems, and other fields.
Limitations and Drawbacks
- Loss of Rank Information:Using only ranks ignores the actual magnitude of performance gaps and may reduce test power;
- Assumes Independent Ranks:Ranks on each dataset should be generated independently; if datasets are correlated (e.g. from the same source), test accuracy may be affected;
- Conservativeness of the Nemenyi Test:The Nemenyi test's critical difference is based on all pairwise comparisons; with many algorithms it can be overly conservative, making significant differences hard to detect;
- Sensitivity to Ties:When several algorithms perform similarly, how ties are handled affects the statistic;
- Small-sample Problem: When the number of datasets N is small, the chi-square approximation may be inaccurate; use the exact distribution or an F-distribution correction;
- Ignores Dataset Difficulty Differences:All datasets receive equal weight, but in practice some datasets are more representative or harder; weighted ranks may be considered.