Logical contradictions in the One-way ANOVA and Tukey-Kramer multiple
comparisons tests with more than two groups of observations
We show that the One-way ANOVA and Tukey-Kramer (TK) tests agree on any sample with two groups. This result is based on a simple identity connecting the Fisher-Snedecor and studentized probabilistic distributions and is proven without any additional assumptions; in particular, the standard ANOVA assumptions (independence, normality, and homoscedasticity (INAH)) are not needed. In contrast, it is known that for a sample with k > 2 groups of observations, even under the INAH assumptions, with the same significance level , the above two tests may give opposite results: (i) ANOVA rejects its null hypothesis , while the TK one, , is not rejected for any pair ; (ii) the TK test rejects for a pair (with ) while ANOVA does not reject . We construct two large infinite pseudo-random families of samples of both types satisfying INAH: in case (i) for any and in case (ii) for some larger . Furthermore, in case (ii) ANOVA, being restricted to the pair of groups , may reject equality with the same . This is an obvious contradiction, since implies for all Similar contradictory examples are constructed for the Multivariable Linear Regression (MLR). However, for these constructions it seems difficult to verify the Gauss-Markov assumptions, which are standardly required for MLR.
View on arXiv