Post-Hoc Tests in One-Way ANOVA: The Case for Normal Distribution

Authors

  • Joel Juarros-Basterretxea Orcid
    Departamento de Psicología y Sociología, Universidad de Zaragoza, Zaragoza, Spain
  • Gema Aonso-Diego Orcid
    Departamento de Psicología, Universidad de Oviedo, Oviedo, Spain
  • Álvaro Postigo Orcid
    Departamento de Psicología, Universidad de Oviedo, Oviedo, Spain
  • Pelayo Montes-Álvarez Orcid
    Departamento de Psicología, Universidad de Oviedo, Oviedo, Spain
  • Álvaro Menéndez-Aller Orcid
    Departamento de Psicología, Universidad de Oviedo, Oviedo, Spain
  • Eduardo García-Cueto Orcid
    Departamento de Psicología, Universidad de Oviedo, Oviedo, Spain

Abstract

When one-way ANOVA is statistically significant, a multiple comparison problem arises, hence post-hoc tests are needed to elucidate between which groups significant differences are found. Different post-hoc tests have been proposed for each situation regarding heteroscedasticity and sample size groups. This study aims to compare the Type I error (α) rate of 10 post-hoc tests in four different conditions based on heteroscedasticity and balance between-group sample size. A Montecarlo simulation study was carried out on a total of 28 data sets, with 10,000 resamples in each, distributed through four conditions. One-way ANOVA tests and post-hoc tests were conducted to estimate the α rate at a 95% confidence level. The percentage of times the null hypothesis was falsely refused is used to compare the tests. Three out of four conditions demonstrated considerable variability among sample sizes. However, the best post-hoc test in the second condition (heteroscedastic and balance group) did not depend on simple size. In some cases, inappropriate post-hoc tests were more accurate. Homoscedasticity and balance between-group sample size should be considered for appropriate post-hoc test selection.