Bayesian Versus Frequentist Approaches in Multilevel Single-Case Designs: On Type I Error Rate and Power

Authors

  • Cristina Rodríguez-Prada Orcid
    Department of Social Psychology and Methodology, School of Psychology, Universidad Autónoma de Madrid, Madrid, Spain
  • José Ángel Martínez-Huertas Orcid
    Department of Methodology of Behavioral Sciences, School of Psychology, Universidad Nacional de Educación a Distancia, Madrid, Spain
  • Ricardo Olmos Orcid
    Department of Social Psychology and Methodology, School of Psychology, Universidad Autónoma de Madrid, Madrid, Spain

Abstract

Single-case designs (SCEDs) assess intervention effects through repeated measurements on one or a few individuals. Multilevel models nest repeated measures within individuals and have gained popularity for inferential analysis in SCEDs, in combination with expert knowledge of the clinicians and applied researchers. However, researchers often face model specification challenges without knowing the true population model underlying their data. This study evaluates how model selection criteria (AIC, BIC, WAIC, LOO) conditioned on the selected model impact statistical power and Type I error rates in intervention effects, reflecting the ecological reality where practitioners do not know the true model. A Monte Carlo simulation modelled data of AB designs varying sample size, measurement points, intervention effects, and random effect structures. Competing multilevel models were then fitted and compared using AIC, BIC, WAIC, and LOO to examine the impact of model selection on statistical power and Type I error rates. Results indicated that frequentist criteria performed well in simpler models in terms of power, while Bayesian approaches showed greater robustness with respect to Type I error control. The findings provide practical insights on multilevel model selection under real-world conditions, highlighting Bayesian methods as a robust alternative for applied researchers handling small sample sizes and complex data structures.