A Monte Carlo simulation of an SCED AB-design with baseline (A) and intervention (B) phases was conducted. The dependent variables included statistical power, Type I error rate, and proportion of true model selection per information criteria. Statistical power indicates the rate of correctly detecting the intervention effect, while the Type I error rate measures the incorrect identification of an effect when none exists. Correct model selection quantifies how often the information criteria identified the true population model during comparison.

SCEDs data were generated based on four models varying in the number of random effects (covariance parameters). The models were:

where Yij is the response variable for individual j at observation i, γ00 represents baseline mean (intercept), γ10 is the average intervention effect (slope), eij is the Level-1 residual error for individual j at observation i and conditionij is a dummy-coded predictor indicating whether observation i for individual j belongs to the baseline phase (0) or the intervention phase (1). This model does not include random effects.

where u0j is the random intercept in the baseline for individual j.

where u1j is the random slope for individual j.

where u0j and u1j are random intercepts and slopes for individual j, respectively.

In all the population models, fixed effects were set at γ10=5, random effects followed normal distributions u0∼N0,2, u1∼N0,2, and residual variance was set to N0,1. Random effects were uncorrelated (r = 0) to minimise model complexity, as suggested by Van den Noortgate and Onghena (2003a) and Moeyaert et al. (2017). In addition, residuals were assumed to be independent over time, and no autocorrelation or secular nor temporal trends were simulated.

The simulation comprised 144 scenarios from various factor combinations:

In each scenario, four model information criteria were evaluated: frequentist criteria (AIC, BIC) and Bayesian criteria (WAIC, LOO), allowing a comparative analysis of selection approaches. Seven different priors were tested to assess their effects on statistical power, Type I error rates for the intervention effect, and the accuracy of model selection using WAIC and LOO. In line with the default parameterisation in brms, weakly informative priors (see Gelman, 2006; McElreath, 2020) were specified on the standard deviations of the random effects (σᵤ₀ and σᵤ₁), not on the variance components (σ²ᵤ₀ and σ²ᵤ₁). Because standard deviations are constrained to be positive, Half-Cauchy and Half-normal priors with scale parameters of 10, 20, and 50 were used, alongside a weakly informative Uniform (0, 100) prior. Following the approach outlined by Moeyaert et al. (2017), who derived these values from reanalyses of empirical SCED studies with normally distributed continuous outcomes, our aim was to reflect plausible ranges for variance components in this context. This approach allows the priors to regularize estimation in small-sample settings while still letting the normally distributed outcome data contribute substantially to the results. Increasing the scale parameter in the Half-Cauchy and Half-Normal distributions makes the prior less informative by allowing for greater variability in the variance components. A smaller scale concentrates the prior mass closer to zero, while a larger scale spreads the distribution and assigns more weight to larger variance values, thus reducing the prior's influence. We adopted this approach because one of the main challenges in this context is selecting the appropriate prior distribution. Although future studies should explore this issue in greater depth, our current aim is to increase variability so that the prior parameterization remains sensible while still allowing the data to provide valuable insight.

Five hundred replications were conducted for each of the 144 simulation conditions in R. Frequentist models used the lme4 package (Bates et al., 2015) for models with random effects and nlme package (Pinheiro et al., 2021) for the minimal model, always using REML as the estimation method and Satterthwaite’s approximation to the denominator degrees of freedom (via lmerTest). Bayesian estimation utilised the brms package (Bürkner, 2017), employing the Hamiltonian Monte Carlo (HMC) algorithm with its NUTS extension for efficient sampling of complex models (Hoffman & Gelman, 2014). Bayesian models were fitted using two chains of 1,000 iterations each, with 400 warm-up iterations per chain, yielding a total of 1,200 post–warm-up draws per model. The adapt_delta parameter was set to 0.95 to reduce divergent transitions and enhance sampling reliability. Extracted data included point estimates, standard deviations, p values (frequentist), and posterior means with 95% credible intervals (Bayesian). Information Criteria fit indices (AIC, BIC, WAIC, LOO) and convergence diagnostics were also extracted (R^, with R^<1.1 indicating satisfactory convergence). According to this criterion, approximately 99% of the replicas showed appropriate convergence. The convergence dataset is available on our OSF project (see Rodríguez-Prada et al., 2026a).

Data processing and analysis were performed in R using RStudio. Power and Type I error rates were calculated based on intervention effects. Frequentist decisions used a significance threshold (p < 0.05), while Bayesian ones relied on 95% credible intervals (CIs). Model selection was assessed using AIC, BIC, WAIC, and LOO by comparing each candidate model (minimal, partial intercepts, partial slopes, maximal) to the population model. In addition, analysis of variance (ANOVA) was conducted to examine the influence of simulation parameters on performance indices, and partial eta-squared (ηp2 ) was reported as a measure of effect size. Partial eta-squared was used as a relative indicator of the impact of simulation conditions. Effect size magnitudes were not interpreted using conventional cut-off values, as the aim was not to compare absolute effect sizes with previous studies but to examine differences across simulation conditions. Partial eta-squared was selected over eta-squared because it provides an unbiased estimate of the unique variance explained by each factor in multifactorial designs (Richardson, 2011). Consistent with prior methodological simulation studies (e.g., Moeyaert et al., 2017), cut-off points of .01, .06, and .14 were adopted to classify small, medium, and large effects, respectively (Cohen, 1988). This approach allowed us to identify the most influential simulation parameters beyond statistical significance. The dataset, prepared for the dissemination of scientific data, is available on the Open Science Framework at Rodríguez-Prada et al. (2026a). This repository contains the scripts for data processing and analysis, which are also available at Rodríguez-Prada (2025).

Seven Bayesian priors were evaluated for estimating variance components across four population multilevel models under the simulated conditions (Table S1, Rodríguez-Prada et al., 2026b). Minimal differences were observed in statistical power among the priors (F6, 215136= 0.76, p<.001,ηp2=0.002), Type I error rates (F(6, 107568) = 0.34, p = 0.916, ηp2 = < .001), and model selection accuracy (F(6, 428638) = 1.50, p = 0.174; ηp2 = < .001). Statistical power remained moderate (≈ 0.62), Type I error rates were conservative (≈ 0.033), and WAIC and LOO correctly identified the population model in 82% of cases. The truncated Cauchy prior (Half−Cauchy∼0,10 was selected for further analyses due to slightly higher power and proximity to the nominal Type I error rate of 0.05, reflecting its suitability for this simulation context.

Correct model selection rates for frequentist (AIC, BIC) and Bayesian (WAIC, LOO) indices showed minimal overall differences (F(1.64, 117212) = 1557.99, p < .001; ηp2 = 0.007). However, there was a significant interaction between the index and population model (F(4.93, 117212.53) = 2883.85, p < .001; ηp2 = .11). Marginal means (Table S2, Rodríguez-Prada et al., 2026b) indicate that WAIC and LOO excel at identifying the maximal model but perform worse with simpler models, selecting them 76%–82% of the time. In contrast, frequentist indices favour simpler models: BIC overwhelmingly selects the minimal model (98%), while AIC demonstrates a more balanced across all population models (Figure 1). Although the interaction effects of the information criteria with both the number of individuals (F(3.28, 117212.53) = 213.47; p < 0.001; ηp2 = 0.002) and repeated measures (F(4.93, 117212.53) = 88.85; p < 0.001; ηp2 = 0.001) were small, they reveal noteworthy patterns. More individuals and repeated measures improve correct model selection. AIC and LOO are less sensitive to these factors, while frequentist indices, especially BIC, show greater variability. Overall, Figure 1 shows that BIC and AIC consistently outperform WAIC and LOO in simpler structures (minimal and partial–intercepts models), whereas their performance decreases when slope variability is included in the model. In contrast, WAIC and LOO provide relatively stable, though lower, accuracy across models. Thus, the results suggest that the choice of information criterion has important implications: BIC and AIC tend to favor parsimony, whereas WAIC and LOO yield more balanced but less accurate selections.

A detailed analysis of errors in BIC, AIC, WAIC, and LOO (i.e., the model selected when the fit index fails to identify the correct one) reveals distinct error patterns. BIC tends to overly penalise complex models and, when incorrect, selects one of the simpler models (minimal or partial-intercepts) in 77.8% of the occasions. In contrast, WAIC and LOO often err by selecting the maximal model nearly half the time. AIC shows the most balanced error distribution, although it slightly prefers the random-intercepts model (Figure 2). When AIC and BIC misidentify the true model, the most frequent error is selecting simpler models (e.g., BIC favours partial-intercepts at 47.8%). In contrast, WAIC and LOO tend to misclassify more complex models, with nearly half of errors involving the maximal specification (46.3% for LOO and 47.9% for WAIC). These results suggest that AIC and BIC are biased toward parsimony, whereas WAIC and LOO are more likely to overfit, highlighting systematic tendencies in model misclassification.

Power and Type I error rates conditioned on model selection were analyzed, simulating scenarios in which the true population model is unknown, as happens in ecological contexts.

On power, ANOVA results showed a significant main effect of the fit index (F(1.27, 60132.31) = 2702.46; p < 0.001; ηp2 = 0.05), with BIC yielding the highest power, followed by AIC, WAIC and LOO (Table S3, Rodríguez-Prada et al., 2026b). However, for BIC, this increase in power was accompanied by substantially inflated Type I error rates in more complex models. Significant effects were also observed for population model (F(3, 47482) = 11393.98; p < 0.001; ηp2 = 0.42), effect size of the intervention (F(1, 47482) = 9713.90; p < 0.001; ηp2 = 0.17), and number of individuals (F(2, 47482) = 2782.18; p < 0.001; ηp2 = 0.10). Larger effects and sample sizes enhance power, whereas higher error variance in models, as in maximal models, diminishes it (Table S4, Rodríguez-Prada et al., 2026b).

The interaction between the population model and effect size (F(3, 47482) = 2309.91; p < 0.001; ηp2 = 0.108) showed stable statistical power with large effect sizes across models. However, there were significant declines for moderate effects, particularly in Bayesian methods, reflecting their conservative nature (Figure 3; Table S4, Rodríguez-Prada et al., 2026b). Overall, across all criteria, statistical power is high and stable for simpler models (minimal and partial–intercepts) but declines sharply when slope variability and maximal structures are introduced. The drop is most pronounced for smaller effects (ES = 1.15), where power often falls below .40, particularly for LOO and WAIC. Larger effects (ES = 2.70) mitigate but do not eliminate this decline. These results indicate that model complexity disproportionately reduces power for detecting smaller effects, with Bayesian indices (LOO, WAIC) being more sensitive to this loss than frequentist ones (AIC, BIC).

The model fit index, conditioned on the selected model, significantly impacts the Type I error rate (F(1.49, 35636.95) = 511.52, p < .001, ηp2 = .02). Although the effect size is small, meaningful differences emerge among indices. Frequentist indices tend to exceed the nominal Type I error rate, while the Bayesian indices demonstrate a more conservative behaviour, staying closer to the nominal value of 0.05 (Table S5, Rodríguez-Prada et al., 2026b). The interaction between population model and model fit indices is statistically significant but small (F(4.47, 35636.95) = 165.28, p < .001, ηp2  = .02). Examining marginal means (Table S5, Rodríguez-Prada et al., 2026b), the BIC index shows an unacceptable Type I error rate for complex models, reaching 13% for the maximal model. The AIC index is less sensitive than BIC, with a Type I error rate of 8.9% for the maximal model. WAIC and LOO remain stable across all population models, aligning closely with the nominal level in complex models (Figure 4). Thus, error rates remain close to the nominal .05 level for minimal and partial–intercepts models across all criteria. However, as complexity increases, AIC and BIC show inflated error rates, particularly under maximal specifications (AIC ≈ .13, BIC ≈ .09). By contrast, Bayesian fit indices (WAIC and LOO) maintain more stable control of Type I error, even in complex models. These findings suggest that AIC and BIC may increase the risk of false positives when applied to highly parameterized structures, whereas WAIC and LOO provide more conservative performance.

In summary, BIC exhibited the highest values for both statistical power and the Type I error rate. WAIC and LOO yielded the lowest values, remaining close to but not exceeding the nominal 0.05 level for false positives. AIC demonstrated a balanced performance between statistical power and the Type I error rate. Interaction effects highlighted the influence of between-subject factors on both power and the Type I error rate, particularly the intervention effect size, sample size, and population model for power; and only the interaction with the population model for the Type I error rate.

The present simulation study evaluated the operating characteristics of Bayesian and frequentist methods in MLMs applied to SCEDs when the true population model is unknown. Findings are specific to simulated conditions and need empirical validation. Our findings should be interpreted conditional on the simulated data-generating mechanisms: an AB design with normally distributed outcomes and no explicit autocorrelation or phase-specific trends, nor cross-level interactions. In applied SCEDs, however, outcomes are often counts and time-related structure is common. These features can change the effective model complexity and the penalty-fit trade-off, potentially altering the relative behavior of AIC/BIC versus WAIC/LOO and frequentist versus Bayesian estimations. This is consistent with recent work extending multilevel SCED models to GLMM settings and more realistic covariance structures (e.g., Li et al., 2024, 2025a, 2025b), and motivates future simulations that jointly manipulate outcome distribution, trends, and autocorrelation to evaluate whether the performance patterns reported here generalize to these more ecological scenarios. These decisions limit the scope of our findings, since model complexity of MLM applications often arises precisely from elements such as trends (general or phase-specific) or autocorrelation (Natesan Batley & Hedges, 2021). We deliberately opted for a simple design to keep the conditions more controlled and to examine the performance of model selection criteria in a focused way, particularly regarding random effects and the detection of random slopes. Including both baseline and intervention trends would have substantially increased the complexity of the simulation, making it more difficult to isolate the behaviour of the information. Future research should explore the behavior of information criteria on model selection in more complex designs (e.g., multiple baseline or reversal designs) and advanced covariance structures, such as autoregressive terms, to better reflect real-world data. Using dependent variables that follow normal distributions, uncommon in SCEDs where ceiling and floor effects and heteroscedasticity tend to feature, also limits generalizability. Future studies should investigate the behavior of both frequentist and Bayesian approaches within generalized linear multilevel models (GLMMs). By incorporating variables with asymmetric distributions, such as Poisson distributions, researchers can address this gap and explore small effect sizes that are frequently overlooked in qualitative analyses. Recent research has demonstrated that linear mixed models can be adapted to handle count outcomes in SCEDs, leading to more accurate estimates, particularly in cases of overdispersion or small sample sizes (Declercq et al., 2019). More recently, Li and colleagues have shown how GLMMs can be effectively applied to various count data scenarios, including zero-inflated and overdispersed distributions (Li, 2024; Li et al., 2025b; Li et al., 2024). They have also provided step-by-step tutorials designed for applied researchers (see Li et al., 2025a). These contributions emphasize the importance of moving beyond normally distributed outcomes to enhance the ecological validity and robustness of statistical inferences in SCED data.

Regarding the Bayesian fit indices, WAIC and LOO, both derived from the conditional likelihood, were chosen for their accessibility and compatibility with the evaluated models. However, future work could benefit from a broader exploration of Bayesian model comparison strategies. Greater emphasis should be placed on marginal rather than conditional likelihood-based approaches, which some authors argue are more suitable for model comparison in Bayesian frameworks (Ariyo et al., 2022; Merkle et al., 2019). Additionally, recent methodological advances demonstrate the growing applicability of Bayes factors within different SCEDs, such as ABAB, alternating treatments and changing criterion designs (Yamada & Okada, 2024, 2025). Incorporating Bayes factors in future analyses could offer complementary insights into model selection decisions. The best results of these information criteria were observed under optimal conditions: more individuals, repeated measures, and larger effect sizes, consistent with previous research (Baek et al., 2020; Moeyaert et al., 2017). While MLMs effectively capture complex realities, their mathematical and statistical demands underscore the importance of clear model specification. Bickel (2007) observes that these models perform best when grounded in robust theories and literature. Thus, advancing MLMs should also promote theoretical development, particularly within psychology. It is important to remember that statistical decisions should never be made without considering the wider context and the insights provided by clinical expertise. An analysed effect might not reach the traditional thresholds of statistical significance but can still be socially relevant, aligning with the concept of social validity (Kazdin, 1977; Snodgrass et al., 2023), and ways of control the antecession of the intervention before the change is fundamental to reassure the efficacy of an intervention (Perone, 1999). Future research should investigate the contingency relations between the decisions proposed by these models and those made by applied researchers, to understand how they are related.

MLM is a particularly versatile option for analysing SCED data, offering general and individual effect quantification, statistical testing, and the ability to model complex effects. This study highlights the strengths of Bayesian methods for highly complex models, where they maintain stable power and Type I error rates across varying effect sizes, sample sizes, and repeated measures. For simpler models, the frequentist REML approach performs equally well or better, particularly when AIC is used for model selection. When protection against Type I errors is paramount, as in assessing therapeutic change, Bayesian methods provide a reliable and robust framework, making them a valuable tool for applied researchers in SCEDs. Given that the data-generating model is typically unknown in applied SCEDs, the practical contribution of this study is to characterise the pros and cons of common model selection fit indices. Under our simulation conditions, REML estimations with AIC offered the most favourable balance between power and Type I error rates in simpler population models. But Bayesian estimations provided a more stable positive performance when model complexity increased. Thus, our recommendation is to treat model selection as risk management under uncertainty, triangulating across plausible random-effects structures, and using sensitivity analyses across AIC/BIC and WAIC/LOO to ensure that conclusions are not criterion-dependent.