^{*}

^{a}

^{b}

Previous research applying multilevel models to single-case data has made a critical assumption that the level-1 error covariance matrix is constant across all participants. However, the level-1 error covariance matrix may differ across participants and ignoring these differences can have an impact on estimation and inferences. Despite the importance of this issue, the effects of modeling between-case variation in the level-1 error structure had not yet been systematically studied. The purpose of this simulation study was to identify the consequences of modeling and not modeling between-case variation in the level-1 error covariance matrices in single-case studies, using Bayesian estimation. The results of this study found that variance estimation was more sensitive to the method used to model the level-1 error structure than fixed effect estimation, with fixed effects only being impacted in the most extreme heterogeneity conditions. Implications for applied single-case researchers and methodologists are discussed.

Single-case designs facilitate the study of intervention effects at the level of the individual by collecting for each individual repeated observations in each of at least two conditions (e.g., baseline and treatment). The most common type of single-case design is the multiple-baseline design (

The application of statistical models to single-case data requires researchers to make assumptions about the errors in the statistical model. It is difficult to assume independence, because the unmeasured factors that give rise to the errors may impact multiple adjacent observations, such as when a child’s illness impacts their behavior several days in a row. When the errors that are closer in time are more similar than errors further apart in time, there is some serial dependency or autocorrelation among the errors. Several autoregressive and moving average models have been considered for the dependency among errors in single-case regression models, such as unstructured, compound symmetry, banded toeplitz or moving average, first-order autoregressive (AR[1]) or independent (σ^{2}I; _{t} = ρ _{t-1} + _{t};

If positive autocorrelation is ignored, the regression coefficients of single-level models are unbiased, but the standard errors of the regression coefficients would be underestimated (

When single-case studies involve multiple cases, like in the multiple-baseline design, multilevel modeling (MLM) has been suggested as a method for analyzing the data from multiple cases. MLM allows for possible dependency of the errors to be taken into account (

The following Equations (1) and (2) are for a two-level model for single-case studies.

Level-1 equation:

Level-2 equation:

where _{ij}_{ij}_{0j} is the baseline intercept for the _{1j} is the difference between the baseline level and the treatment level (shift in level) for the _{2j} is the baseline slope for the _{ij}_{e}_{e}_{00} is the average baseline intercept, θ_{10} is the average shift in level indexed at the time of the first treatment observation, θ_{20} is the average baseline slope, and θ_{30} is the average shift in slope. The level-2 errors, _{0j}, _{1j}, _{2j} and _{3j} are assumed to be multivariate normally distributed _{u}

The effect of ignoring autocorrelation and other misspecifications of the level-1 error structure have been examined for MLM analyses for single-case data. Researchers found that ignoring autocorrelation does not bias the fixed effect estimates, but the inferences about the fixed effects can be inaccurate due to the underestimate of the corresponding standard errors, and the estimates of the variance parameters become more biased (

Although MLM allows autocorrelation among level-1 errors to be taken into consideration in single-case data analyses, this approach holds a critical assumption that the level-1 error structure is the same for all cases. Specifically, it is assumed that (a) autocorrelation is the same for all cases and (b) the level-1 error variance is the same for all cases. Previous single-case research using MLM application as well as misspecification research of level-1 error structures has often assumed the autocorrelation and level-1 error variance to be equal for all cases (e.g.,

However, it is possible that the level-1 error structure may not be homogeneous across cases. For example, the behavior of one child with an emotional/behavioral disorder may vary more substantially from day to day because of intermittent problems in the home, or lapses in medication provision, than that of another child with the same disorder. The findings from previous studies in single-case data support that variations in level-1 error covariance matrices could exist (

Because in some studies the cases appear to have different variances, but commonly adopted models assume homogeneity, it is critical to examine the consequences of modeling (and not modeling) the heterogeneity. Thus, the purpose of this study is to extend the MLM modeling in single-case design to allow between case variation in the level-1 error structure, and to identify the consequences of different modeling methods for the level-1 error structure.

The two level model that allows between case variation in the level-1 error structure in single-case design can still be represented by _{e}_{ej}

_{e}_{.} Assume that there are single-case data with three cases and the AR(1) structure is assumed for the covariance structure ∑_{e}_{e}^{2}_{ej}

Restricted maximum likelihood (REML) estimation is the most commonly used method to analyze multilevel models, and has been implemented by several software procedures that allow easy access. However, REML may encounter estimation problems, such as non-convergence, when analyzing complex multilevel models of SCED data. The Bayesian approach provides a feasible option in computationally intensive scenarios through the use of Markov Chain Monte Carlo (MCMC) procedures (e.g.,

Bayesian inference is the process of fitting a probability model, given the observed data, and summarizing the uncertainty of the model parameters with probability distributions (

A Monte Carlo simulation study was conducted to examine the performance of the proposed Bayesian analysis of single-case design data, which allows between case variation in the level-1 error variances and autocorrelation. Specifically, two level models where the level-1 error structures were modeled in different ways (i.e., not modeling between case variation vs. modeling between case variation) were examined in terms of the accuracy of the estimates of the parameters using Bayesian estimation. In this study, multiple baseline single-case data were generated using _{0j}, _{1j}, _{2j}, and _{3j}, were generated independently from normal distributions with mean 0 and variances of either 0.5, 0.5, 0.05, and 0.05, or 2, 2, 0.2, and 0.2, respectively.

The simulation conditions varied in series length (10 or 20) and the number of cases (4 or 8), to cover the sample sizes that are typical in single case studies. In addition, conditions varied in the level-2 error variance (more or less between case variability), and the method used to analyze the data, either allowing level-1 error structure to vary across cases (Model 2, proposed model) or holding the level-1 error variance and autocorrelation constant across cases (Model 1). For each condition, 1000 data sets were simulated, and parameter relative bias, RMSE and credible interval (CI) coverage and width were estimated. The two different models (Model 1 and Model 2) were used to analyze each generated data set using OpenBUGS software. Although various software programs are available to run Bayesian estimation including SAS, OpenBUGS software was selected due to several advantages, including flexibility to handle the complexity of the models, and possible computational efficiency from using the Gibbs sampling method (

In Bayesian estimation, various choices of priors can be assumed for each parameter. A noninformative prior distribution is considered as one of the reasonable choices of objective prior distribution because noninformative distributions make the data speak for themselves so that posterior inferences are unaffected by external information (_{00}, θ_{10}, θ_{20}, θ_{30}), including the noninformative normal distribution (^{2}), so that posterior inference is not influenced by the choice of variance value. Thus, in this study, all fixed effect parameters were assumed to follow noninformative normal distributions (^{2}).

Unlike priors for fixed effect parameters, priors for random effect parameters have been more difficult to construct. The choice of noninformative prior distribution for the level-2 error variances (i.e.,

Various noninformative and weakly informative prior distributions have been suggested for the variance parameters in multilevel models, including uniform, inverse-gamma family, inverse-Wishart, half-Cauchy, and half-t distributions (

Similarly, a noninformative prior for ρ that follows a normal distribution with

For the proposed model, priors for the level-1 error

The prior for _{σ} and the upper limit of _{σ}. The _{σ} and _{σ} were further assumed to follow a uniform distribution. In addition, the prior for ρ_{j} was assumed to follow the normal distribution with a mean of μ_{ρ} and a variance of _{ρ} and σ_{ρ} were further defined as a normal distribution and a uniform distribution, respectively.

A data set per each condition of the design factors (24 conditions) was first generated to test convergence and to make decisions about the number of iterations, and the burn-in period. Based on the test results, it was decided to use a burn-in of 2,000 iterations and to run an additional 500,000 iterations, but to use only 50,000 samples of the 500,000 iterations after thinning to form the posterior distribution for the main analyses. Various diagnostic criteria were used in monitoring convergence, including trace plots, history plots, Kernel density plots, and Brooks–Gelman–Rubin (BGR) plots for the simulated data sets using two different MCMC chains. No signs for non-convergence were found.

The relative bias values for treatment effects (shift in level and shift in slope) were compared across the two models. For the treatment effects, the average relative bias values were very minimal and RMSE values were comparable for both models. The average relative bias values were smaller than 5% for both treatment parameters (Model 1: shift in level = .001, shift in slope = .017; Model 2: shift in level < .001, shift in slope = .015) which are considered as minimal, according to

The average relative bias values of the variance components across the models are provided in ^{2} values) and identify factors with a medium effect of .06 or larger. The interaction between the true level-1 error structure and the type of model accounted for substantial variation in the average bias (η^{2} = .10) and the average RMSE (η^{2} = .16) of the level-1 error

Outcome | Level-2 error |
Level-1 error |
Autocorrelation |
|||||
---|---|---|---|---|---|---|---|---|

Shift in level |
Shift in slope |
|||||||

Model 1 | Model 2 | Model 1 | Model 2 | Model 1 | Model 2 | Model 1 | Model 2 | |

Bias | 0.89 | 0.88 | 0.99 | 0.97 | 0.10 | 0.06 | -0.60 | -0.48 |

RMSE | 1.23 | 1.21 | 0.42 | 0.41 | 0.22 | 0.18 | 0.26 | 0.25 |

CI coverage | 0.97 | 0.97 | 0.97 | 0.98 | 0.85 | 0.97 | 0.81 | 0.94 |

CI width | 6.43 | 6.44 | 1.62 | 2.16 | 0.47 | 0.81 | 0.74 | 1.10 |

The average CI coverage values of the level-2 error ^{2} = .19, .07, respectively). As illustrated in

In the main study, data having the heterogeneous level-1 error structure had been generated in a way that every case had a unique value of the level-1 error

Factor | Level |
---|---|

Autocorrelation | .2 and .4 or .4 and .2 |

Combination of number of cases and series length per case | 4 and 10 or 8 and 20 |

Method to model the level-1 error structure | Not modeling between case variation (Model 1) or Modeling between case variation (Model 2) |

The results of all of the simulated conditions for the fixed treatment effects and variance components are provided in

Parameter | Bias |
RMSE |
CI coverage |
CI width |
||||
---|---|---|---|---|---|---|---|---|

Model 1 | Model 2 | Model 1 | Model 2 | Model 1 | Model 2 | Model 1 | Model 2 | |

Fixed treatment effects | ||||||||

Shift in level | < -0.01 | < -0.01 | 1.01 | 0.73 | > 0.99 | 0.99 | 7.11 | 5.80 |

Shift in slope | 0.02 | 0.01 | 0.39 | 0.26 | 0.98 | 0.97 | 3.10 | 2.47 |

Variance components | ||||||||

Level-2 error SD | ||||||||

Shift in level | 2.31 | 1.80 | 2.12 | 1.55 | 0.93 | 0.98 | 9.08 | 7.70 |

Shift in slope | 3.74 | 2.67 | 1.09 | 0.74 | 0.91 | 0.98 | 1.68 | 3.42 |

Level-1 error SD | 0.60 | 0.10 | 1.24 | 0.51 | 0.03 | 0.93 | 0.85 | 1.53 |

Autocorrelation | -1.06 | -0.87 | 0.41 | 0.33 | 0.50 | 0.84 | 0.67 | 1.10 |

A published single-case study (

The raw data were extracted from the graphs presented in the article using DataThief III (

The results for Model 1 and Model 2 are compared in _{10}). The treatment effect for the change in trend was -0.31 for Model 1 and -0.69 for the proposed model (θ_{20}). The variance components were also different across the two models. The level-2 error SDs were higher for the proposed model than Model 1. The

Parameter | Model 1 | Model 2 |
---|---|---|

Variance components | ||

Intercept | 14.24 | 12.88 |

Phase | 11.73 | 14.85 |

Time | 0.85 | 1.08 |

Interaction | 1.00 | 1.07 |

Variance | 13.51 | 14.00 |

AR(1) | 0.34 | 0.29 |

Variance (Person 1) | - | 10.66 |

AR(1) | - | 0.13 |

Variance (Person 2) | - | 11.56 |

AR(1) | - | 0.36 |

Variance (Person 3) | - | 13.07 |

AR(1) | - | 0.30 |

Variance (Person 4) | - | 11.35 |

AR(1) | - | 0.32 |

Variance (Person 5) | - | 17.23 |

AR(1) | - | 0.34 |

Fixed effects | ||

Intercept | 27.59 | 24.78 |

Phase | 25.38 | 22.76 |

Time | 0.31 | 0.79 |

Interaction | -0.31 | -0.69 |

This study developed a method for estimating between case heterogeneity in level-1 variances and provides insight into how different modeling approaches for the level-1 error covariance matrices impacts statistical inferences. The results of this study indicate that the different modeling methods in the level-1 error structure can have an impact on the variance components, in some cases, both fixed effects and variance components. This finding is particularly distinguished from previous works that have investigated other misspecifications of the level-1 error structure. The previous studies have found that the fixed effects are generally robust to misspecifications of the level-1 error structure, however, this study found that the misspecification of the level-1 error structure can have an impact on fixed effects when analyzing data that show one or more cases that have substantially different variability than the others. In addition, this study suggests that accuracy and precision of the variance components can be improved by modeling between case variation in the level-1 error structure, and the effectiveness of modeling between case variation increased as the degree of the heterogeneity in the data increased.

The results of this study lead to various implications for applied single-case researchers, as well as for methodologists. Single-case researchers can feel comfortable interpreting the overall treatment effects when they have data that show no or a moderate degree of between case variation in the within-case variance, regardless of whether between-case heterogeneity has been explicitly modeled. However, when the heterogeneity becomes more severe, and particularly when one case has variance that is very different from the others (as in our follow-up study), researchers are encouraged to model the variation in the level-1 error variances. Doing so can lead to less error in the estimate of the average treatment effect, and less bias in the variance estimates. To facilitate modeling between case variation in the level-1 error variances, this study has developed a Bayesian model and corresponding OpenBUGS code, which is accessible to applied researchers for use in their own research (see Supplementary Materials).

This study also provides a few implications for methodologists who study the use of multilevel modeling to conduct single-case data analyses. Although this study was helpful in providing some initial guidance about the use of multilevel modeling for single-case data when there are differences in the within-case variation, this Monte Carlo study was limited by the conditions that were examined. The conditions were chosen based on a review of single-case literature and applied studies that used two-level models to analyze single-case data, but only some of the possible options were included in this study. More simulation work can be done to document the impact of various modeling options across a broader range of heterogeneous data conditions, including those that involve more complex dependent error structures than AR(1). In addition, it is not clear the degree to which biases may be reduced through the choice of alternative priors or may be less pronounced in contexts where simpler models without trends are appropriate. By contrasting conditions and models with and without trends, along with estimation incorporating a wider range of prior distributions, and in particular more informative priors for the random components, future research could make it possible to further refine modeling advice.

The authors have no funding to report.

The authors have declared that no competing interests exist.

This study is initiated based on the first author’s dissertation (

The authors have no support to report.