This paper examined the amount bias in standard errors for fixed effects when the random part of a multilevel model is misspecified. Study 1 examined the effects of misspecification for a model with one Level 1 predictor. Results indicated that misspecifying random slope variance as fixed had a moderate effect size on the standard errors of the fixed effects and had a greater effect than misspecifying fixed slopes as random. In Study 2, a second Level 1 predictor was added and allowed for the examination of the effects of misspecifying the slope variance of one predictor on the standard errors for the fixed effects of the other predictor. Results indicated that only the standard errors of coefficient relevant to that predictor were impacted and that the effect size for the bias could be considered moderate to large. These results suggest that researchers can use a piecemeal approach to testing multilevel models with random effects.
The use of multilevel modeling has become common in social science research because it allows researchers to analyze data that reside at different levels. Multiple levels may include employees nested within teams, students nested within classrooms, or measurements nested within individuals. A strength of multilevel modeling is the flexibility that it allows researchers in testing how variables at higher levels relate to variables at lower levels. A cost of the flexibility is that multilevel models can be complicated to specify. As such, studies have investigated the effects of misspecifying multilevel models. For example, studies have investigated the effects of ignoring (a) the heterogeneity of Level 2 variances (
In multilevel modeling, specifying the random part of the model is often the most difficult because there are rarely strong theories about which effects should vary. Proper specification of the random part of the model is important because it influences the standard errors of the fixed effects (
Despite its importance, there has been little research concerning the effects of misspecifying the random part of the model. One exception is
In the present studies, we examined the consequences of misspecifying the random part of the model on the standard errors of the fixed effects. We were interested in the effects of specifying fixed slopes as varying and varying slopes as fixed. Because closed-form expressions are not possible when there are unequal group sizes or random slopes, we used Monte Carlo simulations to assess the effects of model misspecification. In our simulations, we assessed bias by comparing standard errors for the fixed effects from a model where the slope varied to a model where the slope was fixed. Our goal was to determine which type of misspecification produced the most detrimental effects on the standard errors of the fixed effects.
A succinct way of expressing a basic two-level model where there are
In
Researchers are often advised to test for the significance of the variance components in order to determine what effects should vary. One potential problem with using significance tests for the slope variance components is that the power for these tests is low (
To understand how the misspecification of the random part of the model affects the standard errors for the fixed effects, it is helpful to consider the unrealistic case where the variance components are known, every Level 1 predictor has a random slope, and each group has sufficient size to calculate ordinary least squares (OLS) estimates. Following
where
The error variance matrix for
Finally, pre-multiplying both sides of
At Level 2, the model for β
In
Both
Substituting
The dispersion of
Finally, the dispersion matrix for the fixed effects (γ’s) based on generalized least squares is given by:
The square roots of the diagonals of
standard errors for γ00, γ10, and γ20 are simply the square roots of the diagonals of
In practice, the variance components are not known and need to be estimated with the fixed effects using full or restricted maximum likelihood estimation. Based on the above discussion, we would expect the bias in standard errors would be limited to those fixed effects that are associated with the random effect that is misspecified and that the amount of this bias should decrease as the amount of slope variance decreases. In addition, the amount of bias in standard errors should increase as group size increases. This is because, all things being equal, the elements of the error matrix
The major focus of Study 1 was to examine the effects of omitting a slope from the random part of the model. We also explored the effects of freeing a slope when it should have been fixed. We considered a simple model with only one Level 1 predictor and one Level 2 predictor. As mentioned above, specification of the random part of the model impacts the standard errors for the fixed effects. We assessed the effect of misspecification on the standard errors of three fixed effects: the cross-level effect (γ01), the slope mean for
We used a 3 (number of groups) × 3 (average group size) × 2 (cross-level interaction effect size) × 4 (slope mean) × 3 (slope – intercept correlation) × 5 (slope variance effect size) design. The values chosen for the conditions were based somewhat on the simulations conducted by
We created the Level 2 variable
To generate the
For each sample, we estimated two multilevel models using restricted maximum-likelihood estimation: one where the slope was fixed and one where the slope varied. For each model, we recorded the estimates for γ01, γ10, and γ11, as well as their standard errors. We calculated the bias in standard errors by subtracting the standard error from the data generating model from that of the misspecified model. Thus, a positive value for bias reflects the standard errors for the misspecified model was higher than the data generating model. Cohen’s
Prior to evaluating the results, we inspected how well the population parameters were uncovered by our analyses. We calculated root mean square error of approximation values (RMSEA) for each parameter. RMSEA values ranged from .02 to .03. Thus, all values were in the acceptable range and consistent with prior simulations.
Source | γ01 |
γ10 |
γ11 |
||||||
---|---|---|---|---|---|---|---|---|---|
NJ = 5 | .090 | .000 | 0.01 | .061 | -.009 | -0.52 | .062 | -.009 | -0.47 |
NJ = 10 | .079 | .000 | 0.01 | .046 | -.010 | -0.77 | .046 | -.010 | -0.72 |
NJ = 20 | .075 | .000 | 0.01 | .037 | -.012 | -1.14 | .037 | -.012 | -1.08 |
NG = 50 | .110 | .000 | 0.02 | .066 | -.014 | -0.84 | .067 | -.014 | -0.76 |
NG = 100 | .078 | .000 | 0.02 | .046 | -.010 | -0.87 | .046 | -.010 | -0.82 |
NG = 200 | .055 | .000 | 0.02 | .032 | -.007 | -0.88 | .033 | -.007 | -0.85 |
γ10 = 0.00 | .081 | .000 | 0.01 | .048 | -.010 | -0.59 | .049 | -.011 | -0.56 |
γ10 = 0.10 | .081 | .000 | 0.01 | .048 | -.010 | -0.59 | .049 | -.011 | -0.56 |
γ10 = 0.30 | .081 | .000 | 0.01 | .048 | -.010 | -0.59 | .049 | -.010 | -0.56 |
γ10 = 0.50 | .081 | .000 | 0.01 | .048 | -.010 | -0.59 | .049 | -.010 | -0.56 |
τ11 = 0.00 | .044 | .000 | -0.02 | .035 | .001 | 0.09 | .036 | .001 | 0.09 |
τ11 = 0.025 | .084 | .000 | 0.00 | .041 | -.005 | -0.28 | .042 | -.005 | -0.26 |
τ11 = 0.05 | .084 | .000 | 0.01 | .045 | -.008 | -0.47 | .046 | -.008 | -0.44 |
τ11 = 0.10 | .084 | .000 | 0.01 | .052 | -.014 | -0.78 | .052 | -.014 | -0.74 |
τ11 = 0.15 | .084 | .001 | 0.02 | .058 | -.019 | -1.03 | .058 | -.019 | -0.96 |
τ01 = 0.00 | .076 | .000 | 0.01 | .047 | -.009 | -0.53 | .047 | -.009 | -0.50 |
τ01 = -0.30 | .084 | .000 | 0.01 | .049 | -.011 | -0.64 | .050 | -.011 | -0.60 |
τ01 = -0.70 | .084 | .000 | 0.01 | .049 | -.011 | -0.62 | .049 | -.011 | -0.59 |
γ11 = 0.00 | .081 | .000 | 0.01 | .048 | -.010 | -0.59 | .049 | -.011 | -0.56 |
γ11 = 0.20 | .081 | .000 | 0.01 | .048 | -.010 | -0.59 | .049 | -.010 | -0.56 |
Misspecification of the random part of the model did influence the standard errors for γ10 and γ11. Cohen’s d’s ranged from -1.14 to 0.09 for γ10 and -1.08 to 0.09 to for γ11. The smallest values were found for the no slope variance condition, and the biggest values was found for the largest group size condition. When there was no population slope variance, estimating a model with a random slope produced a bias of .001 (Cohen’s
To get a better sense of the effects of the study conditions on bias, we conducted separate ANOVA’s investigating the impact of the study characteristics on the bias in standard errors for the γ10 and γ11. We included only the conditions where there was slope variance to produce a fully crossed design.
Source | Bias in γ10 |
Bias in γ11 |
|||
---|---|---|---|---|---|
Partial η2 | Partial η2 | ||||
Number of cases per group (NJ) | 2 | 54013.35* | .11 | 50272.89* | .10 |
Number of groups (NG) | 2 | 264087.34* | .38 | 240527.90* | .36 |
Slope Mean (SM) | 3 | 0.54 | .00 | 0.66 | .00 |
Slope variance (SV) | 3 | 521630.03* | .65 | 466481.81* | .62 |
Cross-level interaction (CINT) | 1 | 2.51 | .00 | 3.86* | .00 |
Intercept – slope correlation | 2 | 660.27* | .00 | 576.28* | .00 |
NG × SV | 6 | 18279.90* | .11 | 16739.40 | .10 |
*
Absolute bias of the standard errors for
In Study 2, we assessed the effects of misspecifying the random part of the model when there were multiple random slopes by adding a second Level 1 predictor. There is potential for greater effects as this also influences the correlations between that slope variance with the variances of the intercept and other slopes. We were interested in how misspecifying slope variance for
We used a 3 (number of groups) × 3 (average group size) × 4 (slope correlation) × 5 (slope variance effect size for
The Level 2 variables
As in Study 1, we estimated multiple models. In Model 1, the slope for
Source | γ01 |
γ10 |
γ11 |
γ20 |
||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
NJ = 5 | .076 | .001 | 0.03 | .082 | -.010 | -0.26 | .067 | -.008 | -0.36 | .081 | .002 | 0.06 |
NJ = 10 | .064 | .001 | 0.03 | .060 | -.009 | -0.35 | .050 | -.008 | -0.51 | .059 | .003 | 0.10 |
NJ = 20 | .059 | .001 | 0.03 | .048 | -.010 | -0.47 | .042 | -.010 | -0.63 | .049 | .004 | 0.18 |
NG = 50 | .090 | .001 | 0.06 | .088 | -.013 | -0.38 | .074 | -.012 | -0.59 | .087 | .004 | 0.12 |
NG = 100 | .064 | .001 | 0.07 | .060 | -.009 | -0.36 | .050 | -.008 | -0.60 | .060 | .003 | 0.11 |
NG = 200 | .045 | .000 | 0.07 | .042 | -.006 | -0.35 | .035 | -.006 | -0.62 | .042 | .002 | 0.12 |
τ11 = .000 | .066 | .000 | -0.01 | .051 | .002 | 0.08 | .040 | .001 | 0.07 | .058 | .001 | 0.02 |
τ11 = .025 | .066 | .000 | 0.01 | .058 | -.005 | -0.16 | .047 | -.004 | -0.21 | .063 | .001 | 0.04 |
τ11 = .050 | .066 | .000 | 0.02 | .061 | -.008 | -0.24 | .050 | -.007 | -0.34 | .063 | .002 | 0.07 |
τ11 = .10 | .067 | .001 | 0.04 | .067 | -.012 | -0.38 | .057 | -.011 | -0.54 | .063 | .004 | 0.13 |
τ11 = .15 | .067 | .002 | 0.06 | .071 | -.016 | -0.51 | .062 | -.016 | -0.70 | .064 | .006 | 0.18 |
τ22 = .00 | .066 | .001 | 0.03 | .061 | -.010 | -0.28 | .048 | -.008 | -0.40 | .053 | .001 | 0.03 |
τ22 = .10 | .067 | .001 | 0.03 | .064 | -.010 | -0.30 | .054 | -.009 | -0.42 | .066 | .004 | 0.11 |
τ12 = .00 | .066 | .001 | 0.03 | .062 | -.009 | -0.28 | .050 | -.008 | -0.38 | .059 | .001 | 0.04 |
τ12 = .10 | .067 | .001 | 0.03 | .065 | -.010 | -0.32 | .054 | -.009 | -0.45 | .066 | .003 | 0.08 |
τ12 = .50 | .067 | .001 | 0.03 | .065 | -.010 | -0.31 | .056 | -.010 | -0.46 | .066 | .005 | 0.14 |
τ12 = .90 | .067 | .001 | 0.04 | .065 | -.010 | -0.32 | .056 | -.010 | -0.42 | .066 | .007 | 0.19 |
ICCX1 = .00 | .066 | .000 | 0.02 | .062 | -.010 | -0.30 | .052 | -.009 | -0.43 | .063 | .003 | 0.10 |
ICCX1 = .50 | .067 | .001 | 0.04 | .064 | -.009 | -0.29 | .054 | -.009 | -0.40 | .063 | .003 | 0.09 |
ICCX2 = .00 | .066 | .001 | 0.03 | .063 | -.010 | -0.31 | .053 | -.009 | -0.42 | .061 | .003 | 0.10 |
ICCX2 = .50 | .067 | .001 | 0.03 | .064 | -.010 | -0.29 | .053 | -.009 | -0.41 | .064 | .003 | 0.09 |
rX1X2 = .00 | .068 | .001 | 0.03 | .052 | -.010 | -0.58 | .051 | -.010 | -0.49 | .052 | .001 | 0.06 |
rX1X2 = .10 | .067 | .001 | 0.03 | .056 | -.010 | -0.40 | .051 | -.009 | -0.45 | .056 | .002 | 0.07 |
rX1X2 = .50 | .066 | .001 | 0.03 | .056 | -.010 | -0.47 | .052 | -.010 | -0.50 | .055 | .005 | 0.21 |
rX1X2 = .90 | .065 | .001 | 0.03 | .090 | -.009 | -0.20 | .058 | -.007 | -0.27 | .089 | .005 | 0.10 |
It is interesting to note that varying the amount of slope variance produced little bias in the standard errors for γ20. In general, the amount of bias was similar to that for the standard errors of γ01. In addition, the degree of correlation between slopes did not appear to impact the bias in the standard errors for γ10 and γ11. For example, when the correlation was .00, the bias in the standard errors for γ10 was -.009 with a Cohen’s
We conducted two ANOVA’s investigating the impact of the study characteristics on the bias in standard errors for the average γ10 and γ11. As with Study 1, only conditions where there was variance in the slopes were included. We limited the ANOVA to the main effects and all two-way interactions because of the large number of conditions.
Source | Bias in γ10 |
Bias in γ11 |
|||
---|---|---|---|---|---|
Partial η2 | Partial η2 | ||||
Number of cases per group | 2 | 9961.09 | 0.01 | 50982.95 | 0.04 |
Number of groups | 2 | 427913.47 | 0.27 | 436516.19 | 0.27 |
Predictor correlation | 3 | 23328.24 | 0.03 | 91597.76 | 0.11 |
Slopes correlation | 3 | 121.52 | 0.00 | 3532.56 | 0.00 |
Slope variance for X1 | 3 | 573301.99 | 0.43 | 745638.19 | 0.49 |
ICC for X1 | 1 | 1746.76 | 0.00 | 2459.09 | 0.00 |
ICC for X2 | 1 | 118.75 | 0.00 | 299.74 | 0.00 |
Varying the slope variance for
It is well known that misspecification of the random part of multilevel models impacts the standard errors for the fixed effects. In the present studies, we sought to identify how great this impact is, and what type of misspecification has the larger effect. In Study 1, we found that fixing the slope variance when it should be random has bigger effects than freeing the slope variance when it should be fixed. Large effect sizes for both the Level 1 slope and cross-level interaction standard errors were observed for relatively small levels of slope variance. The bias was such that Type I errors were more likely when the slope was fixed but should have varied, and Type II errors were more likely when the slope varied but should have been fixed. We also found that the number of groups attenuated the biasing effect of slope variance magnitude such that more groups were associated with less of a biasing effect.
In Study 2, we investigated the effects of misspecifying the random part of the model in situations with multiple random slopes and between-group variance for correlated Level 1 predictors. Overall, the results were similar to the single random slope models in that these added conditions did not have a large impact. As expected, the bias in standard errors is limited to the fixed effects associated with the slope that should be random. That is, the only fixed effects that are affected are the direct effect of the Level 1 predictor, and the cross-level interaction of the Level 2 variable predicting the Level 1 slope.
In our simulations, we considered a basic two-level model with relatively few continuous predictors. Thus, it is not clear how well the results would generalize to studies with a larger number of predictors, additional levels, or categorical variables. Similarly, we considered a model with a simple compound symmetry variance/covariance matrix for the Level 1 errors. The assumption of compound symmetry is often relaxed for longitudinal models because of autocorrelation and heterogeneity of variance that occurs for repeated measures. This directly effects the standard errors for Level 1 coefficients (
We see three major recommendations emerging from the results of the present studies. First, we suggest that researchers, when in doubt, give preference to random slopes. Our results suggest that doing so would minimize the amount of bias in the relevant standard errors for the fixed effects. We suspect that it is rare that the variance for a slope is exactly zero. However, even if this were to be the case, our results suggest that allowing that slope to vary would produce very little bias. In contrast, fixing a slope that should be random produces considerably more bias. If researchers desire to gauge the potential effects of misspecifying slope variance, then they could compare models with and without random slopes.
Second, we suggest that researchers should avoid using nonrandomly varying slope models to evaluate cross-level interactions. Cross-level interactions may exist when the variance in slopes is not statistically significant (
Finally, we recommend using a piecemeal approach when there are many random effects that may cause estimation problems or if there are not enough degrees of freedom to estimate all of the random effects. Results from Study 2 suggest that omitting a slope from the random part of the model only affects coefficients concerning the omitted slope. Researchers could test their hypotheses regarding fixed effects allowing for the appropriate random effects. For example, we may hypothesize that the Level 2 variable, Z, predicts the slopes of
The authors have no funding to report.
The authors have declared that no competing interests exist.
The authors have no support to report.