Using the EffectLiteR framework, researchers can test classical null hypotheses about effects of interest via Wald and F-tests, while taking into account the stochastic nature of group sizes. This paper aims at extending EffectLiteR to test informative hypotheses, assuming for example that the average effect of a new treatment is greater than the average effect of an old treatment, which in turn is greater than zero. We present a simulated data example to show two methodological novelties. First, we illustrate how to use the Fbar- and generalized linear Wald test to assess informative hypotheses. While the classical test did not reach significance, the informative test correctly rejected the null hypothesis, indicating the need to take into account the order of the treatment groups. Second, we demonstrate how to account for stochastic group sizes in informative hypotheses using the generalized non-linear Wald statistic. The paper concludes with a short data example.

Empirical research across a broad range of disciplines aims at assessing the effectiveness of a treatment or an intervention. The interest might lie in average effects on a certain outcome or in conditional effects, given values of categorical or continuous covariates. For example, a researcher might be interested in the average effect of an extended day program on student achievement

The EffectLiteR approach (

A limitation of the current EffectLiteR approach is that it does not allow for testing informative hypotheses (

The goal of this paper is to integrate informative hypothesis testing into the EffectLiteR approach. We will show how

In the following, the background of the EffectLiteR framework is explained. This will be illustrated by means of a non-randomized experiment, whose data of size

Group | ||||
---|---|---|---|---|

When all variables are observed, the EffectLiteR approach first estimates a linear regression model. Typically, this regression model contains not only the main effects of

1.667 | -0.285 | 0.029 | 0.300 | -0.252 | 0.591 | 0.261 | -0.546 | 0.128 | 0.322 | 0.032 | -0.234 |

All the effects of interest in the EffectLiteR approach will be written as a function of the estimated regression parameters and the (conditional) expectations of the covariates. In the next section, possible hypotheses of interest are discussed.

With a general linear hypothesis test, hypotheses concerning regression parameters like

The EffectLiteR model is based on intercept and effect functions (see also

To be able to calculate the average effect, we need the unconditional expectation of

Group | Fixed group sizes |
Stochastic group sizes |
||
---|---|---|---|---|

Adjusted mean | Average effect | Adjusted mean | Average effect | |

Control ( |
1.601 (0.099) | 1.601 (0.099) | ||

Conventional therapy ( |
1.683 (0.127) | 0.082 (0.161) | 1.683 (0.128) | 0.082 (0.162) |

Innovative therapy ( |
1.875 (0.074) | 0.274 (0.124) | 1.875 (0.075) | 0.274 (0.124) |

To test the hypotheses of interest, EffectLiteR makes use of the

The

Using the same notation, the linear Wald test can be defined as (

To test

Informative hypotheses reflect prior order expectations regarding means, regression coefficients or combinations thereof. These hypotheses can be constructed by means of different constraints (see, e.g.,

The

If the constraints are linear,

The generalized linear Wald statistic is a generalization of the regular Wald test and can be found in

In our example, the corrected Wald test as well as the

To calculate the

The second approach is more economical, as the mixing weights are estimated first, also by means of simulation, and directly used to calculate the

Estimating the

It can be concluded that integrating informative hypotheses into the EffectLiteR framework enriches testing hypotheses regarding effects of interest. This is because it allows to directly test hypotheses that correspond to the researcher’s prior order expectations of the treatment groups. Using this approach, it is not necessary anymore to follow a cumbersome two-step procedure with potentially increased type I error rates, as is needed in classical null hypothesis testing. In fact, it is even possible to detect significant results that would not have been detected via classical null hypothesis testing, as was the case in our motivating example. Going one step further, the next section illustrates how to account for stochastic group sizes while testing informative hypotheses in the EffectLiteR framework.

The treatment groups (

So far, we have treated the group sizes as fixed. Theoretically, this implies that each time we would replicate the study, the group sizes would remain the same. However, in practice, this is not always the case, and the group sizes may vary from sample to sample. In this case, it is more correct to treat the group sizes as stochastic, and this implies that the group weights become free parameters in our model.

To apply informative hypothesis testing in the EffectLiteR framework while taking into account stochastic group sizes, two changes are needed. First, a new method is needed to obtain

To obtain

The generalized non-linear Wald statistic can be found in

In our simulated data example, we obtain the following results. The columns of

We showed how to account for stochastic group sizes when testing informative hypotheses within the EffectLiteR framework. If group weights are considered as free parameters in the model, uncertainty increases, which manifests itself in increased standard errors of parameters estimates. This seems to be more evident in small samples than it is in large samples. This observation is important especially in the social and behavioral sciences, where treatment group sizes often vary across samples and the exact group sizes are not determined before conducting the experiment. However, at this point, considering stochastic group sizes comes with an increased computational cost, as parameter estimation has to be carried out in a two-step procedure by means of lavaan.

In this section, the illustrated methods will be applied to an artificial data example from the constrained statistical inference literature (

Group | Raw/adjusted Means and relative treatment group frequency |
---|---|

Possible hypotheses regarding the effects of interest are:

The adjusted means and average effects estimates are presented in

Group | Adjusted mean | Average effect |
---|---|---|

No training ( |
-0.920 (0.777) | |

Physical training ( |
0.922 (0.744) | 1.842 (1.076) |

Behavioral therapy ( |
3.341 (0.706) | 4.261 (1.049) |

Physical training & behavioral therapy ( |
4.237 (0.801) | 5.157 (1.116) |

Testing Hypothesis

This paper demonstrated how to integrate informative hypotheses into the EffectLiteR framework, while taking into account stochastic group sizes. We provided R scripts and explanatory documents in the

Considering stochastic group sizes in the approach described in this paper does not seem to impact the results substantially if the sample size is large. However, when dealing with small sample sizes, standard errors might be underestimated when erroneously treating stochastic group sizes as fixed. This is because of adequately accounting for the increased uncertainty that stems from considering group weights as free instead of as fixed parameters in the model.

Two approaches to estimate the

The limitations of this paper and the outlook on future research are the following. First, we only considered manifest variables. In the future, the presented methods should be extended to be able to deal with latent variables. Furthermore, the small-sample properties of the generalized non-linear Wald test are unknown. Thus, testing informative hypotheses in the EffectLiteR framework while accounting for stochastic group sizes should be examined further in future research by means of simulation studies. Moreover, we did not examine the consequences when assumptions of the general linear model like homoscedasticity are not fulfilled. When considering stochastic group sizes, it has already been shown that in the two-group context, erroneously assuming equal variances is only critical regarding standard error estimation of effects when group sizes are unequal (see, e.g.,

Finally, the two-step procedure presented to test informative hypotheses about effects of interest while considering stochastic group sizes needs a lot of manual tuning at this point. To make this approach more efficient and accessible for applied researchers, it might be possible to adapt it using the distance statistic (

The data for this article are freely available (see the

For this article, the following Supplementary Materials are available via PsychArchives repository (for access see

R scripts.

R data sets.

Explanatory documents.

This work has been supported by the Research Foundation Flanders (FWO, grant G020115N to Yves Rosseel and Axel Mayer).

The authors have declared that no competing interests exist.

The authors have no additional (i.e., non-financial) support to report.