Methodology

Extending the Reach of the Common Cause Design Using Meta-Analytic Methods: Applications and Issues

Mon, 23 Dec 2024 00:00:00 -0800

Most meta-analytic methods examine effects across a collection of primary studies. We introduce an application of meta-analytic techniques to estimate effects and homogeneity within a single, primary study consisting of multiple, pretest-intervention-posttest units. This novel assessment was used to validate the recently created “Common Cause” (CC) design. In each case, we established the CC design by eliminating control groups from randomized studies, thereby deconstructing each experiment. This deconstruction enabled us to compare difference-in-difference results in randomized designs with a control group to pretest-posttest differences in a CC design without a control group. Meta-analysis results of multiple OXO effects from the CC designs were compared to meta-analytic effects of multiple randomized studies. This within-study-comparison logic and associated analyses produced consistent similarity between CC and validating-study results when directions of findings and patterns of statistical significance were considered. We provide plausible explanations for varying CC effect-size estimates, describe strengths and limitations, and address future research directions.

Minimum Required Sample Size for Modelling Daily Cyclic Patterns in Ecological Momentary Assessment Data

Robin van de Maat, Johan Lataster, Peter Verboon — Mon, 23 Dec 2024 00:00:00 -0800

Cyclical patterns in ecological momentary assessment (EMA) data on emotions have remained relatively underresearched. Addressing such patterns can help to better understand emotion dynamics across time and contexts. However, no general rules of thumb are readily available for psychological researchers to determine the required sample size for measuring cyclical patterns in emotions. This study, therefore, estimates the minimum required sample sizes—in terms of the number of measurements per time period and subjects—to obtain a power of 80% given a certain underlying cyclical pattern based on input parameter values derived from an empirical EMA dataset. Estimated minimum required sample sizes varied between 50 subjects and 10 measurements per subject for accurately detecting cyclical patterns with a large magnitude, to 60 subjects and 30 measurements per subject for cyclical patterns of small magnitude. The resulting rules of thumb for sample sizes are discussed with a number of considerations in mind.

A Framework for Planning Sample Sizes Regarding Prediction Intervals of the Normal Mean Using R Shiny Apps

Wei-Ming Luh, Jiin-Huarng Guo — Mon, 23 Dec 2024 00:00:00 -0800

Replication is a core principle for research, and the recent recognition of the importance of constructing prediction intervals for precise replications highlights the need for robust sample-size planning methodologies. However, methodological and technical complexities often hinder researchers from efficiently achieving this task. This study addresses this challenge by developing five R Shiny apps specifically tailored to determine sample sizes concerning prediction intervals for the mean of the normal distribution. Two measures of precision, absolute and relative widths, are considered. Additionally, the apps consider unequal sampling unit costs and sample size allocations to achieve optimal results by exhaustive search. Simulation results validate the proposed methodology, demonstrating favorable coverage rates. Two illustrative examples of one-sample and two-sample problems showcase these apps’ versatility and user-friendly nature, providing researchers with a valid and straightforward approach for systematically planning sample sizes.

Maintaining Data Quality When Using Social Media for Recruitment: Risks, Rewards, and Steps Forward

Marissa P. Bartmess, Tamu Abreu — Mon, 23 Dec 2024 00:00:00 -0800

Social media is increasingly used to recruit participants for research studies and has been shown to be an effective means of recruitment, in terms of cost, time, and accessibility. However, researchers often struggle with the challenges of using social media for recruitment, as minimal guidance is available. Without careful consideration of the risks to data quality when using social media as a recruitment tool, the overall results of studies can be compromised. This paper provides three hypothetical scenarios based in part on the real-world experiences of researchers using social media-based recruitment (SMR) methods. The scenarios serve as a discussion and learning opportunity for researchers to identify data quality issues with SMR and postulate how issues can be mitigated. Inexperience with SMR can lead to severe flaws in data collection, which can be mitigated early in the study process with appropriate measures in place. Researchers need to proactively educate themselves and take measures to avoid common pitfalls associated with SMR to achieve robust data quality and research integrity.

How Large Must an Associational Mean Difference Be to Support a Causal Effect?

Michael Höfler, Ekaterina Pronizius, Erin Buchanan — Mon, 23 Dec 2024 00:00:00 -0800

An observational study might support a causal claim if the association found cannot be explained by bias due to unconsidered confounders. This bias depends on how strongly the common predisposition, a summary of unconsidered confounders, is related to the factor and the outcome. For a positive effect to be supported, the product of these two relations must be smaller than the left boundary of the confidence interval for, e.g., a standardised mean difference (d). We suggest means to derive heuristics for how large this product must be to serve as a confirmatory threshold. We also provide non-technical, visual means to express researchers’ assumptions on the two relations to assess whether a finding on d is explainable by omitted confounders. The ViSe tool, available as an R package and Shiny application, allows users to choose between various effect sizes and apply it to their own data or published summary results.

Partitioning Dichotomous Items Using Mokken Scale Analysis, Exploratory Graph Analysis and Parallel Analysis: A Monte Carlo Simulation

Mon, 30 Sep 2024 00:00:00 -0700

Estimating the number of latent factors underlying a set of dichotomous items is a major challenge in social and behavioral research. Mokken scale analysis (MSA) and exploratory graph analysis (EGA) are approaches for partitioning measures consisting of dichotomous items. In this study we perform simulation-based comparisons of two EGA methods (EGA with graphical least absolute shrinkage and selector operator; EGAtmfg with triangulated maximally filtered graph algorithm), two MSA methods (AISP: automated item selection procedure; GA: genetic algorithm), and two widely used factor analytic techniques (parallel analysis with principal component analysis (PApc) and parallel analysis with principal axis factoring (PApaf)) for partitioning dichotomous items. Performance of the six methods differed significantly according to the data structure. AISP and PApc had highest accuracy and lowest bias for unidimensional structures. Moreover, AISP demonstrated the lowest rate of misclassification of items. Regarding multidimensional structures, EGA with GLASSO estimation and PApaf yielded highest accuracy and lowest bias, followed by EGAtmfg. In addition, both EGA techniques exhibited the lowest rate of misclassification of items to factors. In summary, EGA and EGAtmfg showed comparable performance to the highly accurate traditional method, parallel analysis. These findings offer guidance on selecting methods for dimensionality analysis with dichotomous indicators to optimize accuracy in factor identification.

A General Framework for Modeling Missing Data Due to Item Selection With Item Response Theory

Paul A. Jewsbury, Ru Lu, Peter W. van Rijn — Mon, 30 Sep 2024 00:00:00 -0700

In education testing, the items that examinees receive may be selected for a variety of reasons, resulting in missing data for items that were not selected. Item selection is internal when based on prior performance on the test, such as in adaptive testing designs or for branching items. Item selection is external when based on an auxiliary variable collected independently to performance on the test, such as education level in a targeting testing design or geographical location in a nonequivalent anchor test equating design. This paper describes the implications of this distinction for Item Response Theory (IRT) estimation, drawing upon missing-data theory (e.g., Mislevy & Sheehan, 1989, https://doi.org/10.1007/BF02296402; Rubin, 1976, https://doi.org/10.1093/biomet/63.3.581), and selection theory (Meredith, 1993, https://doi.org/10.1007/BF02294825). Through mathematical analyses and simulations, we demonstrate that this internal versus external item selection framework provides a general guide in applying missing-data and selection theory to choose a valid analysis model for datasets with missing data.

Post-Hoc Tests in One-Way ANOVA: The Case for Normal Distribution

Fri, 28 Jun 2024 00:00:00 -0700

When one-way ANOVA is statistically significant, a multiple comparison problem arises, hence post-hoc tests are needed to elucidate between which groups significant differences are found. Different post-hoc tests have been proposed for each situation regarding heteroscedasticity and sample size groups. This study aims to compare the Type I error (α) rate of 10 post-hoc tests in four different conditions based on heteroscedasticity and balance between-group sample size. A Montecarlo simulation study was carried out on a total of 28 data sets, with 10,000 resamples in each, distributed through four conditions. One-way ANOVA tests and post-hoc tests were conducted to estimate the α rate at a 95% confidence level. The percentage of times the null hypothesis was falsely refused is used to compare the tests. Three out of four conditions demonstrated considerable variability among sample sizes. However, the best post-hoc test in the second condition (heteroscedastic and balance group) did not depend on simple size. In some cases, inappropriate post-hoc tests were more accurate. Homoscedasticity and balance between-group sample size should be considered for appropriate post-hoc test selection.

Modelling the Effect of Instructional Support on Logarithmic-Transformed Response Time: An Exploratory Study

Luis Alberto Pinos Ullauri, Wim Van Den Noortgate, Dries Debeer — Fri, 28 Jun 2024 00:00:00 -0700

Instructional support can be implemented in learning environments to pseudo-modify the difficulty or time intensity of items presented to persons. This support can affect both the response accuracy of persons towards items as well as the time persons require to complete items. This study proposes a framework to model response time in learning environments as a function of instructional support. Moreover, it explores the effect of instructional support on response time in assembly tasks training using Virtual Reality. Three models are fitted with real-life data collected by a project that involves both industry and academic partners from Belgium. A Bayesian approach is followed to implement the models, where the Bayes factor is used to select the best fitting model.

Comparison of Lasso and Stepwise Regression in Psychological Data

Di Jody Zhou, Rajpreet Chahal, Ian H. Gotlib, Siwei Liu — Fri, 28 Jun 2024 00:00:00 -0700

Identifying significant predictors of behavioral outcomes is of great interest in many psychological studies. Lasso regression, as an alternative to stepwise regression for variable selection, has started gaining traction among psychologists. Yet, further investigation is valuable to fully understand its performance across various psychological data conditions. Using a Monte Carlo simulation and an empirical demonstration, we compared Lasso regression to stepwise regression in typical psychological datasets varying in sample size, predictor size, sparsity, and signal-to-noise ratio. We found that: (1) Lasso regression was more accurate in within-sample selection and yielded more consistent out-of-sample prediction accuracy than stepwise regression; (2) Lasso with a harsher shrinkage parameter was more accurate, parsimonious, and robust to sampling variability than the prediction-optimizing Lasso. Finally, we concluded with cautious notes and recommendations in practice on the application of Lasso regression.