Methodology

A Comparison of Optimization Algorithms for Forced-Choice Questionnaire Assembly

Scarlett Escudero — 2026-06-30

Forced-choice questionnaires (FCQs) are increasingly favored over traditional Likert-type formats due to their reduced susceptibility to faking and social desirability (SD). Their construction typically involves pairing items from existing single-stimulus banks. This study compares four methods for assembling FCQs: a genetic algorithm (GA), two simulated annealing (SA) strategies (blueprint-based and scale-parameter-optimized), and brute-force (BF) random search. These methods are evaluated via simulation and an empirical example, focusing on trait score recovery. The effects of questionnaire length and SD matching on recovery are also examined. Three item banks varying in the aj-SDj relationship and inclusion of heteropolar blocks were used to assess performance across pairing scenarios. GA consistently produced the most reliable scores, followed by SA with aj optimization. All examined factors significantly affected reliability. GA is recommended for FCQ assembly, especially with short questionnaires, no heteropolar blocks, and high aj-SDj correlation.

How to Use Independent Validation in Python

Thede von Oertzen — 2026-06-30

To statistically test whether two groups or models differ, classifier accuracy is compared. However, common accuracy estimates like cross-validation have unknown distributions, making them unsuitable for statistical inference. Alternatives like permutation tests or train-test splits are computationally expensive and limited to frequentist tests against chance. Independent Validation (IV) is a more flexible alternative providing a known estimate distribution. This enables both conventional hypothesis testing and Bayesian analysis of classifier performance. Although Python is most widely used for machine learning, a Python implementation of IV has been lacking so far. This article introduces such an implementation; beyond the core IV algorithm, the package allows to: (1) plot accuracy against training set size, (2) estimate the posterior distribution of the asymptotic accuracy, and (3) query the posterior for statistics and credible intervals. This makes it easy to apply IV when comparing accuracy posteriors across classes, datasets, or classifiers on the same data.

A Simulation-Based Comparison of Minimization, Rerandomization, and Anticlustering for Creating Experimental Conditions

Martin Papenberg — 2026-06-30

Anticlustering has been used as a novel method to assign subjects to conditions in experiments. Anticlustering can be applied when covariate measurements are available at the beginning of an experiment and minimizes differences in covariates between conditions. In a simulation study implementing a two-group between-subjects design, we compared anticlustering with established methods for minimizing covariate imbalance: rerandomization and minimization. Anticlustering most strongly reduced covariate imbalance, followed by rerandomization and minimization. Lower covariate imbalance increased the precision of the effect size estimate. The average statistical power of the unadjusted analysis (independent t-test) was not improved when using covariate-based assignment as compared to random assignment. However, with random assignment, the statistical power of the unadjusted analysis depended on observed covariate imbalance; with covariate-based assignment, the statistical power of the unadjusted analysis was less affected by covariate imbalance because imbalance was minimized. Statistical adjustment via regression was most important to maximize statistical power.

Beyond Scalar Invariance: Evaluating the Validity of Person-Level Score Comparisons

Gregor Sočan — 2026-06-30

Scalar invariance is widely regarded as essential for comparing test-score means across groups. However, it is less clear when test scores can be meaningfully compared at the individual level — specifically, whether individuals from different groups who share the same observed score have the same expected value of the latent trait. I show that scalar invariance alone is insufficient for meaningful person-level comparisons based on sum scores. In addition to scalar invariance, person comparison invariance requires equality of latent variable means and omega coefficients across groups. Nevertheless, non-invariance effects can be relatively small if the omega coefficients are high and similar in magnitude across groups. I relate person comparison invariance to predictive invariance and provide R code to test person comparison invariance and to visualise the effects of non-invariance.

Controlling for Time-Varying Confounding in the Longitudinal Fixed-Effects Model: A Latent Variable Approach

Baeksan Yu — 2026-03-27

Fixed-effects regression models are commonly used in longitudinal studies as a means to estimate causal effects while controlling for unobserved time-invariant confounders. However, unobserved time-varying confounding remains potentially problematic, and identifying and measuring such confounders can be resource-intensive and costly. We propose the Time-Varying Confounding Structural Equation Model (TVC-SEM), a simple longitudinal model that builds on previous “common factor” models and which can serve as a robustness check for the assumption of no unobserved time-varying confounding in the fixed-effects approach. We posit a model with a latent autoregressive variable Zit, which represents the combined influence of both time-invariant and time-varying unobservables, and which is linked to the independent and dependent variables over time. Through Monte Carlo simulations and analyses of data from the Early Childhood Longitudinal Studies Kindergarten cohort (ECLS-K) and the Rural Substance Abuse and Violence Project (RSVP), we show that, under most conditions, TVC-SEM provides less biased estimates than several variants of the traditional fixed-effects model. Our proposed approach offers applied researchers a practical check for gauging the extent to which the fixed-effects assumption of no time-varying confounding may produce bias in the estimation of causal effects.

Analyzing Group Differences and Measurement Fairness in Process Data: A Sequential Response Model With Covariates

Yuting Han — 2026-03-27

This article introduces the sequential response model with covariates (SRM-C) for analyzing process data, with emphasis on three key capabilities: detecting potential measurement bias in response processes, evaluating group differences in ability distributions and improving parameter estimation precision. The SRM-C combines measurement and structural components, with the measurement component modeling response sequences conditional on abilities and covariates, and the structural component characterizing group-specific ability distributions. Sparsity assumptions implemented through horseshoe prior distributions address identification issues within the Bayesian framework. Monte Carlo simulations demonstrated robust parameter recovery and effective differential item functioning (DIF) detection. An empirical analysis of PISA problem-solving data illustrated the model’s utility in distinguishing ability differences from potential measurement bias. The SRM-C offers a comprehensive framework for understanding group differences in process data while ensuring measurement fairness.

Bayesian Versus Frequentist Approaches in Multilevel Single-Case Designs: On Type I Error Rate and Power

Cristina Rodríguez-Prada — 2026-03-27

Single-case designs (SCEDs) assess intervention effects through repeated measurements on one or a few individuals. Multilevel models nest repeated measures within individuals and have gained popularity for inferential analysis in SCEDs, in combination with expert knowledge of the clinicians and applied researchers. However, researchers often face model specification challenges without knowing the true population model underlying their data. This study evaluates how model selection criteria (AIC, BIC, WAIC, LOO) conditioned on the selected model impact statistical power and Type I error rates in intervention effects, reflecting the ecological reality where practitioners do not know the true model. A Monte Carlo simulation modelled data of AB designs varying sample size, measurement points, intervention effects, and random effect structures. Competing multilevel models were then fitted and compared using AIC, BIC, WAIC, and LOO to examine the impact of model selection on statistical power and Type I error rates. Results indicated that frequentist criteria performed well in simpler models in terms of power, while Bayesian approaches showed greater robustness with respect to Type I error control. The findings provide practical insights on multilevel model selection under real-world conditions, highlighting Bayesian methods as a robust alternative for applied researchers handling small sample sizes and complex data structures.

Bridging the Gap: Introducing Joint Models for Longitudinal and Time-to-Event Data in the Social Sciences

Sophie Potts — 2026-03-27

In time-to-event analyses in social sciences, there often exist endogenous time-varying variables, where the event status is correlated with the trajectory of the covariate itself. Ignoring this endogeneity will result in biased estimates. In the field of biostatistics this issue is tackled by estimating a joint model for longitudinal and time-to-event data as it handles endogenous covariates properly. This method is underused in the social sciences even though it is very useful to model longitudinal and time-to-event processes appropriately. Therefore, this paper provides a gentle introduction to the method of joint models and highlights its advantages for social science research questions. We demonstrate its usage on an example on marital satisfaction and marriage dissolution and compare the results with classical approaches such as a time-to-event model with a time-varying covariate. In addition to demonstrating the method, our results contribute to the understanding of the relationship between marriage satisfaction, marriage dissolution and other covariates.

Evaluating the Standard Error Estimation of the Local Structural-After-Measurement (LSAM) Approach in Structural Equation Modeling

Seda Can — 2025-12-18

Accurate estimation of standard errors (SEs) is essential in SEM as they quantify the uncertainty of parameter estimates, are fundamental to computing test statistics, and ensure robust population inferences. This study evaluated SEs within the Local Structural-After-Measurement (LSAM) framework, a two-step approach to SEM. Two simulation studies examined analytic and resampling-based SE methods under varying conditions, including normal and nonnormal data, different sample sizes, and both correct and misspecified models. The nonparametric bootstrap yielded near-unbiased SEs under nonnormality, even when models were misspecified, while the parametric bootstrap performed well under normal conditions with correct model specification. The analytic two-step method performed well under normal conditions but showed increased bias with nonnormal data and smaller samples. The robust two-step method reduced this bias in larger samples, though some underestimation remained in small-sample and misspecified conditions. To complement SE bias results, 90% coverage rates were assessed. Findings confirm LSAM’s capability for accurate SE estimation in challenging research contexts.

Mixture Multigroup Bayesian SEM With Approximate Measurement Invariance for Comparing Structural Relations Across Many Groups

Hongwei Zhao — 2025-12-18

In social sciences, researchers often compare relations between constructs, referred to as “structural relations”, across a large number of groups. This paper proposes Mixture Multigroup Bayesian SEM (MixMG-BSEM), a novel method for comparing structural relations across many groups while accounting for approximate measurement invariance in factor loadings. Traditional methods often assume exact measurement invariance, which may not reflect real-world data where small differences in measurement parameters commonly occur across many groups. MixMG-BSEM addresses this by using Multigroup Bayesian CFA with small-variance priors to allow for these small differences, and groups are then clustered based on their structural relations using Mixture Modeling. This is done in a stepwise estimation procedure built on the structural-after-measurement approach. By combining cluster-specific structural relations with small between-group differences in measurement parameters, MixMG-BSEM obtains a clustering that is driven only by the structural relations. The robustness and effectiveness of MixMG-BSEM are demonstrated through a simulation study.