A Simulation-Based Comparison of Minimization, Rerandomization, and Anticlustering for Creating Experimental Conditions

The gold standard for conducting experiments is random assignment of subjects to conditions, allowing to establish the causal effect of an experimental intervention on an outcome. While preventing systematic differences between conditions, random assignment does not prevent that experimental groups randomly differ with regard to covariates that influence the outcome (Morgan & Rubin, 2012; Nguyen et al., 2017; Senn, 2013). The need to deal with nuisance variance due to measured covariates has long been recognized. However, there is an ongoing debate about how to address it: Should covariates be actively balanced among experimental conditions — “messing” with the traditional random assignment — or is statistical adjustment of covariates sufficient (Sella et al., 2021; Senn, 2022; Taves, 1974; Toorawa et al., 2009; Treasure & MacRae, 1998)? According to Hu et al. (2014), there are three main advantages of actively balancing covariates. First, balancing allows for a more precise estimate of the effect of an experimental manipulation; second, balancing increases the credibility of the analysis by increasing the comparability of the different conditions; third, with balancing, the analysis is less prone to misspecification of the statistical model (Qin et al., 2024). Consequently, there seems to be a clear intuitive impetus towards active balancing (cf. Senn, 2005). Practitioners as well as researchers tend to view model-based adjustment of covariates with skepticism if considerable imbalance is observed, and prefer to prevent imbalances altogether (Coart et al., 2023).¹ If covariate imbalance occurs as a consequence of random assignment (Nguyen et al., 2017), statistical adjustment seems to leave an “air of uncertainty about the validity of the conclusion[s]” (Treasure & MacRae, 1998, p. 362).

Many methods have been brought forward as potential enhancements of random assignment that increase covariate balance between experimental conditions (Lin et al., 2015). Moreover, the development and investigation of new covariate-balancing techniques is an active area of study (e.g., Ma et al., 2024; Qin et al., 2024; Zhang et al., 2024). When a covariate is categorical (e.g., biological sex or socioeconomic status), stratification techniques such as stratified block randomization are frequently applied in clinical studies, ensuring that combinations of categories — so called strata — are balanced between treatment and control conditions (for overviews, see Kernan et al., 1999; Lin et al., 2015). Stratification is based on assignment rules, for example through a permuted block design, that balances the occurrence of each condition within each stratum (Pocock & Simon, 1975). However, blocking rules cannot be applied when there are many strata or when covariates are numeric (Morgan & Rubin, 2012). Balancing numeric variables typically requires an automated and algorithmic assignment that employs a mathematical notion of covariate imbalance between conditions. In this paper, we investigate anticlustering, which uses an objective function to quantify covariate imbalance and uses an algorithmic assignment to optimize balance. We compared anticlustering to conceptually similar methods (minimization and rerandomization) that also use objective functions to quantify covariate imbalance.

Minimization is a traditional method to achieve balance in numeric covariates between conditions (Pocock & Simon, 1975; Taves, 1974; Treasure & MacRae, 1998). It can be used in sequential designs where subjects enter an experiment one after another. Minimization chooses for each new subject the experimental condition in such a way that covariate imbalance is reduced most effectively. In a sequential design, selecting the imbalance-minimizing condition will often be a deterministic choice, which may be undesirable in clinical experiments where the experimenter should not have definitive knowledge of the condition to which a subject is assigned (Pocock & Simon, 1975; Sella et al., 2021). It is therefore possible to induce a “biased coin” procedure where the balance-maximizing condition is only selected with a probability p < 1. Whether such a random element is necessary, however, depends on the circumstances of a particular experiment, for example on regulatory expectations.

In nonsequential experiments, covariate measurements for all subjects may be available at the beginning of an experiment. Hence, the conditions can be allocated using all covariate information, thereby potentially reducing covariate imbalance more strongly than would be possible in sequential experiments (Kapelner et al., 2021). Rerandomization is a popular method developed by Morgan and Rubin (2012) that can be used to allocate conditions in nonsequential experiments. Rerandomization first formalized sentiments about repeating a random assignment when a first attempt “went wrong” and resulted in high imbalance among conditions. Rerandomization consists of the following steps (Morgan & Rubin, 2015): (a) recruit experimental subjects and and collect all covariate measurements; (b) define a criterion of covariate imbalance that is deemed “acceptable”; (c) randomize subjects to conditions; (d) compute covariate balance; (e) if covariate balance is not acceptable, repeat randomization until it is; (f) start the experiment. Morgan and Rubin (2012) used the mahalanobis distance of covariate means between treatment and control group as criterion of covariate imbalance that should be minimized.

A critical choice for rerandomization is selecting a maximum amount of imbalance that is deemed “acceptable”. The decision criterion has usually been formalized as an acceptance probability. The acceptance probability roughly corresponds to the extremeness of the observed mahalanobis distance as compared to the distribution of all mahalanobis distances, which is usually approximated using the ${\chi }^2$ distribution (e.g., Zhang et al., 2024). A lower acceptance probability leads to less imbalance in covariates. However, low acceptance probabilities potentially lead to longer run times of the rerandomization method, and possibly prevent finding acceptable solutions altogether — especially when the number of covariates increases (Qin et al., 2024).

Though not developed for this purpose, anticlustering has recently been used as a new method to achieve covariate balance in nonsequential experiments (e.g., Stadelmann et al., 2025). Anticlustering can be used to partition a pool of experimental subjects into disjunct groups with the aim of maximizing similarity between groups, based on the available covariates (Späth, 1986). Due to the availability of the free and open source software package anticlust (Papenberg & Klau, 2021), anticlustering has been applied increasingly in a wide range of research fields, including psychology (Nagel et al., 2024; Schaper et al., 2023), chemistry (Rahu et al., 2024), machine learning (Mauri et al., 2023), and cell biology (Papenberg, Wang, et al., 2025). Even though the original introduction of anticlust did not discuss using it for allocating experimental conditions, it has already been used for this purpose (Even et al., 2023; Stadelmann et al., 2025; Tuti et al., 2022).

Anticlustering is characterized by employing objectives of between-group balance that are known from cluster analysis. In the context of anticlustering, subjects are assigned to groups in such a way that these objectives are maximized; in cluster analysis, the same objectives would be minimized (Brusco et al., 2020). As already recognized by Späth (1986), when maximizing the variance — i.e., the objective function used in k-means clustering — it is possible to create groups that are similar to each other. In particular, k-means anticlustering equates the mean values of the covariates between groups. Thereby, k-means anticlustering is conceptually similar to rerandomization, which also minimizes differences in covariate means. However, rerandomization uses the mahalanobis distance between covariate means, thereby controlling for the covariance between covariates: Covariates that are correlated with other covariates obtain a lower weight than covariates that provide unique information. K-means anticlustering uses the same weighting for all covariates regardless of their covariances.

Using the mahalanobis distance is an appealing feature of rerandomization that so far has not been incorporated into anticlustering. However, anticlustering offers several advantages over rerandomization, which may make its use beneficial in certain circumstances. First, anticlustering is naturally defined for an arbitrary number of groups and not only two. While Morgan and Rubin (2012) discuss that criteria used in MANOVA testing can be adapted for rerandomization for more than two groups, we are not aware of actual implementations for three or more groups. The extension to three and more groups would also require investigating suitable values for the acceptance probability, which are probably different from suitable values in the context of just two groups. The acceptance probability is a necessary parameter in rerandomization, but not required in anticlustering. Moreover, anticlustering can also employ other objective functions than just the k-means criterion, which only minimizes differences in covariate means. For example, the k-plus objective incorporates higher order moments such as the variance, skewness and kurtosis, in addition to means (Papenberg, 2024). This feature may particularly relevant in the case of nonlinear covariate effects (see Ma et al., 2024).

To summarize, when measurements are available at the beginning of the experiment, anticlustering can be applied in a wide range of settings: It can be used to assign subjects to an arbitrary number of conditions; it handles an arbitrary number of covariates; it scales from small to large experiments (Papenberg & Klau, 2021); it handles categorical as well as numeric covariates (Papenberg, Wang, et al., 2025); and it can be used to balance different properties of the data such as means and variances of covariates (Papenberg, 2024). However, the benefits of using anticlustering for allocating experimental conditions have not been previously investigated. Because anticlustering has however already been applied for this purpose, such an evaluation is much-needed. We close the gap in the current paper using a simulation study that allowed to establish a ground-truth for the presence of covariate effects on the outcome to be predicted. We compared anticlustering with standard random assignment, minimization and rerandomization, which are well-known methods for minimizing covariate imbalance (Coart et al., 2023; Morgan & Rubin, 2012; Sella et al., 2021; Treasure & MacRae, 1998). In the simulation, we implemented the most simple design — but arguably one of the most important designs as well — that can be used to adjust for covariates in experimental research: It implemented one experimental factor that varied in two levels, one numeric covariate,² and one numeric dependent variable. Such a design can be analysed using a simple t-test when the covariate is not adjusted, and ANCOVA or regression when the covariate is adjusted. As previous research has amplified the need to carefully select prognostic covariates for analysis (Raab et al., 2000; Wysocki et al., 2022), we included an additional condition where five “noise variables” were generated that were unrelated to the outcome. We implemented (a) random assignment, (b) assignment via anticlustering based on the covariate(s), (c) assignment via minimization based on the covariate(s) (Sella et al., 2021; Treasure & MacRae, 1998), and (d) assignment via rerandomization based on the covariate(s) (Morgan & Rubin, 2012). We applied both an unadjusted analysis (i.e., the t-test) and a covariate-adjusted analysis (i.e., regression) for all four assignment methods. As dependent variables, we investigated the degree of covariate imbalance between conditions.³

Method

We implemented a simulation study using the R programming language (Version 4.5.1, R Core Team, 2022). An accompanying Open Science Framework (OSF) repository contains all code and data required to fully reproduce the simulation and its analysis (see Papenberg & Angelike, 2026). The simulation was conducted to compare four different methods for assigning subjects to experimental conditions: standard random assignment, anticlustering, rerandomization, and minimization. For anticlustering, we used the implementation provided in the anticlust package (Version 0.8.10-1, Papenberg & Klau, 2021). We applied k-means anticlustering, which minimizes the mean values of covariates between conditions, via the default exchange algorithm. The exchange algorithm is a non-deterministic optimization method based on pairwise interchanges. It starts by randomly initializing the groups under the restriction of equal-sized conditions. It then iterates through all subjects and attempts to improve balance by swapping each subject with a subject that is currently assigned to a different condition. A subject is not swapped if no exchange would improve balance according to the objective function. The process stops after all possible exchanges have been evaluated for each subject. Balance could potentially be further improved by not stopping after a single iteration through all subjects; instead it is possible to repeat the iteration process until no single exchange is able to further improve the anticlustering objective, i.e., until a local maximum is found (Weitz & Lakshminarayanan, 1998). From the viewpoint of minimizing covariate imbalance, the local maximum method is unambiguously better. However, in the current study, we opted for using the non-locally optimal exchange method, which may be preferable in the context of designing experiments: Even when explicitly minimizing covariate imbalance, a remaining degree of randomness is desirable (Kapelner et al., 2021), and the non-locally optimal algorithm maintains a higher similarity to the initial random assignment. In any case, it is not possible to predict the final group allocation without knowledge about the random initialization. Therefore, anticlustering would arguably be in line with potential regulatory expectations demanding unpredictability of treatment assignment.

For minimization, we used the implementation provided by Sella et al. (2021), which uses the variance in covariate means as the objective to be minimized (Pocock & Simon, 1975). It can be used as a deterministic procedure or with a “biased coin” rule. We used the deterministic version that always selected the best possible condition with regard to covariate balance. While a random component may be preferable in some practical applications, using the deterministic procedure allowed us to investigate the best possible performance of minimization. Employing a random component would decrease observed balance, but offers no statistical advantages that would be useful in a simulation study. Thus, using the deterministic implementation of minimization yielded a more fair comparison to anticlustering and rerandomization that had the inherent advantage of being nonsequential methods. Even though all observations are available simultaneously in the simulation, the minimization method is based on sequentially assigning subjects to conditions. Hence, the $i$th subject is assigned to a condition using the information on covariate imbalance computed via the previous $i-1$ subjects. For rerandomization, we used an implementation of the standard procedure of Morgan and Rubin (2012), provided by Zhang et al. (2024). Rerandomization requires to define an acceptance probability used as the criterion for accepting an assignment based on covariate imbalance. Lower probabilities imply stricter requirements regarding observed balance between conditions. We set a strict acceptance probability of $p=.001$ following Morgan and Rubin (2015).

Results

Covariate Imbalance

Figure 1 shows the distribution of covariate imbalance by assignment technique (random, minimization, anticlustering, rerandomization), depending on whether additional noise variables were included or not. As expected, minimization reduced covariate imbalance as compared to random assignment. Rerandomisation and anticlustering exhibited an even more pronounced reduction in covariate imbalance, as shown by two highly peaked distributions of $d_{cov}$ in the left panel of Figure 1. Without noise variables, the standard deviation of the distribution of $d_{cov}$ was SD = 0.001 for anticlustering, SD = 0.000 for rerandomization, SD = 0.053 for minimization, SD = 0.217 for random assignment. When noise variables were included, the covariate-based assignments had more difficulties in balancing the prognostic covariate. This result was expected because achieving balance on multiple variables simultaneously is more difficult than achieving balance on only one variable alone (Papenberg, 2024). In this condition, anticlustering exhibited the most pronounced reduction in covariate imbalance, as shown by the single peaked distribution of $d_{cov}$ in the right panel of Figure 1. With noise variables, the standard deviation of the distribution of $d_{cov}$ was SD = 0.032 for anticlustering, SD = 0.053 for rerandomization, SD = 0.105 for minimization, and SD = 0.216 for random assignment.

Click to enlarge

Figure 1

Distribution of Covariate Imbalance for the Four Assignment Techniques When Noise Variables Were Included or Not

Note. Covariate imbalance was computed using the standardized effect size $d$ (Cohen, 1988) for the standard normally distributed covariate. Note that the histograms were cut at $d_{cov}=-0.2$ and $d_{cov}=0.2$ to properly portray the largest portion of the distribution. Only 7.6% of all values were outside of this interval, but there were outliers of up to $d_{cov}=-2.85$ and $d_{cov}=2.42$.

Statistical Power

We computed the average number of significant results while excluding all simulation runs where the effect of the experimental condition was zero (${\beta }_T$ = 0) to obtain estimates of statistical power. Table 1 shows the global average statistical power for all methods, aggregated across all simulation conditions. Using covariate-based assignment alone did not improve power as compared to random assignment. Statistical power was however substantially increased by statistically adjusting for the covariate via regression. Using covariate-based assignment in addition to statistically adjusting for the covariate increased power marginally by less than 1 percentage point.

Table 1

Global Simulation Results

Analysis	Power	95% CI	n
Anticlustering + unadjusted analysis	.669	[.668,.669]	900000
Rerandomization + unadjusted analysis	.669	[.668,.670]	900000
Minimization + unadjusted analysis	.670	[.669,.671]	900000
Random Assignment + unadjusted analysis	.670	[.669,.671]	900000
Random Assignment + adjusted analysis	.712	[.711,.713]	900000
Minimization + adjusted analysis	.719	[.718,.720]	900000
Anticlustering + adjusted analysis	.721	[.720,.722]	900000
Rerandomization + adjusted analysis	.721	[.720,.722]	900000

Note. Statistical power of all methods, aggregated across all simulation conditions when there was an effect of the condition (${\beta }_T>0$); $n$ refers to the numer of simulation runs in which ${\beta }_T>0$.

Figure 2 shows that statistical power of the unadjusted analyses decreased when the effect of the prognostic covariate on the dependent variable increased. Statistically adjusting for the covariate mitigated the reduction in power completely regardless of how experimental groups were assigned. When noise variables were included in the analysis, statistical power was reduced slightly for the adjusted analysis. Interestingly, using covariate-based assignment techniques prevented this reduction in power due to the presence of noise variables. Average power for the covariate-adjusted analyses with noise variables was .720 [.719, .721] with assignment via anticlustering, .720 [.718, .721] with assignment via rerandomization, .717 [.716, .719] with assignment via minimization, and .706 [.705, .707] with random assignment. When no noise variables were present, there was basically no difference in power between the assignment techniques: Average power for the covariate-adjusted analyses was .722 [.721, .724] with assignment via anticlustering, .723 [.722, .724] with assignment via rerandomization, .721 [.720, .723] with assignment via minimization, and .719 [.717, .720] with random assignment. Hence, the overall increase in power due to balancing shown in Table 1 was due to the presence of additional non-prognostic noise variables.

Click to enlarge

Figure 2

Statistical Power by Covariate Effect and Noise Condition

Note. Error bars indicate the 95% confidence interval.

Table 1 showed that balancing covariates by assignment did not improve statistical power, even though nuisance variance was better balanced between conditions than when random assignment was applied (see Figure 1). This result probably conflicts with the intuition of many researchers who might assume that balancing covariates increases statistical efficiency. Why do our results conflict with this intuition? In fact, we must differentiate, (a) how the long-term properties (i.e., statistical power) of statistical tests are influenced by balancing, and (b) how the properties of an individual study are affected by balancing. As our results indicate, the long-term frequency of significant results is not influenced by actively balancing. However, the probability that an individual study finds a significant result is affected by covariate imbalance. To corroborate this sentiment, we predicted the probability that a t-test found a significant result. In particular, we computed a logistic regression predicting statistical significance (1 = significant; 0 = not significant) using covariate imbalance $d_{cov}$ as predictor. Again, we only used the conditions when there was an effect of the treatment (${\beta }_T>0$).

The logistic regression coefficient of covariate imbalance was significant, $b=1.39$, 95% CI $[1.34,1.44]$, $z=56.80$, $p<.001$, showing that covariate imbalance was related to the probability of an individual study to find a treatment effect – if no statistical adjustment was applied. Figure 3 illustrates the relationship between covariate imbalance and statistical power, using the coefficients obtained from the logistic regression. It depicts a quite strong relationship between covariate imbalance and the probability to find an effect. Depending on the direction of the covariate imbalance, power can be increased or decreased. That is, if the treatment has a positive effect on the outcome and the covariate also has a positive effect, finding a significant effect is facilitated if there are higher average values of the covariate in the treatment condition. However, finding a significant effect is aggravated if there are lower average values of the covariate in the treatment condition. This result explains why the long-term power is not negatively affected by covariate imbalance; imbalance can also facilitate finding an effect if the imbalance “favors” the treatment condition. Note that covariate imbalance did not predict statistical power of the regression analysis that adjusted for the covariate, as shown by another logistic regression, $b=-0.02$, 95% CI $[-0.07,0.03]$, $z=-0.80$, $p=.427$.

Click to enlarge

Figure 3

Relationship Between Statistical Power and Covariate Imbalance for the Unadjusted t-Test

Note. The coloured vertical lines indicate the $1^{st}$ and $99^{th}$ percentile of observed covariate imbalance, depending on assignment method and the presence of noise variables. For random assignment, the dashed and solid lines overlap because covariate imbalance is not affected by the presence of noise variables if the assignment is entirely random. When no noise variables were included, anticlustering and rerandomization practically negated covariate imbalance. When noise variables were included, anticlustering led to the most pronounced reduction in covariate imbalance.

Alpha Error Probability

The following analysis only considered simulation conditions in which ${\beta }_T=0$ to assess the occurrence of false positive results, i.e., alpha errors. Table 2 illustrates alpha error probability for each method. Interestingly, only using covariate-assignment without applying statistical adjustment decreased the probability of committing an alpha error. The other methods did not depart from the nominal alpha level of.05. Figure 4 illustrates the alpha error probability in dependence of the covariate effect: When variation due to the covariate increased, using covariate-based assignment decreased the alpha error probability. Alpha error probability was most attenuated when the covariate effect was maximal (${\beta }_C=1$). While lower alpha errors may at first seem beneficial, they are indicative of a generally more conservative behaviour of the unadjusted test based on covariate-based condition assignment (e.g., Shao et al., 2010). In fact, the conservativeness obscured true statistical power, which was only revealed when using covariate-adjustment during analysis (see Table 1 and Figure 2).

Click to enlarge

Figure 4

Probability of Alpha Errors by Covariate Effect

Note. Error bars indicate the 95% confidence interval.

Table 2

Global Simulation Results

Analysis	% Alpha Error	95% CI	n
Anticlustering + adjusted analysis	.051	[.050,.052]	180000
Anticlustering + unadjusted analysis	.029	[.028,.030]	180000
Minimization + adjusted analysis	.051	[.050,.052]	180000
Minimization + unadjusted analysis	.030	[.030,.031]	180000
Random Assignment + adjusted analysis	.051	[.050,.052]	180000
Random Assignment + unadjusted analysis	.051	[.050,.052]	180000
Rerandomization + adjusted analysis	.051	[.050,.052]	180000
Rerandomization + unadjusted analysis	.029	[.029,.030]	180000

Note. Alpha error probability of all methods, aggregated across all simulation conditions when there was no effect of the condition (${\beta }_T=0$); $n$ refers to the number of simulation runs in which ${\beta }_T=0$.

Figure 5 illustrates the distribution of p-values in the condition with ${\beta }_C=1$ (i.e., the largest effect of the covariate) and ${\beta }_T=0$ (i.e., no condition effect) for the unadjusted analysis when using anticlustering assignment. Interestingly, the distribution of p-values was highly skewed, favoring larger p-values, thus explaining the low occurrence of alpha errors. With standard random assignment, p-values are uniformly distributed; that p-values diverge from the uniform distribution when using covariate-based assignment is quite striking and requires an explanation. In our simulation, when there was no treatment effect, variation in the dependent variable was generated based on the influence of the covariate (${\beta }_C$) and random error (${\mathrm{ϵ}}_i$). Because the covariate was actively balanced between conditions, all of the systematic portion of variance in the outcome was evenly distributed between conditions. For this reason, there was less difference in the outcome between conditions than would be expected under random assignment, skewing the distribution of p-values, and decreasing the probability of committing an alpha error. Though the resulting conservativeness of the unadjusted test has also previously been proven theoretically (e.g., Ma et al., 2015, 2024; Shao et al., 2010), we believe that it is of educational value to reproduce the consequence of such proofs in an empirical simulation.

Click to enlarge

Figure 5

Distribution of p-Values of the Unadjusted t-Test

Note. Illustrates the distribution of all p-values when using anticlustering-based assignment, and a large covariate effect (${\beta }_C=1$) and no condition effect (${\beta }_T=0$) were present.

Discussion

Our study set out to investigate anticlustering as a novel method to balance covariates among experimental conditions, which researchers have recently started doing (Even et al., 2023; Stadelmann et al., 2025; Tuti et al., 2022). Using a simulation study, we compared it with established methods for allocating experimental subjects to conditions: rerandomization and minimization, which are alternative methods to achieve balance in covariates (Morgan & Rubin, 2012; Treasure & MacRae, 1998), and standard random assignment. In doing so, we were able to locate the unfamiliar method of anticlustering in respect to these well-known and established methods. Moreover, we also provide some insight regarding the general question on whether actively balancing covariates or statistical adjustment of covariates is more important in the analysis of experiments.

In our simulation, we implemented the arguably most basic design used in experimental research: the independent two-group design, which can be analysed using the ubiquitous t-test or a regression (or ANCOVA) when a measured covariate is controlled for. Our results can be summarized as follows: When a covariate is prognostic of the outcome variable, statistical adjustment is crucial for maximizing statistical power (see Figure 2). Balancing covariates by itself does not increase power, and using it on top of statistical adjustment only marginally improves power if the analysis includes covariates that are not prognostic of the outcome. These results for example confirm an investigation by Coart et al. (2023). Regarding statistical power alone, our results therefore attest to the importance of statistical adjustment and not necessarily to the usefulness of balancing covariates. Hence, our results confirm the points repeatedly made by Senn (1989, 2005, 2013, 2022).

However, there are advantages to actively balancing covariates that could be shown by our study. Our results confirm that balancing via minimization, rerandomization and especially via anticlustering leads to less observed imbalance in covariates (see Figures 1 and 3). If researchers worry that observed covariate imbalance will decrease the credibility of their findings (cf. Senn, 2005; Treasure & MacRae, 1998), balancing does provide a potent counter-measure. Moreover, we find that balancing decreases the risk of having “bad luck” when no statistical adjustment is employed: There is less variance in statistical power if covariates have been balanced (see Figure 3). Imbalance in covariates can facilitate as well as aggravate detecting existing effects of the treatment — depending on whether the direction of the imbalance is consistent with the direction of the treatment effect. However, researchers may prefer to exert control on the level of imbalance instead of hoping for good luck that the true effect of interest is not cancelled out by the effect of the covariate. Moreover, we find that the standardized effect size $d$ of the treatment effect has less variability when covariates were balanced (see Table 3). Observed estimates of $d$ will tend to be closer to the true value when balancing was applied. This finding confirms previous theoretical results according to which balancing covariates increases the precision of the treatment effect (Hu et al., 2014; Morgan & Rubin, 2012).

Previously, researchers have highlighted that actively balancing covariates may invalidate the usual interpretation of statistical tests: These tests assume that true random sampling is used to allocate experimental subjects to experimental conditions (e.g., Moulton, 2004; Senn, 2013). When covariate-based assignment is used and the covariate is correlated with the dependent variable, the classical statistical tests are more conservative than required (Morgan & Rubin, 2012; Qin et al., 2024). Our study confirms these previous theoretical results. For the unadjusted analysis, we find that the actual alpha error departs from the nominal alpha error when covariates have been balanced. Balancing decreased the probability of alpha errors, which is arguably less problematic than an increase would be. However, we do not recommend regarding the reduced alpha error rate as beneficial, because it was due to a more conservative test behaviour that came at the cost of statistical power. Instead, we recommend using statistical adjustment of the covariate in combination with covariate-based assignment, which maximizes power and maintains the nominal alpha error level. An alternative to statistical adjustment via regression would be to implement a randomization test that incorporates the logic of the covariate-based assignment process into the significance test (Bugni et al., 2018; Ma et al., 2015; Morgan & Rubin, 2012; Shao et al., 2010). These tests can increase statistical power and maintain the nominal alpha level without employing statistical adjustment. In practice, however, it seems that randomization tests are rarely used (Lin et al., 2015) and they are not readily available in standard software (Ohashi, 1990). In particular, randomization tests must incorporate the specific procedure that was used for condition assignment, which cannot be captured by software defaults. In contrast, statistical control of a covariate is always possible and — as confirmed by our results — highly useful (e.g., Raab et al., 2000; also see Senn, 1989). Moreover, for anticlustering, a randomization test must still be developed, which is certainly an interesting venue for future research.

In general, all balancing techniques that we investigated (minimization, rerandomization and anticlustering) provided rather comparable results across the dependent variables. Anticlustering and rerandomization were more successful at minimizing observed covariate imbalance than minimization. This is not surprising, given that anticlustering and rerandomization use all information simultaneously to maximize balance. Minimization is a sequential method that assigns each subject one-by-one after another. It operates using less information. Rerandomization and anticlustering obtained highly similar results across all dependent variables. Again, this is not surprising, given that their objective functions are basically equivalent for uncorrelated covariates — at least when using the k-means objective for anticlustering, which was applied in the simulation study. When there were several covariates (in the noise condition) anticlustering achieved increased balance as compared to rerandomization. This is because anticlustering uses a more potent optimization algorithm than rerandomization, which relies on repeated resampling without systematically searching for improved balance.

With regard to statistical power and the probability of alpha errors, minimization, rerandomization and anticlustering provided very similar results (see Table 1 and Table 2). Therefore, we expect that the question whether to apply anticlustering, rerandomization or minimization will not depend on general superiority of either method; instead, it probably depends on the conditions of the data inquiry: If information on all subjects is available at the start, researchers have the opportunity to apply anticlustering or rerandomization. If subjects are recruited sequentially, however, anticlustering or rerandomization cannot be applied, but minimization can.

With regards to the relative merits of anticlustering versus rerandomization, the present study establishes practical equivalence between the two methods. Again, which method should be applied may depend on the circumstances. The anticlustering method is readily available for researchers via the open source and free software package anticlust (Papenberg & Klau, 2021) that has been maintained for several years and includes much documentation and convenience functionality. Anticlustering also has the particular advantage that it can be applied for more than two experimental conditions, which is so far lacking for rerandomization. Moreover, anticlustering can easily be adapted for more advanced use cases such as balancing higher order moments instead of only covariate means (Papenberg, 2024), or including constraints on group assignments (Papenberg, Breuer, et al., 2025; Papenberg, Wang, et al., 2025). Rerandomization on the other hand has received more scrutiny on a theoretical level for the purpose of balancing covariates in experiments (Johansson et al., 2021; Morgan & Rubin, 2012); the origin of anticlustering has not been in the realm of statistical inference (Brusco et al., 2020; Späth, 1986). We therefore call for further investigation of rerandomization and anticlustering in different contexts to establish additional evidence on the relative usefulness of the two methods.

Based on our findings, we provide the following guidance on when anticlustering is likely to be most beneficial in practice. First, our findings suggest that balancing covariates in general is useful and should be done if possible. In nonsequential designs, when covariate measurements are available at the beginning of an experiments, anticlustering is a suitable method to achieve balance: It increases the precision of the effect size estimation; by increasing balance between conditions it potentially increases interpretability and trustworthiness of the results; when no statistical adjustment is feasible, it reduces variability of statistical power of an unadjusted statistical analyses. However, we generally recommend using covariate-adjustment during analysis whenever possible, while adhering to good practice in covariate selection (e.g., Raab et al., 2000; Wysocki et al., 2022). Balancing via anticlustering is not a substitute for statistical adjustment. Using both covariate-based assignment and statistical adjustment of covariates provides the best of two worlds. Our results are in line with official guidelines for medical research (e.g., European Medicines Agency, 2015) that explicitly recommend: (1) assignment methods that ensure covariate balance, and (2) covariate-adjusted analyses.