Mixture Multigroup Structural Equation Modeling (MMG-SEM; Perez Alonso et al., 2024) combines mixture modeling with MG-SEM. In general, the mixture multigroup approach (De Roover, 2021; De Roover et al., 2022), aims to find a clustering that focuses on specific parameters of interest. In MMG-SEM, the clustering focuses on the structural relations, while MM differences are accounted for by group-specific parameters, so they do not affect the clustering.

For its estimation, Perez Alonso et al. (2024) used the ‘Structural-After-Measurement’ (SAM; Rosseel & Loh, 2022) approach, which estimates a SEM model in two steps. In the first step, the MM is estimated, whereas the structural model (SM; including the structural relations) is estimated in the second step. The estimation of MMG-SEM is briefly described below (for more details, see Perez Alonso et al., 2024).

A common challenge for clustering methods is selecting an appropriate number of clusters. Researchers have tried to solve this problem based on different approaches, such as balancing model fit and complexity (Akaike, 1974; Akogul & Erisoglu, 2016; Schwarz, 1978), considering relative fit improvement (Ceulemans & Kiers, 2006), cluster separation (Biernacki et al., 2000), and/or substantive interpretation (van den Bergh et al., 2017). Given the popularity of clustering, new methods for model selection keep emerging. However, we focus on commonly used approaches for model selection in the context of mixture SEM methods, which are detailed below.

The aim of the simulation study was to compare the performance of six model selection measures in selecting the number of clusters for MMG-SEM: AIC, AIC3, BICG, BICN, CHull, and ICL. To this end, we used a Monte-Carlo simulation with six manipulated factors that are expected to affect the model selection performance. The factors and their corresponding levels are described below:

Size of regression parameters β: 0.3, 0.4;

The size of the regression parameters (i.e., 0.3 and 0.4) was inspired by previous simulation studies in SEM (e.g., Guenole & Brown, 2014; Perez Alonso et al., 2024) and cross-national empirical studies with many groups (e.g., Bastian et al., 2014; Kuppens et al., 2008). Similarly, the number of groups was 24 and 48, which correspond to a range of groups that is generally found in empirical large-scale international surveys (Rutkowski & Svetina, 2014). The number of clusters (i.e., 2 and 4)⁶6

A smaller simulation where K = 1 was also performed but not included in the main text due to space constraints. More details can be found in Perez Alonso et al. (2025)

In total, the design included 2 (size of regression parameters) × 2 (number of groups) × 3 (within-group sample size) × 2 (number of clusters) × 2 (cluster size) × 3 (within-cluster differences) = 144 data generation conditions. For all conditions, 100 different data sets were generated, for a total of 14400 data sets. To evaluate the model selection measures, each data set was analyzed six times with MMG-SEM from one to six clusters, for a total of 14400 × 6 = 86400 analyses. Note that we added non-invariances to the loadings (see the Data Generation section for more information) and that such non-invariances are correctly modeled in MMG-SEM. All the data generation and analyses were done using R Version 4.3.3 (R Core Team, 2024), and the code is openly shared at Perez Alonso (2024). The data generation procedure is described below.

Each data set was generated according to the SEM model in Figure 1, with four LVs, each one measured by five indicators, for a total of 20 observed variables. The structural relations, which are the parameters of interest for the clustering, are represented by four regression parameters. F1 and F2 served as exogenous variables, while F3 and F4 were endogenous variables. Note that F3 acts as a ‘mediator’ and is, thus, an exogenous and endogenous variable at the same time.

The sample size per Group N_g, the number of Groups G, and the number of clusters K were defined according to the manipulated factors ‘within-group sample size’, ‘number of groups’, and ‘number of clusters’, respectively. The number of groups per cluster was manipulated according to the ‘cluster size’. In the balanced condition, the groups were equally divided per cluster. For instance, for G = 48 and K = 4, each cluster contained 12 groups. In contrast, in the unbalanced condition, there was one larger cluster with 75% of the groups, and the remaining clusters were equally sized. For example, for G = 48 and K = 4, the first cluster would contain 36 groups and the remaining three clusters would contain four groups each.

The 20 observed variables were generated from a MVN(μg,Σgk), where the mean vector μg was a vector of zeros and the covariance matrices Σgk were generated according to Equation 9. Thus, to generate the data, the parameters in Equation 9 (i.e., Λ,Θg,Bk, and Ψgk) must be defined. The Λ and Θg matrices were generated aiming to obtain a total variance per item around 1 and a reliability (R²) of 0.6 for each observed indicator. To do this, the non-zero values in Λ were set to 0.6 while the residual variances in Θg were drawn from a uniform distribution U(0.3, 0.5). Note that the residual variances in Θg were sampled for each group, allowing group-specific differences as specified in Equation 9.

To evaluate the effect of loading non-invariances on the model selection, we also added between-group differences to the Λ matrices. In particular, 50% of the groups presented non-invariances. For each non-invariant group, we applied the non-invariance to the second and third loading of each factor (the first loading is fixed to 1). The non-invariances were randomly sampled from a uniform distribution around 0.4, i.e., U(0.3, 0.5), and it was randomly decided whether the non-invariance was added or subtracted to the original loading (i.e., 0.6). As a result, each non-invariant loading was different for each non-invariant group.

The setup of the regression parameters in Bk can be seen in Figure 2. The manipulated factor ‘size of the regression parameters’ (β) indicated the size of the coefficients β₁, β₂, β₃, and β₄. The difference between the clusters was created by setting one of those coefficients to zero in each cluster. Thus, the size of the regression parameters also defined the size of the difference between the clusters. Note that, when K = 2, Models 3 and 4 in Figure 2 were not applicable. The parameters in Bk were also affected by the manipulated factor ‘within-cluster differences’ (σβ). To simulate empirically realistic conditions, we added small differences in the coefficients β to each Group g within a Cluster k. To do this, within each cluster, the regression parameter of each group was drawn from a normal distribution N(β,σβ), where the β and the σβ acted as the mean and the standard deviation of the distribution, respectively. The values of σβ implied no within-cluster differences (σβ=0), small differences (σβ = 0.05), or large differences (σβ=0.1). Note that we expect 99% of the sampled values to be within three standard deviations of the mean when drawing from a normal distribution. For instance, if we consider β = 0.3 and σβ=0.05 and we focus on β₁ (see Figure 2), the values of β₁ in Cluster 1 will be sampled from N(0, 0.05), whereas they will be sampled from N(0.3, 0.05) in Cluster 2. We expect 99% of the values to lie between -0.15 and 0.15 in Cluster 1 and between 0.15 and 0.45 in Cluster 2. Thus, the small differences level (σβ=0.05) led to almost no overlap between clusters, whereas the large level (σβ=0.1) ensured overlap between the clusters.

Finally, the elements in Ψgk were defined by sampling the variance of the exogenous variables F1 and F2 from a uniform distribution U(0.75, 1.25), and their covariance from U(−0.3,0.3) for all groups. Similarly, the total variance of the endogenous factors F3 and F4 was sampled from U(0.75, 1.25) for each group, and their residual variance depended on the cluster-specific regression parameters. For example, if the total variance of F3 for Group g was Var_tot, the residual variance Var_res for Group g and Cluster k was Varres=Vartot−(β22Var(F1)+β32Var(F2)+2β2β3Cov(F1,F2)).

To determine the cluster separation in the simulated datasets, we applied the R² Entropy, an Entropy-based measure. Specifically, the R² Entropy indicates how well the observed responses predict the cluster memberships z^gk (for details on its calculation, see Vermunt & Magidson, 2016). It takes on a value of 1 when the clusters are perfectly separated (i.e., no classification uncertainty) and a value of 0 when there is no separation at all.

To get an overview of how the cluster separation was affected by the simulation conditions, we evaluated the R² Entropy at (an approximated) population level. For this, we generated data for each simulation condition with a large sample size. Given that MMG-SEM’s clustering is at the group-level, the relevant sample size for the clustering and R² Entropy is the number of groups G, which was set to 192 groups. Subsequently, we computed the cluster memberships z^gk based on the true parameters values for Λ,Θg,Bk, and Ψgk (i.e., they were used as starting values in an MMG-SEM analysis where no parameter updates were performed). Per condition, 300 replications were generated.

The R² Entropy ranged from 0.76 to 1 with an average of 0.93 across all data sets, which indicated well-separated clusters overall. The R² Entropy was mostly influenced by the within-cluster differences σβ, the regression coefficients β, and the within-group sample size N_g. Their main effects can be seen in Table 1, and the interaction between σβ and β is shown in Figure 3. Lower R² Entropy values were found in difficult conditions involving large within-cluster differences (σβ=0.1), lower regression coefficients (β = 0.3) and/or low within-group sample size (Ng=50). It is expected that the model selection measures will struggle to choose the correct number of clusters in conditions where the clusters are not well separated.

For each model selection measure, we assessed how often it correctly selected the true number of clusters. To gain a deeper understanding of each measure, we also inspected how often it over- and under-selected the number of clusters. The main effects of the manipulated factors of the simulation can be seen in Table 2. In total, the best-performing method was the CHull, followed by the AIC, AIC3, BICG, ICL, and BICN, with a proportion of correctly selected models of 0.77, 0.66, 0.64, 0.63, 0.61, and 0.54, respectively. Note that the similarity between the results of AIC3 and BICG can be partially explained by the similarity of their penalties (see Equations 11 and 12); that is, AIC3’s penalty is 3, whereas BICG’s penalty is 3.18 and 3.87 when G is 24 and 48, respectively.

The within-group sample size N_g was most influential on the model selection performance. Specifically, on average, the model selection measures were correct 47% and 74% of the times when Ng=50 and Ng=100, respectively. Such dramatic improvement did not hold when N_g increased to 200, for which the measures were correct 72% of the time. The improvement from Ng=50 to Ng=100 aligns with common sample size requirements in SEM, since 100 is considered the minimum for consistent estimates (Gorsuch, 1983). The slight decrease in performance when Ng=200 can be explained by the trends of under- and over-selection of the number of clusters. Generally, when choosing the incorrect model, the measures tended to under-select. However, over-selection was more prominent in the case of a large within-group sample size (Ng=200). Such results are unsurprising, considering that larger sample sizes give more power to identify smaller differences (leading to more clusters). This is more likely in case of larger within-cluster differences (σβ=0.1), which can be identified as additional clusters. This can be clearly seen in Figure 4, where the model selection performance dramatically drops when Ng=200 and σβ=0.1.

The model selection performance was also greatly affected by the number of clusters K. On average, all model selection measures found it more difficult to identify the correct model when more clusters were underlying the data. From Table 2, the proportion of correctly selected models was better when K = 2 (0.82) than when K = 4 (0.46). Such results are in line with previous research where an increase in the true number of clusters dramatically decreased the performance of the model selection measures (Bulteel et al., 2013; Lukočienė et al., 2010) even when the cluster separation is high (Steinley & Brusco, 2011). Similarly, the effect of the cluster size on model selection showed a comparable trend. On average, the model selection measures were correct 72% of the time when the cluster size was balanced and 57% when it was unbalanced. The effect of both the number of clusters and cluster size can be explained by lower within-cluster sample sizes in case of more and/or unbalanced clusters, which reduced the power to detect the appropriate model. The model selection was also, to a lesser extent, affected by the regression coefficients β. Specifically, a lower regression coefficient and unbalanced cluster sizes lowered the proportion of correctly selected models.

The interaction between two of the most important factors can be seen in Figure 5. The plot clearly shows that more clusters and large within-cluster differences led to a dramatic decrease in the performance of all model selection measures. Specifically, the AIC, AIC3, BICG, and ICL presented a substantial decrease of the correctly selected number of clusters. In contrast, the CHull and BICN presented a lower decrease in performance when K = 4 and/or σβ=0.1. BICN's performance was generally worse than all other model selection measures, however.

From Figures 4 and 5, we also learn that the performance sometimes improved when going from σβ=0 to σβ=0.05, especially for the CHull. For the CHull, this can be explained by the saturation effect, which happens when adding more clusters results in a negligible increase in the log-likelihood. This can lead to an artificially large scree ratio (because the denominator approaches zero, see Equation 13), whereas, when looking at the scree plot, a virtually horizontal line, rather than an elbow, is visible at this point. In empirical practice, one can remedy this problem by looking for a clear elbow in the scree plot instead of just relying on the scree ratios.

The comparison between CHull and the other measures can be considered unfair, since CHull selects a model with at least two clusters. Thus, when K = 2, the other model selection measures may select a one-cluster model when the cluster separation is low, while CHull will select at least two clusters ⁷7

As can be seen in Table 2, CHull never under-selects the number of clusters when K = 2

Finally, for a more comprehensive understanding of the outcomes, we examined how often the model selection measures correctly identified the number of clusters when considering the two best models (e.g., how often is the correct model among the two models with the lowest AIC values). The correct model was among the two best models 78.5%, 72.7%, 73.1%, 72.9%, 73.1%, and 69.3% of the times for CHull, AIC, AIC3, BICG, ICL, and BICN, respectively. The BICG, BICN, and ICL showed the largest improvements in performance compared to when we focus only on the best model.

The model selection results must be interpreted in light of the cluster recovery (i.e., to what extent MMG-SEM assigns groups to the correct clusters when the correct number of clusters is specified). To this end, we computed the Adjusted Rand Index (ARI; Hubert & Arabie, 1985) for the model with the true number of clusters. For instance, if K = 2, we computed the ARI for the model with two clusters (regardless of what the model selection measures chose). The ARI compares two partitions (i.e., modal assignments of the groups to a cluster), taking on a value of 1 for complete agreement and 0 when agreement does not exceed that between two random partitions. Note that it can take on negative values if the agreement is less than what is expected at random. The average ARI across all conditions was good (0.93), but it ranged from -0.10 to 1. Per model selection measure, we also inspected the average ARI across data sets depending on whether the number of clusters was under-, over-, or correctly selected (Table 3). Clearly, selecting the incorrect model (i.e., under- or over-selection) was related to the ARI being lower for the correct model. Specifically, the average ARI was below 0.89 for all measures when the selected model was incorrect, while it was above 0.97 when the correct model was selected.

Before drawing conclusions, it is important to note that, strictly speaking, there are as many clusters as there are groups when σβ>0 (i.e., in case of within-cluster differences). As MMG-SEM does not capture within-cluster differences, the model selection could suggest extracting more clusters or even capturing each group as a separate cluster, especially when there is enough power to find small differences. However, in this study, we still assumed the true number of clusters to be K, since it is desirable to assign groups with very similar regression parameters to the same cluster (see Introduction). Considering every group as a separate cluster would boil back down to an MG-SEM with group-specific relations and the pesky pairwise comparisons thereof. As this is what we wanted to avoid, we did not include the MG-SEM model in this study.

In an extensive simulation study, we assessed six different model selection measures for MMG-SEM. As expected, lower within-group sample sizes (Ng=50), large within-cluster differences (σβ=0.1), more clusters (K = 4), and unbalanced cluster sizes decreased the performance of all measures. Overall, the best-performing measure was the CHull and the worst was the BICN. AIC, AIC3, BICG, and ICL presented similar performances with minor differences depending on specific conditions.

Considering our results, we suggest using the CHull when performing model selection for MMG-SEM. Since it cannot select the minimum or maximum number of clusters, we suggest also inspecting CHull’s scree plot and looking for an elbow to confirm the number of clusters. If no elbow is visible, the most appropriate number of clusters is likely one or the maximum number evaluated. Furthermore, since CHull was not universally best across all data sets⁸8

For instance, out of the total 14400 data sets: (1) AIC and CHull were both correct in 8377 cases; (2) AIC was incorrect and CHull correct in 2754 cases; and (3) AIC was correct and CHull incorrect in 1105 cases.