A General Framework for Planning the Number of Items/Subjects for Evaluating Cronbach’s Alpha: Integration of Hypothesis Testing and Confidence Intervals

Cronbach’s alpha (Cronbach, 1951) stands as one of the most widely used coefficients reflecting the interrelatedness of items (Sijtsma, 2009; Sijtsma & Pfadt, 2021). Despite considerable debate surrounding its utilization (Cho & Kim, 2015; Cortina, 1993; Green & Yang, 2009; Kelley & Pornprasertmanit, 2016; McNeish, 2018; Raykov & Marcoulides, 2019; Schmitt, 1996; Sijtsma & Pfadt, 2021), the estimation and the testing of the alpha coefficient have gained critical attention in applied settings. Ensuring the reliability of a measure remains crucial for correctly interpreting the effects of experimental variables. However, it has been evidenced that most reliabilities fall short of ideal standards in making precise, confident decisions (Charter, 2003a). Furthermore, as investigative tasks heavily rely on sample information, determining sample size in the initial stage of research design is pivotal in reducing sampling errors. While systemic studies have discussed sample size calculation, particularly in Intraclass Correlation Coefficient (ICC) studies (Donner & Eliasziw, 1987; Shieh, 2014a; 2014b; Shoukri et al., 2004; Walter et al., 1998), the discussion regarding Cronbach’s alpha remains relatively scarce.

Estimating the necessary number of participants to yield meaningful results proves challenging (Charter,1999; Cocchetti,1999; Flight & Julious, 2016; Peterson & Kim, 2013). On one hand, the number of subjects might be too small to produce sufficiently precise reliability coefficients or enough statistical power for hypothesis tests (Charter, 2003b; Heo et al., 2015; Yurdugül, 2008). On the other hand, the number of measurements (items, raters) might be too large to lack cost-effectiveness (Hsu, 1994; Overall & Dalal, 1965). Note that the magnitude of the coefficient alpha is contingent upon the number of items, with a curvilinear relationship (Komorita & Graham, 1965). This aspect necessitates further investigation into subject/item size determination. Specifically, this involves considerations of constructing confidence intervals and assessing cost-effectiveness in reliability estimation.

The conventional practice of reporting alpha coefficients as point estimates impedes interpretation and replication (Terry & Kelley, 2012) since the alpha estimate is influenced by variance sources and contains unknown-direction sampling errors. Recommending the reporting confidence intervals (CIs) aims to enhance the trustworthiness of reliability (American Psychological Association [APA], 2001; Bonett & Wright, 2015; Fan & Thompson, 2001; Iacobucci & Duhachek, 2003; Kelley et al., 2003) and to convey information related to precision and reproducibility, especially in cases of very large or small sample sizes (Mendoza & Stafford, 2001). However, the mere use of CIs does not inherently enhance statistical practice (Cumming, 2014; Morey et al., 2016) without proper sample size planning (Liu, 2009). Liu (2012) also noted that current sample size planning typically aims to achieve the power of a statistical test under specified alternative hypotheses, rather than constructing precise confidence intervals. Notably, sample size influences CI width (Charter, 1999). The process of planning sample sizes to obtain CI precision has some parallels to planning for statistical power but often results in significantly different sample size requirements (Borenstein et al., 2001; Goodman & Berlin, 1994). Moreover, researchers lack well-established criteria for determining CI widths (Smithson, 2003), and may overlook the stochastic nature of interval width, i.e., a CI width is a random variable, where approximately half the time, the computed CI width exceeds the desired width in repeated sampling (Terry & Kelley, 2012), potentially underestimating required sample sizes (Liu, 2009).

In light of the various applications of integration of hypothesis testing and confidence intervals for obtaining the needed sample sizes, the present study employed the concept of event rejection, validity, and width (Jiroutek et al., 2003), defined as follows: an event rejection (R) is said to occur if the null value of Cronbach’s alpha is rejected; an event validity (V) is said to occur if Cronbach’s alpha is contained between the upper and lower CI limits, and an event width (W) is said to occur if the width of the CI is no larger than the desired width w. The probabilities of the aforementioned events are then denoted as $P(R)$, $P(V)$, and $P(W)$, respectively. There are combinations of these events for various scenarios. Consider a scenario where a researcher aims to limit the width of a CI, conditional upon Cronbach’s alpha falling between the lower and upper CI bounds. Determining the necessary sample size in this case ensures $P(W|V)$ achieving a probability at a desired level of $1-\beta$. Another instance arises when, alongside rejecting the null hypothesis, there's a desire to construct a confidence interval within a specified width. Calculating the required sample size for reporting Cronbach’s alpha aims to warrant that $P(W\cap R)$ can achieve a desired probability of $1-\beta$. This integrative approach to sample size planning, addressing multiple conditional probabilities, remains infrequently explored in literature, apart from the work by Terry and Kelley (2012), who focused on $P(W)\ge 1-\beta$ for composite reliability coefficients. Other studies have discussed various conditional probabilities (Beal, 1989; Jiroutek et al., 2003; Liu, 2012). Our study addresses a total of nine unconditional/conditional probabilities (cases), namely:

In this regard, being able to consider these probabilities can provide researchers with a thoughtful and comprehensive scenario that was not treated in much detail in the past. Also, combining hypothesis testing and a CI estimate is essential because a power-based approach can help avoid the ethical issues raised by recruiting too many/few participants (Cesana, 2013; Maxwell & Kelley, 2011).

Yet another crucial consideration within reliability studies involves balancing the cost of data acquisition against the precision/accuracy of estimates. Surprisingly, a cost-efficient design, rooted in health economics, often remains neglected (Rezagholi & Mathiassen, 2010). In practical applications, the acquisition of raters or certain measurements entails a considerable expense. When prioritizing budget constraints, obtaining substantial information at minimal cost necessitates optimizing the configuration of both the number of measurements (i.e., items, raters) and subjects (or observations) (Shoukri et al., 2003). To address these pivotal concerns, this study proposes the development of a framework that integrates hypothesis testing and confidence intervals. The aim is to determine the optimal number of measurements/subjects, guided by two key objectives: (a) minimizing the total cost for a desired probability, and (b) maximizing the probability of interest within a predefined total cost. Additionally, to enhance accessibility for researchers, this proposed procedure has been translated into two R Shiny apps (Diedenhofen & Musch, 2016; R Development Core Team, 2020). Leveraging rapid advancements in computing technology, there is a renewed opportunity to stimulate interest in sample size planning, specifically exploring various conditional probabilities.

The subsequent sections of this study are organized as follows. The Measurement Model section delves into an elucidation of Cronbach’s alpha using a measurement model. In the section, Method for Acquiring Number of Measurements and Subjects, we detail the methodology for acquiring pairs of measurements/subjects for $P\text{(}R\text{)}$, $P\text{(}V\text{)}$, and $P\text{(}W\text{)}$, respectively. The Proposed Apps section showcases the functionality of the proposed apps concerning objectives (a) or (b) with an illustrative example. Moving to the Tables and Simulations section, we present three tables and simulation results. Finally, the Discussion and Conclusions section encapsulates the study with some best-practice suggestions.

The Measurement Model

Cronbach’s $\alpha$ describes the reliability of a sum (or average) of m measurements (test items, raters, occasions, or alternative forms). To evaluate the Cronbach’s $\alpha$ coefficient, a model for the parallel- measurements score $y_{ij}$ is given as:

where $t_i$ is the true score of subject $i$ and $e_{ij}$ is the error of measurement $j$ for subject $i$, $i=1,\mathrm{...},n$ (subjects); $j=1,\mathrm{...},m$ (measurements). We also assume that { $t_i$} are normally and identically distributed with mean 0 and variance ${\sigma }_s^2$, { $e_{ij}$} are normally and identically distributed with mean 0 and variance ${\sigma }_e^2$; and { $t_i$} and { $e_{ij}$} are independent. That is, the random vector $(y_{i1,}\mathrm{...},y_{im})$ is distributed as a multi-normal distribution with mean 0 and a covariance of $\Sigma ={\sigma }_s^{2}11^{\prime }+{\sigma }_e^{2}I$. Based on Feldt (1965, 1969) and Kraemer (1981), the estimated Cronbach’s coefficient can be expressed as

where $M{S}_{M\times S}$ is the mean square for measurement by subject, $M{S}_S$ is for subjects, and $\widehat{\alpha }$ estimates the population value of Cronbach’s alpha $\alpha$ ( $0<\alpha <1$) as

Based on Yurdugül (2008) and Heo et al. (2015), Equations (2) and (3) can be re-expressed as

where S is the unbiased sample covariance matrix; $1^{\prime }$ is the transpose of column vector $1$ with m unit elements; $\Sigma$ is the variance-covariance matrix of the population; and $tr(\Sigma )$ is the sum of the diagonal elements of the square matrix $\Sigma$. Note that under the assumption of parallel measures (two measures have identical true scores and equal error variances), we can obtain $\Sigma ={\sigma }_s^{2}11^{\prime }+{\sigma }_e^{2}I$, where $I$ is an identity matrix; hence, Equation (4) is identical to Equations (2) and (3). Also note that the coefficient alpha is satisfactory if the less restrictive essentially tau-equivalent assumption (i.e., unequal variances but equal covariances) holds (Sijtsma & Pfadt, 2021) in the case of approximate unidimensionality.

From Kristof (1963) and Feldt (1965), we know that $(1-\alpha )/(1-\widehat{\alpha })$ is distributed as a central F-distribution with ( $n-1$) and $(n-1)(m-1)$ degrees of freedom. Therefore, we define the test statistic

which is distributed as an F-distribution with $vn=(n-1)(m-1)$ and $vd=(n-1)$ degrees of freedom. In the present study, to find the number of measurements/subjects, we applied distribution F, based on the distributional theory derived by Feldt (1965) and Feldt et al. (1987), described in the Method for Acquiring Number of Measurements and Subjects section.

Method for Acquiring Number of Measurements and Subjects

In this section, we consider events rejection, validity, and width and their corresponding $P\text{(}R\text{)}$, $P\text{(}V\text{)}$, and $P\text{(}W\text{)}$, respectively to acquire pairs of measurements/subjects by using F distribution. First, to enhance the clinical interpretation of testing Cronbach’s alpha, the null hypothesis against the right-tailed alternative hypothesis is considered in the following manner:

Note that the hypothesis testing described here does not invoke the nil null hypothesis that the score reliability is 0 (Fan & Thompson, 2001) but instead ${\alpha }_0$ is a golden standard or a particular criterion (Kuijpers et al., 2013; Nunnally & Bernstein, 1994). For a significance level $\delta$, to test the null hypothesis $H_0:1-\alpha \ge 1-{\alpha }_0$, we have

where $F=(1-\widehat{\alpha })/(1-{\alpha }_0)$ and $F_{\delta ,(vn,vd)}$ is the ${\delta }^{th}$ quantile of distribution F with $vn=(n-1)(m-1)$ and $vd=(n-1)$ degrees of freedom. Hence, based on $F<F_{\delta ,(vn,vd)}$, $H_0$ can be rejected when

which can be defined as an event rejection (R). For the alternative hypothesis with a specified value ${\alpha }_1>{\alpha }_0$, the power function will coincide with P(R), the probability of the event rejection, as

where $F=(1-{\widehat{\alpha }}_1)/(1-{\alpha }_1)$. To achieve the desired power $1-\beta ,$ we must set

Then, we can find various pairs of measurement m with its corresponding number of subjects n to satisfy Equation (10), that is, to satisfy $P(R)\ge 1-\beta$.

Second, for $P\text{(}V\text{)}$, various numbers of measurements/subjects for constructing two-sided and one-sided CIs of coefficient ${\alpha }_1$ with a desired probability are described. A confidence level is set to $1-\delta$ (i.e., the probability of the event validity, $P(V)=1-\delta$). To form a two-sided CI, from Equation (5) and by Feldt et al. (1987), it can be shown that

where $F=(1-{\widehat{\alpha }}_1)/(1-{\alpha }_1)$. The lower confidence limit (LCL) and the upper confidence limit (UCL) are denoted as $LCL=1-(1-{\widehat{\alpha }}_1)/F_{\delta /2,(vn,vd)}$ and $UCL=1-(1-{\widehat{\alpha }}_1)/F_{1-\delta /2,(vn,vd)}$, respectively. Hence, a ( $1-\delta$) 100% two-sided CI is [LCL, UCL], for which the width is

an increasing function of $(1-{\widehat{\alpha }}_1)$. In some contexts, there is rationale in acquiring a one-sided CI, structured as [LCL, 1]. It can be shown that

Here, $LCL=1-(1-{\widehat{\alpha }}_1)/F_{\delta ,(vn,vd)}$ and $UCL=1$. The width of the one-sided CI is defined as

From Equations (11) and (12), it is known that the width of a CI is an increasing function of $(1-{\widehat{\alpha }}_1)$ given $\delta$, $m$, and $n$. Hence, the width is a random variable.

Third, we define the event width ( $W$) as $UCL-LCL\le w$ for a two-sided CI, or $1-LCL\le (1-{\alpha }_{\text{1}})+w/2$ for a one-sided CI, where w is the desired width chosen as sensibly as $0<w\le (1-{\alpha }_0)$. The probability of the event width is

where $F=(1-{\widehat{\alpha }}_1)/(1-{\alpha }_1)$; $w^a=w$ and $F_{{}_{(vn,vd)}}^a$= $(1/F_{\delta /2,(vn,vd)}-1/F_{1-\delta /2,(vn,vd)})$ for a two-sided CI; and $w^a=(1-{\alpha }_1)+w/2$ and $F_{{}_{(vn,vd)}}^a$= $1/F_{\delta ,(vn,vd)}$ for a one-sided CI, respectively. Then, to achieve $P\text{(}W\text{)}\ge 1-\beta$, we must set

by replacing ${\alpha }_1$ with a planning value obtained from expert opinion or prior research (Bonett, 2002). Then, pairs of (m, n) can be obtained to satisfy Equation (14). For other unconditional/conditional probabilities, the pairs can be obtained by using the proposed apps demonstrated in The Proposed Apps section.

The Proposed Apps

In the framework of integration of hypothesis testing and CIs for nine unconditional/ conditional cases, when the budget is the primary concern, we derived an optimal pair (m, n) under cost constraint. Let $c_m$ represent the cost per measurement, $c_s$ represent the cost per subject, and $c_{ms}$ represent the cost per observation. The total cost can be expressed as

when a pair of ( $m,n$) is given (Eliasziw & Donner, 1987). To be ethically and economically feasible for objectives (a) and (b) outlined in the introduction, we have developed two R Shiny apps described as follows to either minimize the total cost for a desired probability or to maximize the probability of interest within a predefined total cost.

For objective (a), App (I) (see Luh, 2024a) is designed based on the section Method for Acquiring Number of Measurements and Subjects, employing an exhaustive search method. To use App (I), researchers start by selecting the specific event of interest (case), setting a significance level, a desired probability, a planning value for an alternative ${\alpha }_1$, and determining the number of measurements up to which all outcomes will be printed. For cases related to the event (R), researchers input a null hypothesis value which should be smaller than ${\alpha }_1$. Regarding cases linked to the event (W), users specify the desired width of CI and whether it’s one- or two-sided. Additionally, if there is a cost constraint, they can specify the unit cost of measurement, subject, and observation can be specified (refer to Figure 1). Upon entering these values, clicking “Submit” executes App (I), displaying a list of measurement-subject pairs that satisfy the desired probability. Among the pairs with the minimal total cost, an optimal pair with the maximal probability for objective (a) is highlighted at the bottom of the output.

For objective (b), the probability of interest is maximal for a given cost (C). From Equation (15), we have $C-c_{m}m=(c_s+c_{ms}m)n$. Thus, for given measurements $m$, the corresponding number of subjects is $n=(C-c_{m}m\text{)}/(c_s+c_{ms}m)$. The optimal pair that has a maximal probability can be derived by using an exhaustive algorithm. We offer App (II) (see Luh, 2024b) for a user-friendly application. Researchers need to specify the event of interest and the corresponding parameters. Additionally, the fixed total cost (C), as well as the unit cost of measurement, subject, and observation, are required. The output presents the optimal measurement-subject pair along with its corresponding total cost and the maximum attainable probability within this total cost.

In the following, we utilized an example from Bonett (2002) to demonstrate the functionality of the proposed apps. For objective (a), to test $H_0:\alpha \le {\alpha }_0=.7$ versus $H_1:\alpha ={\alpha }_1=.8$ at $\delta =.025$, with a desired probability (power) of .9 and a given number of measurements $m=4$, App (I) indicated the required number of subjects for $P(R)$ as $n=170$, a result close to Bonett’s 173. Additionally, aiming for a desired precision, $P(W)$, Bonett (2002) set the planning value ${\alpha }_1=0.7$, the desired absolute precision of 0.2 with 95% confidence for a two-sided CI, which yielded a total of 99 subjects. However, our simulation revealed an empirical probability of .5429, roughly equal to a probability of 1/2. Using the statpsych R package, the sample size was determined as 95 by the command size.ci.cronbach(.05, .7, 4, 0.2), still falling short of the desired probability (1- $\beta$) of .8. Contrastingly, based on App (I), the required number of subjects was 123, resulting in an empirical probability of .8004 from our 10,000 simulations.

Subsequently, if the primary concern is the total cost, assuming the costs of obtaining a single measurement, $c_m$, and a single subject, $c_s$, are both $1, while a single observation, $c_{ms}$, is $0. App (I) derived the optimal pair (11, 101) with a minimal cost of $112 for objective (a) in the case of $P(W)$. Finally, for objective (b), with a fixed cost of $112, employing App (II) with $\delta =.05$ and 1- $\beta =.8$ for a two-sided CI, we obtained the optimal number of measurements and subjects as $m=11$ and $n=101$, respectively, resulting in a maximal probability of .804072.

Tables and Simulations

Discussion and Conclusion

The determination of sample size stands as a crucial and integral part of study planning, vital to achieving robust statistical power and precise estimation—a cornerstone of sound statistical practice. In addressing the need for enhanced clinical interpretation and cost-effectiveness, our study aimed to contribute to this evolving field by establishing the required number of measurements/subjects for evaluating Cronbach’s alpha within a comprehensive framework encompassing hypothesis testing and confidence intervals. The introduction of our proposed apps represents a novel advancement, enabling researchers to identify optimal configurations of measurements and corresponding subjects across various events of width, validity, and rejection, crucial for achieving desired probabilities. Our empirical findings underscore the accuracy of the obtained optimal numbers, exhibiting excellent coverage rates and near-identical empirical probabilities to the desired ones. Significantly, our study illuminates the intricate interplay among the number of measurements, subjects, and costs.

It is important to note that the calculations presented here are tailored for parallel measurements and normal distributions. We acknowledge prior research (Liu et al, 2010; Olvera Astivia et al., 2020) highlighting the consequence of violating distributional assumptions and the presence of outliers. However, studies by Raykov (1997), Yuan and Bentler (2002), Osburn (2000), and our observations from simulations indicate that under specific yet verifiable conditions, coefficient alpha remains minimally affected by population deviations from scale reliability. Hence, our work stands as an initial guide for sample size planning and lays a foundation for future investigations. Subsequent studies might expand to encompass intraclass correlation coefficient (ICC) cases, facilitating adaptability across a spectrum of research designs.