This is an open access article distributed under the terms of the Creative Commons
Attribution License (Making decisions about the generalization of research findings is a cognizant and reflective action for the researchers and research audience (Onwuegbuzie & Collins, 2007). This action entails a careful examination and interpretation of research findings drawn from specific samples or cases and extrapolation of those findings to other diverse populations, cases, or settings (Polit & Beck, 2010). When thinking about generalization, research users ask questions such as: Can these findings be useful for our settings or populations? What aspects of these findings are more pertinent to our practice settings? Seemingly, these questions are simple and could be answered in a dichotomous response. However, generalizing and drawing general implications is a complex and iterative process and involves critical inductive (specific study to general population/setting or context), deductive (general population/context to specific setting/population), and abductive (inferring the likeliest explanation from a set of observations to explain specific or general population/context) reasoning (Baskerville & Lee, 1999; Fisher et al., 2018; Smaling, 2003; Tashakkori et al., 2021). Many authors have discussed the principles and models of generalization in quantitative and qualitative research. Mixed methodologists agree that generalization of mixed methods findings require robust designs; well-formulated questions; strategic data collection, analysis, and interpretation; and integration of qualitative and quantitative strands (Bazeley, 2018; Creswell & Plano Clark, 2018; Tashakkori et al., 2021). Nevertheless, an in-depth discussion about the nature of generalization in mixed methods research (MMR), the generalization of MMR findings, and strategies for achieving generalization in MMR have received limited attention. Since there has been a great expansion in the usage of MMR across clinical, educational, and health sciences and the findings of MMR conducted within these disciplines must be translated into practice and policymaking (Fiorini et al., 2016; Guetterman et al., 2019; Younas et al., 2019), there is a need to explore and discuss the idea of generalization in MMR more comprehensively.
Purpose
The purpose of this paper is to explore the logics of generalization in MMR and offer strategies to enhance generalization. The outline of the paper is as follows. First, we provide an overview of the logics and purposes of generalization and how it is achieved in quantitative and qualitative research. Second, we explore the nature of generalization in MMR and outline strategies for enhancing generalization in mixed methods. Classical and contemporary literature on generalization in qualitative, quantitative and MMR was used to support our arguments. Published mixed methods studies were used to expand on the presented arguments.
Background: Generalization in Research
Generalizing and drawing implications is a complex process for both researchers and the research audience because it entails the critical appraisal of research methods and the evaluation of study findings for a particular setting, population, and context. Researchers assess the statistical (external and internal) and case-to-case generalizability of research. Instead, the research audience assesses the research findings based on their individual and collective experiences (Campbell, 1986; Onwuegbuzie et al., 2009). This type of generalization is referred to as naturalistic generalization (Stake, 1995). The usefulness of quantitative research is usually measured in terms of the generalization of findings rather than the information about the individual participants (Altman & Bland, 1998; Fisher et al., 2018; Norman, 2017). The generalization of quantitative research findings requires statistical conclusion validity, construct validity, internal validity, and external validity. The definitions of these concepts are presented in Table 1.
Table 1
Key Concepts Related to Generalization in Quantitative and Qualitative Research
| Concepts | Description |
|---|---|
| Quantitative Research | |
| Statistical conclusion validity | Conclusions drawn from a study are based on relevant and roust statistical testing (García-Pérez, 2012). |
| Construct validity | It refers to the extent to which a data collection instrument adequately measures the construct or theory that it purports to measure (Streiner et al., 2015). |
| Internal validity | It refers to the idea of whether the processes and methods used in a study are robust and relevant and limits systematic errors enabling the researchers to achieve the study purpose (Andrade, 2018). |
| External validity | External validity pertains to the extent to which study findings can be confidently generalized to other contexts, populations, and cases (Andrade, 2018). |
| Qualitative Research | |
| Credibility | The extent to which qualitative findings are congruent with or true to reality (Lincoln & Guba, 1985). |
| Transferability | The degree to which qualitative findings can be applied to other contexts and settings (Lincoln & Guba, 1985). |
| Dependability | The extent to which qualitative findings are consistent and can be reproducible (Lincoln & Guba, 1985). |
| Confirmability | It refers to the neutrality of qualitative findings and the extent to which the findings are biased free (Lincoln & Guba, 1985). |
| Contextualization | Explicating the situatedness of qualitative research in terms of the researchers’ context and the context of the phenomenon under consideration (Levitt et al., 2018). |
| Methodological Integrity | It is the extent to which the processes used in a qualitative study are consistent with the phenomenon under study and the processes are relevant to generate answers or achieve the study aims (Levitt et al., 2018). |
The idea of generalization of research findings in quantitative research is not a new enterprise. It is seen as one of the criteria of the rigor of quantitative research more so than qualitative research (Norman, 2017; Polit and Beck, 2010). Authors have also discussed some approaches to generalization of research findings in qualitative (Firestone, 1993; Larsson, 2009; Levitt, 2021; Levitt et al., 2018; Maxwell, 2021; Mayring, 2007) and case study research (Goeken & Börner, 2012; Tsang, 2014). The generalization of qualitative research is often assessed in terms of the trustworthiness criteria, entailing credibility, transferability, dependability, and confirmability (Lincoln & Guba, 1985). However, contemporary scholars have proposed other approaches to generalization in qualitative research. Levitt et al. (2018) highlighted the importance of contextualization and methodological integrity of qualitative methods for enhancing the transfer and use of qualitative findings across different settings and populations (Table 1). Levitt (2021) introduced the concept of qualitative generalization and argued that qualitative research is concerned with the transferability of research findings to the phenomenon under study rather than the population. Maxwell (2021) elaborated that internal (i.e., transferring qualitative findings to the studied group, setting, and context) and external generalization (i.e., applying qualitative findings to a distinct group, setting, or context) are suited for qualitative research. A detailed discussion of all these types and arguments of generalization in quantitative and qualitative research is beyond the scope of this paper and researchers should refer to the cited primary sources.
Shadish (1995) outlined two kinds of generalizations, namely operational and conceptual generalizations. The operational generalizations pertain to different operational aspects of a study, such as interventions, programs, and settings (e.g., generalizing the findings from a randomized control trial and using the tested intervention in another context). The conceptual generalizations can be about different variables and constructs associated with the operational aspects (e.g., generalization of the finding that socioeconomic determinants of health affect self-care behaviours) (Shadish, 1995).
Firestone (1993) presented a basic typology of research generalization, which includes, three models: statistical, analytical, and case-to-case generalizations. According to Firestone (1993) quantitative research generalizations are statistical, that is, the generalization of sample-based research findings to a particular population. Such generalizations are based on the sampling and probability theory and are considered more plausible if a large and random sample is used. Tashakkori et al. (2021) further elaborated on the notion of ‘generalizing to’ and ‘generalizing across’ in quantitative research. The former refers to generalizing from a sample statistic to an unknown population parameter, and the latter refers to generalizing widely across many diverse groups, settings, populations, and places. Such a strict view of generalization in quantitative research is not applicable to qualitative research (Larsson, 2009). Qualitative research is context-bound and involves the use of small yet rich and diverse samples. Therefore, there seems to be a consensus that qualitative research generalizations are analytical or case-to-case in nature (Firestone, 1993; Mayring, 2007; Polit & Beck, 2010). Analytical generalizations refer to the extrapolation of research findings to a broader theory or model to support the assumptions of theory about the studied phenomenon and the applicability of the theoretical assumptions across a wide range of settings (Firestone, 1993). Analytical generalizations can also enable researchers to identify the scope of a theory by identifying the range of circumstances to which a theory is applicable (Firestone, 1993; Polit & Beck, 2010). Case-to-case generalizations refer to the transfer of research findings from one setting or context to an entirely different setting or context (Firestone, 1993). Such generalizations are made after assessing a case in terms of its material facts, appropriateness, and rationale for generalization for the case or population to which generalization is intended (Firestone, 1993; Polit & Beck, 2010).
Method
We completed a critical review of methodological literature to identify literature sources such as peer-reviewed articles, book chapters, and research texts about generalization in research in general and mixed methods in particular. A critical review is an account of a specific topic to develop a theoretical understanding (Paré & Kitsiou, 2017). Such reviews aim at developing theories, frameworks, and models (Grant & Booth, 2009). We completed a literature search in Scopus, Web of Science, and Ovid general keywords, MESH terms, and subject headings such as “generalization in research”, “research methods”, “mixed methods generalization”, and “translation of mixed methods research findings”. We also conducted searches in the Journal of Mixed Methods Research, International Journal of Multiple Research Approaches, and Sage Research Methods database to locate relevant sources. This literature search was not a systematic search as done in systematic reviews. Therefore, we did not report the search results using PRISMA reporting guidelines. We selected literature sources using purposive sampling based on our reading and knowledge of papers about generalization in research and experiences of designing and conducting MMR. The inclusion and exclusion criteria for the selected sources were: a) discussion papers about generalization in qualitative, quantitative, and MMR, b) book chapters or books on generalization in research, and c) empirical studies that could be used as examples to illustrate various types of generalization, logics, and principles in practice.
Analysis of Literature Sources
We did not complete a formal synthesis of literature as done in systematic or scoping reviews. Instead, we presented our arguments in a logical and systematic manner to contribute to our boarder argument on enhancing generalization in MMR. First, the selected literature sources specific to generalization in qualitative, quantitative, and mixed methods research were thoroughly reviewed and interpreted. The authors conducted weekly meetings to discuss the selected sources and identify supporting information to illustrate our arguments. Second, after the interpretation of the arguments in the literature sources, it was deemed relevant to present an overview of various types of generalization and their definitions. Therefore, we extracted information about various types of generalizations and presented it in tables. Third, we identified and selected any available frameworks and models about generalization in research that served as the basis to build our discussion on generalization in MMR. Finally, using the selected frameworks, generalization types, definitions, and descriptions, we framed our discussion.
The Purposes and Logics of Generalization
Generalization aims at the translation and implementation of gained knowledge for real life and evidence-informed practice and policymaking (Knottnerus et al., 2020; Polit & Beck, 2010). Shadish (1995) expanded upon the work of Cook (1990) regarding causal inferences in experimental research and outlined five overarching principles concerning the logic of research generalization, which are discussed as follows.
The Five Overarching Principles
The Principle of Proximal Similarity
The principle of proximal similarity posits that generalization occurs when there are explicit similarities between the demographics of research participants, and the characteristics of settings. For example, if a researcher is interested in applying findings of a study conducted on family caregivers in Europe to North American context, the demographic characteristics (e.g., age group, ethnic backgrounds) of caregivers and their context (e.g., family structure, living arrangements, & social protection services available to caregivers) should be comparable to North America.
The Principle of Heterogeneity of Irrelevancies
The principle of heterogeneity of irrelevancies suggests that generalization of a finding is strengthened when it holds in other studies across differences in conceptually irrelevant entities such as individuals, settings, treatments, outcome measures, and times (Shadish, 1995). It means generalization requires identifying what is and what is not required for qualifying a generalization; that is, generalizations are strongest or easiest when they do not require qualification. For example, a researcher applying research findings about the stress and coping in teenagers from European to North American context may need to assess and examine if the stress and coping patterns differed in teenagers living independently or with their families, or teenagers’ schooling and social network or other variables which may be considered irrelevant in one context.
The Principle of Discriminant Validity
The principle of discriminant validity posits that generalization is legitimate when the research finding/s are driven by a specific construct and not by an alternative construct. It means that when it can be demonstrated that the research findings are the outcomes of specific variables and not the result of other variables, not accounted for during the research. For example, if a researcher aims to apply a research finding that perceived social support of informal caregivers is explained by their mental health. Before generalizing, it should be carefully assessed whether improved mental health is contributing to increased perceived social support and not other factors such as the strength of family network, or caregiver resilience.
The Principle of Empirical Interpolation and Extrapolation
The principle of empirical interpolation and extrapolation posits that we “generalize most confidently when we can specify the range of persons, settings, treatment, outcomes, and times over which the finding holds more strongly, less strongly, or not at all” (Shadish, 1995, pp. 425–426). For example, a researcher discovered that medical students develop long term friendships and collaboration after graduation if they studied at least three courses together. Generalizing this finding to a new context may require assessing how this changes if medical students studied three courses that involved theoretical concepts only compared to practical/hands-on skills. While this data may not be available in the study, the researcher could interpolate this from the type of courses described. For instance, if the studied courses were pathophysiology, anatomy, and health assessment. It can be inferred that health assessment included more hands-on/practical skills compared to pathophysiology and anatomy. Extrapolation would require an assessment of the type of interactions among these students and the number of friends included in each social network. This data will be difficult to extract if not described in the sample, hence affecting the generalizability.
The Principle of Explanation
The principle of explanation posits that generalization occurs when researchers are fully aware and knowledgeable about the essential variables to be generalized in parts or as a whole, a study’s distinct parts, and moderating and mediating processes to achieve generalization. For example, if a researcher intends to apply a broad finding that experiential learning can enhance the emotional intelligence of students, it will require a breakdown of finding in terms of the type, nature, and duration of teaching and learning strategies used for experiential learning and what aspects of emotional intelligence (e.g., internal motivation, self-regulation, self-awareness, or empathy) were mainly affected.
Generalization in MMR
Generalizing research findings is a complex (Shadish, 1995) and a pragmatic process (Larsson, 2009), contingent upon the nature of the research question and the rigor of individual qualitative or quantitative studies (Firestone, 1993; Shadish, 1995). However, generalizations cannot be made with absolute certainty because they go beyond the datasets upon which generalizations are made (Polit & Beck, 2010). One possible way to achieve generalizable findings requires using qualitative and quantitative approaches that are integrated to achieve a meaningful understanding of a phenomenon (Shadish, 1995).
Robust MMR (involving clear integration of qualitative and quantitative phases and data) is often considered better suited for valid generalizations because it involves the combination of qualitative and quantitative paradigms, methodologies, and methods for several purposes (Johnson & Onwuegbuzie, 2004). The quantitative data from MMR could enable a general level of statistical generalization, and the qualitative data allow one to generalize a process and understand how a finding operates in different contexts. Some of the purposes of MMR are: to explore and understand a phenomenon, to enhance the results of experiments, to compare cases and settings, to involve participants throughout the research, and to develop theories and test those in subsequent phases (Creswell & Plano Clark, 2018). In addition, integration is an essential feature of MMR (Bazeley, 2018; Fetters, 2019), therefore it can allow the researchers to combine the strengths of quantitative and qualitative approaches and use multiple lines of reasoning to generalize single or multiple research findings (Creswell & Plano Clark, 2018). The use of large and random samples in the quantitative phase of an MMR holds promise for making plausible statistical generalizations. A more detailed description of qualitative findings and credible inferences can help in making analytical generalizations (Polit & Beck, 2010).
Drawing from and building upon the three types of generalizations (Firestone, 1993) and five principles of generalization (Shadish, 1995), we explicate generalization in MMR and propose different strategies to enhance the generalization of MMR findings. For clarity, we have provided the definitions of various types of generalization in Table 2.
Table 2
Various Kinds of Generalization
| Kind of Generalization | Description |
|---|---|
| Operational | Generalizing operational aspects of a study such as interventions, programs, and features of the settings. |
| Conceptual | Generalizing findings about different variables and constructs associated with the operational aspects of a study. |
| Statistical | Generalizing research findings of sample to a studied population. |
| Generalizing to | Generalizing from a sample statistic to an unknown population parameter. |
| Generalization across | Generalizing widely across many different groups, settings, populations, and places. |
| Analytical | Generalizing research findings to a broader theory or model to support the assumptions of theory or model about the studied phenomenon. |
| Case-to-Case | Generalizing findings from one research setting or context to an entirely different context or setting. |
Logic and Models of Generalization in Mixed Methods Designs
There are several typologies of MMR designs. For example, parallel, sequential, conversion, multilevel, and fully integrated designs (Tashakkori et al., 2021) and simultaneous, sequential, and complex; qualitatively driven, quantitatively driven, and multi-method designs (Morse & Niehaus, 2009), and sequential exploratory, sequential explanatory, and convergent designs (Creswell & Plano Clark, 2018). Irrespective of the MMR typology, every mixed methods study involves the collection of qualitative and quantitative data and the use of both methods. Onwuegbuzie et al. (2009) discussed that five types of generalization (external statistical, internal statistical, analytical, case-to-case transfer and/or naturalistic generalization) can be applicable to various MMR designs. Their internal and external statistical generalizations are merged under statistical generalization and their naturalistic generalization was excluded from this discussion. They described that naturalistic generalizations are made by research consumers. Therefore, since our focus is on how researchers can enhance the generalizability of their studies, we excluded this type from our discussion. Building upon the previous work on generalization and augmenting it with the logics of generalization, we discuss generalization in MMR in accordance with the sequence of data collection and the purpose of MMR. The relevant logics and models of generalization for each type of core mixed methods design are illustrated in Figure 1 and discussed as follows. The most relevant logics and models for each type of MMR design are based on our interpretation and logical reasoning and may not be supported due to a lack of discussion of generalization in MMR. Therefore, we acknowledge that this discussion may appear to be speculative to some readers. Nevertheless, we have tried to offer literature support where available.
Figure 1
Framework for Generalization in Mixed Methods
Some mixed methods design (e.g., convergent and simultaneous) involves the collection of qualitative and quantitative data in a parallel manner. The primary purpose of such designs is to compare and contrast the results of both qualitative and quantitative strands (Creswell & Plano Clark, 2018; Fetters, 2019). Other purposes include developing a comprehensive understanding of a phenomenon, validating the qualitative findings with quantitative or vice versa, determining causal relationships between different variables, and identifying diverse facets of any phenomenon (Creswell & Plano Clark, 2018; Younas & Durante, 2022).
Sequential mixed methods design has varying purposes. For example, a sequential exploratory design involves the collection of qualitative data followed by a subsequent quantitative phase (Creswell & Plano Clark, 2018; Fetters, 2019). The findings of the qualitative phase inform the development of a survey or tool, identification of new research variables, development of a program or intervention, and the selection of research participants for the subsequent phase (Creswell & Plano Clark, 2018; Younas & Durante, 2022). The primary purpose of this design is to explore a phenomenon using a dominant qualitative phase and then assess how the quantitative results provide a comprehensive understanding of the initial qualitative phase (Creswell & Plano Clark, 2018). Conversely, a sequential explanatory design comprises a quantitative phase and a subsequent qualitative phase to explain the results of the quantitative phase (Fetters, 2019). The primary purpose of this design is to understand the results of the studied phenomenon after a certain time and, therefore, can be used to assess the feasibility, effectiveness, challenges, issues, and future implications of any program and intervention (Creswell & Plano Clark, 2018).
When qualitative and quantitative data are collected in a parallel manner, we argue that statistical generalizations and the principle of proximal similarity appear to be quite relevant generalization principles. This is because of the need to carefully assess the demographic characteristics of the participants and the features of the study setting as well as the robustness and relevance of statistical tests used to draw inferences (conclusions drawn from individual strand in MMR) in the quantitative phase in convergent MMR designs (Almeida, 2018; Younas et al., 2022). For example, a convergent mixed methods design aims to develop a comprehensive understanding and explanation of a phenomenon through a comparison of qualitative and quantitative findings (Creswell & Plano Clark, 2018) or aims to assess causal relations among different phenomena (Pedersen et al., 2018; Younas et al., 2022). Pedersen et al. (2018) applied a convergent study to explain multifaceted mechanisms and drivers of social inequality in Cardiac Rehabilitation (CR) attendance. For the quantitative strand, they used questionnaires and identified possible mechanisms and drivers using the Odds Ratio (OR) and logistic regression. For the qualitative strand, they used individual and dyadic interviews to further explain and support the results of the quantitative phase and analyzed the data using qualitative framework analysis. In order to develop an integrated whole of qualitative and quantitative inferences, they used joint displays (a visual or tabular display of qualitative, quantitative, and mixed methods findings) entailing quantitative data, graphs, qualitative themes, and participants’ quotes. In such designs, if a large and random sample is employed using probability sampling, there is a greater chance that the sample is an accurate representation of the target population, therefore making statistical generalizations plausible. The plausibility of statistical generalizations may depend on two things: robust point or parameter estimates using the sample means and confidence intervals respectively of the quantitative dataset and credible and robust mixed inferences and meta-inferences generated after the integration of qualitative and quantitative datasets. If these two goals are achieved, researchers can confidently generalize the findings to other similar populations after assessing the characteristics of the sample, settings, and the methods used in the study (i.e., the principle of proximal similarity) (Shadish, 1995).
When qualitative and quantitative data are collected in a sequential manner, we argue that the relevant logic of generalization could be the principles of proximal similarity and explanation, and the models are analytical and statistical generalizations. For example, a sequential exploratory design is often used when there is a lack of cultural and context-specific instruments, interventions, or a program. Therefore, researchers comprehensively explore the phenomenon and then develop the required instrument or tool (Creswell & Plano Clark, 2018; Fetters, 2019). An instrument or an intervention developed for a specific context can only be used for another context if there are definite similarities among participants and the settings (i.e., proximal similarity). For developed instruments, statistical generalization holds promise if a large sample is used for psychometric testing, which allows generalization to the population used in the subsequent quantitative phase as well as other similar populations in different contexts. For example, Rasheed et al. (2020) used an exploratory sequential design to develop a scale for measuring self-awareness of nurses. In the qualitative phase, they interviewed 13 nurses to explore their perspectives about self-awareness in nursing practice. Based on thematic analysis, they generated four aspects of self-awareness namely personal, professional, contextual, and conscientious. These four aspects were used as the domains of the self-awareness in nursing scale entailing 25 items. The 25-item scale was validated in a pilot phase (n = 252) and a subsequent quantitative phase (n = 216) using content and face validity, exploratory factor analysis, and reliability testing. Rasheed et al. (2020) used a power analysis to determine the reasonable sample for pilot and quantitative phase for scale testing. This testing resulted in the finalization of 18-item self-awareness in nursing practice scale. The developed scale was more relevant to the nursing professionals compared to other scales available to measure self-awareness of the general public.
When a sequential explanatory design is used, we argue that the principle of proximal similarity and interpolation and extrapolation can be applied along with the models of statistical and analytical generalizations. The focus of such designs is to understand the outcomes of quantitative phases through subsequent qualitative phases (Creswell & Plano Clark, 2018; Fetters, 2019), therefore if the results are to be generalized, it seems plausible to examine the contexts, settings, demographic characteristics, and the underlying theoretical assumptions (i.e., proximal similarity). The similarities across these components can enable analytical generalizations. For example, Nørgaard et al. (2018) determined patients' experiences with the effect of visualization during atrial fibrillation (AF) and its association with pain, anxiety, pain medication, and procedure length. They used the qualitative phase to explore patient experiences of receiving the intervention. The quantitative results did not support the use of visualization to reduce pain intensity during the ablation of AF. However, the qualitative results substantiated that patients experienced pain, and upon stimulation to use their own resources by using visualization as a pain management strategy, patients were able to cope with the pain.
When qualitative and quantitative data are collected in a parallel or sequential manner in mixed methods experimental designs (Creswell & Plano Clark, 2018), we argue that both statistical and analytical generalizations are plausible based on the logic of the principle of discriminant validity, interpolation and extrapolation, and explanation. The model of generalization depends upon the purpose and features of the experiment and intervention (Leviton, 2017). Mixed methods experimental designs can have two overarching purposes. First, testing different theories and underlying theoretical assumptions by developing logic models or frameworks of the relevant variables. Second, to test the feasibility, efficacy, or effectiveness of the interventions (Lucas, 2003; Younas et al., 2022). An experimental study designed to achieve the former purpose can help to make analytical generalizations. It is because the theory adopted to guide the study is used to make predictions about its own applicability under various conditions and different settings (Firestone, 1993). Johnson and Christensen (2019) refer to this as “replication logic” where the same result is seen in different contexts with different kinds of people. For example, if a researcher intends to test the assumptions of any theory, the analytical generalization will aim to apply the theory in different populations, settings, or situations based on the results obtained from theory testing. In contrast, if the purpose of an intervention study is to determine the efficacy, effectiveness, or feasibility of an intervention in a given population, statistical generalizations are more plausible (Leviton, 2017; Younas et al., 2022). Similarly, if the researcher assesses the usefulness, implementation, and sustainability of intervention from participants’ perspectives in explanatory designs, the logic of interpolation and extrapolation and proximal similarity applies. This is because the features of intervention and the characteristics of those to whom it was implemented and to those to whom it will be applied too must bear adequate similarities (Leviton, 2017).
When a case study mixed methods design is used, we argue that the most relevant logic of generalization are principles of proximal similarity and explanation, and the models are analytical and case-to-case generalization (Yin, 2013). The most important aspect of case study generalization is the presence of a clear definition of a case (a person, an organization, or a community) (Firestone, 1993; Yin, 2013). Therefore, the characteristics of the case should be adequately outlined to allow transferring the findings to another context or setting and may allow researchers to assess that the reported outcomes or findings are in fact a result of the constructs or variables studied in a case study (i.e., proximal similarity and explanation). Gomm et al. (2000) argued that empirical generalization (defined as from cases to unstudied cases) could be achieved in case study research by examining the extent to which studied case/s can be characteristic or non-characteristic in relevant respects to the unstudied cases. Their account of empirical generalization is consistent with the notion of case-to-case generalization, hence lending some support to our claim that case-to-case generalization is a relevant model in mixed methods case study. Generalizations in case study research also require researchers to extrapolate the results at a concrete level for other similar cases and an abstract level for newer cases (Yin, 2013). For the former case, the principle of proximal similarity is applicable. It allows the researchers to match the operational aspects of research to the target of generalization (Shadish, 1995). For the latter case, the principle of explanation is relevant because it emphasizes breaking down the operational and conceptual components and processes of research and generalizing the most essential and applicable components (Shadish, 1995).
Strategies to Enhance Generalization in MMR
Several authors have proposed strategies to enhance generalization in qualitative and quantitative research (Firestone, 1993; Mayring, 2007; Polit & Beck, 2010; Tsang, 2014). Similar strategies are applicable to the individual phases of MMR designs. Adapting from and building upon these strategies, some strategies which are more pertinent to MMR are discussed. The strategies discussed below can be useful to enhance any type of generalization elaborated above.
Integration at Multiple Levels
Integration is at the core of MMR (Creswell & Plano Clark, 2018; Fetters & Molina-Azorin, 2017) and is one of the greatest strengths of MMR and an essential element for the plausible generalization of MMR. A common misunderstanding is that merely the collection of qualitative and quantitative data constitutes MMR, and integration occurs after distinct qualitative and quantitative analyses have been completed. This notion has been challenged, and authors recommend that integration should occur at the design, methods, interpretation, and reporting levels (Bazeley, 2018; Creswell & Plano Clark, 2018; Tashakkori et al., 2021).
Bazeley (2018) emphasized that the integration of qualitative and quantitative data must occur before drawing final conclusions. Fetters and Molina-Azorin (2017) described 15 dimensions of MMR integration namely:
-
philosophical
-
theoretical
-
researcher
-
team
-
literature review
-
rationale
-
study purpose, aims, research questions
-
research design
-
sampling
-
data collection
-
data analysis
-
interpretation
-
rhetorical
-
dissemination
-
research integrity
A brief overview of these dimensions is presented in Table 3. However, readers should refer to the primary source (Fetters & Molina-Azorin, 2017) for a detailed understanding of these dimensions.
Table 3
Dimensions of Integration in Mixed Methods Research
| Dimension | Description |
|---|---|
| Philosophical | Explicating the underlying research paradigm/s and their assumptions underpinning different stages of research. |
| Theoretical | Integrating concepts and principles from broad theories and frameworks or middle-range theories. |
| Researcher | Utilizing and integrating personal skills, experiences, and paradigmatic and methodological stances to conduct MMR. |
| Team | Involving researchers from various backgrounds, participants, and other stakeholders throughout the research and integrating their experiences to guide the study. |
| Literature review | Reviewing and building an MMR study on an in-depth qualitative and quantitative literature review concerning the topic of interest. |
| Rationale | Providing a clear rationale for conducting a mixed methods study. |
| Study purpose, aims, and research | Developing an overarching mixed methods question and providing clear purposes and questions for qualitative and quantitative phases. |
| Research design | Using and outlining the MMR design to achieve the MMR purpose. |
| Sampling | Identifying relevant sampling types and the timing of sampling. |
| Data collection | Integrating different qualitative and quantitative data collection methods and outlining how methods contribute to achieving the aim of MMR (in terms of comparing, matching domains and constructs, expanding, connecting, building, and validating). |
| Data analysis | Analyzing qualitative and quantitative data separately using relevant methods and integrating the qualitative and quantitative data through mixed methods integration techniques and methods. |
| Interpretation | Interpreting the individual qualitative and quantitative results, the integrated results, and drawn inferences in order to generate meta-inferences about the studied phenomenon. |
| Rhetorical | Using terminologies that reflect MMR methodology and methods and present the integrated approaches and stances and findings which support the use of MMR. |
| Dissemination | Using various methods and outlets for the publication of a complete MMR and additional findings from the individual qualitative and quantitative phases, which ass to the body of knowledge about the phenomenon. |
| Research integrity | Establishing rigor of qualitative and quantitative phases and mixed methods study as a whole using relevant approaches to ensure rigor. |
Note. Adapted from Fetters and Molina-Azorin (2017) with permission from SAGE.
Although all of these 15 dimensions may not be applicable to every MMR, integration at multiple levels in a single MMR study can increase the credibility and strength of the study (Creamer, 2017). For instance, using a large and random sample for the quantitative phase and clearly outlining the characteristics and features of the sample through the qualitative findings can enhance statistical, analytical, and case-to-case generalization (Bazeley, 2018). Onwuegbuzie et al. (2009) coined the fundamental principle of data analysis which pertains to the idea that qualitative and quantitative data analysis methods in MMR should be incorporated in such as way that at least one kind of generalization is possible. This principle also highlights the importance of integration, particularly at the level of analysis, to enhance generalizability.
Providing in-depth and rich descriptions of a phenomenon from the integration of the qualitative and quantitative results can enhance the understanding of similarities and differences in the datasets, thereby allowing case-to-case generalization (Bazeley, 2018; Polit & Beck, 2010). During integration, it is also essential to focus on discordant qualitative, quantitative, and mixed results as those results could serve as the precursors for exploring alternative explanations, determining the effect of diverse samples, and expansion of theoretical and conceptual knowledge about the phenomenon under consideration (Bazeley, 2018; Fetters et al., 2013; Younas et al., 2023). If discordant findings are missed it could subsequently lead to missed alternative explanations, thereby influencing the principle of discriminant validity and all three types of generalizations.
Comprehensive Description of Mixed Methods and Study Findings
There is a consensus that researchers need to provide a detailed description of the study methods and findings to allow replication (Polit & Beck, 2010) as well as the description of the study participants and their characteristics (Tashakkori et al., 2021) can enhance the transferability of findings (Lincoln & Guba, 1985; Yin, 2013). This, in turn, can promote case-to-case and analytical generalizations (Firestone, 1993). A comprehensive description of study methods allows the readers and policymakers to replicate the study in other contexts and assess the consistency of findings over time and across cross-cultural contexts. If the results are consistent in other settings and contexts, it can promote valid generalizations (Firestone, 1993). The replication of studies in diverse contexts, times, and populations can help determine the variations of and diversity in findings (Shadish, 1995), thus making analytical generalization effective.
If researchers provide a comprehensive description of the characteristics of study participants, the demographics, the contexts, the time frames, the features of the interventions, and the feasibility, effectiveness, and efficacy of programs and interventions; researchers and policymakers can make more cognizant decisions about the generalizability of the findings. A detailed description of study findings allows researchers to assess the quality of interpretations of the qualitative and quantitative datasets and the integrated conclusions (i.e., metainferences) of both datasets (Tashakkori & Teddlie, 2008). In MMR, the researchers should also provide detailed descriptions of the qualitative and quantitative methods and the strategies used to integrate qualitative and quantitative strands as well as the time and intent of integration (Creswell & Plano Clark, 2018; Younas et al., 2022). For example, in sequential explanatory designs connecting quantitative results with the qualitative strand through sampling is critical for enhancing the overall research rigor (Fetters, 2019; Younas & Durante, 2022. If researchers can show how and why specific participants were selected to explain the quantitative results, the generalizability of mixed methods findings is improved (Ivankova, 2014). Providing an adequate description of the nature, type, and characteristics of the sample is critical to generate various types of generalizations (Onwuegbuzie et al., 2009). Sykes et al. (2018) demonstrated that sampling errors, incongruent sampling techniques within mixed methods designs, and non-representative samples could threaten the generalizability of mixed methods findings. Therefore, the researchers should provide a detailed description of the study methods as well as the analysis of the quantitative findings, the qualitative findings, and the conclusions drawn from the comparison of both datasets. For a better and more comprehensive description of integration and analysis of qualitative and quantitative data, researchers could use contiguous (i.e., presenting qualitative, quantitative, and mixed analyses in separate sections) and weaving (i.e., presenting qualitative and quantitative results on a theme-by-theme basis) approach (Fetters et al., 2013; Fetters & Freshwater, 2015). The use of weaving or a contiguous approach is contingent upon the purpose and nature of mixed methods design and the aims of the researchers. For example, for qualitatively driven MMR involving phenomenology, grounded theory, or ethnography as the dominant qualitative phase; the weaving technique may allow the researchers to provide a comprehensive overview of research findings. This strategy to provide a comprehensive description of study methods and findings is directly related to the following strategy.
Generating Strong and Plausible Inferences and Metainferences
Inference refers to interpreting the research findings and the conclusions drawn to answer the mixed research question (Creswell & Plano Clark, 2018). Tashakkori and Teddlie (2008) indicated that inferences (i.e., these are the conclusions drawn from individual qualitative and quantitative strands of a mixed methods study) and metainferences (i.e., these are the conclusions drawn after the integration of qualitative and quantitative findings) should be evaluated in terms of their quality to capture the study findings (inference quality) and the degree to which the conclusions can be generalized to other settings (inference transferability). Strong and plausible inferences are those that are consistent with the operational as well as the theoretical and conceptual knowledge base and aspects of the study (Plano Clark & Ivankova, 2015). The inferences which meet these quality checks can be considered more useful to researchers and policymakers (Tashakkori & Teddlie, 2008). Therefore, quality inferences and metainferences can enhance all three modes of generalization in MMR (Gibson, 2017; Onwuegbuzie et al., 2009; Younas et al., 2023).
Generating quality mixed inferences is an essential element of MMR (Creswell & Plano Clark, 2018). It requires researchers to fully know and explore the qualitative and quantitative data using sound methods and integrate both datasets (Plano Clark & Ivankova, 2015). Immersion in the qualitative data helps researchers to fully understand participants’ perspectives (Polit & Beck, 2010) and capture their meanings concerning the studied phenomenon (Tashakkori & Teddlie, 2008), which promotes effective analytical generalization (Polit & Beck, 2010). Polit and Beck (2010) argue that quantitative researchers are often not fully immersed in their data and often think in terms of variables and relationships among variables and mediating and moderating variables for process. This type of thinking serves as an obstacle to successful and insightful analysis and undermines the quality of generalizations based on variables. However, such thinking does not allow for analytical generalizations about the complexities of the phenomena. Therefore, if the quality of mixed inferences is to be enhanced and the generalization of findings are to be made plausible, researchers should strive to know the quantitative data well. Knowing the quantitative data allows researchers to understand their sample and sample characteristics and select appropriate and powerful means of analysis to enhance the quality of statistical generalizations. Onwuegbuzie et al. (2009) cautioned against drawing statistical generalizations from MMR if the quantitative sample in MMR design is not identical to the qualitative sample and is nested (sample for one phase is a subset of the sample for the second phase), parallel (different samples from same population), or multilevel (two or more samples obtained from distinct levels of inquiry). Therefore, a better description of the nature and type of sample affect the quality of drawn metainferences and hence generalizations. Knowing both the qualitative and quantitative data provides opportunities to conduct intensive data analysis for individual cases, efficiently compare the findings, transform the datasets, and perform cross-case analyses (Bazeley, 2018), thereby enhancing the rigor and plausibility of mixed methods inferences and meta-inferences (Tashakkori & Teddlie, 2008).
Conclusions
Despite different interpretations of generalization across methodologies and fields, it can be agreed upon that making decisions about generalizability is a complex process. Likewise, generalizing mixed-methods findings is a complex process requiring knowledge about mixed methods and abilities to critically reflect and evaluate the findings from individual strands and the findings of the mixed methods. Drawing from the work about generalization in research and the logic and typology of generalization, we aimed to extend the discussion on generalization in MMR. We argued for using pertinent models and logic of generalization in different mixed methods designs and offered three key strategies to enhance generalization across all of these modes and logics. We argued for multilevel integration in MMR designs, provision of a detailed and comprehensive description of the design, study, sample features and study findings, and generating inferences and metainferences that are well-grounded in the data for making plausible claims for generalizing. It is important to highlight that our proposed framework for generalization in MMR is not the only way to think about MMR generalization and this framework can be further refined and revised. Nevertheless, we hope this framework can generate further debate and discussion on the applicability and relevance of each logic and model of generalization across various MMR designs.