Sampling and Fieldwork Practices in Europe: Analysis of Methodological Documentation From 1,537 Surveys in Five Cross-National Projects, 1981-2017

Scope of the Analysis

We analysed the documentation of five cross-national comparative survey projects of established scientific standing, as measured by the number of publications recorded in the Web of Science Core Collection¹ database. The inventory includes: (1) all seventeen autumn editions of the Eurobarometer (EB) from the years between 2001-2017, including three autumn rounds conducted in the years 2001-2003 in the Candidate Countries Eurobarometer (CCEB) project, (2) four editions of the European Quality of Life Survey (EQLS) from 2003, 2007, 2011 and 2016, (3) eight rounds of the European Social Survey (ESS) carried out biennially since 2002, (4) four editions of the European Values Study (EVS) from 1981, 1990, 1999 and 2008, and (5) thirty-one editions of the International Social Survey Programme (ISSP) carried out between 1985 and 2015. Each of the national survey samples in these projects aimed to be representative of the entire adult population of the given country. In total, the methodological inventory involved 64 different project editions (also called waves or rounds) and 1,537 national surveys (project*edition*country). The analysis was limited to surveys conducted in European countries, even if the survey project itself was covering countries outside Europe. Table 1 presents basic information about the analysed projects (for detailed information about the projects’ geographical coverage see also Table S1 in Supplementary Materials).

Sampling Procedures

Sample Design

Detailed information about the sample design, in particular about stratification and clustering, is necessary for understanding how exactly the sampling was carried out. Additionally, we searched for information about non-probabilistic selection at any stage – as a potential threat to sample quality, and about the size of the design effect, which can be used to compare the effective sample size between samples drawn following different designs (cf. Lynn et al., 2007).

A review of the available survey documentation showed that full information about all elements of sample design is practically present only in the ESS. Hence, while we include the presence of information on sample design in the assessment of the survey documentation quality, we abstain from examining the differences in this regard across projects in our main analysis.

Fieldwork Procedures

Among fieldwork procedures, we first focus on the interview mode, distinguishing between Paper-And-Pencil Interviewing (PAPI), Computer-Assisted Personal Interviewing (CAPI), Face-to-Face Interviewing (where the distinction between PAPI and CAPI was not possible), Computer-Assisted Telephone Interviewing (CATI), Computer-Assisted Web Interviews (CAWI), Postal Surveys and Drop-off surveys. Next, we record information about the length of fieldwork (based on the dates of the beginning and end of fieldwork) as a proxy for the overall fieldwork effort. Related to the quality of the sample, we note whether substitutions were allowed. Finally, we record whether the surveys used response-enhancing procedures, such as incentives, advance letters, or refusal conversion, and whether they employed back-checks as a way of verifying the quality of fieldwork.

The presence or absence of these fieldwork procedures can influence the quality of the data and the survey outcome rates, as shown by methodological studies discussing the mode effect (Bethlehem, Cobben, & Schouten, 2011), the negative correlation between the length of fieldwork and the fraction of nonresponse units (Vandenplas, Loosveldt, & Beullens, 2015), as well as the positive impact on the measurement quality of: advance letters (von der Lippe, Schmich, Lange, & Koch, 2011), incentives (Grauenhorst, Blohm, & Koch, 2016), refusal conversion (Stoop et al., 2010), back-checking procedures (Kohler, 2007) and the negative impact of fieldwork substitutions on nonresponse bias (Elliot, 1993).

Survey Outcome Rates

Reporting on the effects of fieldwork consists of establishing the number of respondents, the number of non-respondents, and the evaluation of several standard survey outcome rates, e.g. the response rate, cooperation rate, refusal rate and contact rate. If the survey outcome rates are to be compared, they need to be calculated according to the same definitions (Smith, 2002). We relied on the standard of the American Association for Public Opinion Research (AAPOR, 2016). For each survey, we attempted to establish the number of: (1) respondents, i.e. of complete and partial interviews, (2) non-contacts, (3) refusals and break-offs, (4) cases to which other reasons for nonresponse apply, (5) persons with unknown eligibility, (6) not eligible units. Based on this information, we calculated survey outcome rates based on the AAPOR (2016) standard.

Response rates are now commonly provided by survey projects and increasingly calculated following standard procedures. While response rates are associated with sample representativeness (Cornesse & Bosnjak, 2018), they remain “second best quality indicators” (Mohler, 2019; cf. Couper & de Leeuw 2003), substituting for nonresponse bias, which is notoriously difficult to quantify and thus not addressed in the documentation of most of the analysed survey projects. The exception is the ESS, which in recent rounds documents nonresponse bias based on neighbourhood characteristics as well as population register data on age and gender (Beullens et al., 2016, pp. 70-73; Wuyts & Loosveldt, 2019, pp. 86-99).

Quality Assessment of Methodological Documentation

Before presenting cross-project differences in survey practices based on survey documentation, we assess the information content of the project documentation itself. Whether the survey documentation contains all the essential information about the key stages of the survey process has a direct bearing on survey usability, which is an important – if underappreciated – dimension of survey quality (Biemer, 2010). Our evaluation draws from an earlier study of survey documentation quality that included fewer and more general indicators, also covering questionnaire development (Kołczyńska & Schoene, 2018).

With a focus on sample design and representativeness, we constructed indicators in four areas: (1) sampling procedures, (2) sampling design, (3) fieldwork procedures, and (4) reported survey outcome rates, each with between two and six pieces of information that are crucial for forming an opinion about a survey’s quality. In the area of sampling, we looked for information that would allow us to judge the representativeness of the drawn sample, referring to the target population, the sampling frame, the type of the sample, and within-household selection procedures. Regarding sampling design, we searched for information that would allow us to reconstruct the exact sampling process, i.e., on the presence or absence of stratification, clustering, whether the sample was non-probabilistic, and on the design effect. Among fieldwork procedures, we focused on the survey mode, information on whether substitutions were allowed, as well as on procedures aimed at increasing survey response and data integrity: whether the survey employed incentives, advance letters, refusal conversion techniques, and back-checking or fieldwork control. Finally, pertaining to survey outcomes, we collected information about the response rate and information breaking down the drawn sample depending on eligibility, contact, and response, as necessary for the calculation of different outcome rates (for full schema see Table S2 in Supplementary Materials).

We score each national survey on whether the documentation contains information on the selected aspect of methodology (coded 1) or not (coded 0). The proportion of positive scores in each area constitutes the value in that area, ranging from 0 if the project documentation did not contain that kind of information, to 1 if all pieces of information we searched for were included.

According to the documentation quality scores shown in Table 2, the best-documented project – in all editions – is the ESS, which kept a consistently high level of reporting on the methodology of national surveys, and the EQLS. The quality of survey documentation conducted as part of EVS and ISSP was much lower and much more differentiated. EB and CCEB have the weakest methodological documentation. The documentation for both these projects is only available for entire editions (project*edition), and not for individual national survey samples, which explains the lack of variability in documentation quality within editions. The documentation content of EB and CCEB changed beginning with the 2004 edition when an additional piece of information on within-household selection procedures was included. In order to show this, we represent EB and CCEB in Table 2 with two rows.

Figure 1 presents the median documentation quality for each project edition, showing the clear superiority of ESS compared to all the other projects. It should be noted that in the comparatively weakest-described second round of ESS from 2004, the quality of documentation was still substantially higher than that of the best-described edition from among all the other projects. It should also be pointed out that the later editions of the two longest-lasting projects, EVS and ISSP, saw substantial improvement in the quality of documentation in the late 1990s, following the seminal report on the consistencies and differences between national surveys in ISSP Round 1995 (Park & Jowell, 1997), which set the standard for future survey documentation efforts of ESS and EQLS.

On the contrary, EB and CCEB – not shown in the graph due to lack of changes over time – clearly stand out as not only exhibiting no improvement and consistently withholding information on outcome rates but also as not providing documentation on the level of national surveys. Similar patterns – both in terms of cross-project differences and within-project changes – were found in Kołczyńska and Schoene (2018).

Cross-Project Differences in Target Populations

Cross-national survey projects typically have project-wide definitions of target populations that apply to most national surveys. According to general definitions, target populations in EB and CCEB include the resident populations aged 15 and over (European Commission, 2017). EQLS targeted “all people aged 18 and over whose usual place of residence is in the territory of the countries included in the survey” (EQLS, 2013, p. 3). ESS surveys “all persons aged 15 and over resident within private households, regardless of their nationality, citizenship, language or legal status” (ESS, 2017, p. 6). In EVS the definition is practically the same as for ESS; the only difference is the lower age limit of 18 (EVS, 2016). ISSP included “persons aged 18 years and older” (ISSP, 2017, p. VII).

These standard definitions are not uniformly enforced. ISSP and – to a lesser extent – EVS allow certain deviations in the target populations. In some surveys in ISSP the lower age cut-off is as low as 14, while in others it is as high as 21; if the surveys have an upper age cut-off, it ranges from 65 to 94. EVS conforms to the project-wide minimum age of 18, but a few surveys employ an upper age cut-off at 74, 75, or 80.

Apart from age cut-offs and the exclusion of institutionalized populations, the other potential restrictions of target population definitions are in terms of geographic coverage. These geographic exclusions typically pertain to sparsely populated or distant territories that are unlikely to have a substantial influence on the results of cross-national analyses. However, they may also include regions where conducting fieldwork would be associated with increased risk, e.g., due to security concerns.

Cross-Project Differences of Sampling Procedures and Sample Types

Table 3 shows distributions of sampling frames, types of survey samples, and procedures used to select respondents within households in the analysed survey projects. The presented comparisons show significant cross-project variation mainly resulting from the different sampling strategies employed by projects. For example, EB and CCEB use area samples, where households are chosen primarily using random route procedures, and target respondents are selected using one of the different birthday rules. EQLS relies on either personal or address-based registers, but area samples using random route protocols (for household selection) are also common. EVS is characterised by a large proportion of non-probability quota samples in the early waves of the project. ISSP relies primarily on probabilistic samples based on address or – less frequently – individual registers, with a small proportion of quota samples and multistage samples of an undetermined type. ESS uses probabilistic samples exclusively. EQLS, ISSP and ESS more often use the birthday rules than the Kish grid for the within-household selection of respondents. In EVS, both methods are used equally.

Cross-Project Differences in Fieldwork Length and Fieldwork Procedures

The next part of the analysis deals with differences in fieldwork length and fieldwork procedures. Figure 2 indicates significant cross-project differences in the length of fieldwork. EB and CCEB have by far the shortest fieldwork periods, with the median in all the analysed editions not exceeding 29 days, and in the editions conducted after 2008 dropping below 20 days. Other projects have considerably longer fieldwork periods than EB and CCEB. Specifically, the last edition of EQLS and all the editions of ESS, saw a median length of fieldwork of between 114 and 151 days (between 16 and 22 weeks). In EVS median fieldwork length ranged between 60 and 92 days (between 9 and 13 weeks), and between 30 and 102 days (between 4 and 15 weeks) in ISSP. It is worth noting that the specification of ESS requires that fieldwork not be shorter than four weeks to avoid a high rate of non-contacts, while the maximum recommended fieldwork length is four months (Koch & Blohm, 2006).

The length of fieldwork is not without consequences for survey outcome rates. One of the simplest methods for improving the response rate is by extending the time devoted to fieldwork (Stoop et al., 2010). Additionally, longer fieldwork results in broadening the scope of available procedures that can increase the chance of conducting interviews with people who spend less time at home and with reluctant respondents. Recognizing this, the ESS, as of Round 10, decided to extend the minimum length of the fieldwork from four to six weeks (European Social Survey, 2019, p. 5). At the same time, extended duration may be a symptom of difficulty during fieldwork execution, including an insufficient number of interviewers or low interviewer motivation.

An examination of the variation in fieldwork procedures (Table 4) provides more context for the short fieldwork periods in EB and CCEB. Both projects, in all their national surveys, allow fieldwork substitutions (cf. Kohler, 2007). Allowing substitutions significantly reduces the effort required to obtain a successful interview, as the interviewers do not need to make repeated attempts to contact the particular individual selected for the survey. Instead, after a failed interview attempt, they can choose a substitute respondent. Such practices in no way improve the quality of the survey sample, but they certainly shorten the length of fieldwork (Chapman, 2003; Vehovar, 1999). In EQLS and ESS, substitutions of unavailable respondents were prohibited in all the national surveys and in the ESS they were treated as interview falsification (cf. Stoop, Koch, Halbherr, Loosveldt, & Fitzgerald, 2016). Substitutions were allowed in some national surveys in EVS and ISSP.

As shown in Table 4, by far the most common mode of data collection in all the cross-country projects was the face-to-face interview. CAPI was more common in the younger projects, ESS and EQLS, while PAPI dominates the ISSP and – to a lesser extent – also EVS. In EB and CCEB, the documentation does not allow us to establish whether PAPI or CAPI was employed. In a small percentage of national surveys in EVS also CATI, CAWI and postal surveys were used. ISSP occasionally employed multiple modes, in most cases consisting of PAPI or CAPI supplemented by postal or drop-off surveys. It is worth mentioning that ISSP questionnaires are often administered with another survey to reduce the cost of data collection. For example, the 2011 wave of ISSP in Germany was fielded with the German General Survey 2012 ALLBUS (ISSP, 2013, p. 38)² .

An analysis of the use of performance-enhancing procedures provides information about the effort survey projects put into data collection. Advance letters were employed in all the national surveys conducted in EQLS and in most of those conducted in ESS, but to a lesser extent in EVS and ISSP. Monetary or material incentives were used primarily in ESS, far less frequently in other projects. Refusal conversions, that is procedures used to convince people refusing to be respondents to participate after all, were employed in 3/4 of all ESS surveys, and in over half of EQLS surveys, but in EVS and ISSP refusal conversion was used much less frequently, in only 1 in 5 national surveys. Finally, back-checking procedures were used in almost all ESS surveys, slightly less often in EQLS, and significantly less so (although still in more than half of the national surveys) in EVS and ISSP.

Cross-Project Differences in Survey Outcome Rates

Fieldwork efforts have an impact on the final results, including on the survey outcome rates (cf. Sturgis, Williams, Brunton-Smith, & Moore, 2017). Figure 3 presents three survey outcome rates: the response rate (RR2), the contact rate (CON1), and the refusal rate (REF1), achieved in surveys in consecutive editions of the compared projects. In line with the AAPOR (2016) definitions, RR2 is the number of complete and partial interviews divided by the number of interviews (complete and partial) plus the number of non-interviews (refusal and break-off plus non-contacts plus others) plus all cases of unknown eligibility (unknown if housing unit, plus unknown, other). The CON1 treats all cases of indeterminate eligibility as eligible, and measures the proportion of all cases in which the survey reached any member of the housing unit. REF1 is the proportion of cases in which a housing unit or respondent refuses to give an interview, or breaks off an interview among all potentially eligible cases. Data that would allow calculating the values of RR2, CON1, and REF1 were not available for EB and CCEB, nor for some editions of the EVS (1981-1990) or the ISSP³ (1985-1990 and 1994-1995).

Data shown in Figure 3 confirm reports about the growing difficulty faced by researchers conducting surveys (de Leeuw, Hox, & Luiten, 2018). In all analysed projects, later editions noted a significant rise in refusal rates, and a corresponding decline in contact and response rates. The latter is particularly evident in the ISSP: at the beginning of the 1990s the median response rate was about 80 percent, but in the last archived edition from 2015 (at the time data were prepared for this paper) it dropped to around 45 percent. This decline was caused by a significant reduction in contact rates, and to a lesser degree by the rise of refusals. Comparing editions of the projects that were conducted in the same year, the highest values of RR2 and CON1 were usually noted in ESS, and the lowest in ISSP. The highest refusal rates were noted in EQLS, and the lowest in ISSP.