In addition to the challenges already discussed, an additional difficulty arises when these scales are adapted to languages other than English. Literal translations of extreme Likert scale anchors — such as “Strongly disagree” and “Strongly agree” — often produce awkward or unfamiliar phrases that deviate from natural, everyday language. This incongruence can compromise the clarity, interpretability, and validity of responses, particularly when respondents struggle to relate to or visualize the extremities of these anchors. For instance, in the Slovenian language, the literal translation of “Strongly disagree” and “Strongly agree” as “Močno se ne strinjam” and “Močno se strinjam, ” respectively, sounds somewhat unnatural and is rarely encountered in everyday conversations. However, phrases like “Sploh se ne strinjam” (“Do not agree at all”) and “Se popolnoma strinjam” (“I completely agree”) are more commonly used and may be perceived as more intuitive and relatable. However, this translation is both linguistically asymmetrical and thus potentially lacks intra-scale symmetry, as the lower extremity (“Sploh se ne strinjam”) and the upper extremity (“Se popolnoma strinjam”) differ in their linguistic structure, as well as stray away from the direct translation from the original (English language). Similar issues have been observed in other languages, such as German, where natural expressions like “Stimme überhaupt nicht zu” and “Stimme ganz und voll zu” deviate from the structurally symmetrical form of Likert anchor points. Furthermore, a lack of symmetry between Likert scale extremities can introduce bias by affecting how respondents perceive and interpret the scale. If the lower and upper extremities differ in intensity, this can also result in a difference in the perceived distance between anchor points. Unequal intervals between response options may lead to inconsistent responses and challenges in accurately measuring constructs. These issues are especially problematic in cross-linguistic research, where natural but asymmetrical translations might exacerbate these effects and result in greater perceived distances between anchor points.

This paper aims to answer the following research question:

Addressing this research question is important for ensuring the validity and reliability of cross-linguistic research, since the choice of anchor points may inadvertently introduce systematic biases or influence the interpretation of results. To explore this, we designed an online experiment to explore how varying anchor points influence responses and, consequently, the results of the study.

We addressed this problem in the context of information security, drawing upon Protection Motivation Theory (PMT). PMT was originally introduced by Rogers (1975) and is a widely recognized theoretical framework that explains how individuals are motivated to adopt protective behaviours in response to perceived threats – in relation to the perceived severity of a threat, their vulnerability to it, and the efficacy of certain protective behaviour. Several studies confirmed the correlation of the said constructs with security motivation intention (Li et al., 2022; Mou et al., 2022; Zuwita & Rahmatullah, 2021).

To answer the research question, we designed an online experiment to evaluate how different translations of Likert scale anchor points influence survey responses. First, we have evaluated which phrases used in the Slovenian Likert end-points are more familiar to the Slovenian population. We analysed Google search engine results (using Google.si for the Slovene language, with personalization of the results disabled) for each phrase by calculating the familiarity score as suggested by Weijters et al. (2013), i.e., by dividing the count for each endpoint label (e.g., “strongly agree”) by the count of the label name without the modifier (e.g., “agree”) and computing the natural logarithm of the ratio: (ln⁡hits("strongly agree")hits("agree")). The endpoints used in GroupCA exhibited higher familiarity score (–2.7 for “Do not agree at all” and –5 for “Completely agree”) compared to their (positive or negative) counterparts used in GroupSA (– 4.1 for “Strongly disagree” and – 6.6 for “Strongly agree”).

The items used in the survey were adapted from Thompson et al. (2024) and adapted for secure smartphone use. The items were translated from English to Slovene language using a standard back-translation process (Brislin, 1980). Perceived vulnerability, perceived severity, and self-efficacy were measured by six items, and response efficacy, and security intentions by four items. Items were measured on a 5-point Likert scale.

To investigate the effects of structurally symmetrical versus linguistically natural but asymmetrical translations of anchor points on participant answers, we conducted an experiment in which a survey randomly assigned participants to one of the two groups: GroupSA for the linguistically less natural but structurally symmetrical translations of anchor points, and GroupCA for the linguistically natural but asymmetrical translations (see Table 1).

Descriptive statistics per item in each study group are summarized in Table 3. There are no major differences in means or standard deviations per item between the study groups. Slightly higher means are present in the GroupCA for the perceived vulnerability. Slightly higher variation is present in some of the items in the GroupCA. The distribution of answers to all items is approximately normal.

To test the familiarity hypothesis (Weijters et al., 2013), we computed the total number of high-anchor responses (‘Strongly agree’ and ‘Completely agree’) across all 26 items and compared them using an Independent Samples T-Test. The mean number of “Strongly agree” responses per item was M = 51.7, SD = 23.9, and the mean number of “Completely agree” responses was M = 66.2, SD = 28.0. The difference in means (T-Test for Independent Variables) is statistically significant, t = 2.01, p < .05. Hence, we conclude that respondents more frequently chose the anchor point that uses phrases more commonly used in their daily language.

Next, to assess how different anchor points influence the validity and reliability of the questionnaire, we conducted a Confirmatory Factor Analysis (CFA). The results of CFA per study group are summarized in Table 4. The results are very similar. Both CFA models exhibit good overall fit, factor loadings above 0.5 and AVE above 0.50, supporting convergent validity for both models. The reliability, as measured by composite reliability measure, indicates sufficient measurement reliability for all PMT scales in both study groups. However, slightly higher values for CR and AVE were observed in GroupCA.

Multigroup analysis was performed to test the measurement invariance of PMT scales across the two groups (Table 5). The results indicate that the global latent factor structure is the same across groups (configural invariance). Metric invariance was not supported by the data, suggesting that factor loadings differ between the two groups. Further analysis showed that measurement non-invariance is due to SI-2 and SI-3 indicators measuring security intentions. Partial measurement invariance was supported in the model, allowing the two indicators to have different factor loadings across groups. Scalar invariance was supported by the overall model fit and an insignificant chi-square difference test between the partial metric and scalar invariance model. Finally, the model's fit presupposing different latent means across groups was not significantly better than the model with the same latent means, suggesting the five latent factors do not differ in the mean values between the two groups.

Finally, we tested the structural equation model in the two groups (Table 6). The model presupposing the same structural loading between each independent variable (perceived vulnerability, perceived severity, response efficacy, and self-efficacy) and security intentions did not differ significantly from the model suggesting equal structural weights across the two groups (∆χ² = 2.59; p = .629). However, the model fit was slightly better for the model with different structural weights across groups (RMSEA = 0.07; CFI = 0.96; IFI = 0.96, SRMR = 0.09, NFI = 0.93, NNFI = 0.96). Although non-significantly different, the weight between perceived vulnerability and security intentions is higher and statistically significant in the CA group, while in SA group, the relationship is not statistically significant. It could be due to chance, but the relationship should be further explored when using SA or CA scale.

According to the finding from the T-Test that “Completely agree” was, on average, selected more often than “Strongly agree”, we confirm the familiarity hypothesis assuming that respondents will feel more comfortable choosing the more familiar endpoint, despite its semantically higher extremity. This is in accordance with the research findings (Weijters et al., 2013) that respondents tend to choose more familiar labels more frequently, even if these labels convey greater intensity (and should, theoretically, be chosen less often for that reason).

When comparing the mean values for the constructs, perceived vulnerability values were statistically significantly higher in the GroupCA compared to the GroupSA with greater variability in the GroupCA. This finding also aligns with the familiarity hypothesis (Weijters et al., 2013), suggesting that idiomatic and relatable translations encourage respondents to engage more with the extreme points of the scale. However, mean differences for other constructs were not statistically significant, though the GroupCA showed consistently higher variances across several scales. Furthermore, differences in skewness and kurtosis also emphasize the impact of anchor wording on response distributions. The GroupCA showed flatter distributions with reduced kurtosis and skewness values closer to zero for some scales, such as self-efficacy. These patterns suggest that idiomatic anchors in the GroupCA promoted more balanced and diverse response distributions.

CFA (Table 3) showed good model fit and convergent validity across both groups. Both groups achieved factor loadings above 0.5, AVE values above 0.50, and CR values above 0.7. This indicates sufficient measurement reliability and validity for both groups. However, nearly all CR and AVE values are slightly higher in the GroupCA compared to the GroupSA. While these differences are not large, their consistency suggests a slight advantage in measurement reliability and validity for the GroupCA. These results may reflect the influence of more idiomatic and familiar anchor translations in the GroupCA, which could have improved the respondents’ ability to consistently interpret and respond to the scale. This aligns with prior research that emphasized the role of anchor familiarity in enhancing response fluency since more familiar word combinations allow respondents to process them more quickly and perceive them as more reliable (Conklin & Schmitt, 2008; Weijters et al., 2013). Although the differences in CR and AVE values are very minor, their near-uniformity across constructs highlights the potential benefits of selecting anchors that resonate more naturally with the target population.

The test of measurement invariance using multigroup analysis provided sufficient support for configural invariance, but not for full metric and scalar invariance across GroupSA and GroupCA. Metric and scalar invariance were established for all factors but Security intentions, where two items showed differing loadings and intercepts across the groups. Partial metric and scalar invariance was achieved for this factor, as the remaining two items were invariant across groups. This suggests that, in some cases, differences in response scale wording may influence how certain items are understood and answered. However, having at least two invariant items per factor, enables meaningful comparisons of latent means across groups, as conducted in our study. We recommend caution when using different response scales across languages, especially when constructs are measured with fewer than three items. Such practices may threaten measurement invariance and lead to potentially erroneous conclusions about cross-cultural differences, which could steam from differences in item interpretation rather than true construct differences.

The results of multigroup SEM models indicate that for both groups, perceived severity, response efficacy, and self-efficacy were significantly associated with security intentions, consistent with the predictions of Protection motivation theory. The standardized path coefficients are stable across the groups, suggesting that these constructs consistently influence behavioural intentions regardless of the scale format used. However, the relationship between perceived vulnerability and security intentions was significant only in the GroupCA. This suggests that the anchor points in GroupCA may have enabled respondents to visualize their responses regarding their perceived vulnerability better. The difference may have caused it to have a significant impact on respondents’ behavioural intentions, compared to GroupSA, where this association was not statistically significant. The differences in regression results between the groups may also reflect the influence of anchor familiarity on response behaviour. As indicated in Weijters et al. (2013), more familiar anchors can enhance cognitive fluency, which leads to more confident and potentially extreme responses. Theoretically, all PMT constructs are recognized as the key determinants of protective behaviours or security intentions. Consequently, significant correlations among these constructs are expected, as they have been consistently empirically confirmed in previous studies (Li et al., 2022; Mou et al., 2022; Zuwita & Rahmatullah, 2021). This suggests that results from GroupCA yielded more valid conclusions.

The consistent differences observed between the groups show the nuanced influence of anchor phrasing on the measurement properties of survey instruments. The higher variance and more normally distributed response patterns in the GroupCA suggest that idiomatic phrasing may encourage respondents to engage more intensively with the full spectrum of response options. This might also potentially contribute to mitigating biases like central tendency or acquiescence bias. These findings support the idea that familiar anchors create cognitive ease and enable respondents to better differentiate between response categories and express their attitudes with higher precision. This may also explain the stronger association between perceived vulnerability and security intentions in the GroupCA in the regression model. When respondents are presented with familiar and intuitive phrasing, they may feel more confident in articulating their perceptions of vulnerability, resulting in a more pronounced connection between their attitudes and behavioural intentions.

We should note that even though the observed differences in means, variances, and regression outcomes were not large, their consistency across multiple constructs and items may indicate that the linguistic relatability of anchors does not merely affect isolated responses but can influence the overall structure of the data. For example, skewness and kurtosis values of the constructs in the GroupCA reflect a tendency toward more balanced engagement with the scale. Participants have spread their responses more evenly across the available options. These findings suggest that anchor phrasing should not be considered a neutral design choice but an important part of survey design that shapes how respondents interpret and interact with survey items. This can impact results on the item level, construct level, and the associations between constructs.