Do Culture and Reading Literacy Associate with Inconsistent Responding on Mixed-Worded Scales?

Abstract

Questionnaire scales often include a combination of both positively and negatively worded items such as “I make friends easily at school” and “I feel lonely at school”. In this case, a respondent with a high sense of belonging to school is expected to agree with the positively worded item and disagree with the negatively worded item. However, studies show that some of the respondents fail to switch the side of their responses according to item wording, a behavior we call inconsistent responding. The proportion of inconsistent respondents varies between countries in international assessments. This study shows that on a scale from PISA 2018 across 75 countries, between 4% to 30% of respondents show this behavior. By looking at two variables for culture and one variable for reading literacy, I expected to find significant associations between all of the variables and inconsistency. But I found that when having them all in the same model, only reading is significantly different from zero. I conclude that researchers should be cautious when using these types of scales in international assessments, especially when there are countries with low reading achievement levels.