Authors:
Pohl, Nada Zahra, pohln@hdm-stuttgart.de, Hochschule der Medien
Schmidt, Heiko, heiko.schmidt@gesis.org, GESIS – Leibniz-Institut für Sozialwissenschaften
Kern, Dagmar, dagmar.kern@gesis.org,GESIS – Leibniz-Institut für Sozialwissenschaften
Gottschling, Steffen, gottschling@hdm-stuttgart.de, Hochschule der Medien
Kammerer, Yvonne, kammerer@hdm-stuttgart.de, Hochschule der Medien
Keywords: Negated questionnaire items, Reading comprehension ability, Detection of comprehension difficulties, Eye-tracking
Abstract:
Questionnaires are essential for collecting self-report data, but items that are difficult to comprehend threaten reliability and validity (Lenzner, 2012). Thus, advancing questionnaire pretesting is crucial, and eye-tracking offers a promising way to detect readers’ comprehension difficulties (Graesser et al., 2006). Negated items are often used to mitigate biases like acquiescence bias (Johnson et al., 2004), yet they impose greater cognitive demands on comprehension and response processes (Christensen, 2009). This study investigates comprehension difficulties associated with negated (compared to affirmative) questionnaire items using eye-tracking and subjective ratings and examines whether reading comprehension ability moderates these effects. While prior research suggests that negated items require more processing time during comprehension and response phases (Baumgartner et al., 2018; Koutsogiorgi & Michaelides, 2022), studies have mostly focused on university students with high and uniform reading proficiency, limiting generalizability Using a within-subjects design, 49 participants (aged 18–50) with varying reading comprehension abilities completed 32 Likert-type questionnaire items (16 affirmative, 16 negated). Items (with scale) were presented separately in random order, with affirmative and negated versions counterbalanced across participants. Eye-tracking measures included total fixation duration and number of regressions in item statements (normalized per syllable), total fixation duration on the response scale, and transitions between statements and response scales. Participants also provided subjective comprehensibility ratings. Results showed significantly longer fixation durations and increased regressions for negated items, indicating higher processing demands. Participants also fixated longer on response scales and made more transitions between statements and response scales when answering negated items. Reading comprehension ability was significantly negatively related to total fixation duration on statements and response scale but did not moderate the effects of item type, suggesting that negation imposes additional cognitive demands regardless of reading ability. Yet, better readers retrospectively rated negated items as less comprehensible than poorer readers, indicating greater awareness of comprehension difficulties. These findings highlight the added difficulty negation poses in items. While negation is sometimes used to counteract response biases, its effects on comprehension and response processes warrant caution. Future research should explore alternative strategies to mitigate biases without compromising comprehensibility, ultimately improving questionnaire design and data accuracy.