Authors:
Shubi, Omer, shubi@campus.technion.ac.il, Technion - Israel Institute of Technology
Avraham Hadar, Cfir, kfir-hadar@campus.technion.ac.il, Technion - Israel Institute of Technology
Berzak, Yevgeni, berzak@technion.ac.il, Technion - Israel Institute of Technology
Keywords: eye tracking, information seeking, question answering, multimodal models
Abstract:
Readers can have different goals with respect to the text that they are reading. Can these goals be decoded from their eye movements over the text? In this work, we use question answering as a versatile paradigm for studying how eye movements in reading reveal reader’s goals, and examine for the first time whether it is possible to distinguish between two types of common reading goals: information seeking and ordinary reading for comprehension. Using OneStop, a large-scale eye tracking dataset, we address this task with a wide range of models that cover different architectural and data representation strategies, and further introduce a new model ensemble. We find that transformer-based models with scanpath representations coupled with language modeling solve it most successfully, and that accurate predictions can be made in real time, long before the participant finished reading the text. We further introduce a new method for model performance analysis based on mixed effect modeling. Combining this method with rich textual annotations, such as the length of the text and the difficulty of the question, question answering behavior, and reading comprehension outcomes, reveals key properties of textual items, participants, and their eye movements in reading that contribute to the difficulty of the task. This approach improves our understanding of the variability in eye movement patterns across the two reading regimes.