Authors:
Lopes Rego, Adrielli, a.t.lopesrego@vu.nl, Department of Education Sciences - Vrije Universiteit Amsterdam
Snell, Joshua, j.j.snell@vu.nl, Department of Cognitive Psychology - Vrije Universiteit Amsterdam
Meeter, Martijn, m.meeter@vu.nl, Department of Education Sciences - Vrije Universiteit Amsterdam
Keywords: semantics;embeddings;saliency;eye-movements;OB1-reader
Abstract:
During reading, readers perform forward and backward eye movements through text, called saccades. Although such movements are intuitive to readers, the mechanisms underlying such behaviour is not yet fully known, particularly regarding the role of higher-order linguistic processes in guiding reading behaviour. It is of general agreement that more surprising words are read more slowly. Yet, the mechanism through which inter-word processing influences how readers move through text remains largely unaccounted for. One possibility is that readers tend to target the closest most informative word in the surrounding context when moving through text. In this study, we investigate the influence of semantics on saccade targeting, i.e., determining where to move our eyes next. Using contextualized embeddings from GPT-2 small and large, we measure pairwise semantic similarity between the fixated word and its surrounding words within a context window. We then analyse to what extent contextualized semantic similarity predicts the next fixation target, beyond word length, frequency and surprisal. In addition, we explore the mechanism through which this relation may take place by performing simulations with the OB1-reader, a model of eye-movement control in reading. For this end, we develop a saliency map based on semantic similarity and positional distance between the fixated word and the surrounding word, whereby the closer and less similar to the fixated word, the more likely the surrounding word is to attract attention and thus be the next fixation target. Furthermore, saliency of words in upcoming positions is also modulated by predictability and orthographic input from the parafovea to account for the uncertainty about the identity of upcoming, not yet recognised words.