Publication Overview

Maria Staudte, Matthew W. Crocker, Alexis Heloir, Michael Kipp

The influence of speaker gaze on listener comprehension: Contrasting visual versus intentional accounts

Previous research has shown that listeners follow speaker gaze to mentioned objects in a shared environment to ground referring expressions, both for human and robot speakers. What is less clear is whether the benefit of speaker gaze is due to the inference of referen- tial intentions (Staudte and Crocker, 2011) or simply the (reflexive) shifts in visual atten- tion. That is, is gaze special in how it affects simultaneous utterance comprehension? In four eye-tracking studies we directly contrast speech-aligned speaker gaze of a virtual agent with a non-gaze visual cue (arrow). Our findings show that both cues similarly direct listeners’ attention and that listeners can benefit in utterance comprehension from both cues. Only when they are similarly precise, however, does this equality extend to incongru- ent cueing sequences: that is, even when the cue sequence does not match the concurrent sequence of spoken referents can listeners benefit from gaze as well as arrows. The results suggest that listeners are able to learn a counter-predictive mapping of both cues to the sequence of referents. Thus, gaze and arrows can in principle be applied with equal flexi- bility and efficiency during language comprehension.

Published in: Cognition
Year: 2014
Volume: 133
Pages: 317-328