Publication Overview

Nikolina Koleva, Sabrina Hoppe, Mehdi Moniri, Maria Staudte, Andreas Bulling

On the interplay between spontaneous spoken instructions and human visual behaviour in an indoor guidance task

We report on an indoor guidance study to explore the interplay between spontaneous spoken instructions and listeners' eye movement behaviour. The study involves a remote speaker (the instructor) to verbally guide a listener (the walker) to complete nine everyday tasks in different locations inside a room. We collect a multi-modal dataset of 12 pairs of users consisting of the scene view videos from the listener's perspective, their gaze data, and instructors' verbal instructions. We analyse the impact on instructions and listener gaze when the speaker can see 1) only the video, 2) the video and the gaze cursor, or 3) the video and gaze cursor with artificial noise. Our results show that listener gaze behaviour mostly depends on utterance presence (or absence) and, further, that it varies significantly before and also after instructions. Moreover, speakers give more negative feedback when listener gaze position is available to them. These findings suggest that speakers use gaze information as an indication of what referent the listener is effectively considering, which lead listeners to deliberately use their gaze as a pointer. On the other hand listener gaze is a symptom of language comprehension since no differences during instructions were observed.

Published in: Proceedings of the 37th Annual Conference of the Cognitive Science Society
Year, Month: 2015, 7