RA9: Multimodal Dialog Systems

Vision and Research Strategy

Multimodal dialog systems exploit one of the major characteristics of human-human interaction: the coordinated use of different modalities. During the reporting period, we have been working on new interaction paradigms for smart spaces, such as instrumented cars, shops, homes and meeting rooms. Our vision is to lay the foundations for the next generation of symmetric multimodal dialog systems that support flexible, efficient, adaptive and robust means for human-environment interaction, while being easy to learn and use. We are investigating the foundations and the software for multimodal dialog systems that allow the average person to interact with smart spaces anytime and anywhere without special skills or training. To reach this ambitious goal, our research strategy is mainly focused on the development of advanced methods to allow for affective, collaborative, multiparty multimodal communication. In addition, we have worked on strategies needed for coordinating input and output modalities, for resolving integration and synchronization issues among modes, and for using information in one input mode to disambiguate noisy or error-prone input in another. Our work on tangible multimodal interfaces, in which humans can interact with physical and virtual objects in their surrounding environment via coordinated speech, gesture, eye gaze and physical manipulation, exploits the full richness of modalities known from everyday human-human communication as shown in the multimodal in-car demonstrator.

Research Topics and Achievements

Responsible Investigators and Personal Development


Prizes and Awards