Multimodal In-Car Dialog Demonstrator

This demonstrator implements a multimodal human-machine interface that fuses haptic interaction between the user and the car with speech, gaze, and gesture recognition. The scenario has already been intensively worked on at DFKI for several years. The demonstrator has been tailored to the objectives and needs of the Cluster and is meant to support collaborative use. It is based on a Mercedes R-Class SUV, which has been donated to DFKI by Daimler AG.

The demonstrator was used in several studies on the design of multimodal interaction models; on role-based access to devices, applications, and services, and the development thereof; on the development of speaker classification methods; on the development of libraries with presentation strategies; and on post-ASR error correction using eye gaze. The demonstrator is a continuation of long-standing cooperation with M. Pinkal's group on multimodal in-car dialog (TALK project). Cluster-internal collaborations related to the demonstrator include the improvement of a free-text dictation system in embedded mobile scenarios (D. Klakow); investigation of the influence of verbal references to buildings on the gaze behavior of a driver (M. Crocker); the use of advanced video encoding protocols for mobile live streaming via UMTS/HSUPA (T. Herfet); and the study of the use of animated characters in the car for persuasive systems (M. Kipp's JRG). The prototype system was demonstrated to Karl-Theodor zu Guttenberg, at that time German Federal Minister of Economics and Technology, in 2009.


Positioning mechanism. Sensor data in the environment (car) is transmitted to the user device. There, it is evaluated and matched against the stored user profile.

Hardware Equipment of the Demonstrator

Research Based on the Demonstrator within RA9

Collaborations with Use of the Demonstrator