This demonstrator implements a multimodal human-machine interface that fuses haptic interaction between the user and the car with speech, gaze, and gesture recognition. The scenario has already been intensively worked on at DFKI for several years. The demonstrator has been tailored to the objectives and needs of the Cluster and is meant to support collaborative use. It is based on a Mercedes R-Class SUV, which has been donated to DFKI by Daimler AG.

The demonstrator was used in several studies on the design of multimodal interaction models; on role-based access to devices, applications, and services, and the development thereof; on the development of speaker classification methods; on the development of libraries with presentation strategies; and on post-ASR error correction using eye gaze. The demonstrator is a continuation of long-standing cooperation with M. Pinkal's group on multimodal in-car dialog (TALK project). Cluster-internal collaborations related to the demonstrator include the improvement of a free-text dictation system in embedded mobile scenarios (D. Klakow); investigation of the influence of verbal references to buildings on the gaze behavior of a driver (M. Crocker); the use of advanced video encoding protocols for mobile live streaming via UMTS/HSUPA (T. Herfet); and the study of the use of animated characters in the car for persuasive systems (M. Kipp's JRG). The prototype system was demonstrated to Karl-Theodor zu Guttenberg, at that time German Federal Minister of Economics and Technology, in 2009.

Positioning mechanism. Sensor data in the environment (car) is transmitted to the user device. There, it is evaluated and matched against the stored user profile.

The demonstrator is based on a fully-equipped Mercedes R-Class luxury SUV, which has been donated in 2008 to DFKI by Daimler AG to support our research in a realistic environment. Being a prototype, the car cannot be approved for on-road use, although it is technically fully functional.
In order to read status information from the car's comfort systems and to trigger actions, we connected a multimodal dialog manager to the car's CAN (Controller Area Network) bus via a USB interface by Vector Informatik. Based on this hardware, we implemented a software interface that provides our HMI with information about the status of the car's windows, seat heating, climate controls, multi-functional steering wheel (including its buttons and rotation angle), and brake pedal. The interface also allows the HMI to physically manipulate the state of these devices through a set of simple commands, e.g. in order to open and close the windows.
The steering wheel and brake pedal function as input devices for a driving simulation, which is used to measure the distraction induced by interacting with the system. The driving simulation is rear-projected to the windshield of the car using an ultra-wide-screen projector mounted on top of the hood.
For multimodal interaction with the driver and passengers, four seats have been equipped with individual automotive touch screens: two 7-inch touch screens installed on a dashboard above the center console for the driver's and front passenger seats, and two 10-inch touch screens for the rear seats. They are connected to individual compact PCs located in the trunk. In addition, all seats are equipped with hi-fidelity directional microphones (Sennheiser ME 105-NI) that are used to capture the corresponding passenger's speech. The microphones are connected over XLR to a MOTU, which in turn is connected to the server PC.
Gaze information can be obtained using a Tobii IS-Z1 Eyetracker, which is specifically tailored to embedded scenarios like the automotive set-up in terms of size and features. Furthermore, a video stream showing the driver's face is recorded using a standard webcam.

Development of Evaluation Measures

For evaluation of multimodal interaction concepts in the context of driving and driver distraction, we provide Lane Change Task (LCT). LCT is a widely accepted measure in automotive research. Hence, using LCT allows a direct comparison of results. However, we are aware of the fact that LCT's scope is limited and it will therefore not be a suitable evaluation measure for subsequent phases. Therefore, we began to develop our own solution. The experiments reported are already based on that solution. A formal comparison with LCT has been conducted, revealing a high correlation.

Development of a Driving Simulator

Our driving simulator solution was realized with jMonkey Engine (jME), a high performance scene graph based graphics API. This open-source framework was written entirely in Java and has built up a reputation in game development. Furthermore, an abstraction layer for using physics libraries like the Open Dynamics Engine (ODE) is included in the jME Physics 2 framework. The combination of these two components enables a remarkable driving simulation using self-built traffic scenarios. In order to facilitate a realistic simulation, our solution provides an interface for game controllers. For example, considering use in the Mercedes R-Class, the simulator is controlled via the CAN bus, which provides the information about the car's steering wheel and pedals. Further interfaces to a 2D traffic simulator, as well as the abovementioned eye tracker, are in development. The simulator collects car information (position, direction, speed, etc.) that can be analyzed according to the LCT after the test drive has been finished. Moreover, this data can be sent at run time to an external visualization system, which locates the virtual car in a much more detailed copy of the model and simultaneously provides the driver with a high-resolution projection on the windshield.

Design of multimodal interaction models

A study was conducted that addressed the question of how speech and tangible interfaces can be combined in order to provide effective multimodal interaction in vehicles, taking into account the special requirements induced by the circumstances of driving. In this study, speech was used to set the interaction context (determine the object which is to be manipulated) and a turn-and-push dial is used to manipulate/adjust. The distraction induced by manual (conventional), speech-only, and multimodal interaction (combination of speech and turn-and-push dial) was measured using the standardized lane change task (LCT). Results show that while subjects were able to perform more tasks in the manual condition, their driving was significantly safer when using speech-only or multimodal dialog.

Role-based access to devices, applications, and services

A study was conducted that addressed the question of how communication between passengers can be supported by transmitting speech (and later also video) of the communication partners back and forth. In particular, the following question was addressed: is listening to noisy speech coming from the back seat really distracting to the driver? Subjects rated the truth of common-sense statements played from the back of the car (clear, noisy) while driving with a drive simulator. NOISY was rated significantly more distracting than CLEAR, while objective driving performance degrades only for men and not for women.

Development of speaker classification methods

We developed a GMM-SVM-supervector system for speaker age recognition and conducted an experimental study with the aim being to evaluate the performance of the system and explore the selection of parameters. Due to changes in the definition of the age classes and the use of particularly short utterances for testing, the underlying task is considered more difficult than the one underlying previous studies (by the authors and others). Nevertheless, a comparable accuracy has been obtained.

Development of models for role-based access to devices, applications and services

We developed a concept for passenger localization and identification that especially takes into account information security and privacy issues. According to this approach, information that is stored on nomadic devices (mobile, personal devices that are brought into the car) is used to decode and locally store personal information.
 Each seat is equipped with a microphone, pressure sensor, and a screen. Voice profiles are generated and pressure is measured at each position. For each seat, icons on the built-in screens indicate if speech was detected or if pressure information is available. This sensory information is made available to the users' personal mobile devices, which also contain their user profiles. Passengers that are not regularly using the car can connect to it using their mobile phone, which obtains the current measurements for all seats and performs a comparison locally on the device. Using speaker identification technology and sensor fusion networks, a match likelihood is computed for each seat. If the profile on the device matches some seat's sensor data above a certain threshold, the device informs the user about the successful positioning and asks whether he wants to share certain parts of his user profile with the car. Accepting this enables advanced personalization scenarios on the built-in displays, such as resuming a movie that was previously interrupted. For those passengers who have traveled with the car before, the positioning comes into effect without requiring any further interaction from the user. A personalized greeting indicates that a successful match has been made. For the driver in particular, permitting the device to share personal information with the car allows it to retrieve schedule data and link it to automotive functions like the on-board navigation system.

Development of libraries with presentation strategies

The effectiveness of five modality variants (speech, text-only, icon-only, two combinations of text and icons) for presenting local danger warnings for drivers was experimentally investigated. We focused on suddenly-appearing road obstacles within a maximum up-to-date scenario as it is envisaged in Car2Car communication research. Effectiveness was measured by the minimum time necessary for fully interpreting the content. Results show that text-only requires the most time, while icon-only is perceived the fastest. The two combined versions lay in between. The minimum length for speech was determined by the duration of the utterance, which is longer than the perception time of text-only in this case. However, speech could be decoded reliably by nearly all subjects. Results indicate further that a blinking visual cue provided through the peripheral visual channel was able to enhance the salience of visual modalities. Subjective judgments by the subjects furthermore suggest a combined use of visual and auditory modalities.
In a second study, two presentation factors were selected: modality and level of assistance. The modality factor had 4 variants: speech warning, visual and speech warning, visual warning with blinking cue, and visual warning with sound cue. The level of assistance varied: action suggestions (AS) were sometimes given. In accordance with the ISO usability model, 6 measurements were derived to assess the effectiveness and efficiency of the warnings and the drivers' satisfaction. Results indicate that the combination of speech and visual modality leads to the best performance as well as the highest satisfaction. In contrast, purely auditory and purely visual modalities were both insufficient for presenting high-priority warnings. AS generally improved the usability of the warnings, especially when they were accompanied by supporting information so that drivers could validate the suggestions.
Finally, in a third study, we investigated possible behavioral effects. We compared driving performance as well as driver stress measured with physiological sensors in the following experimental conditions: NO ASSISTANCE, WORKING SYSTEM and UNANTICIPATED FAILURE. Results show that the driving performance and stress level was severely affected by system failures.

Post-ASR error correction using eye gazes

We conducted another study on correcting speech recognition while driving. Following the ASR post-correction paradigm, we considered ASR as a black box. For dictating (and editing) a text message while driving, the black-box scenario is very likely as car manufacturers are in the process of moving from owned on-board ASR solutions to off-board third party services. Obvious means for correcting errors that we know of from personal computers at home will not be immediately applicable here, because in mobile situations a number of constraints apply. Moreover, correcting speech recognition errors by adding another speech recognition step bears the danger of ending up in cascading errors that frustrate the user. We therefore believe that the problem of post-ASR correction is a suitable testbed for the proposed gaze-based interaction concept.
In our experiment, we used eye-gaze tracking in combination with a button on the steering wheel as explicit input. This approach combines the advantages of direct interaction on visual displays but avoids the drawbacks of a touchscreen. Particular advantages are freedom of placement for the screen (even out of the user's reach) and that both of the driver's hands remain on the steering wheel. The results showed that this interaction modality is slightly slower and more distracting than a touchscreen, while being significantly faster than automated speech interaction.

Summary of finished and on-going work for the multimodal in-car dialog demonstrator.

Implicit gaze-based input on the example of post-ASR error correction.

Internal Collaborations

With RA1, we continued long-standing cooperation with Manfred Pinkal's group on multimodal in-car dialog (TALK project). Together with Dietrich Klakow (RA1), we compared different methods for improving a given free-text dictation system with respect to its efficiency in embedded mobile scenarios, where distraction, interaction cost, and hardware limitations enforce strict constraints over traditional scenarios. A corpus-based evaluation measures the trade-off between the improvement of the word error rate versus the interaction steps that are required under various parameters. The best combination of parameters results in a 55% relative improvement in WER, whereas using a “black box” ASR and a domain-specific LM, 35% could be achieved.
In collaboration with Matthew Crocker (RA1), we carried out a data collection on how verbal references to buildings influence the gaze-behavior of a driver. In particular, descriptions based on the spatial characteristics ("the building on the right hand side") vs. descriptions based on features of the building ("the building with the blue roof") were compared under realistic driving conditions. Eye gazes of the subject were tracked using the equipment of the demonstrator. The data is still in the process of being analyzed. We expect insights into the influence of such descriptions on driving safety. Furthermore, we plan to supplement the findings with a driving experiment in which natural language descriptions of objects outside the car are investigated under controlled conditions.
The telecommunications lab led by Thorsten Herfet (RA2, RA4) has developed a fault-tolerant video encoding protocol (AL-HEC = Application Layer Hybrid Error Coding) for mobile live video streaming via UMTS/HSUPA. The video content is send to a central server, which transcodes the incoming stream to various formats as required by the receiving clients, e.g. Flash video. We will integrate such a video client in our car's HMI so that the driver is able to receive live video streams from other cars. One particular useful scenario would be one in which a car at the beginning of a traffic jam broadcasts information about the current road status, e.g. the progress of construction work. In case of an accident, such live video from nearby cars would provide valuable information for police and/or emergency medical personnel to control their operations.
Gerhard Weikum's group (RA5) developed a query processing and reasoning system that allows the application of background knowledge in order to retrieve a set of most likely destinations from a database. These results can also be used to improve the accuracy of a speech recognizer through dynamic grammars. The system presents evidence to the user as to why a destination has been suggested. The system is being integrated into the car demonstrator, which will then be able to automatically infer a destination based on current day, time of the day, and knowledge about the user's goals based on emails, calendar etc.. Furthermore, the reasoning system is applied to provide a list of possible destinations at a given date (day/month/year) based on the personal data set. The reasoner uses hard- and soft rules to infer likely destinations, which are rated with a confidence value. Additionally, relations between passengers in the car are derived, which can be recognized by comparing speech profiles with profiles stored on the mobile devices of passengers.
 In collaboration with Michael Kipp (JRG Embodied Agents), we are studying the use of animated characters in the car for persuasive systems. Carefully controlling the driving conditions (e.g. animated characters are allowed only when the wheels are not turning), we are particularly interested in the question of whether animated characters are a suitable means for persuading the driver to drive in a more energy efficient manner. The motivation behind this scenario is that research in the automotive industry revealed that while innovation in the area of engine and drive chain leads to average savings of 0.5 to 1 %, the driver is the major factor (25 % to 30 % savings).
In collaboration with Alexander Koller (JRG Efficient Algorithms in Computational Linguistics), we are investigating how route descriptions for driving tasks can be improved through references to landmark objects. An ongoing master's thesis aims to develop a navigational system that utilizes information about static and dynamic objects that are suitable as landmarks, i.e. salient buildings and other cars. The navigational system is based on the driving simulator that is developed in cooperation with Prof. Slusallek's group, so that navigational tasks can be safely simulated and evaluated in a virtual city environment.

External Collaborations

Tim Schwartz (M2CI Cluster of Excellence) and Christian Müller (DFKI) co-organized the workshop series Multimodal Interfaces for Automotive Applications (MIAA) in conjunction with the International Conference on Intelligent User Interfaces (IUI). The first workshop took place in February 2009 on Sanibel Island, Florida. The second workshop took place in February 2010 in Hong Kong, China.
From April to July 2009, Yujia Cao visited. Yujia is a PhD student at the Human Media Interaction group of the University of Twente (Enschede, the Netherlands). Her research topic is cognitive-aware multimodal information presentation, focusing on the design of multimodal presentations that can be more efficiently perceived and processed by the human cognitive systems. She designed and conducted a user experiment investigating how local danger warnings can be better presented to drivers using the demonstrator.
In August 2009, Dagmar Kern from the University of Duisburg-Essen visited. She conducted an experiment on using eye gazes to replace or supplement touch-based interaction in the car using the demonstrator. The prototype of our joint system was demonstrated to Dr. Karl-Theodor zu Guttenberg, at that time German Federal Minister of Economics and Technology.