Research Areas

The research areas RA1 Text and Speech Processing and RA2 Visual Computing form the basis of any advanced scientific work on multimodality, since they contribute core expertise in handling the most prominent multimodal data types: text, speech, images and video. In both research areas, we advance processing methods for raw data to meet the requirements of the envisioned applications concerning efficiency, robustness, and reliability. The research areas also jointly explore cross-modal computation techniques, for example, linguistic processing using visual context information like lip movement or gaze.

The investigation of processing methods for linguistic and visual data is done in collaboration with RA3 Algorithmic Foundations. More generally, RA3 provides efficient processing techniques for all other areas, in particular, algorithms for handling massive data sets, computational geometry techniques, efficient and effective indexing methods, and general methods for obtaining reliable implementations. Furthermore, computing and interaction is increasingly distributed and needs to satisfy stringent availability, security and privacy requirements. These requirements are common to most application domains, and are addressed in RA4 Security, Privacy, and Accountability.

The application-oriented research areas RA5 to RA9 depend on and stimulate foundational research in RA1 to RA4. The methods and demonstrator systems stemming from RA5 to RA9 provide important building blocks for the envisioned future multimodal environments.

RA5 Knowledge Management aims to facilitate meaningful answers to advanced queries by collecting, organizing, and semantically understanding information drawn from distributed sources. While RA5 focuses on raw data of comparably high quality, e.g., scientific publications, RA6 Information Processing in the Life Sciences concentrates on noisy raw data and its curation, and presentation and visualization in the context of computational biology. RA7 Large-Scale Virtual Environments develops the necessary techniques and software to realize visually rich virtual environments, while RA8 Synthetic Virtual Characters focuses on the creation of realistic virtual characters with respect to appearance and behavior. RA9 Multimodal Dialog Systems focuses on symmetric multimodal dialog systems.