RA1: Text and Speech Processing


Prof. Dr. Manfred Pinkal - UdS-LST, UdS-CS

Prof. Dr. Matthew Crocker - UdS-LST, UdS-CS

Prof. Dr. Hans Uszkoreit - UdS-CS, DFKI


Vision and Research Strategy

Language is the most natural and expressive medium for human communication and interaction. The richness of language derives from its grounding in our knowledge of the world and the immediate linguistic and non-linguistic context. In Research Area 1, we have focused on structurally informed models of distributional semantics, using unsupervised and minimally supervised approaches that combine linguistic knowledge with linguistic information. We substantially extended our work by going beyond intra-sentential context in two directions. On the one hand, we created text-level models of discourse relations and script knowledge, and applied these models to improve text-level comprehension. On the other hand, we have advanced cross-modal methods that deeply integrate language processing with both extra-linguistic knowledge and visual information, and lead to more naturalistic interaction with dialog systems and avatars. Furthermore, our methodological contributions regarding neurophysiological and pupillometric measures have enabled us to investigate situated human-computer interaction in greater detail. Our results have significantly contributed to improvements in several types of natural language processing tasks: the offline extraction of complex knowledge from text to feed knowledge bases; the online semantic interpretation (disambiguation and composition) required for deep text understanding, enrichment of text documents, and deep question-answering; the understanding of pictures and visual scenes; and the integration of speech and vision modalities to enrich spoken-language understanding and generation technologies in virtual interactive environments.

Research Topics and Achievements


Awards and Honors