Sign Language Synthesis and Interaction

Vision and Research Strategy

Sign Languages are the native language for the large cultural group of the Deaf. Studies show that reading/writing skills of Deaf individuals is significantly lower compared to hearing people which puts Deaf people at a severe disadvantage in many contexts, including education, work and internet.

Animated sign language avatars on the internet or on mobile devices have the potential to significantly improve the situation for the Deaf. However, producing comprehensible sign language avatars is a difficult, interdisciplinary problem involving linguistics, AI, HCI and animation. State-of-the-art systems reach comprehensibility levels of only 60%. The main reason is the highly multimodal nature of sign language. The main challenges are: the modeling of nonmanual features (face, head, torso) and prosody (pause, motion dynamics), usable tools for the creation of animation databases and flexible deployment capabilities.

Research in this junior research group aims at developing signing avatars that reach comprehensibility rates of 80-90% and that can be flexibly deployed (web, mobile, 3D displays). This involves the creation of notation systems for nonmanual features and prosodic aspects, the implementation of animation methods that produce natural looking signs and the systematic validation of output quality using comprehensibility tests.

The research requires close contact with and the direct involvement of Deaf people for advise, quality control and validation tests. The outcome of this research will encompass novel notations, flexible animation tools and proven validation methods. This will advance the state of the art in sign language synthesis, facilitate general research in sign linguistics and pave the way for concrete accessibility applications for the Deaf.

Intuitive Sign Language Animation Authoring Environment for the Deaf

Projects and Collaborations

Dr. Alexis Heloir

Dr. Alexis Heloir

Alexis Heloir is leading the Junior Research Group Sign Language Synthesis and Interaction.

Fon: +49 681 302 3393