RA8: Synthetic Virtual Characters

Vision and Research Strategy

Our long-term vision is to build virtual characters that are indistinguishable from real people. They should look and speak like human beings, show emotions, and mimic the behavior of real people in an individual and character-specific way. Our virtual characters will be able to engage in a multimodal interaction with individual human users, a group of human users, or even among themselves for a limited task-oriented domain in a mixed-reality, instrumented environment. Sensory input from cameras (to recognize the person, gesture, emotion) and microphones (for speech recognition) will allow human-like perception, enhanced by input from the instrumented environment (locations, movements, actions of physical objects and people). The virtual characters can behave like celebrities (film stars, politicians, talk show hosts) or imitate everyday people (you can create your personal virtual representative) on the basis of exchangeable persona modules. These modules contain empirically derived, mathematical models of the original person's appearance and behavior (gesture, posture changes, head movement) which can be created by naïve users by selecting suitable video clips of the human originals and using intuitive tools to extract the key factors that make up the person's behavioral shell. The user can store, visualize, tweak and merge persona models to achieve perfect imitation or create new personalities using a number of pre-defined style dimensions. The persona modeling toolkit performs automatic video and speech analysis, exploiting computer vision and speech technology, machine learning and classification techniques in a human-in-the-loop editing environment. Highly accurate motion capture data can be imported and is fully compatible with these models. Mechanisms for high level control are available to fulfill the complex needs of real-time behavior control for virtual characters, from walking through a door to changing the current topic of the dialog.

Research Topics and Achievements

Responsible Investigators and Personal Development

Collaborations

Prizes and Awards

Publications