ERC Consolidator Grant for MMCI investigator

Currently, it takes a great deal of effort to transfer real scenes into virtual three-dimensional worlds. The Saarbrücken computer science professor Christian Theobalt hopes to simplify and considerably accelerate this process. With the help of just one camera, in the future computers should be able to independently develop computational methods and continually optimize them in order to capture scenes realistically, both spatially and temporally. For this, Theobalt has now been awarded a two million euro “ERC Consolidator Grant” from the European Union’s Research Council. Thus, he is honored by the ERC for the second time, having already won an ERC Starting Grant in 2013. His research will benefit not only autonomous systems and robots, but also virtual and augmented reality applications, medicine and engineering.

Professor Christian Theobalt is working on a quantum leap in the area of virtual reality.

When American researchers wanted to make a 3D-printed portrait of former US President Barack Obama, they had to build a device out of 50 LED lights and 14 special cameras. In addition, they mounted ten different flash lights and specially developed light scanners in order to translate Obama’s face to a digital image. In the future Christian Theobalt, head of the Graphics, Vision & Video research group at the Max Planck Institute for Informatics in Saarbrücken and computer science professor at Saarland University, wants to do this with only one camera. “Nobody wants to carry around a multi-camera system for such 3D applications – that is just not reasonable,” he explains. In research, major advances have been achieved in recent years. “However, these were mostly related to very specific applications, like capturing facial expressions or hand movements. Usually, they required optimal conditions and special camera systems,” continues Theobalt.

To capture general, complex, moving real-world scenes in three dimensions, and also include the time component as the so-called fourth dimension, current methods are insufficient. “This can be described well using the example of everyday types of scenes: In many real environments, people move around in complex ways, get hidden behind other people or things, and often interact in the subtlest ways with various objects in the scene. It’s similar with street traffic – in an intersection, at any second, vehicles can cross lanes, occlude one another, or suddenly turn in different directions. Autonomous robots (which should cooperate with humans efficiently and safely), as well as autonomous vehicles, must deal with such situations, and they need to be able to reconstruct them in detail. 4D reconstruction is the basis for their perceptual capabilities, but the problem is very complex,” explains Theobalt. Nonetheless, to make this possible, the Saarbrücken computer scientist wants to connect two special fields systematically in a new way. On the one hand are so-called model-based methods, which in the past have been used for very specific scenes, such as in capturing faces. On the other, Theobalt relies on machine learning and in particular so-called deep learning. This neural network-based learning process has been causing a sensation in science for some time now. In any case, these neural networks must also be trained, and they have to be supervised during the training phase. “That in turn requires a great deal of training data, for which the correct solution – the correct 4D reconstruction – is known. However, for complex everyday scenes, this is not practical,” says Theobalt.

Together with his group he therefore wants to research the fundamentals, so that in the future, given a large amount of image data, computers will be able to independently choose a model upon which the digital representation is based, in terms of geometry, movement, and material and lighting properties of the visible surface. They should then also determine the methods by which they will reconstruct the scene, and constantly improve these on the basis of newly recorded data. In this way, it would no longer be necessary to program robots; they could simply be presented with the methods from which to start. They could then observe and learn from them. “Any application that depends on the analysis of real scenes will benefit from our results,” says Theobalt, “not only autonomous systems and augmented or virtual reality, but also applications in sports, biomechanics, medicine, human-computer interaction, and engineering.”

With the two million euro in funding from the European Union, Christian Theobalt will create positions for doctoral and postdoctoral researchers at the Max Planck Institute for Informatics. That is the aim of the European Research Council (ERC), which, through its research awards, promotes outstanding researchers chosen solely on the basis of applicants’ scientific excellence and the pioneering role of their proposed research. They are supported in the expansion of their research groups and can advance their own research projects.


Background on Christian Theobalt:

Christian Theobalt is Professor of Computer Science and leader of the research group “Graphics, Vision & Video” at the Max Planck Institute for Informatics in Saarbrücken. He is also Professor of Computer Science at Saarland University. From 2007–2009 he was a visiting professor at Stanford University. He received his Master of Science in Artificial Intelligence at the University of Edinburgh, his diploma in computer science at Saarland University, and his doctorate at the Max Planck Institute for Informatics.

His research focuses on foundational algorithmic questions in the interdisciplinary areas of computer vision, computer graphics, human-computer interaction, and machine learning. In particular, he researches algorithms for 3D reconstruction of static and dynamic scenes, virtual and augmented reality, markerless motion capture techniques, computer animation methods, methods for inverse rendering (estimation of material and lighting properties), machine learning techniques to support 3D and 4D reconstruction, new cameras and sensors, methods for semantic video editing, and methods for image-based rendering.

For his work he has been honored with many awards, among them the Otto Hahn Medal of the Max Planck Society in 2007, the EUROGRAPHICS Young Researcher Award in 2009, and the German Pattern Recognition Award in 2012. He has received two ERC grants from the European Union, an ERC Starting Grant in 2013 and now an ERC Consolidator Grant in 2017. The magazine “Capital” selected him in 2015 as one of the “Top 40 Innovation Leaders under 40” in Germany. He also one of the founders of the Captury GmbH in Saarbrücken, a spin-off of his research group, which has received multiple awards and sells a revolutionary new method for markerless motion capture.


Further Information:


Questions can be directed to:

Professor Christian Theobalt
Max-Planck-Institut für Informatik
Saarland Informatics Campus
Tel.: +49 681 9325 4028



Gordon Bolduan
Competence Center Computer Science Saarland
Saarland Informatics Campus E1.7
Tel: +49 681 302-70741

newsarchive >>