Training Deep Networks for Real-world Computer Vision Scenarios with Rendered Data

• Applicant: Professor Dr.-Ing. Thomas Brox
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2018 to 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 401269959

Approaching computer vision tasks with deep learning requires large datasets to exploit the full potential of this approach. For some computer vision tasks, such as motion estimation or depth estimation, however, it is impossible to provide such large datasets by manual annotation of images. One possible solution is the use of synthetic, rendered image data. In this case, both the input images and the desired outputs are rendered and are available for the training process. However, in the end, the network should provide good results on real data. In this project we want to develop methods that will allow us to train networks on synthetic data but also achieve optimal results on real data. To this end, we will develop methods that integrate non-annotated real data besides the annotated synthetic data into the training process. The focus will be on optical flow estimation and depth estimation from videos. Moreover, we want to investigate to what extent the developed concepts can be employed for image based control, too.

• DFG Programme: Research Grants