Transferring Deep Neural Networks from Simulation to Real-World

• Applicant: Professor Dr. Vasileios Belagiannis
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 458972748

Computer vision contributes in creating visual priors as self-contained tasks or input to another system. In the context of autonomous navigation, the system can be a mobile agent that not only relies on the raw sensory inputs, but also on computer vision algorithms for understanding the environment. Recent studies on embodied agents show that an agent acts more accurately when visual priors such as semantic segmentation, depth estimation are provided next to the raw input data. Producing the visual priors though comes at the cost of data collection and annotation. The latest approaches build on deep neural networks, which are trained with supervision. For that propose, a large pool of data and annotations has to be created prior to training the model. To address this limitation, simulation is an alternative source for data and annotation generation. In the context of deep neural networks, it can be considered for the replacing the real-world, where a large amount of synthetic data is created according to the task in place. Although, the data simulation has clear advantages over the real-world datasets, there is also a clear limitation. Training a deep neural network with synthetic data does not result in good performance on real-world data.In this research project, we are going to conduct research on closing the performance drop when transferring deep neural network models from the simulation to real-world applications. Our testbed for measuring the performance will be semantic image segmentation and depth estimation from a single image. In our research, we will propose algorithms that teach a deep neural network how to fast learn adapting into new environments. This concept is widely known as meta-learning. In this project, it will be explored for learning a model in simulation and then transferring it to the real-world. Meta-learning has never been seen as a way to tackle model transfer, but its formulation suits well to the problem.

• DFG Programme: Research Grants