Planning and executing robotic actions using simulated image sequences created by generative deep neural networks

• Applicant: Professor Dr. Florentin Wörgötter
• Subject Area: Automation, Mechatronics, Control Systems, Intelligent Technical Systems, Robotics ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2019 to 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 417069796

Common methods for robotic action planning rely on symbolic processing. For example: AI planning requires domain knowledge and planning algorithms. Similarly, physics simulations, on which planning can be based too, have to rely on explicitly encoded programming instructions. Different from this, the goal of this work is to create a computer vision-based system that is able to generate short plans for robotic manipulation actions based on implicitly simulated image sequences. To this end, we will first determine different selected manipulation affordances in a visual scene using an encoder-decoder DNN (deep neural network). The resulting ‘affordance map’, thus, represents permissive action preconditions. Then we will use a different, generative DNN, which takes the scene as input and creates a new 3D-output scene by “imagining” one of the manipulation actions for which an affordance exists. Hence, this new scene shows how the situation would change if this action would actually be performed. As this is done in 3D, the resulting scene can be checked for geometric consistency and – if ok – the system has very likely arrived at a permitted action post-condition. Using this ‘imagined’ output scene for a next action, short planning sequences will be generated, which are then executed with our robots. This allows for rigorous quantification of real action outcome against “imagined” outcome. The central hypothesis that underlies this work is that it should be possible to create (short) executable plans relying entirely on sub-symbolic information. The main benefit of this is that simple everyday tasks for service robots might become plannable with a much reduced effort for explicit representations.

• DFG Programme: Research Grants
• Co-Investigator: Dr. Tomas Kulvicius