Differentiable Physics Simulators for Realistic 3D Reconstruction

• Applicant: Professor Dr. Daniel Cremers
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 413611294

The reconstruction of 3D structure from camera data is among the classical problems in computer vision. Even though methods have made enormous progress over the last years and now entire actions can be reconstructed in 3D from synchronous videos, most methods are limited to reconstructing the surface of the object. In contrast, the goal of the present project is to reconstruct the entire underlying physics directly from the camera data - this means that we want to determine not only the shape of the object but also its physical properties such as mass, elasticity, momentum and internal and external forces. We believe that this will lead to a more complete understanding of the filmed action. This should make it possible to obtain much more complete reconstructions from significantly fewer observations. In addition, the estimated physical simulation should also allow better extrapolations of the action into the future and thus more reliable predictions of how the action will develop. Methodologically, we want to solve the problem both as an inverse problem and with the help of differentiable physics simulators. The core idea of the latter approach is that we use an encoder-decoder network in which the decoder represents the differentiable simulation of the video from simulation parameters and the encoder, which assigns suitable simulation parameters to an input video, is to be learned from training data.

• DFG Programme: Research Units
• Subproject of: FOR 2987: Learning and Simulation in Visual Computing