Reconstruction of Complex Deformations in 3D Scenes from Color and Depth Images

• Applicant: Professor Dr.-Ing. Reinhard Koch
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing ; Measurement Systems
• Term: from 2014 to 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 257332386

While the 3D reconstruction of rigid environments has been investigated very thoroughly in the last two decades, the analysis of deforming scenes is still subject to intense research. Most reconstruction algorithms for deformable objects utilize feature point tracking in a sequence of input images, however, feature tracking requires a sufficient amount of texture or structure in the image as well as a certain level of unambiguity of these features. This does not only apply to color images but also to depth images, since depth based reconstruction requires structural information for a stable tracking result.Since the input data of feature based tracking methods is already the result of an interpretation, a number of disadvantages are inherent to this class of algorithms:1. The reduction of complex image data to a set of feature coordinates does discard a lot of information that can potentially be used to solve the reconstruction problem.2. Optimization does not employ the measurement data directly, which is inaccurate and entails error prone propagation of image noise.3. The existence of outliers in the feature correspondences requires the algorithm to identify wrong information and discard it. This leads to a weaker convergence behavior of the overall optimization problem.This issues can be circumvented by using methods operating directly on the input data: 'Analysis by Synthesis' (AbS) based methods generate a 3D model of the tracked object and use it to synthesize a virtual camera image. The difference between the synthesized images and the real input images is then used to analyze the object parameters. A disadvantage of this system is the need for an extensive global optimization process, since the problem can not be linearized in its object parameters. However, AbS makes use of the complete image data available.The main goal of this research project is to combine the advantages of existing AbS methods and feature based algorithms while maintaining the useful properties of direct optimization. This is going to be achieved by using information about feature movement to provide 'guidance' to the direct optimization process without contaminating the direct optimization goal. In addition to this, an adaptive deformation model will be constructed to ensure versatility while reducing the computational optimization costs.

• DFG Programme: Research Grants