3D-Modelling of seafloor structures from ROV-based video sequences

• Applicant: Professor Dr.-Ing. Reinhard Koch
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2008 to 2016
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 68506282

• Final Report Year: 2015

Final Report Abstract

During the two phases of the project, influences of water on image formation and possible models for the effects were investigated. It was found that the two major effects, on image color and image geometry have a negative influence on methods from computer vision, especially 3D reconstruction, and make adaptations necessary. Therefore, a calibration approach for calibrating the distance between glass and camera and a possible tilt of the glass with respect to the image sensor was developed. It is a non-linear optimization based on an Analysis-by-Synthesis approach and allows to determine the glass parameters accurately. Based on the calibration method, a new approach for refractive Structure from Motion was developed including methods for estimating relative and absolute pose. The greatest challenge was to avoid projections of 3D points into the image due to the need of solving a 12th polynomial. Especially in non-linear optimization, the great number of such projections causes the classic approach of minimizing the reprojection error to be infeasible in terms of run-time otherwise. The solution to the problem was found by introducing a virtual camera for each image pixel into which 3D points can be projected perspectively. This allows for the first time to compute refractive bundle adjustment efficiently and thus to optimize reconstruction with more than a handful of images. After determining camera poses and 3D points during SfM, dense depth maps need to be computed. This was achieved by introducing a refractive plane sweep algorithm that does not rely on homographies or projections of 3D points into the image and runs on the GPU. Based on the dense depth maps, the effects of water on color in the input images can be corrected using the physical model on underwater light propagation resulting in color-corrected textures for the 3D model. In summary, a whole pipeline for underwater 3D reconstruction was developed, where geometrical effects of water on image formation were modeled explicitly in every step. Effects on color can then be corrected as well, allowing to perform reconstruction of objects below water as if they were retrieved from the seafloor.

Publications

• Simulating deep sea underwater images using physical models for light attenuation, scattering, and refraction. In P. Eisert, J. Hornegger, and K. Polthier, editors, VMV 2011: Vision, Modeling & Visualization, number 978- 3-905673-85-2, pages 49–56, Berlin, Germany, 2011. Eurographics Association
  A. Sedlazeck and R. Koch
  
• Perspective and non-perspective camera models in underwater imaging - overview and error analysis. In F. Dellaert, J.-M. Frahm, M. Pollefeys, L. Leal-Taixé, and B. Rosenhahn, editors, Outdoor and Large-Scale Real-World Scene Analysis, volume 7474 of Lecture Notes in Computer Science, pages 212–242. Springer Berlin Heidelberg, 2012
  A. Sedlazeck and R. Koch
  
• Refractive calibration of underwater cameras. In A. Fitzgibbon, S. Lazebnik, P. Pietro, Y. Sato, and C. Schmid, editors, Computer Vision - ECCV 2012, volume 7576 of Lecture Notes in Computer Science, pages 846– 859. Springer Berlin Heidelberg, 2012
  A. Jordt-Sedlazeck and R. Koch
  
• Refractive plane sweep for underwater images. In J. Weickert, M. Hein, and B. Schiele, editors, Pattern Recognition, volume 8142 of Lecture Notes in Computer Science, pages 333–342. Springer Berlin Heidelberg, 2013
  A. Jordt-Sedlazeck, D. Jung, and R. Koch
  
• Refractive structure-from-motion on underwater images. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 57–64, 2013
  A. Jordt-Sedlazeck and R. Koch
  
• Gas bubble shape measurement and analysis. In J. Hornegger X. Jiang and R. Koch, editors, Pattern Recognition, volume 8753 of Lecture Notes in Computer Science, pages 743–749. Springer International Publishing, 2014
  C. Zelenka
  
• Blind deconvolution on underwater images for gas bubble measurement. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XL-5/W5:239–244, 2015
  C. Zelenka and R. Koch
  (See online at https://doi.org/10.5194/isprsarchives-XL-5-W5-239-2015)