All-Optical Off-Pixel Resonant Demodulation for Very-High-Resolution Time-of-Flight Imaging

• Applicants: Privatdozentin Dr.-Ing. Shanshan Gu-Stoppel; Privatdozent Dr.-Ing. Miguel Heredia Conde
• Subject Area: Communication Technology and Networks, High-Frequency Technology and Photonic Systems, Signal Processing and Machine Learning for Information Technology ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 513531607

3D-imaging systems are of paramount importance in emerging fields, such as autonomous driving or mobile robotics. Lidar systems and Time-of-Flight (ToF) cameras, also known as flash Lidar, are among the best-performing alternatives in real-world scenarios. Lidar makes use of one or several laser beams, which are steered to cover the desired Field of View (FoV). Sequential measurements for different beam directions slow down acquisition, limiting the frame rate. Differently, ToF cameras densely illuminate the entire FoV and make use of an array of time-resolving pixels to acquire measurements concurrently for all scene points. Unfortunately, the use of special pixels with complex architectures limits the resolution of the generated depth images, which is up to two orders of magnitude lower than that of conventional cameras. In this work, we capitalize on the idea that both Lidar and "flash Lidar" are two extreme cases of a more abstract manifold of possible solutions for 3D sensing, which remains largely unexplored. Lidar systems rely on single fast photodetectors, while ToF cameras exploit pixel arrays. Similarly, Lidars exploit motion to time-multiplex the spatial domain, while ToF cameras do not leverage motion-based multiplexing. Differently from both conventional Lidar and ToF imaging, in this project we propose exploiting fast oscillatory motion to shift the light demodulation problem in the ToF configuration from the time domain to the spatial domain. Consequently, ToF pixels no longer need to act as time-domain correlators, as this process is translated to spatial domain by a mechano-optical setup. This allows for substituting the complex time-resolving ToF pixels by conventional pixels, thus opening the door for very-high-resolution (HR) depth imaging, up to 10-100 Mpix. A key enabling factor will be the development of an ultra-fast MEMS mirror, with very short displacement, but very large oscillation frequencies. This requires both analytic modelling and FEM simulations. Operation at resonant frequencies is expected to allow for sufficient displacement at very large frequencies. Fabrication of such a device and development of the associated electronics call for specialized know-how and equipment, which will be provided by Fraunhofer ISIT. Spatially-multiplexing the demodulation process opens a new type of inverse problem to be studied, i.e. retrieving a depth image from the raw data of the camera. This largely differs from the classical ToF imaging problem, since correspondence between pixels and scene points is no longer static, but dynamic. Changing the oscillation frequency of the micromirror, ToF measurements at different frequencies can be obtained. This will be leveraged to resolve multiple return paths per pixel. Additionally, the multi-frequency data can be used for per-pixel material recognition. This will be leveraged to demonstrate the first HR "trimodal camera", delivering intensity, depth, and material images in real time.

• DFG Programme: Research Grants