End to End Deep Learning Image Reconstruction and Pathology Detection

• Applicants: Professor Dr. Bernhard Kainz; Professor Dr. Florian Knoll
• Subject Area: Medical Informatics and Medical Bioinformatics ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing ; Medical Physics, Biomedical Technology ; Radiology
• Term: since 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 513220538

Medical imaging, especially magnetic Resonance Imaging (MRI), is essential for modern healthcare, providing tools for noninvasive diagnostics, guidance, and unique monitoring options for treatment and understanding of disease. The majority of diagnostic medical imaging pipelines follow the same principles: raw measurement data is acquired by scanner hardware, processed by image reconstruction algorithms, and then evaluated for pathology by human radiology experts. Under this paradigm, every step has traditionally been optimized to generate images that are visually pleasing and easy to interpret for human experts. However, raw sensor information that could maximize patient-specific diagnostic information may get lost in this process. This problem is amplified by recent developments in MRI reconstruction with machine learning (ML). ML methods allow to generate visually pleasing images from very limited sensor data. Reduced measurement data is often desired for accelerated imaging protocols, providing more patient comfort through reducing scan times from hours to minutes. Though, the less data is acquired, the more likely it is that ML image reconstruction methods will hide signs of disease, replacing pathology-defining features with more likely healthy image features from the training data. ML has also been tremendously successful for analyzing medical images and to detect patterns of disease robustly across modalities. So far, these tools have been disjointed from the image acquisition process and are usually only applied to reconstructed image data. In this project we will seize the opportunity to fuse ML for image reconstruction and ML for image-based disease localization, thus providing an end-to-end learnable image reconstruction and joint pathology detection approach that operates directly on raw measurement data. We expect that this combination can maximize diagnostic accuracy while providing optimal images for both human experts and diagnostic ML models. To achieve this, our project includes three goals and work packages. (1) we will develop a ML reconstruction approach for accelerated MRI that is robust towards different scanners, sequences, and anatomy. (2) we will develop disease detection methods that can learn from human-generated annotations in image space and at the same time extract information directly from accelerated raw measurement data. (3) we will bring these developments together and establish end-to-end training of image reconstruction and image analysis. Our hypothesis is that such an approach will prevent ML-based MRI interpretation from missing important signs of disease and provide tools to communicate interpretation uncertainties in image regions with limited sensor data. We will use our fastMRI+ dataset for our developments. The dataset contains fully sampled raw measurement data, images and 20.000 specialist expert bounding box annotations for more than 50 pathology categories from more than 8.000 patients.

• DFG Programme: Research Grants