TRR 125:  Cognition-Guided Surgery

• Subject Area: Medicine ; Computer Science, Systems and Electrical Engineering ; Mathematics
• Term: from 2012 to 2017
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 211469636

• Final Report Year: 2017

Final Report Abstract

The brightest minds of our generation are working on digitizing and integrating all areas of life to find smart solutions for the challenges of the 21st century. While economy and society have since experienced groundbreaking innovations, minimal progress has been made in the field of surgery. Due to the technological leeway, the success of surgical treatment remains highly dependent on the knowledge and experience levels of surgical teams. The vision of the Transregional Collaborative Research Center 125 “Cognition-Guided Surgery” was therefore to develop a technical-cognitive system that behaves like a humanoid assistant, supports the surgeon during the course of the operation and learns from experience. For this purpose all information along the surgical treatment path should be gathered (perception) to derive the current state of treatment plans, tissue characteristics or the condition of patients (interpretation) and to generate an assistant function for the surgeon (action). After completion, each action was to be evaluated and stored in the knowledge base so that it could be reused for future operations (learning). Over the course of the funding period, basic methods for the concept of “Cognition-Guided Surgery” were explored. Innovative concepts for quantification of preoperative perfusion parameters (radiological imaging) as well as intraoperative geometric patient parameters (mitral valve size, small bowel length) for objective diagnosis and therapy of surgical diseases were investigated. For the first time it was possible to collect and incorporate different forms of surgical information. This was achieved by means of a common knowledge base, by formalizing surgical factual knowledge and by developing new methods for data annotation (crowd sourcing, sparse annotations). With regard to interpretation, it was possible for the first time to fuse stand-alone solutions into a knowledge-based main framework for surgical applications. This was achieved by means of semantic modeling of patients and processes (patient factor model, OntoSPM) and by integrating image-processing algorithms and simulation processes. Based on these methods, clinical applications could be developed that performed cognitive actions. Applications that were internationally awarded were an assistance system for treatment planning in liver surgery that incorporates heterogenous information, knowledge-based selection of different mitral valve prosthesis in cardiothoracic surgery and a learning camera robot in laparoscopic colorectal surgery. The SFB/Transregio 125 investigated cognition-guided surgery in terms of universal, technical, cognitive assistance systems. Through development of prototypes for cognitive systems, surgical decision making could be facilitated in the future by receiving individual, precise, context aware treatment suggestions that result in safer and more effective treatment for patients.

Publications

• (2012). Dense GPU-enhanced surface reconstruction from stereo endoscopic images for intraoperative registration. Med Phys 39, 1632–1645
  Röhl S, Bodenstedt S, Suwelack S, Kenngott H, Müller-Stich BP, Dillmann R and Speidel S
  (See online at https://doi.org/10.1118/1.3681017)
• (2013) Development and validation of automatic tools for interactive recurrence analysis in radiation therapy: Optimization of treatment algorithms for locally advanced pancreatic cancer, Radiation Oncology, 08/2013; 8(1)
  Kessel KA, Habermehl D, Jäger A, Floca RO, Zhang L, Bendl R, Debus J, and Combs SE
  (See online at https://doi.org/10.1186/1748-717x-8-138)
• (2013). "Fibrosis and Pancreatic Lesions Counterintuitive Behavior of the Diffusion Imaging-Derived Structural Diffusion Coefficient D." Invest Radiol.48 (3):129-33
  Klauss M, Gaida MM, Lemke A, Grunberg K, Simon D, Wente MN, Delorme S, Kauczor HU, Grenacher L, and Stieltjes B
  (See online at https://doi.org/10.1097/rli.0b013e31827ac0f1)
• (2013). Real-time image guidance in laparoscopic liver surgery: first clinical experience with a guidance system based on intraoperative CT imaging. Surg Endosc 28, 933–940
  Kenngott HG, Wagner M, Gondan M., Nickel F, Nolden M, Fetzer A, Weitz J, Fischer L, Speidel S, Meinzer H-P, Böckler D, Büchler MW and Müller-Stich, BP
  (See online at https://doi.org/10.1007/s00464-013-3249-0)
• (2013). The Medical Imaging Interaction Toolkit: Challenges and Advances. Int J CARS, 8(4):607-620
  Nolden M, Zelzer S, Seitel A, Wald D, Müller M, Franz AM, Maleike D, Fangerau M, Baumhauer M, Maier-Hein L, Maier-Hein KH, Meinzer HP, Wolf I
  (See online at https://doi.org/10.1007/s11548-013-0840-8)
• 2013. “Improving Accuracy of Markerless Tracking of Lung Tumours in Fluoroscopic Video by Incorporating Diaphragm Motion.” In Abstract Book of the International Conference on the Use of Computers in Radiation Therapy, 100
  Schwarz M, Teske H, Stoll M, Bendl R
  (See online at https://doi.org/10.1088/1742-6596/489/1/012082)
• (2014). “3D Morphometry Using Automated Aortic Segmentation in Native MR Angiography: An Alternative to Contrast Enhanced MRA?” Cardiovascular Diagnosis and Therapy 4(2): 80–87
  Müller-Eschner M, Müller T, Biesdorf A, Wörz S, Rengier F, Böckler D, Kauczor H-U, Rohr K, and von Tengg-Kobligk H
  (See online at https://doi.org/10.3978/j.issn.2223-3652.2013.10.06)
• (2014). “3D Regression Voting on CT- Volumes of the Human Liver for SSM Surface Appearance Modeling.” Proceedings of Shape – Symposium on Statistical Shape Models and Applications
  NorajitraT, Meinzer HP, Maier-Hein KH
  
• (2014). “Augmented Reality-Enhanced Endoscopic Images for Annuloplasty Ring Sizing.” In Augmented Environments for Computer-Assisted Interventions, edited by C. A. Linte, Z. Yaniv, P. Fallavollita, P. Abolmaesumi, and D. R. H. III, 128–37. Lecture Notes in Computer Science 8678. Springer
  Engelhardt S, Simone RD, Zimmermann N, Al-Maisary S, Nabers D, Karck M, Meinzer H-P, and Wolf I
  (See online at https://doi.org/10.1007/978-3-319-10437-9_14)
• (2014). “State-of-the-Art Aortic Imaging: Part II - Applications in Transcatheter Aortic Valve Replacement and Endovascular Aortic Aneurysm Repair.” Vasa. 43(1): 6–26
  Rengier F, Geisbüsch P, Schoenhagen P, Müller-Eschner M, Vosshenrich R, Karmonik C, von Tengg-Kobligk H, and Partovi S
  (See online at https://doi.org/10.1024/0301-1526/a000324)
• (2014). „Real-time image guidance in laparoscopic liver surgery: first clinical experience with a guidance system based on intraoperative CT imaging.” Surg Endosc 28(3):933-40
  Kenngott HG, Wagner M, Gondan M, Nickel F, Nolden M, Fetzer A, Weitz J, Fischer L, Speidel S
  (See online at https://doi.org/10.1007/s00464-013-3249-0)
• „Can masses of non-experts train highly accurate image classifiers? A crowdsourcing approach to instrument segmentation in laparoscopic images.“ In Med Image Comput Comput Assist Interv. – MICCAI 2014, 17(Pt 2):438-45
  Maier-Hein L, Mersmann S, Kondermann D, Bodenstedt S, Sanchez A, Stock C, Kenngott HG, Eisenmann M, Speidel S
  (See online at https://doi.org/10.1007/978-3-319-10470-6_55)
• (2015) "Correlation of Histological Vessel Characteristics and Diffusion- Weighted Imaging Intravoxel Incoherent Motion-Derived Parameters in Pancreatic Ductal Adenocarcinomas and Pancreatic Neuroendocrine Tumors." Invest Radiol. 50(11):792-7
  Klauss M, Mayer P, Bergmann F, Maier-Hein KH, Hase J, Hackert T, Kauczor HU, Grenacher L, and Stieltjes B
  (See online at https://doi.org/10.1097/rli.0000000000000187)
• (2015) Evaluation of inter- and intrafractional motion of liver tumors using interstitial markers and implantable electromagnetic radiotransmitters in the context of image-guided radiotherapy (IGRT) - the ESMERALDA trial. Radiat Oncol. 10: 143
  Habermehl D, Naumann P, Bendl R, Oelfke U, Nill S, Debus J, Combs SE
  (See online at https://doi.org/10.1186/s13014-015-0456-y)
• (2015). "A Newton-Galerkin Method for Fluid Flow Exhibiting Uncertain Periodic Dynamics". Journal on Uncertainty Quantification, 2(1):153–173
  Schick M, Le Maitre OP, and Heuveline V
  (See online at https://doi.org/10.1137/130908919)
• (2015). “3D Statistical Shape Models incorporating 3D Random Forest Regression Voting for Robust CT Liver Segmentation.” Proceedings of SPIE Medical Imaging, 2015
  Norajitra T, Meinzer HP, Maier-Hein KH
  (See online at https://doi.org/10.1117/12.2082909)
• (2015). “Correlation of Quantitative Dual-energy CT Iodine Maps and Abdominal CT-perfusion Measurements: Are Single-acquisition DECT Iodine Maps More Than a Reduced-Dose Surrogate of Conventional CT Perfusion?” Invest Radiol. 50 (10): 703-708
  Stiller W, Skornitzke S, Fritz F, Klauß M, Hansen J, Pahn G, Grenacher L, and Kauczor H.-U.
  (See online at https://doi.org/10.1097/rli.0000000000000176)
• (2015). “Learning Surgical Know-How: Dexterity for a Cognitive Endoscope Robot”. In: 7th IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and the 7th IEEE International Conference on Robotics, Automation and Mechatronics (RAM). 137-142
  Bihlmaier A and Wörn H
  (See online at https://doi.org/10.1109/ICCIS.2015.7274610)
• (2015). “Qualitative and quantitative evaluation of rigid and deformable motion correction algorithms using dual-energy CT images in view of application to CT perfusion measurements in abdominal organs affected by breathing motion.” Br J Radiol. 88(1046):20140683
  SkornitzkeS, Fritz F, Klauß M, Pahn G, Hansen J, Hirsch J, Grenacher L, Kauczor H-U, and Stiller W
  (See online at https://doi.org/10.1259/bjr.20140683)
• (2015). „Knowledge-based workspace optimization of a redundant robot for minimally invasive robotic surgery (MIRS).“ In: IEEE International Conference on Robotics and Biomimetics (ROBIO). 1403-1408
  Hutzl J, Bihlmaier A, Wagner M, Kenngott HG, Müller BP and Wörn H
  (See online at https://doi.org/10.1109/ROBIO.2015.7418967)
• (2015).„Gender-specific differences in outcome of ascending aortic aneurysm surgery.“ PLoS One. 10(4): e0124461
  Beller CJ, Farag M, Wannaku S, Seppelt P, Arif R, Ruhparwar A, Karck M, Weymann A, and Kallenbach K
  (See online at https://doi.org/10.1371/journal.pone.0124461)
• 2015. “Real-Time Markerless Lung Tumor Tracking in Fluoroscopic Video: Handling Overlapping of Projected Structures.” Medical Physics 42 (5): 2540–49
  Teske H, Mercea P, Schwarz M, Nicolay NH, Sterzing F, Bendl R
  (See online at https://doi.org/10.1118/1.4917480)
• „Toward Knowledge-Based Liver Surgery: Holistic Information Processing for Surgical Decision Support“.Int J Comput Assist Radiol Surg. (Special Issue: IPCAI), 7. April 2015, 1–11
  März K, Hafezi M, Weller T, Saffari A, Nolden M, Fard N, Majlesara A, Zelzer S, Maleshkova M, Volovyk M, Gharabaghi N, Wagner M, Emami G, Engelhardt S, Fetzer A, Kenngott H, Rezai N, Rettinger A, Studer R, Mehrabi A, Maier-Hein L
  (See online at https://doi.org/10.1007/s11548-015-1187-0)
• "Toward Cognitive Pipelines of Medical Assistance Algorithms.", Int J Comput Assist Radiol Surg., Vol. 11, No. 9, pp. 1743 - 1753
  Philipp P, Maleshkova M, Katic D, Weber C, Götz M, Rettinger A, Speidel S, Kämpgen B, Nolden M, Wekerle AL, Dillmann R, H. Kenngott, B. Müller, and R. Studer
  (See online at https://doi.org/10.1007/s11548-015-1322-y)
• (2016) “Dual-energy perfusion-CT in recurrent pancreatic cancer – preliminary results / [Dual-energy Perfusions-CT bei Pankreaskarzinomrezidiven – Vorläufige Ergebnisse].” Fortschritte auf dem Gebiet der Röntgenstrahlen und der bildgebenden Verfahren, 188(6): 559-565
  FritzF, Skornitzke S, Hackert T, Kauczor HU, Stiller W, Grenacher L, and Klauss M
  (See online at https://doi.org/10.1055/s-0042-105765)
• (2016). "Bridging the gap between formal and experience-based knowledge for context-aware laparoscopy", Int J Comput Assist Radiol Surg., 11(6): 881-888
  Katić D, Schuck J, Wekerle A, Kenngott H, Müller-Stich B, Dillmann R, Speidel S
  (See online at https://doi.org/10.1007/s11548-016-1379-2)
• (2016). "Bringing Data-Driven Process Analysis into Surgical Practice - the Surgical Process Analyzer", Proceedings Surgical Data Science, Springer, Heidelberg, Juni 2016
  Wagner M, Weller T, Ternes L-M, Rempel R, Maleshkova M, Sure-Vetter Y, Kenngott H
  
• (2016). "Comprehensive patient-specific information preprocessing for cardiac surgery simulations". Int J Comput Assist Radiol Surg., Special Issue: IPCAI2016
  Schoch N, Kißler F, Stoll M, Engelhardt S, de Simone R, Wolf I, Bendl R, Heuveline V
  (See online at https://doi.org/10.1007/s11548-016-1397-0)
• (2016). “3D Statistical Shape Models incorporating Landmark-wise Random Regression Forests for Omni-directional Landmark Detection.” IEEE Trans Med Imaging. 36(1):155-168
  Norajitra T, Maier-Hein KH
  (See online at https://doi.org/10.1109/TMI.2016.2600502)
• (2016). “A Learning-Based, Fully Automatic Liver Tumor Segmentation Pipeline Based on Sparsely Annotated Training Data.” In SPIE Medical Imaging
  Goetz M, Heim E, Maerz K, Norajitra T, Hafezi M, Fard N, Mehrabi A, Knoll M, Weber C, Maier-Hein L, Maier-Hein KH
  (See online at https://doi.org/10.1117/12.2217655)
• (2016). “Accuracy Evaluation of a Mitral Valve Surgery Assistance System Based on Optical Tracking.” Int J Comput Assist Radiol Surg. 11 (10): 1891-904
  Engelhardt S, De Simone R, Al Maisary S, Kolb S, Karck M, Meinzer H-P, and Wolf I
  (See online at https://doi.org/10.1007/s11548-016-1353-z)
• (2016). “DALSA: Domain Adaptation for Supervised Learning From Sparsely Annotated MR Images.” IEEE Trans Med Imaging, 35(1)
  Goetz M, Weber C, Binczyk F, Polanska J, Tarnawski R, Bobek-Billewicz B, Koethe U, Kleesiek J, Stieltjes B, and Maier-Hein KH
  (See online at https://doi.org/10.1109/tmi.2015.2463078)
• (2016). “Intraoperative Quantitative Mitral Valve Analysis Using Optical Tracking Technology.” Ann Thorac Surg.101 (5):1950-6
  Engelhardt S, Wolf I, Al Maisary S, Meinzer H-P, Karck M, and De Simone R
  (See online at https://doi.org/10.1016/j.athoracsur.2016.01.018)
• (2016). “IVIM DW-MRI of autoimmune pancreatitis: therapy monitoring and differentiation from pancreatic cancer.” Eur Radiol. 26(7):2099-106
  Klauss M, Maier-Hein K, Tjaden C, Hackert T, Grenacher L, Stieltjes B
  (See online at https://doi.org/10.1007/s00330-015-4041-4)
• (2016). “ROS-based Cognitive Surgical Robotics”. In: Robot Operating System (ROS) - The Complete Reference. Springer. 317-342
  Bihlmaier A, Beyl T, Nicolai P, Kunze M, Mintenbeck J, Schreiter L, Brennecke T, Hutzl J, Raczkowsky J and Wörn H
  (See online at https://doi.org/10.1007/978-3-319-26054-9_12)
• (2016). „Machine-learning based comparison of CT-perfusion maps and dual energy CT for pancreatic tumor detection“. In Proceedings of SPIE Medical Imaging
  Goetz M, Skornitzke S, Weber C, Fritz F, Mayer P, Koell M, Stiller W, und Maier-Hein K
  (See online at https://doi.org/10.1117/12.2216645)
• (2016). „Towards an open-source semantic data infrastructure for integrating clinical and scientific data in cognition-guided surgery.” In: SPIE Medical Imaging
  Fetzer A, Metzger J, Katic D, März K, Wagner M, Philipp P, Engelhardt S, Weller T, Zelzer S, Franz AM, Maleshkova M, Rettinger A, Speidel S, Wolf I, Kenngott H, Müller B, Maier-Hein L, Meinzer HP, Nolden M
  (See online at https://doi.org/10.1117/12.2217163)
• 2016. Worst case optimization for interfractional motion mitigation in carbon ion therapy of pancreatic cancer. Radiation Oncology, 11(1), p.134
  Steitz, J., Naumann, P., Ulrich, S., Haefner, M.F., Sterzing, F., Oelfke, U. and Bangert, M.
  (See online at https://doi.org/10.1186/s13014-016-0705-8)
• (2017) Comparison of Safety Margin Generation Concepts in Image Guided Radiotherapy to account for Daily Head and Neck Pose Variations. PLoS One. 2016 Dec 29;11(12):e0168916
  Stoll M, Stoiber E, Grimm S, Debus J, Bendl R, Giske K
  (See online at https://doi.org/10.1371/journal.pone.0168916)
• 2017). "Towards Cognition-Guided Patient-Specific FEM- based Cardiac Surgery Simulation". Proceedings of the "Functional Imaging and Modeling of the Heart (FIMH) 2017", Springer's LNCS Lecture Notes in Computer Science
  Schoch N, Heuveline V.
  (See online at https://doi.org/10.1007/978-3-319-59448-4_12)