Tobias Ross1, David Zimmerer2, Anant Vemuri3, Fabian Isensee2, Manuel Wiesenfarth4, Sebastian Bodenstedt5, Fabian Both6, Philip Kessler6, Martin Wagner7, Beat Müller7, Hannes Kenngott7, Stefanie Speidel5, Annette Kopp-Schneider4, Klaus Maier-Hein2, Lena Maier-Hein3. 1. Computer Assisted Medical Interventions, German Cancer Research Center, Im Neuenheimer Feld 581, 69210, Heidelberg, Germany. t.ross@dkfz-heidelberg.de. 2. Medical Image Computing, German Cancer Research Center, Im Neuenheimer Feld 581, 69210, Heidelberg, Germany. 3. Computer Assisted Medical Interventions, German Cancer Research Center, Im Neuenheimer Feld 581, 69210, Heidelberg, Germany. 4. Division of Biostatistics, German Cancer Research Center, Im Neuenheimer Feld 581, 69210, Heidelberg, Germany. 5. Translational Surgical Oncology, National Center for Tumor Diseases (NCT), Fetscherstrasse 74, 01307, Dresden, Germany. 6. understand.ai, Hirschstr. 71, 76133, Karlsruhe, Germany. 7. Department of General, Visceral and Transplant Surgery, University of Heidelberg, Im Neuenheimer Feld 110, 69210, Heidelberg, Germany.
Abstract
PURPOSE: Surgical data science is a new research field that aims to observe all aspects of the patient treatment process in order to provide the right assistance at the right time. Due to the breakthrough successes of deep learning-based solutions for automatic image annotation, the availability of reference annotations for algorithm training is becoming a major bottleneck in the field. The purpose of this paper was to investigate the concept of self-supervised learning to address this issue. METHODS: Our approach is guided by the hypothesis that unlabeled video data can be used to learn a representation of the target domain that boosts the performance of state-of-the-art machine learning algorithms when used for pre-training. Core of the method is an auxiliary task based on raw endoscopic video data of the target domain that is used to initialize the convolutional neural network (CNN) for the target task. In this paper, we propose the re-colorization of medical images with a conditional generative adversarial network (cGAN)-based architecture as auxiliary task. A variant of the method involves a second pre-training step based on labeled data for the target task from a related domain. We validate both variants using medical instrument segmentation as target task. RESULTS: The proposed approach can be used to radically reduce the manual annotation effort involved in training CNNs. Compared to the baseline approach of generating annotated data from scratch, our method decreases exploratively the number of labeled images by up to 75% without sacrificing performance. Our method also outperforms alternative methods for CNN pre-training, such as pre-training on publicly available non-medical (COCO) or medical data (MICCAI EndoVis2017 challenge) using the target task (in this instance: segmentation). CONCLUSION: As it makes efficient use of available (non-)public and (un-)labeled data, the approach has the potential to become a valuable tool for CNN (pre-)training.
PURPOSE: Surgical data science is a new research field that aims to observe all aspects of the patient treatment process in order to provide the right assistance at the right time. Due to the breakthrough successes of deep learning-based solutions for automatic image annotation, the availability of reference annotations for algorithm training is becoming a major bottleneck in the field. The purpose of this paper was to investigate the concept of self-supervised learning to address this issue. METHODS: Our approach is guided by the hypothesis that unlabeled video data can be used to learn a representation of the target domain that boosts the performance of state-of-the-art machine learning algorithms when used for pre-training. Core of the method is an auxiliary task based on raw endoscopic video data of the target domain that is used to initialize the convolutional neural network (CNN) for the target task. In this paper, we propose the re-colorization of medical images with a conditional generative adversarial network (cGAN)-based architecture as auxiliary task. A variant of the method involves a second pre-training step based on labeled data for the target task from a related domain. We validate both variants using medical instrument segmentation as target task. RESULTS: The proposed approach can be used to radically reduce the manual annotation effort involved in training CNNs. Compared to the baseline approach of generating annotated data from scratch, our method decreases exploratively the number of labeled images by up to 75% without sacrificing performance. Our method also outperforms alternative methods for CNN pre-training, such as pre-training on publicly available non-medical (COCO) or medical data (MICCAI EndoVis2017 challenge) using the target task (in this instance: segmentation). CONCLUSION: As it makes efficient use of available (non-)public and (un-)labeled data, the approach has the potential to become a valuable tool for CNN (pre-)training.
Authors: Lena Maier-Hein; Sven Mersmann; Daniel Kondermann; Sebastian Bodenstedt; Alexandro Sanchez; Christian Stock; Hannes Gotz Kenngott; Mathias Eisenmann; Stefanie Speidel Journal: Med Image Comput Comput Assist Interv Date: 2014
Authors: Andru P Twinanda; Sherif Shehata; Didier Mutter; Jacques Marescaux; Michel de Mathelin; Nicolas Padoy Journal: IEEE Trans Med Imaging Date: 2016-07-22 Impact factor: 10.048
Authors: Luis C Garcia-Peraza-Herrera; Lucas Fidon; Claudia D'Ettorre; Danail Stoyanov; Tom Vercauteren; Sebastien Ourselin Journal: IEEE Trans Med Imaging Date: 2021-04-30 Impact factor: 10.048
Authors: Lena Maier-Hein; Matthias Eisenmann; Duygu Sarikaya; Keno März; Toby Collins; Anand Malpani; Johannes Fallert; Hubertus Feussner; Stamatia Giannarou; Pietro Mascagni; Hirenkumar Nakawala; Adrian Park; Carla Pugh; Danail Stoyanov; Swaroop S Vedula; Kevin Cleary; Gabor Fichtinger; Germain Forestier; Bernard Gibaud; Teodor Grantcharov; Makoto Hashizume; Doreen Heckmann-Nötzel; Hannes G Kenngott; Ron Kikinis; Lars Mündermann; Nassir Navab; Sinan Onogur; Tobias Roß; Raphael Sznitman; Russell H Taylor; Minu D Tizabi; Martin Wagner; Gregory D Hager; Thomas Neumuth; Nicolas Padoy; Justin Collins; Ines Gockel; Jan Goedeke; Daniel A Hashimoto; Luc Joyeux; Kyle Lam; Daniel R Leff; Amin Madani; Hani J Marcus; Ozanan Meireles; Alexander Seitel; Dogu Teber; Frank Ückert; Beat P Müller-Stich; Pierre Jannin; Stefanie Speidel Journal: Med Image Anal Date: 2021-11-18 Impact factor: 13.828
Authors: Fatemeh Haghighi; Mohammad Reza Hosseinzadeh Taher; Zongwei Zhou; Michael B Gotway; Jianming Liang Journal: IEEE Trans Med Imaging Date: 2021-09-30 Impact factor: 11.037