Oliver Maier1, Steven Hubert Baete2, Alexander Fyrdahl3, Kerstin Hammernik4,5, Seb Harrevelt6, Lars Kasper7,8,9, Agah Karakuzu10, Michael Loecher11, Franz Patzig7, Ye Tian12,13, Ke Wang14, Daniel Gallichan15, Martin Uecker16,17,18,19, Florian Knoll2. 1. Institute of Medical Engineering, Graz University of Technology, Graz, Austria. 2. Center for Biomedical Imaging, New York University School of Medicine, New York, NY, USA. 3. Department of Clinical Physiology, Karolinska University Hospital, and Karolinska Institutet, Stockholm, Sweden. 4. Department of Computing, Imperial College London, London, UK. 5. Institute of Computer Graphics and Vision, Graz University of Technology, Graz, Austria. 6. Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands. 7. Institute for Biomedical Engineering, ETH Zurich and University of Zurich, Zurich, Switzerland. 8. Translational Neuromodeling Unit, Institute for Biomedical Engineering, University of Zurich and ETH Zurich, Zurich, Switzerland. 9. Techna Institute, University Health Network, Toronto, ON, Canada. 10. NeuroPoly Lab, Institute of Biomedical Engineering, Polytechnique Montréal, Montréal, QC, Canada. 11. Department of Radiology, Stanford University, Stanford, CA, USA. 12. Utah Center for Advanced Imaging Research (UCAIR), Department of Radiology and Imaging Sciences, University of Utah, Salt Lake City, UT, USA. 13. Ming Hsieh Department of Electrical and Computer Engineering, Viterbi School of Engineering, University of Southern California, Los Angeles, CA, USA. 14. Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA, USA. 15. Cardiff University Brain Research Imaging Centre, Cardiff, UK. 16. Institute for Diagnostic and Interventional Radiology, University Medical Center Göttingen, Göttingen, Germany. 17. German Centre for Cardiovascular Research (DZHK), Berlin, Germany. 18. Cluster of Excellence "Multiscale Bioimaging: from Molecular Machines to Networks of Excitable Cells" (MBExC), University of Göttingen, Göttingen, Germany. 19. Campus Institute Data Science (CIDAS), University of Göttingen, Göttingen, Germany.
Abstract
PURPOSE: The aim of this work is to shed light on the issue of reproducibility in MR image reconstruction in the context of a challenge. Participants had to recreate the results of "Advances in sensitivity encoding with arbitrary k-space trajectories" by Pruessmann et al. METHODS: The task of the challenge was to reconstruct radially acquired multicoil k-space data (brain/heart) following the method in the original paper, reproducing its key figures. Results were compared to consolidated reference implementations created after the challenge, accounting for the two most common programming languages used in the submissions (Matlab/Python). RESULTS: Visually, differences between submissions were small. Pixel-wise differences originated from image orientation, assumed field-of-view, or resolution. The reference implementations were in good agreement, both visually and in terms of image similarity metrics. DISCUSSION AND CONCLUSION: While the description level of the published algorithm enabled participants to reproduce CG-SENSE in general, details of the implementation varied, for example, density compensation or Tikhonov regularization. Implicit assumptions about the data lead to further differences, emphasizing the importance of sufficient metadata accompanying open datasets. Defining reproducibility quantitatively turned out to be nontrivial for this image reconstruction challenge, in the absence of ground-truth results. Typical similarity measures like NMSE of SSIM were misled by image intensity scaling and outlier pixels. Thus, to facilitate reproducibility, researchers are encouraged to publish code and data alongside the original paper. Future methodological papers on MR image reconstruction might benefit from the consolidated reference implementations of CG-SENSE presented here, as a benchmark for methods comparison.
PURPOSE: The aim of this work is to shed light on the issue of reproducibility in MR image reconstruction in the context of a challenge. Participants had to recreate the results of "Advances in sensitivity encoding with arbitrary k-space trajectories" by Pruessmann et al. METHODS: The task of the challenge was to reconstruct radially acquired multicoil k-space data (brain/heart) following the method in the original paper, reproducing its key figures. Results were compared to consolidated reference implementations created after the challenge, accounting for the two most common programming languages used in the submissions (Matlab/Python). RESULTS: Visually, differences between submissions were small. Pixel-wise differences originated from image orientation, assumed field-of-view, or resolution. The reference implementations were in good agreement, both visually and in terms of image similarity metrics. DISCUSSION AND CONCLUSION: While the description level of the published algorithm enabled participants to reproduce CG-SENSE in general, details of the implementation varied, for example, density compensation or Tikhonov regularization. Implicit assumptions about the data lead to further differences, emphasizing the importance of sufficient metadata accompanying open datasets. Defining reproducibility quantitatively turned out to be nontrivial for this image reconstruction challenge, in the absence of ground-truth results. Typical similarity measures like NMSE of SSIM were misled by image intensity scaling and outlier pixels. Thus, to facilitate reproducibility, researchers are encouraged to publish code and data alongside the original paper. Future methodological papers on MR image reconstruction might benefit from the consolidated reference implementations of CG-SENSE presented here, as a benchmark for methods comparison.
Authors: Agah Karakuzu; Stefan Appelhoff; Tibor Auer; Mathieu Boudreau; Franklin Feingold; Ali R Khan; Alberto Lazari; Chris Markiewicz; Martijn Mulder; Christophe Phillips; Taylor Salo; Nikola Stikov; Kirstie Whitaker; Gilles de Hollander Journal: Sci Data Date: 2022-08-24 Impact factor: 8.501
Authors: Nikou L Damestani; Owen O'Daly; Ana Beatriz Solana; Florian Wiesinger; David J Lythgoe; Simon Hill; Alfonso de Lara Rubio; Elena Makovac; Steven C R Williams; Fernando Zelaya Journal: Hum Brain Mapp Date: 2021-03-17 Impact factor: 5.399