Alexia Tran1, Louis Lassalle2, Pascal Zille3, Raphaël Guillin4, Etienne Pluot5, Chloé Adam3, Martin Charachon3, Hugues Brat6, Maxence Wallaert3, Gaspard d'Assignies3,7, Benoît Rizk6. 1. Department of Radiology, Hôpital Européen Georges Pompidou, Assistance Publique-Hôpitaux de Paris; Université de Paris, 20 Rue Leblanc, 75015, Paris, France. tranalexia@yahoo.com. 2. Réseau d'imagerie Sud Francilien, Evry, France. 3. Incepto Medical, Paris, France. 4. Department of Radiology, Centre Hospitalier Universitaire de Rennes, Rennes, France. 5. Department of Radiology, Radiologie B, Hôpital Cochin, Assistance Publique-Hôpitaux de Paris; Université de Paris, Paris, France. 6. Institut de Radiologie de Sion, Groupe 3R, Sion, Switzerland. 7. Department of Radiology, Centre Hospitalier Départemental Vendée, La Roche-sur-Yon, France.
Abstract
OBJECTIVES: To develop a deep-learning algorithm for anterior cruciate ligament (ACL) tear detection and to compare its accuracy using two external datasets. METHODS: A database of 19,765 knee MRI scans (17,738 patients) issued from different manufacturers and magnetic fields was used to build a deep learning-based ACL tear detector. Fifteen percent showed partial or complete ACL rupture. Coronal and sagittal fat-suppressed proton density or T2-weighted sequences were used. A Natural Language Processing algorithm was used to automatically label reports associated with each MRI exam. We compared the accuracy of our model on two publicly available external datasets: MRNet, Bien et al, USA (PLoS Med 15:e1002699, 2018); and KneeMRI, Stajduhar et al, Croatia (Comput Methods Prog Biomed 140:151-164, 2017). Receptor operating characteristics (ROC) curves, area under the curve (AUC), sensitivity, specificity, and accuracy were used to evaluate our model. RESULTS: Our neural networks achieved an AUC value of 0.939 for detection of ACL tears, with a sensitivity of 87% (0.875) and a specificity of 91% (0.908). After retraining our model on Bien dataset and Stajduhar dataset, our algorithm achieved AUC of 0.962 (95% CI 0.930-0.988) and 0.922 (95% CI 0.875, 0.962) respectively. Sensitivity, specificity, and accuracy were respectively 85% (95% CI 75-94%, 0.852), 89% (95% CI 82-97%, 0.894), 0.875 (95% CI 0.817-0.933) for Bien dataset, and 68% (95% CI 54-81%, 0.681), 93% (95% CI 89-97%, 0.934), and 0.870 (95% CI 0.821-0.913) for Stajduhar dataset. CONCLUSION: Our algorithm showed high performance in the detection of ACL tears with AUC on two external datasets, demonstrating its generalizability on different manufacturers and populations. This study shows the performance of an algorithm for detecting anterior cruciate ligament tears with an external validation on populations from countries and continents different from the study population. KEY POINTS: • An algorithm for detecting anterior cruciate ligament ruptures was built from a large dataset of nearly 20,000 MRI with AUC values of 0.939, sensitivity of 87%, and specificity of 91%. • This algorithm was tested on two external populations from different other countries: a dataset from an American population and a dataset from a Croatian population. Performance remains high on these two external validation populations (AUC of 0.962 and 0.922 respectively).
OBJECTIVES: To develop a deep-learning algorithm for anterior cruciate ligament (ACL) tear detection and to compare its accuracy using two external datasets. METHODS: A database of 19,765 knee MRI scans (17,738 patients) issued from different manufacturers and magnetic fields was used to build a deep learning-based ACL tear detector. Fifteen percent showed partial or complete ACL rupture. Coronal and sagittal fat-suppressed proton density or T2-weighted sequences were used. A Natural Language Processing algorithm was used to automatically label reports associated with each MRI exam. We compared the accuracy of our model on two publicly available external datasets: MRNet, Bien et al, USA (PLoS Med 15:e1002699, 2018); and KneeMRI, Stajduhar et al, Croatia (Comput Methods Prog Biomed 140:151-164, 2017). Receptor operating characteristics (ROC) curves, area under the curve (AUC), sensitivity, specificity, and accuracy were used to evaluate our model. RESULTS: Our neural networks achieved an AUC value of 0.939 for detection of ACL tears, with a sensitivity of 87% (0.875) and a specificity of 91% (0.908). After retraining our model on Bien dataset and Stajduhar dataset, our algorithm achieved AUC of 0.962 (95% CI 0.930-0.988) and 0.922 (95% CI 0.875, 0.962) respectively. Sensitivity, specificity, and accuracy were respectively 85% (95% CI 75-94%, 0.852), 89% (95% CI 82-97%, 0.894), 0.875 (95% CI 0.817-0.933) for Bien dataset, and 68% (95% CI 54-81%, 0.681), 93% (95% CI 89-97%, 0.934), and 0.870 (95% CI 0.821-0.913) for Stajduhar dataset. CONCLUSION: Our algorithm showed high performance in the detection of ACL tears with AUC on two external datasets, demonstrating its generalizability on different manufacturers and populations. This study shows the performance of an algorithm for detecting anterior cruciate ligament tears with an external validation on populations from countries and continents different from the study population. KEY POINTS: • An algorithm for detecting anterior cruciate ligament ruptures was built from a large dataset of nearly 20,000 MRI with AUC values of 0.939, sensitivity of 87%, and specificity of 91%. • This algorithm was tested on two external populations from different other countries: a dataset from an American population and a dataset from a Croatian population. Performance remains high on these two external validation populations (AUC of 0.962 and 0.922 respectively).
Authors: Christoph Germann; Giuseppe Marbach; Francesco Civardi; Sandro F Fucentese; Jan Fritz; Reto Sutter; Christian W A Pfirrmann; Benjamin Fritz Journal: Invest Radiol Date: 2020-08 Impact factor: 10.065