Ziba Gandomkar1, Kevin Tay2, Patrick C Brennan1, Emma Kozuch3, Claudia Mello-Thoms1,4. 1. Discipline of Medical Imaging and Radiation Sciences, Image Optimisation and Perception Group (MIOPeG), The University of Sydney, Sydney, NSW, Australia. 2. Medical Imaging Department, Prince of Wales Hospital, Randwick, NSW, Australia. 3. University of Notre Dame, Notre Dame, Indiana, 46556, USA. 4. Department of Radiology, The University of Iowa, Iowa City, IA, 52242, USA.
Abstract
PURPOSE: To propose a framework for optimal pairing of radiologists when reading mammograms based on their search patterns. MATERIALS AND METHODS: Four experienced and four less-experienced radiologists were asked to assess 120 cases (59 with cancers) while their eye positions were tracked. Fourteen eye-tracking metrics were extracted to quantify the differences among radiologists' visual search pattern. For each radiologist and metric, less-experienced radiologists and expert readers were ranked based on the level of similarities in gaze patterns (from the most different to the most similar). Less-experienced readers and experts were also ranked based on the values of area under the receiver operating characteristic curve (AUC) after pairing (the best possible way of ranking). Using the Kendall's tau distance, rankings based on different metrics were compared with the best possible ranking. Using paired Wilcoxon signed-rank test, the AUC values when pairing in the best way were compared with pairing based on different metrics. Finally, we investigated the robustness of pairing strategies against the small sample size. RESULTS: For ranking the experienced radiologists, results from eight metrics were as good as the best possible ranking. For the less-experienced ones, only one metric resulted in a ranking comparable to the best possible way of ranking. The AUC values of pairings based on these metrics did not differ significantly from the best pairing scenario. Compared to the pairings based on the cognitive metrics, the ranking based on AUC values varied more greatly with the sample size, suggesting that it is less robust against the small sample size compared to the cognitive metrics. CONCLUSION: Different pairings may have different effects on performance; some are detrimental while some improve the performance of the pair. Using the suggested cognitive metrics, we can optimize the pairings even with a small dataset.
PURPOSE: To propose a framework for optimal pairing of radiologists when reading mammograms based on their search patterns. MATERIALS AND METHODS: Four experienced and four less-experienced radiologists were asked to assess 120 cases (59 with cancers) while their eye positions were tracked. Fourteen eye-tracking metrics were extracted to quantify the differences among radiologists' visual search pattern. For each radiologist and metric, less-experienced radiologists and expert readers were ranked based on the level of similarities in gaze patterns (from the most different to the most similar). Less-experienced readers and experts were also ranked based on the values of area under the receiver operating characteristic curve (AUC) after pairing (the best possible way of ranking). Using the Kendall's tau distance, rankings based on different metrics were compared with the best possible ranking. Using paired Wilcoxon signed-rank test, the AUC values when pairing in the best way were compared with pairing based on different metrics. Finally, we investigated the robustness of pairing strategies against the small sample size. RESULTS: For ranking the experienced radiologists, results from eight metrics were as good as the best possible ranking. For the less-experienced ones, only one metric resulted in a ranking comparable to the best possible way of ranking. The AUC values of pairings based on these metrics did not differ significantly from the best pairing scenario. Compared to the pairings based on the cognitive metrics, the ranking based on AUC values varied more greatly with the sample size, suggesting that it is less robust against the small sample size compared to the cognitive metrics. CONCLUSION: Different pairings may have different effects on performance; some are detrimental while some improve the performance of the pair. Using the suggested cognitive metrics, we can optimize the pairings even with a small dataset.
Authors: Abdulla Al Suman; Carlo Russo; Ann Carrigan; Patrick Nalepka; Benoit Liquet-Weiland; Robert Ahadizad Newport; Poonam Kumari; Antonio Di Ieva Journal: PLoS One Date: 2021-12-02 Impact factor: 3.240