AIMS: Early diagnosis of developmental dysplasia of the hip (DDH) using ultrasound (US) is safe, effective and inexpensive, but requires high-quality scans. The effect of scan quality on diagnostic accuracy is not well understood, especially as artificial intelligence (AI) begins to automate such diagnosis. In this paper, we developed a 10-point scoring system for reporting DDH US scan quality, evaluated its inter-rater agreement and examined its effect on automated assessment by an AI system-MEDO-Hip. METHODS: Scoring was based on iliac wing straightness and angulation; visibility of labrum, os ischium and femoral head; motion; and other artifacts. Four readers from novice to expert separately scored the quality of 107 scans with this 10-point scale and with holistic grading on a scale of 1-5. MEDO-Hip interpreted the same scans, providing a diagnostic category or identifying the scan as uninterpretable. RESULTS: Inter-rater agreement for the 10-point scale was significantly higher than holistic scoring ICC 0.68 vs 0.93, p < 0.05. Inter-rater agreement on the categorisation of individual features, by Cohen's kappa, was highest for os ischium (0.67 ± 0.06), femoral head (0.65 ± 0.07) and iliac wing (0.49 ± 0.12) indices, and lower for the presence of labrum (0.21 ± 0.19). MEDO-Hip interpreted all images of a quality > 7 and flagged 13/107 as uninterpretable. These were low-quality images (3 ± 1.2 vs. 7 ± 1.8 in others, p < 0.05), with poor visualization of the os ischium and noticeable motion. AI accuracy in cases with quality scores < = 7 was 57% vs. 89% on other cases, p < 0.01. CONCLUSION: This study validates that our scoring system reliably characterises scan quality, and identifies cases likely to be misinterpreted by AI. This could lead to more accurate use of AI in DDH diagnosis by flagging low-quality scans likely to provide poor diagnosis up front.
AIMS: Early diagnosis of developmental dysplasia of the hip (DDH) using ultrasound (US) is safe, effective and inexpensive, but requires high-quality scans. The effect of scan quality on diagnostic accuracy is not well understood, especially as artificial intelligence (AI) begins to automate such diagnosis. In this paper, we developed a 10-point scoring system for reporting DDH US scan quality, evaluated its inter-rater agreement and examined its effect on automated assessment by an AI system-MEDO-Hip. METHODS: Scoring was based on iliac wing straightness and angulation; visibility of labrum, os ischium and femoral head; motion; and other artifacts. Four readers from novice to expert separately scored the quality of 107 scans with this 10-point scale and with holistic grading on a scale of 1-5. MEDO-Hip interpreted the same scans, providing a diagnostic category or identifying the scan as uninterpretable. RESULTS: Inter-rater agreement for the 10-point scale was significantly higher than holistic scoring ICC 0.68 vs 0.93, p < 0.05. Inter-rater agreement on the categorisation of individual features, by Cohen's kappa, was highest for os ischium (0.67 ± 0.06), femoral head (0.65 ± 0.07) and iliac wing (0.49 ± 0.12) indices, and lower for the presence of labrum (0.21 ± 0.19). MEDO-Hip interpreted all images of a quality > 7 and flagged 13/107 as uninterpretable. These were low-quality images (3 ± 1.2 vs. 7 ± 1.8 in others, p < 0.05), with poor visualization of the os ischium and noticeable motion. AI accuracy in cases with quality scores < = 7 was 57% vs. 89% on other cases, p < 0.01. CONCLUSION: This study validates that our scoring system reliably characterises scan quality, and identifies cases likely to be misinterpreted by AI. This could lead to more accurate use of AI in DDH diagnosis by flagging low-quality scans likely to provide poor diagnosis up front.
Authors: Jacob L Jaremko; Myles Mabee; Vimarsha G Swami; Lucy Jamieson; Kelvin Chow; Richard B Thompson Journal: Radiology Date: 2014-06-25 Impact factor: 11.105