Frank W Samuelson1. 1. U.S. Food and Drug Administration, 10903 New Hampshire Ave, Building 62, Room 3102, Silver Spring, MD, 20993-0002, USA. frank.samuelson@fda.hhs.gov
Abstract
RATIONALE AND OBJECTIVES: Before using a new diagnostic imaging device regularly in a clinic, it should be studied using patients and radiologists. Often such studies report diagnostic performance in terms of sensitivity, specificity, area under the receiver operating characteristic curve (AUC), or differences thereof. In this report we look at how these studies differ from actual future clinical practice and how those differences may affect reported performance measures. MATERIALS AND METHODS: We review signal detection (receiver operating characteristic) theory and decision theory. We compare diagnostic measures from several published studies in medical imaging and examine how they relate to theory and each other. RESULTS: We see that clinical decisions can be modeled using signal detection and decision theories. Sensitivity and specificity are inextricably linked with clinical factors, such as prevalence and costs. Imaging devices are used in many different ways in clinical practice, so that sensitivities, specificities, and AUCs measured in studies of new diagnostic imaging devices will differ from those in actual future clinical use. CONCLUSIONS: Measured sensitivities, specificities, and the directions of changes thereof are not necessarily consistent or reproducible across studies of new diagnostic devices. A change in the AUC, which should be independent of clinical costs or prevalence, is a consistent measure across similar studies, and a positive change in AUC is indicative of additional diagnostic information that will be available to radiologists in a future clinical environment. Published by Elsevier Inc.
RATIONALE AND OBJECTIVES: Before using a new diagnostic imaging device regularly in a clinic, it should be studied using patients and radiologists. Often such studies report diagnostic performance in terms of sensitivity, specificity, area under the receiver operating characteristic curve (AUC), or differences thereof. In this report we look at how these studies differ from actual future clinical practice and how those differences may affect reported performance measures. MATERIALS AND METHODS: We review signal detection (receiver operating characteristic) theory and decision theory. We compare diagnostic measures from several published studies in medical imaging and examine how they relate to theory and each other. RESULTS: We see that clinical decisions can be modeled using signal detection and decision theories. Sensitivity and specificity are inextricably linked with clinical factors, such as prevalence and costs. Imaging devices are used in many different ways in clinical practice, so that sensitivities, specificities, and AUCs measured in studies of new diagnostic imaging devices will differ from those in actual future clinical use. CONCLUSIONS: Measured sensitivities, specificities, and the directions of changes thereof are not necessarily consistent or reproducible across studies of new diagnostic devices. A change in the AUC, which should be independent of clinical costs or prevalence, is a consistent measure across similar studies, and a positive change in AUC is indicative of additional diagnostic information that will be available to radiologists in a future clinical environment. Published by Elsevier Inc.
Authors: Catriona M Steele; Rajat Mukherjee; Juha M Kortelainen; Harri Pölönen; Michael Jedwab; Susan L Brady; Kayla Brinkman Theimer; Susan Langmore; Luis F Riquelme; Nancy B Swigert; Philip M Bath; Larry B Goldstein; Richard L Hughes; Dana Leifer; Kennedy R Lees; Atte Meretoja; Natalia Muehlemann Journal: Dysphagia Date: 2019-01-05 Impact factor: 3.438