Application of treatment thresholds to diagnostic-test evaluation: an alternative to the comparison of areas under receiver operating characteristic curves.

Diagnostic tests are often evaluated by comparison of the areas under receiver operating characteristic (ROC) curves. In this study the authors compared this approach with a more direct method that takes into account consequences of a diagnosis. Data from a prospective study of diagnosis of pulmonary embolism were used for a motivating example. Using multivariable logistic regression analysis, three diagnostic models were built and compared based on their ROC curves. Although model 1 (0.706) and model 2 (0.702) had the same ROC-curve area, they performed differently when risks and benefits of subsequent decisions were considered by applying the treatment probability threshold. Models 1 and 3 (0.611) had substantially different ROC-curve areas but performed similarly taking into account the therapeutic consequences. This demonstrates that comparison of diagnostic tests using the areas under the ROC curves may lead to erroneous conclusions about therapeutic usefulness. To correspond to daily practice, it would be more appropriate to also consider the clinical implications in evaluating diagnostic tests. This is made feasible by explicit definition and application of a treatment threshold.