Evaluating physicians' probabilistic judgments.

Physicians increasingly are challenged to make probabilistic judgments quantitatively. Their ability to make such judgments may be directly linked to the quality of care they provide. Many methods are available to evaluate these judgments. Graphic means of assessment include the calibration curve, covariance graph, and receiver operating characteristic (ROC) curve. Statistical tools can measure the significance of departures from ideal calibration, and measure the area under ROC curve. Modeling the calibration curve using linear or logistic regression provides another method to assess probabilistic judgments, although these may be limited by failure of the data to meet the model's assumptions. Scoring rules provide indices of overall judgmental performance, although their reliability is difficult to gauge for small sample sizes. Decompositions of scoring rules separate judgmental performance into functional components. The authors provide preliminary guidelines for choosing methods for specific research in this area.