Global judgments versus decision-model-facilitated judgments: are experts internally consistent?

A widely used method for evaluating the appropriateness of medical procedures and practices is the "modified Delphi" approach using expert panelists' global ratings. However, several difficulties in the assignment of global ratings have led to a search for alternative methods, including the use of decision models. To examine the potential impact of using decision models with an expert panel, the authors compared a panel's global ratings for the appropriateness of carotid endarterectomy with the results of a decision-analytic model in which expert panelists estimated probabilities and utilities that were used as inputs for the model. For 17 different patient scenarios, the nine expert panelists showed variability in "calibration" between the two methods, with their expected utilities calculated from the model generally being higher than their global ratings. However, the correlation between the two methods was excellent. When the panel's median global utility was compared with the panel's median expected utility calculated from the model, the Spearman correlation coefficient was 0.88. This study demonstrated that an expert panel's appropriateness ratings and their expected utilities were highly correlated. In addition, the panelists appeared to be internally consistent in that their judgments about individual probabilities and utilities were correlated with their global judgments. These results should encourage additional efforts to incorporate decision models into the process of clinical guideline development. The authors believe that decision models can help improve a panel's capacity to understand and reconcile discordance, and increase their satisfaction that the process reflects the best possible judgments.