Scoring patient management problems: external validation of expert consensus.

This study determined the extent to which medical faculty agreed when rating options in two written patient management problems in diabetes mellitus. Another purpose was to determine whether option weights (used for scoring) based on the consensus ratings of faculty actually predict the choices of well-qualified physicians. Experts showed better-than-chance agreement but with considerable variation from one part of a problem to another (31% to 73%). Nevertheless, consensus ratings were very accurate predictors of the decisions of endocrinology fellows (correlations from .60 to .97). When scoring weights are assigned to options in patient management problems, consensus (average) ratings of experts are likely to demonstrate high concurrent validity for well-qualified clinicans.