OBJECTIVE: We explored the agreement among radiologists in their evaluation of the appropriateness of individual requests for imaging procedures. MATERIALS AND METHODS: We reviewed 318 noninterventional CT, sonographic, MR imaging, and nuclear medicine procedures ordered at a general internal medicine clinic during 8 months in 1995. Five subspecialty radiologists used data from the radiology request from and clinic notes to independently rate the appropriateness of each requested imaging procedure on a four-point scale. The radiologists were unaware of the results achieved by each procedure. Each case was reviewed by at least three radiologists, of whom at least one had relevant subspecialty expertise. Agreement among radiologists was analyzed using Cohen's kappa statistic and weighted kappa statistics and Cronbach's alpha statistic. RESULTS: Nonchance agreement (kappa) was .19 +/- .05; weighted kappa was .24 +/- .05. Interrater agreement was significantly greater than that expected from chance alone (p < .01). The composite score, defined as the average of the radiologists' scores for each case, showed moderate reliability, as evidenced by a value for Cronbach's alpha of 70. CONCLUSION: In the absence of explicit criteria, we found modest but statistically significant agreement among radiologists about the appropriateness of individual requests for imaging procedures. The disagreement among radiologists highlights the importance of developing well-reasoned, explicit criteria by which to judge the appropriateness of diagnostic radiology procedures. Further study is needed to elucidate the relationship between appropriateness and actual patient outcomes.
OBJECTIVE: We explored the agreement among radiologists in their evaluation of the appropriateness of individual requests for imaging procedures. MATERIALS AND METHODS: We reviewed 318 noninterventional CT, sonographic, MR imaging, and nuclear medicine procedures ordered at a general internal medicine clinic during 8 months in 1995. Five subspecialty radiologists used data from the radiology request from and clinic notes to independently rate the appropriateness of each requested imaging procedure on a four-point scale. The radiologists were unaware of the results achieved by each procedure. Each case was reviewed by at least three radiologists, of whom at least one had relevant subspecialty expertise. Agreement among radiologists was analyzed using Cohen's kappa statistic and weighted kappa statistics and Cronbach's alpha statistic. RESULTS: Nonchance agreement (kappa) was .19 +/- .05; weighted kappa was .24 +/- .05. Interrater agreement was significantly greater than that expected from chance alone (p < .01). The composite score, defined as the average of the radiologists' scores for each case, showed moderate reliability, as evidenced by a value for Cronbach's alpha of 70. CONCLUSION: In the absence of explicit criteria, we found modest but statistically significant agreement among radiologists about the appropriateness of individual requests for imaging procedures. The disagreement among radiologists highlights the importance of developing well-reasoned, explicit criteria by which to judge the appropriateness of diagnostic radiology procedures. Further study is needed to elucidate the relationship between appropriateness and actual patient outcomes.