J J Norcini1, S C Day. 1. American Board of Internal Medicine, Philadelphia, Pennsylvania 19104-2675.
Abstract
OBJECTIVE: To provide national norms for indicators of residency-training program quality and information on their reproducibility. PARTICIPANTS: The 364 residency-training programs that had 4 or more candidates take the 1989 to 1991 certifying examination in internal medicine for the first time. DESIGN: Within each residency, program directors' ratings of medical knowledge, certifying examination scores, and certification status (pass or fail) were available for each candidate from 1989 to 1991. Means of these data were calculated for each program for each year of the study. To provide a way of comparing an individual program with all other programs, percentiles are reported for each year. To assess the precision of the measures, generalizability theory was applied and confidence intervals for all data are reported for programs of various size (1 to 25 residents taking the examination) and over the years (1 to 3). RESULTS: Over the 3 years of the study, knowledge ratings, certification rates, and composite scores declined slightly. The correlations between program ratings of medical knowledge and the composite scores ranged from .47 to .60 and certification rates ranged from .44 to .55. The confidence intervals around all of the program performance indicators are relatively large and are affected most by the number of residents in the program. There is little variability across the years. CONCLUSIONS: In smaller programs the precision of the performance indicators is poor; in programs with only a few residents they are virtually meaningless. On the positive side, programs are relatively stable and aggregating indicators over years is a reasonable way to increase their precision in assessing program performance. If the goal of program evaluation is to identify programs at the extremes, especially those at the low end, then such data may help guide program directors and educators.
OBJECTIVE: To provide national norms for indicators of residency-training program quality and information on their reproducibility. PARTICIPANTS: The 364 residency-training programs that had 4 or more candidates take the 1989 to 1991 certifying examination in internal medicine for the first time. DESIGN: Within each residency, program directors' ratings of medical knowledge, certifying examination scores, and certification status (pass or fail) were available for each candidate from 1989 to 1991. Means of these data were calculated for each program for each year of the study. To provide a way of comparing an individual program with all other programs, percentiles are reported for each year. To assess the precision of the measures, generalizability theory was applied and confidence intervals for all data are reported for programs of various size (1 to 25 residents taking the examination) and over the years (1 to 3). RESULTS: Over the 3 years of the study, knowledge ratings, certification rates, and composite scores declined slightly. The correlations between program ratings of medical knowledge and the composite scores ranged from .47 to .60 and certification rates ranged from .44 to .55. The confidence intervals around all of the program performance indicators are relatively large and are affected most by the number of residents in the program. There is little variability across the years. CONCLUSIONS: In smaller programs the precision of the performance indicators is poor; in programs with only a few residents they are virtually meaningless. On the positive side, programs are relatively stable and aggregating indicators over years is a reasonable way to increase their precision in assessing program performance. If the goal of program evaluation is to identify programs at the extremes, especially those at the low end, then such data may help guide program directors and educators.