Validity evidence of resident competency ratings and the identification of problem residents.

OBJECTIVES: This study examined validity evidence of end-of-rotation evaluations used to measure progress toward mastery of core competencies in residents. In addition, this study investigated whether end-of-rotation evaluations can be used to detect problem residents during their training.
METHODS: Historical data for a 4-year period (2009-2012), containing 4986 observations of 291 internal medicine residents, were examined. Residents were observed and assessed by fellows, faculty members and programme directors on nine domains, including the six Accreditation Council for Graduate Medical Education core competencies, as part of their end-of-rotation evaluations. Descriptive statistics were used to collect evidence of the response process. Correlations between competencies and a generalisability study were used to examine the internal structure of the end-of-rotation evaluations. Hierarchical regression was used to estimate the increase in scores across years of training. Scores on end-of-rotation evaluations were compared with trainees identified as problem residents by programme directors.
RESULTS: Compared with fellows, faculty and programme directors had significantly greater variability in assigning scores across different competencies. Correlations between competencies ranged from 0.69 to 0.92. The reliability of end-of-rotation evaluations was adequate (fellows, phi coefficient [φ] = 0.68; faculty [including programme directors], φ = 0.71). Mean scores increased by 0.21 points (95% confidence interval 0.18-0.24) per postgraduate year. Mean scores were significantly correlated with classification as a problem resident (r = 0.33, p < 0.001); problem residents also had significantly lower ratings across all competencies during PGY-1 compared with all other residents.
CONCLUSIONS: End-of-rotation evaluations are a useful method of measuring the growth in resident performance associated with core competencies when sufficient numbers of end-of-rotation evaluation scores are used. Furthermore, end-of-rotation evaluation scores provide preliminary evidence with which to detect and predict problem residents in subsequent postgraduate training years.