OBJECTIVE: The in-training evaluation report (ITER) is widely used to assess clinical skills, but has limited validity and reliability. The purpose of our study was to assess the feasibility, validity, reliability, and effect on feedback of using daily evaluation forms to evaluate residents in ambulatory rheumatology clinics. METHODS: An evaluation form was developed based on the Royal College of Physicians and Surgeons of Canada CanMEDS roles. There were 12 evaluation items including overall clinical competence. They were rated on a 5-point scale from unsatisfactory to outstanding. All internal medicine residents rotating on rheumatology were strongly encouraged to provide the form to their preceptor at the end of each clinic. A questionnaire was administered to residents and faculty. RESULTS: Seventy-three internal medicine residents completed a 1-month rotation at University of Ottawa (n=26) and McMaster University (n=47). Faculty members completed a total of 637 evaluation forms. The number of evaluation forms ranged from 2 to 16 (mean 8.73) per resident. At an average of 8.73 forms per resident the reliability was 0.71 for the composite score. Fourteen forms would be required for a reliability of 0.8. The correlation between the objective structured clinical examination scores and the forms was 0.48 (p=not significant). Faculty and residents reported increased feedback following implementation of the forms. CONCLUSION: The use of daily evaluation forms is feasible and provides very good reliability. Use of the evaluation forms increases feedback to residents on their performance. The forms were well received by faculty and residents.
OBJECTIVE: The in-training evaluation report (ITER) is widely used to assess clinical skills, but has limited validity and reliability. The purpose of our study was to assess the feasibility, validity, reliability, and effect on feedback of using daily evaluation forms to evaluate residents in ambulatory rheumatology clinics. METHODS: An evaluation form was developed based on the Royal College of Physicians and Surgeons of Canada CanMEDS roles. There were 12 evaluation items including overall clinical competence. They were rated on a 5-point scale from unsatisfactory to outstanding. All internal medicine residents rotating on rheumatology were strongly encouraged to provide the form to their preceptor at the end of each clinic. A questionnaire was administered to residents and faculty. RESULTS: Seventy-three internal medicine residents completed a 1-month rotation at University of Ottawa (n=26) and McMaster University (n=47). Faculty members completed a total of 637 evaluation forms. The number of evaluation forms ranged from 2 to 16 (mean 8.73) per resident. At an average of 8.73 forms per resident the reliability was 0.71 for the composite score. Fourteen forms would be required for a reliability of 0.8. The correlation between the objective structured clinical examination scores and the forms was 0.48 (p=not significant). Faculty and residents reported increased feedback following implementation of the forms. CONCLUSION: The use of daily evaluation forms is feasible and provides very good reliability. Use of the evaluation forms increases feedback to residents on their performance. The forms were well received by faculty and residents.
Authors: Lisa L Willett; Carlos A Estrada; Terry C Wall; Heather L Coley; Julius Ngu; William Curry; Amanda Salanitro; Thomas K Houston Journal: J Grad Med Educ Date: 2011-06