Background: The Accreditation Council for Graduate Medical Education's Next Accreditation System requires residency programs to semiannually submit composite milestone data on each resident's performance. This report describes and evaluates a new assessment review procedure piloted in our departmental Clinical Competency Committee (CCC) semi-annual meeting in June 2016. Methods: A modified Delphi technique was utilized to develop key performance indicators (KPI) linking milestone descriptors to clinical practice. In addition, the CCC identified six specific milestone sub-competencies that would be prescored with objective data prior to the meeting. Each resident was independently placed on the milestones by 3 different CCC faculty members. Milestone placement data of the same cohort of 42 residents (Clinical Anesthesia Years 1-3) were collected to calculate inter-rater reliability of the assessment procedures before and after the implemented changes. A survey was administrated to collect CCC feedback on the new procedure. Results: The procedure assisted in reducing meeting time from 8 to 3.5 hours. Survey of the CCC members revealed positive perception of the procedure. Higher inter-rater reliability of the milestone placement was obtained using the implemented KPIs (Intraclass correlation coefficient [ICC] single measure range: before=.53-.94, after=.74-.98). Conclusion: We found the new assessment procedure beneficial to the efficiency and transparency of the assessment process. Further improvement of the procedure involves refinement of KPIs and additional faculty development on KPIs to allow non-CCC faculty to provide more accurate resident evaluations.
Background: The Accreditation Council for Graduate Medical Education's Next Accreditation System requires residency programs to semiannually submit composite milestone data on each resident's performance. This report describes and evaluates a new assessment review procedure piloted in our departmental Clinical Competency Committee (CCC) semi-annual meeting in June 2016. Methods: A modified Delphi technique was utilized to develop key performance indicators (KPI) linking milestone descriptors to clinical practice. In addition, the CCC identified six specific milestone sub-competencies that would be prescored with objective data prior to the meeting. Each resident was independently placed on the milestones by 3 different CCC faculty members. Milestone placement data of the same cohort of 42 residents (Clinical Anesthesia Years 1-3) were collected to calculate inter-rater reliability of the assessment procedures before and after the implemented changes. A survey was administrated to collect CCC feedback on the new procedure. Results: The procedure assisted in reducing meeting time from 8 to 3.5 hours. Survey of the CCC members revealed positive perception of the procedure. Higher inter-rater reliability of the milestone placement was obtained using the implemented KPIs (Intraclass correlation coefficient [ICC] single measure range: before=.53-.94, after=.74-.98). Conclusion: We found the new assessment procedure beneficial to the efficiency and transparency of the assessment process. Further improvement of the procedure involves refinement of KPIs and additional faculty development on KPIs to allow non-CCC faculty to provide more accurate resident evaluations.