Background: A key component of competency-based medical education (CBME) is direct observation of trainees. Direct observation has been emphasized as integral to workplace-based assessment (WBA) yet previously identified challenges may limit its successful implementation. Given these challenges, it is imperative to fully understand the value of direct observation within a CBME program of assessment. Specifically, it is not known whether the quality of WBA documentation is influenced by observation type (direct or indirect). Methods: The objective of this study was to determine the influence of observation type (direct or indirect) on quality of entrustable professional activity (EPA) assessment documentation within a CBME program. EPA assessments were scored by four raters using the Quality of Assessment for Learning (QuAL) instrument, a previously published three-item quantitative measure of the quality of written comments associated with a single clinical performance score. An analysis of variance was performed to compare mean QuAL scores among the direct and indirect observation groups. The reliability of the QuAL instrument for EPA assessments was calculated using a generalizability analysis. Results: A total of 244 EPA assessments (122 direct observation, 122 indirect observation) were rated for quality using the QuAL instrument. No difference in mean QuAL score was identified between the direct and indirect observation groups (p = 0.17). The reliability of the QuAL instrument for EPA assessments was 0.84. Conclusions: Observation type (direct or indirect) did not influence the quality of EPA assessment documentation. This finding raises the question of how direct and indirect observation truly differ and the implications for meta-raters such as competence committees responsible for making judgments related to trainee promotion.
Background: A key component of competency-based medical education (CBME) is direct observation of trainees. Direct observation has been emphasized as integral to workplace-based assessment (WBA) yet previously identified challenges may limit its successful implementation. Given these challenges, it is imperative to fully understand the value of direct observation within a CBME program of assessment. Specifically, it is not known whether the quality of WBA documentation is influenced by observation type (direct or indirect). Methods: The objective of this study was to determine the influence of observation type (direct or indirect) on quality of entrustable professional activity (EPA) assessment documentation within a CBME program. EPA assessments were scored by four raters using the Quality of Assessment for Learning (QuAL) instrument, a previously published three-item quantitative measure of the quality of written comments associated with a single clinical performance score. An analysis of variance was performed to compare mean QuAL scores among the direct and indirect observation groups. The reliability of the QuAL instrument for EPA assessments was calculated using a generalizability analysis. Results: A total of 244 EPA assessments (122 direct observation, 122 indirect observation) were rated for quality using the QuAL instrument. No difference in mean QuAL score was identified between the direct and indirect observation groups (p = 0.17). The reliability of the QuAL instrument for EPA assessments was 0.84. Conclusions: Observation type (direct or indirect) did not influence the quality of EPA assessment documentation. This finding raises the question of how direct and indirect observation truly differ and the implications for meta-raters such as competence committees responsible for making judgments related to trainee promotion.
Authors: William F Iobst; Jonathan Sherbino; Olle Ten Cate; Denyse L Richardson; Deepak Dath; Susan R Swing; Peter Harris; Rani Mungroo; Eric S Holmboe; Jason R Frank Journal: Med Teach Date: 2010 Impact factor: 3.650
Authors: Christopher I Doty; Lynn P Roppolo; Shellie Asher; Jason P Seamon; Rahul Bhat; Stephanie Taft; Autumn Graham; James Willis Journal: Acad Emerg Med Date: 2015-10-16 Impact factor: 3.451
Authors: Walter Tavares; Walter Eppich; Adam Cheng; Stephen Miller; Pim W Teunissen; Christopher J Watling; Joan Sargeant Journal: Acad Med Date: 2020-07 Impact factor: 6.893
Authors: Brent Thoma; Andrew K Hall; Kevin Clark; Nazanin Meshkat; Warren J Cheung; Pierre Desaulniers; Cheryl Ffrench; Allison Meiwald; Christine Meyers; Catherine Patocka; Lorri Beatty; Teresa M Chan Journal: J Grad Med Educ Date: 2020-08