Dana Dunne1, Katherine Gielissen2, Martin Slade3, Yoon Soo Park4, Michael Green2. 1. Department of Internal Medicine, Section of Infectious Diseases, Yale School of Medicine, 15 York Street LMP 1074, New Haven, CT, 065111, USA. dana.dunne@yale.edu. 2. Department of Internal Medicine, Section of General Medicine, Yale School of Medicine, New Haven, CT, 06511, USA. 3. Occupational Medicine, Yale School of Medicine, New Haven, CT, 06511, USA. 4. Harvard Medical School, Boston, MA, USA.
Abstract
BACKGROUND: Reliable assessments of clinical skills are important for undergraduate medical education, trustworthy handoffs to graduate medical programs, and safe, effective patient care. Entrustable professional activities (EPAs) for entering residency have been developed; research is needed to assess reliability of such assessments in authentic clinical workspaces. DESIGN: A student-driven mobile assessment platform was developed and used for clinical supervisors to record ad hoc entrustment decisions using the modified Ottawa scale on 5 core EPAs in an 8-week internal medicine (IM) clerkship. After a 12-month period, generalizability (G) theory analysis was performed to estimate the reliability of entrustment scores and determine the proportion of variance attributable to the student and the other facets, including particular EPA, evaluator type (attending versus resident), or case complexity. Decision (D) theory analysis determined the expected reliability based on the number of hypothetical observations. A g-coefficient of 0.7 was used as a generally agreed upon minimum reliability threshold. KEY RESULTS: A total of 1368 ratings over the 5 EPAs were completed on 94 students. Variance attributed to person (true variance) was high for all EPAs; EPA-5 had the lowest person variance (9.8% across cases and four blocks). Across cases, reliability ranged from 0.02 to 0.60. Applying this to the Decision study, the estimated number of observations needed to reach a reliability index of 0.7 ranged between 9 and 11 for all EPAs except EPA5 which was sensitive to case complexity. CONCLUSIONS: Work place-based clinical skills in IM clerkship students were assessed and logged using a convenient mobile platform. Our analysis suggests that 9-11 observations are needed for these EPA workplace-based assessments (WBAs) to achieve a reliability index of 0.7. Note writing was very sensitive to case complexity. Further reliability analyses of core EPAs are needed before US medical schools consider wider adoption into summative entrustment processes and GME handoffs.
BACKGROUND: Reliable assessments of clinical skills are important for undergraduate medical education, trustworthy handoffs to graduate medical programs, and safe, effective patient care. Entrustable professional activities (EPAs) for entering residency have been developed; research is needed to assess reliability of such assessments in authentic clinical workspaces. DESIGN: A student-driven mobile assessment platform was developed and used for clinical supervisors to record ad hoc entrustment decisions using the modified Ottawa scale on 5 core EPAs in an 8-week internal medicine (IM) clerkship. After a 12-month period, generalizability (G) theory analysis was performed to estimate the reliability of entrustment scores and determine the proportion of variance attributable to the student and the other facets, including particular EPA, evaluator type (attending versus resident), or case complexity. Decision (D) theory analysis determined the expected reliability based on the number of hypothetical observations. A g-coefficient of 0.7 was used as a generally agreed upon minimum reliability threshold. KEY RESULTS: A total of 1368 ratings over the 5 EPAs were completed on 94 students. Variance attributed to person (true variance) was high for all EPAs; EPA-5 had the lowest person variance (9.8% across cases and four blocks). Across cases, reliability ranged from 0.02 to 0.60. Applying this to the Decision study, the estimated number of observations needed to reach a reliability index of 0.7 ranged between 9 and 11 for all EPAs except EPA5 which was sensitive to case complexity. CONCLUSIONS: Work place-based clinical skills in IM clerkship students were assessed and logged using a convenient mobile platform. Our analysis suggests that 9-11 observations are needed for these EPA workplace-based assessments (WBAs) to achieve a reliability index of 0.7. Note writing was very sensitive to case complexity. Further reliability analyses of core EPAs are needed before US medical schools consider wider adoption into summative entrustment processes and GME handoffs.
Authors: Debra Pugh; Rodrigo B Cavalcanti; Samantha Halman; Irene W Y Ma; Maria Mylopoulos; David Shanks; Lynfa Stroud Journal: J Grad Med Educ Date: 2017-04
Authors: Michele A Favreau; Linda Tewksbury; Carla Lupi; William B Cutrer; Janet A Jokela; Lalena M Yarris Journal: Acad Med Date: 2017-06 Impact factor: 6.893
Authors: Anna Chang; Judith L Bowen; Raquel A Buranosky; Richard M Frankel; Nivedita Ghosh; Michael J Rosenblum; Sara Thompson; Michael L Green Journal: J Gen Intern Med Date: 2012-09-21 Impact factor: 5.128
Authors: Joseph R Geraghty; Raechelle G Ocampo; Sherry Liang; Kimberly E Ona Ayala; Kathleen Hiltz; Haley McKissack; Abbas Hyderi; Michael S Ryan Journal: Acad Med Date: 2021-02-01 Impact factor: 6.893
Authors: Karen E Hauer; Krishan Soni; Patricia Cornett; Jeff Kohlwes; Harry Hollander; Sumant R Ranji; Olle Ten Cate; Eric Widera; Brook Calton; Patricia S O'Sullivan Journal: J Gen Intern Med Date: 2013-08 Impact factor: 5.128
Authors: Michael S Ryan; Alicia Richards; Robert Perera; Yoon Soo Park; J K Stringer; Elizabeth Waterhouse; Brieanne Dubinsky; Rebecca Khamishon; Sally A Santen Journal: Acad Med Date: 2021-01-12 Impact factor: 7.840