Oria Mahmood1, Julia Dagnæs2, Sarah Bube3, Malene Rohrsted4, Lars Konge5. 1. Copenhagen Academy for Medical Education and Simulation, Copenhagen, Capital Region, Denmark; University of Copenhagen, Copenhagen, Denmark. Electronic address: Oriamahmood@gmail.com. 2. Copenhagen Academy for Medical Education and Simulation, Copenhagen, Capital Region, Denmark; University of Copenhagen, Copenhagen, Denmark; Department of Urology, Rigshospitalet, Copenhagen, Denmark. 3. University of Copenhagen, Copenhagen, Denmark; Department of Urology, University Hospital Zealand, Roskilde, Denmark. 4. Department of Urology, Rigshospitalet, Copenhagen, Denmark. 5. Copenhagen Academy for Medical Education and Simulation, Copenhagen, Capital Region, Denmark; University of Copenhagen, Copenhagen, Denmark.
Abstract
BACKGROUND: Competency-based learning has become a crucial component in medical education. Despite the advantages of competency-based learning, there are still challenges that need to be addressed. Currently, the common perception is that specialist assessment is needed for evaluating procedural skills which is difficult owing to the limited availability of faculty time. The aim of this study was to explore the validity of assessments of video recorded procedures performed by nonspecialist raters. METHODS: This study was a blinded observational trial. Twenty-three novices (senior medical students) and 9 experienced doctors were video recorded while each performing 2 flexible cystoscopies on patients. The recordings were anonymized and placed in random order and then rated by 2 experienced cystoscopists (specialist raters) and 2 medical students (nonspecialist raters). Flexible cystoscopy was chosen as it is a simple procedural skill that is crucial to master in a resident urology program. RESULTS: The internal consistency of assessments was high, Cronbach's α = 0.93 and 0.95 for nonspecialist and specialist raters, respectively (p < 0.001 for both correlations). The interrater reliability was significant (p < 0.001) with a Pearson's correlation of 0.77 for the nonspecialists and 0.75 for the specialists. The test-retest reliability showed the biggest difference between the 2 groups, 0.59 and 0.38 for the nonspecialist raters and the specialist raters, respectively (p < 0.001). CONCLUSION: Our study suggests that nonspecialist raters can provide reliable and valid assessments of video recorded cystoscopies. This could make mastery learning and competency-based education more feasible.
BACKGROUND: Competency-based learning has become a crucial component in medical education. Despite the advantages of competency-based learning, there are still challenges that need to be addressed. Currently, the common perception is that specialist assessment is needed for evaluating procedural skills which is difficult owing to the limited availability of faculty time. The aim of this study was to explore the validity of assessments of video recorded procedures performed by nonspecialist raters. METHODS: This study was a blinded observational trial. Twenty-three novices (senior medical students) and 9 experienced doctors were video recorded while each performing 2 flexible cystoscopies on patients. The recordings were anonymized and placed in random order and then rated by 2 experienced cystoscopists (specialist raters) and 2 medical students (nonspecialist raters). Flexible cystoscopy was chosen as it is a simple procedural skill that is crucial to master in a resident urology program. RESULTS: The internal consistency of assessments was high, Cronbach's α = 0.93 and 0.95 for nonspecialist and specialist raters, respectively (p < 0.001 for both correlations). The interrater reliability was significant (p < 0.001) with a Pearson's correlation of 0.77 for the nonspecialists and 0.75 for the specialists. The test-retest reliability showed the biggest difference between the 2 groups, 0.59 and 0.38 for the nonspecialist raters and the specialist raters, respectively (p < 0.001). CONCLUSION: Our study suggests that nonspecialist raters can provide reliable and valid assessments of video recorded cystoscopies. This could make mastery learning and competency-based education more feasible.
Authors: Joshua D Knopf; Rahul Kumar; Michael Barats; Paul Klimo; Frederick A Boop; L Madison Michael; Jonathan E Martin; Markus Bookland; David S Hersh Journal: World Neurosurg Date: 2020-09-02 Impact factor: 2.104