BACKGROUND: The application of crowdsourcing to surgical education is a recent phenomenon and adds to increasing demands on surgical residency training. The efficacy, range, and scope of this technology for surgical education remains incompletely defined. OBJECTIVE: A systematic review was performed using the PubMed database of English-language literature on crowdsourced evaluation of surgical technical tasks up to April 2017. METHODS: Articles were reviewed, abstracted, and analyzed, and were assessed for quality using the Medical Education Research Study Quality Instrument (MERSQI). Articles were evaluated with eligibility criteria for inclusion. Study information, performance task, subjects, evaluative standards, crowdworker compensation, time to response, and correlation between crowd and expert or standard evaluations were abstracted and analyzed. RESULTS: Of 63 unique publications initially identified, 13 with MERSQI scores ranging from 10 to 13 (mean = 11.85) were included in the review. Overall, crowd and expert evaluations demonstrated good to excellent correlation across a wide range of tasks (Pearson's coefficient 0.59-0.95, Cronbach's alpha 0.32-0.92), with 1 exception being a study involving medical students. There was a wide range of reported interrater variability among experts. Nonexpert evaluation was consistently quicker than expert evaluation (ranging from 4.8 to 150.9 times faster), and was more cost effective. CONCLUSIONS: Crowdsourced feedback appears to be comparable to expert feedback and is cost effective and efficient. Further work is needed to increase consistency in expert evaluations, to explore sources of discrepant assessments between surgeons and crowds, and to identify optimal populations and novel applications for this technology.
BACKGROUND: The application of crowdsourcing to surgical education is a recent phenomenon and adds to increasing demands on surgical residency training. The efficacy, range, and scope of this technology for surgical education remains incompletely defined. OBJECTIVE: A systematic review was performed using the PubMed database of English-language literature on crowdsourced evaluation of surgical technical tasks up to April 2017. METHODS: Articles were reviewed, abstracted, and analyzed, and were assessed for quality using the Medical Education Research Study Quality Instrument (MERSQI). Articles were evaluated with eligibility criteria for inclusion. Study information, performance task, subjects, evaluative standards, crowdworker compensation, time to response, and correlation between crowd and expert or standard evaluations were abstracted and analyzed. RESULTS: Of 63 unique publications initially identified, 13 with MERSQI scores ranging from 10 to 13 (mean = 11.85) were included in the review. Overall, crowd and expert evaluations demonstrated good to excellent correlation across a wide range of tasks (Pearson's coefficient 0.59-0.95, Cronbach's alpha 0.32-0.92), with 1 exception being a study involving medical students. There was a wide range of reported interrater variability among experts. Nonexpert evaluation was consistently quicker than expert evaluation (ranging from 4.8 to 150.9 times faster), and was more cost effective. CONCLUSIONS: Crowdsourced feedback appears to be comparable to expert feedback and is cost effective and efficient. Further work is needed to increase consistency in expert evaluations, to explore sources of discrepant assessments between surgeons and crowds, and to identify optimal populations and novel applications for this technology.
Authors: Mary K Powers; Aaron Boonjindasup; Michael Pinsky; Philip Dorsey; Michael Maddox; Li-Ming Su; Matthew Gettman; Chandru P Sundaram; Erik P Castle; Jason Y Lee; Benjamin R Lee Journal: J Endourol Date: 2015-12-30 Impact factor: 2.942
Authors: Melina C Vassiliou; Liane S Feldman; Christopher G Andrew; Simon Bergman; Karen Leffondré; Donna Stanbridge; Gerald M Fried Journal: Am J Surg Date: 2005-07 Impact factor: 2.565
Authors: Darcy A Reed; David A Cook; Thomas J Beckman; Rachel B Levine; David E Kern; Scott M Wright Journal: JAMA Date: 2007-09-05 Impact factor: 56.272
Authors: Khurshid R Ghani; David C Miller; Susan Linsell; Andrew Brachulis; Brian Lane; Richard Sarle; Deepansh Dalela; Mani Menon; Bryan Comstock; Thomas S Lendvay; James Montie; James O Peabody Journal: Eur Urol Date: 2016-01-02 Impact factor: 20.096
Authors: Emily Boyle; Musallam Al-Akash; Anthony G Gallagher; Oscar Traynor; Arnold D K Hill; Paul C Neary Journal: Postgrad Med J Date: 2011-06-03 Impact factor: 2.401
Authors: Dimitrios Stefanidis; Sonal Arora; David M Parrack; Giselle G Hamad; Jeannette Capella; Teodor Grantcharov; David R Urbach; Daniel J Scott; Daniel B Jones Journal: Am J Surg Date: 2012-01 Impact factor: 2.565
Authors: Christopher J Brady; Andrea C Villanti; Jennifer L Pearson; Thomas R Kirchner; Omesh P Gupta; Chirag P Shah Journal: J Med Internet Res Date: 2014-10-30 Impact factor: 5.428
Authors: Geb W Thomas; Steven Long; Marcus Tatum; Timothy Kowalewski; Dominik Mattioli; J Lawrence Marsh; Heather R Kowalski; Matthew D Karam; Joan E Bechtold; Donald D Anderson Journal: Iowa Orthop J Date: 2020