Jaime Jordan1,2, Laura R Hopson3, Caroline Molins4, Suzanne K Bentley5, Nicole M Deiorio6, Sally A Santen6,7, Lalena M Yarris8, Wendy C Coates1, Michael A Gisondi9. 1. Department of Emergency Medicine David Geffen School of Medicine at UCLA Los Angeles California USA. 2. Department of Emergency Medicine Ronald Reagan UCLA Medical Center Los Angeles California USA. 3. Department of Emergency Medicine University of Michigan Ann Arbor Michigan USA. 4. AdventHealth Emergency Medicine Residency Orlando Florida USA. 5. Icahn School of Medicine at Mount Sinai New York New York USA. 6. Virginia Commonwealth University School of Medicine Richmond Virginia USA. 7. University of Cincinnati College of Medicine Cincinnati Ohio USA. 8. Department of Emergency Medicine Oregon Health & Science University Portland Oregon USA. 9. Department of Emergency Medicine Stanford University Palo Alto California USA.
Abstract
BACKGROUND: Research abstracts are submitted for presentation at scientific conferences; however, criteria for judging abstracts are variable. We sought to develop two rigorous abstract scoring rubrics for education research submissions reporting (1) quantitative data and (2) qualitative data and then to collect validity evidence to support score interpretation. METHODS: We used a modified Delphi method to achieve expert consensus for scoring rubric items to optimize content validity. Eight education research experts participated in two separate modified Delphi processes, one to generate quantitative research items and one for qualitative. Modifications were made between rounds based on item scores and expert feedback. Homogeneity of ratings in the Delphi process was calculated using Cronbach's alpha, with increasing homogeneity considered an indication of consensus. Rubrics were piloted by scoring abstracts from 22 quantitative publications from AEM Education and Training "Critical Appraisal of Emergency Medicine Education Research" (11 highlighted for excellent methodology and 11 that were not) and 10 qualitative publications (five highlighted for excellent methodology and five that were not). Intraclass correlation coefficient (ICC) estimates of reliability were calculated. RESULTS: Each rubric required three rounds of a modified Delphi process. The resulting quantitative rubric contained nine items: quality of objectives, appropriateness of methods, outcomes, data analysis, generalizability, importance to medical education, innovation, quality of writing, and strength of conclusions (Cronbach's α for the third round = 0.922, ICC for total scores during piloting = 0.893). The resulting qualitative rubric contained seven items: quality of study aims, general methods, data collection, sampling, data analysis, writing quality, and strength of conclusions (Cronbach's α for the third round = 0.913, ICC for the total scores during piloting = 0.788). CONCLUSION: We developed scoring rubrics to assess quality in quantitative and qualitative medical education research abstracts to aid in selection for presentation at scientific meetings. Our tools demonstrated high reliability.
BACKGROUND: Research abstracts are submitted for presentation at scientific conferences; however, criteria for judging abstracts are variable. We sought to develop two rigorous abstract scoring rubrics for education research submissions reporting (1) quantitative data and (2) qualitative data and then to collect validity evidence to support score interpretation. METHODS: We used a modified Delphi method to achieve expert consensus for scoring rubric items to optimize content validity. Eight education research experts participated in two separate modified Delphi processes, one to generate quantitative research items and one for qualitative. Modifications were made between rounds based on item scores and expert feedback. Homogeneity of ratings in the Delphi process was calculated using Cronbach's alpha, with increasing homogeneity considered an indication of consensus. Rubrics were piloted by scoring abstracts from 22 quantitative publications from AEM Education and Training "Critical Appraisal of Emergency Medicine Education Research" (11 highlighted for excellent methodology and 11 that were not) and 10 qualitative publications (five highlighted for excellent methodology and five that were not). Intraclass correlation coefficient (ICC) estimates of reliability were calculated. RESULTS: Each rubric required three rounds of a modified Delphi process. The resulting quantitative rubric contained nine items: quality of objectives, appropriateness of methods, outcomes, data analysis, generalizability, importance to medical education, innovation, quality of writing, and strength of conclusions (Cronbach's α for the third round = 0.922, ICC for total scores during piloting = 0.893). The resulting qualitative rubric contained seven items: quality of study aims, general methods, data collection, sampling, data analysis, writing quality, and strength of conclusions (Cronbach's α for the third round = 0.913, ICC for the total scores during piloting = 0.788). CONCLUSION: We developed scoring rubrics to assess quality in quantitative and qualitative medical education research abstracts to aid in selection for presentation at scientific meetings. Our tools demonstrated high reliability.
Authors: Rudolf W Poolman; Lucien C M Keijser; Maarten C de Waal Malefijt; Leendert Blankevoort; Forough Farrokhyar; Mohit Bhandari Journal: Acta Orthop Date: 2007-04 Impact factor: 3.717
Authors: Susan E Farrell; Gloria J Kuhn; Wendy C Coates; Phillip H Shayne; Jonathan Fisher; Lauren A Maggio; Michelle Lin Journal: Acad Emerg Med Date: 2014-11 Impact factor: 3.451
Authors: Michelle Lin; Jonathan Fisher; Wendy C Coates; Susan E Farrell; Philip Shayne; Lauren Maggio; Gloria Kuhn Journal: Acad Emerg Med Date: 2014-03 Impact factor: 3.451
Authors: Nicole M Dubosh; Jaime Jordan; Lalena M Yarris; Edward Ullman; Joshua Kornegay; Daniel Runde; Amy Miller Juve; Jonathan Fisher Journal: AEM Educ Train Date: 2018-12-14
Authors: Christopher R Stephenson; Brianna E Vaa; Amy T Wang; Darrell R Schroeder; Thomas J Beckman; Darcy A Reed; Adam P Sawatsky Journal: BMC Med Educ Date: 2017-11-09 Impact factor: 2.463
Authors: Breda H Eubank; Nicholas G Mohtadi; Mark R Lafave; J Preston Wiley; Aaron J Bois; Richard S Boorman; David M Sheps Journal: BMC Med Res Methodol Date: 2016-05-20 Impact factor: 4.615