Brent Thoma1, Stefanie S Sebok-Syer2, Keeth Krishnan3, Marshall Siemens4, N Seth Trueger5, Isabelle Colmers-Gray6, Rob Woods4, Emil Petrusa7, Teresa Chan8. 1. Department of Emergency Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada; Health Professions Education Program, Massachusetts General Hospital Institute of Health Professions, Boston, MA. Electronic address: brent.thoma@usask.ca. 2. Centre for Education Research & Innovation, University of Western Ontario, London, Ontario, Canada. 3. Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada. 4. Department of Emergency Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada. 5. Department of Emergency Medicine, Northwestern University, Chicago, IL. 6. Department of Emergency Medicine, University of Alberta, Edmonton, Alberta, Canada. 7. Health Professions Education Program, Massachusetts General Hospital Institute of Health Professions, Boston, MA. 8. Division of Emergency Medicine, Department of Medicine, McMaster University, Hamilton, Ontario, Canada.
Abstract
STUDY OBJECTIVE: Open educational resources such as blogs are increasingly used for medical education. Gestalt is generally the evaluation method used for these resources; however, little information has been published on it. We aim to evaluate the reliability of gestalt in the assessment of emergency medicine blogs. METHODS: We identified 60 English-language emergency medicine Web sites that posted clinically oriented blogs between January 1, 2016, and February 24, 2016. Ten Web sites were selected with a random-number generator. Medical students, emergency medicine residents, and emergency medicine attending physicians evaluated the 2 most recent clinical blog posts from each site for quality, using a 7-point Likert scale. The mean gestalt scores of each blog post were compared between groups with Pearson's correlations. Single and average measure intraclass correlation coefficients were calculated within groups. A generalizability study evaluated variance within gestalt and a decision study calculated the number of raters required to reliably (>0.8) estimate quality. RESULTS: One hundred twenty-one medical students, 88 residents, and 100 attending physicians (93.6% of enrolled participants) evaluated all 20 blog posts. Single-measure intraclass correlation coefficients within groups were fair to poor (0.36 to 0.40). Average-measure intraclass correlation coefficients were more reliable (0.811 to 0.840). Mean gestalt ratings by attending physicians correlated strongly with those by medical students (r=0.92) and residents (r=0.99). The generalizability coefficient was 0.91 for the complete data set. The decision study found that 42 gestalt ratings were required to reliably evaluate quality (>0.8). CONCLUSION: The mean gestalt quality ratings of blog posts between medical students, residents, and attending physicians correlate strongly, but individual ratings are unreliable. With sufficient raters, mean gestalt ratings provide a community standard for assessment.
STUDY OBJECTIVE: Open educational resources such as blogs are increasingly used for medical education. Gestalt is generally the evaluation method used for these resources; however, little information has been published on it. We aim to evaluate the reliability of gestalt in the assessment of emergency medicine blogs. METHODS: We identified 60 English-language emergency medicine Web sites that posted clinically oriented blogs between January 1, 2016, and February 24, 2016. Ten Web sites were selected with a random-number generator. Medical students, emergency medicine residents, and emergency medicine attending physicians evaluated the 2 most recent clinical blog posts from each site for quality, using a 7-point Likert scale. The mean gestalt scores of each blog post were compared between groups with Pearson's correlations. Single and average measure intraclass correlation coefficients were calculated within groups. A generalizability study evaluated variance within gestalt and a decision study calculated the number of raters required to reliably (>0.8) estimate quality. RESULTS: One hundred twenty-one medical students, 88 residents, and 100 attending physicians (93.6% of enrolled participants) evaluated all 20 blog posts. Single-measure intraclass correlation coefficients within groups were fair to poor (0.36 to 0.40). Average-measure intraclass correlation coefficients were more reliable (0.811 to 0.840). Mean gestalt ratings by attending physicians correlated strongly with those by medical students (r=0.92) and residents (r=0.99). The generalizability coefficient was 0.91 for the complete data set. The decision study found that 42 gestalt ratings were required to reliably evaluate quality (>0.8). CONCLUSION: The mean gestalt quality ratings of blog posts between medical students, residents, and attending physicians correlate strongly, but individual ratings are unreliable. With sufficient raters, mean gestalt ratings provide a community standard for assessment.
Authors: Jonie J Hsiao; Ryan Pedigo; Shirley W Bae; JooYeon Jung; Lisa Zhao; Nathan S Trueger; Teresa M Chan; Andrew Grock Journal: AEM Educ Train Date: 2021-08-01
Authors: Daniel K Ting; Brent Thoma; S Luckett-Gatopoulos; Adam Thomas; Shahbaz Syed; Michael Bravo; Fareen Zaver; Eve Purdy; Edmund S H Kwok; Teresa M Chan Journal: AEM Educ Train Date: 2018-11-02