Lori R Newman1, Dara Brodsky, Richard N Jones, Richard M Schwartzstein, Katharyn Meredith Atkins, David H Roberts. 1. Ms. Newman: Principal Associate in Medicine, Harvard Medical School; Director of Professional Development in Medical Education, Department of Medical Education, Boston Children's Hospital, Boston, MA, and, at the time of this study, Director of Faculty Education, Carl J. Shapiro Institute for Education and Research at Harvard Medical School and Beth Israel Deaconess Medical Center, Boston, MA. Dr. Brodsky: Assistant Professor of Pediatrics, Department of Pediatrics, Harvard Medical School, and Associate Director, Neonatal Intensive Care Unit, Beth Israel Deaconess Medical Center, Boston, MA. Dr. Jones: Associate Professor of Psychiatry and Human Behavior, Warren Alpert Medical School of Brown University, Providence, RI. Dr. Schwartzstein: Professor of Medicine and Medical Education, Department of Medicine, Harvard Medical School, and Executive Director, Carl J. Shapiro Institute for Education and Research at Harvard Medical School and Beth Israel Deaconess Medical Center, and Vice President for Education, Beth Israel Deaconess Medical Center, Boston, MA. Dr. Atkins: Assistant Professor of Obstetrics, Gynecology, and Reproductive Biology, Harvard Medical School, and Director, Undergraduate Medical Education, Beth Israel Deaconess Medical Center, Boston, MA. Dr. Roberts: Associate Professor of Medicine, Department of Medicine, and Dean for External Education, Harvard Medical School, Boston, MA.
Abstract
INTRODUCTION: Frame-of-reference (FOR) training has been used successfully to teach faculty how to produce accurate and reliable workplace-based ratings when assessing a performance. We engaged 21 Harvard Medical School faculty members in our pilot and implementation studies to determine the effectiveness of using FOR training to assess health professionals' teaching performances. METHODS: All faculty were novices at rating their peers' teaching effectiveness. Before FOR training, we asked participants to evaluate a recorded lecture using a criterion-based peer assessment of medical lecturing instrument. At the start of training, we discussed the instrument and emphasized its precise behavioral standards. During training, participants practiced rating lectures and received immediate feedback on how well they categorized and scored performances as compared with expert-derived scores of the same lectures. At the conclusion of the training, we asked participants to rate a post-training recorded lecture to determine agreement with the experts' scores. RESULTS: Participants and experts had greater rating agreement for the post-training lecture compared with the pretraining lecture. Through this investigation, we determined that FOR training is a feasible method to teach faculty how to accurately and reliably assess medical lectures. DISCUSSION: Medical school instructors and continuing education presenters should have the opportunity to be observed and receive feedback from trained peer observers. Our results show that it is possible to use FOR rater training to teach peer observers how to accurately rate medical lectures. The process is time efficient and offers the prospect for assessment and feedback beyond traditional learner evaluation of instruction.
INTRODUCTION: Frame-of-reference (FOR) training has been used successfully to teach faculty how to produce accurate and reliable workplace-based ratings when assessing a performance. We engaged 21 Harvard Medical School faculty members in our pilot and implementation studies to determine the effectiveness of using FOR training to assess health professionals' teaching performances. METHODS: All faculty were novices at rating their peers' teaching effectiveness. Before FOR training, we asked participants to evaluate a recorded lecture using a criterion-based peer assessment of medical lecturing instrument. At the start of training, we discussed the instrument and emphasized its precise behavioral standards. During training, participants practiced rating lectures and received immediate feedback on how well they categorized and scored performances as compared with expert-derived scores of the same lectures. At the conclusion of the training, we asked participants to rate a post-training recorded lecture to determine agreement with the experts' scores. RESULTS:Participants and experts had greater rating agreement for the post-training lecture compared with the pretraining lecture. Through this investigation, we determined that FOR training is a feasible method to teach faculty how to accurately and reliably assess medical lectures. DISCUSSION: Medical school instructors and continuing education presenters should have the opportunity to be observed and receive feedback from trained peer observers. Our results show that it is possible to use FOR rater training to teach peer observers how to accurately rate medical lectures. The process is time efficient and offers the prospect for assessment and feedback beyond traditional learner evaluation of instruction.
Authors: Sarang Kim; Liesel Copeland; Elizabeth Cohen; James Galt; Carol A Terregino; Archana Pradhan Journal: J Gen Intern Med Date: 2021-05-03 Impact factor: 6.473