Ricardo Riveros1, Stephen Kimatian2, Pilar Castro2, Vipul Dhumak3, Hooman Honar4, Edward J Mascha5, Daniel I Sessler4. 1. Department of Pediatric Anesthesia and Outcomes Research, Cleveland Clinic. Electronic address: riveron@ccf.org. 2. Department of Pediatric Anesthesia, Cleveland Clinic. 3. Department of Anesthesia, University of Iowa Hospitals and Clinic. 4. Department of Outcomes Research, Cleveland Clinic. 5. Departments of Quantitative Health Sciences and Outcomes Research, Cleveland Clinic.
Abstract
STUDY OBJECTIVE: To assess professionalism in anesthesiology residents, it is important to obtain evaluations from people with whom they interact on daily basis. The purpose of this study was to evaluate the effect of a Multisource feedback (MSF) on resident's professional behavior and to assess the effect of faculty feedback on resident performance. DESIGN: This study was a two-group randomized clinical trial. SETTING:Residents were recruited from Cleveland Clinic Children's Hospital. PATIENTS: Participants included twenty eight residents doing a two-month rotation in Pediatric Anesthesia. INTERVENTIONS:Multisource feedback questionnaires were developed and then validated using face and content validity. Residents were randomly assigned to a feedback group or a control group. Both groups received the MSF evaluation. Only the group assigned to feedback had a 'coaching meeting' every month creating strategies for improvement. MEASUREMENTS: MSF questionnaires were validated using a face validation and expert content validity. The effect of MSF on a professionalism questionnaire was assessed using analysis of covariance and linear mixed effects regression models. MAIN RESULTS: Observed test-retest agreement was greater than 0.90 for all items, with more than half of kappa statistics greater than 0.50. Cronbach's alpha was 0.71.The MSF increased the self-assessment score with an estimated effect of 0.21 (95% CI 0.06, 0.37), P=.015. There was no detected effect on patient family evaluation, with mean difference (CI) in change from baseline of 0.03 (-0.15, 0.21), P=.77, faculty evaluation, 0.21 (-0.02, 0.44), P=.08, or coworker evaluation 0.13 (-0.11, 0.37). CONCLUSIONS: Our new multi-source feedback questionnaire to assess professionalism had good reliability and internal consistency. Using our validated questionnaire we assessed the effect of a monthly feedback to improve professionalism in anesthesia residents. While we did see improvement in anesthesiology residents' self-assessment, we did not see a similar effect on patient family, faculty or coworker evaluations.
RCT Entities:
STUDY OBJECTIVE: To assess professionalism in anesthesiology residents, it is important to obtain evaluations from people with whom they interact on daily basis. The purpose of this study was to evaluate the effect of a Multisource feedback (MSF) on resident's professional behavior and to assess the effect of faculty feedback on resident performance. DESIGN: This study was a two-group randomized clinical trial. SETTING: Residents were recruited from Cleveland Clinic Children's Hospital. PATIENTS: Participants included twenty eight residents doing a two-month rotation in Pediatric Anesthesia. INTERVENTIONS: Multisource feedback questionnaires were developed and then validated using face and content validity. Residents were randomly assigned to a feedback group or a control group. Both groups received the MSF evaluation. Only the group assigned to feedback had a 'coaching meeting' every month creating strategies for improvement. MEASUREMENTS: MSF questionnaires were validated using a face validation and expert content validity. The effect of MSF on a professionalism questionnaire was assessed using analysis of covariance and linear mixed effects regression models. MAIN RESULTS: Observed test-retest agreement was greater than 0.90 for all items, with more than half of kappa statistics greater than 0.50. Cronbach's alpha was 0.71.The MSF increased the self-assessment score with an estimated effect of 0.21 (95% CI 0.06, 0.37), P=.015. There was no detected effect on patient family evaluation, with mean difference (CI) in change from baseline of 0.03 (-0.15, 0.21), P=.77, faculty evaluation, 0.21 (-0.02, 0.44), P=.08, or coworker evaluation 0.13 (-0.11, 0.37). CONCLUSIONS: Our new multi-source feedback questionnaire to assess professionalism had good reliability and internal consistency. Using our validated questionnaire we assessed the effect of a monthly feedback to improve professionalism in anesthesia residents. While we did see improvement in anesthesiology residents' self-assessment, we did not see a similar effect on patient family, faculty or coworker evaluations.
Authors: John W Ragsdale; Andrea Berry; Jennifer W Gibson; Christiane R Herber-Valdez; Lauren J Germain; Deborah L Engle Journal: Med Educ Online Date: 2020-12