John M Green1. 1. Department of Surgery, Carolinas HealthCare System, Charlotte, North Carolina. Electronic address: greenjm14@gmail.com.
Abstract
PURPOSE: Timely performance evaluation and feedback are critical to resident development. However, formulating and delivering this information disrupts physician workflow, leading to low participation. This study was designed to determine if a locally developed smartphone platform would integrate regular evaluation into daily processes and thus increase faculty participation in timely resident evaluation. METHODS: Formal, documented resident operative and patient interaction evaluations were compiled over an 8-month study period. The study was divided into two 4-month phases. No changes to the existing evaluation methods were made during Phase 1. Phase 2 began after a washout period of 2 weeks and coincided with the launch of a smartphone-based platform. The platform uses a combination of Likert scale questions and the Dreyfus model of skill acquisition to describe competence levels in technical and nontechnical skills. The instrument inflicts minimal effect on surgeon workflow, with the aim of integrating resident evaluation into daily processes. The number of different faculty members performing evaluations, resident level (postgraduate year), type of interaction or procedure, and competency data were compiled. All evaluations were tracked by the program director as they were automatically uploaded into a database. Faculty members were introduced to the new platform at the beginning of Phase 2, and previous methods of evaluation continued to be encouraged and were considered valid throughout both phases of the study. Data were analyzed using Fisher exact test for specific PGY level, and chi-square test was used for overall program analysis. Statistical significance was set at p < 0.05. RESULTS: Total faculty engagement, that is, number of faculty members completing evaluations, increased from 13% (5/38) in Phase 1 to 53% (20/38) in Phase 2. During Phase 1, all evaluations consisted of online forms through the department's established system or e-mails to the program director. Evaluations were completed in 0.9% (15/1599) of cases residents completed in Phase 1 versus 12% (217/1812) of those in Phase 2. During Phase 2, evaluations were conducted exclusively using the new platform. This was done based on participant's choice. Total numbers of residents and core faculty members did not change between Phases 1 and 2. CONCLUSIONS: A smartphone-based platform can be created with existing technology at no cost. It is adaptable and can be updated in real-time and can employ validated scales to build an evaluation portfolio for learners assessing technical and nontechnical skills. Furthermore, and perhaps most importantly, it can be designed to integrate into existing workflow patterns to increase faculty participation. Copyright Â
PURPOSE: Timely performance evaluation and feedback are critical to resident development. However, formulating and delivering this information disrupts physician workflow, leading to low participation. This study was designed to determine if a locally developed smartphone platform would integrate regular evaluation into daily processes and thus increase faculty participation in timely resident evaluation. METHODS: Formal, documented resident operative and patient interaction evaluations were compiled over an 8-month study period. The study was divided into two 4-month phases. No changes to the existing evaluation methods were made during Phase 1. Phase 2 began after a washout period of 2 weeks and coincided with the launch of a smartphone-based platform. The platform uses a combination of Likert scale questions and the Dreyfus model of skill acquisition to describe competence levels in technical and nontechnical skills. The instrument inflicts minimal effect on surgeon workflow, with the aim of integrating resident evaluation into daily processes. The number of different faculty members performing evaluations, resident level (postgraduate year), type of interaction or procedure, and competency data were compiled. All evaluations were tracked by the program director as they were automatically uploaded into a database. Faculty members were introduced to the new platform at the beginning of Phase 2, and previous methods of evaluation continued to be encouraged and were considered valid throughout both phases of the study. Data were analyzed using Fisher exact test for specific PGY level, and chi-square test was used for overall program analysis. Statistical significance was set at p < 0.05. RESULTS: Total faculty engagement, that is, number of faculty members completing evaluations, increased from 13% (5/38) in Phase 1 to 53% (20/38) in Phase 2. During Phase 1, all evaluations consisted of online forms through the department's established system or e-mails to the program director. Evaluations were completed in 0.9% (15/1599) of cases residents completed in Phase 1 versus 12% (217/1812) of those in Phase 2. During Phase 2, evaluations were conducted exclusively using the new platform. This was done based on participant's choice. Total numbers of residents and core faculty members did not change between Phases 1 and 2. CONCLUSIONS: A smartphone-based platform can be created with existing technology at no cost. It is adaptable and can be updated in real-time and can employ validated scales to build an evaluation portfolio for learners assessing technical and nontechnical skills. Furthermore, and perhaps most importantly, it can be designed to integrate into existing workflow patterns to increase faculty participation. Copyright Â
Keywords:
Interpersonal and Communication Skills; Medical Knowledge; Practice-Based Learning and Improvement; Professionalism; competency; evaluation; feedback; non-technical skills; smartphone