Feasibility of Smartphone Application in Plastic Surgery Operative Assessments.

BACKGROUND: The evaluation model of operative competence is based on aggregate tabulations of procedures and end-of-rotation feedback from faculty members. Procedural tabulations do not detail the level of resident involvement in the case, and end of rotation feedback is infrequent and inaccurate due to the necessity of long-term recall. Smart phone-based evaluation systems provide residents with immediate and permanent feedback for surgical encounters. In this study, we examine the feasibility of smart phone-based evaluations in plastic surgery residency.
METHODS: This was a 6-month prospective, single institution pilot study at three teaching hospitals, assessing all PGY levels. We utilized our department mobile application (Wayne State University Surgery Department application), which includes intraoperative evaluations based on the Zwisch scale. Prestudy and poststudy surveys were conducted. An unstructured interview of the Clinical Competency Committee provided feedback for the new evaluation tool against the previous evaluation forms.
RESULTS: Eleven physicians participated in the study, resulting in 126 encounters and 184 procedures. A 10-question prestudy survey was given with answers ranging from 1 (strongly disagree) to 5 (strongly agree). The Clinical Competency Committee faculty ranked the prestudy resident assessment tool 2.82 of 5, whereas the poststudy survey scored 4.64 of 5.
CONCLUSIONS: Residents and faculty both rated the smartphone application as a useful tool for evaluating residents. The success of the application proves its feasibility within plastic surgery residency and may play an important role in rating resident operative competency in the future.

Takeaways

Question: Can smartphone-based real-time operative evaluation be employed in plastic surgery training?

Findings: In this prospective study, the use of smartphone application was evaluated with high satisfaction and reproducibility among plastic surgery trainees and faculty members.

Meaning: Smartphone-based operative evaluation is feasible in plastic surgery training.

INTRODUCTION

The primary objective of any surgery residency is to have its trainees develop the medical knowledge and surgical skills that are required to function as successful independent physicians. Residents achieve this objective through self-directed learning, directly caring for a high volume of patients, and through the sharing of knowledge by their predecessors. The first successful training programs in the United States during the 19th century utilized the Halstedian approach to teach their trainees, which was a time-based approach to residency that has remained the dominant structure for training for many years.[1] Today, the Halstedian framework for training remains; however, there have been many sociopolitical changes that make preparedness for independent practice more difficult to attain. These changes include work hour restrictions, decreased operative autonomy, increased productivity pressures among staff physicians, and a cultural shift toward measuring and improving medical outcomes. To overcome these challenges, the ACGME has developed a set of competencies for surgical residents which should be met by the time of graduation, to better individualize training in the hopes of ensuring surgeons are adequately prepared for transition to independent practice.

Operative competence is an assessment of particular interest within surgery. Technical skill will likely directly impact patient outcomes, and autonomy is frequently cited as a factor for resident readiness.[2] Autonomy is more frequently granted when a resident has demonstrated proficiency at an operation. Intraoperative evaluations are a tool which can be used to tailor autonomy and responsibility to the skill of the resident, across varying operations and attending surgeons. In the current training model, competence is determined by self-logged cases where residents must meet a minimum case number to graduate. The present general surgical guidelines indicate that residents must meet a minimum of 850 major cases to graduate.[3,4,5] This metric does not establish a resident’s level of involvement and performance on the case.[8] Additionally, verbal feedback on case performance traditionally occurs in an end-of-rotation aggregate. This is less desirable as long-term recall is less accurate, is usually presented in a generalized manner nonspecific to single-operative cases, and cannot be stored for review at a later date. Several tools of evaluation have already been described in the literature to fill the void of operative evaluations. These include the Zwisch scale, OSATS, and Ottawa scoring..[6,7,9,10] Advantages of evaluation tools include that they can be stored for an extended period of time, can be performed for each case, provide insight into the level of assistance necessary for residents intraoperatively, and provide a standardized method for evaluating all residents which can later be translated to assess overall operative competency. Our institution utilizes the System for Improving and Measuring Procedural Learning (SIMPL) application, which is based on the Zwisch scale. A SIMPL evaluation requires three pieces of information: the level of help the trainee required rated by the Zwisch scale, a performance rating, and a rating of the complexity of the case. (See figure, Supplemental Digital Content 1, which displays (A) home screen of the Wayne State University Surgery Department application; (B) start of an intraoperative evaluation requiring patient identifier to associate with an operative case; (C) multiple procedures may be evaluated at once after selection; (D) screenshot of application illustrating procedure complexity and the level of help required by the resident during primary cleft repair; (E) screenshot of application illustrating procedure complexity and the level of help required by the resident during secondary cleft repair and rhinoplasty; and (F) screenshot of application illustrating procedure complexity and the level of help required by the resident during craniomaxillofacial reconstruction, http://links.lww.com/PRSGO/B908.) Validity was not the focus of this article; however, both the Zwisch scale and SIMPL app have previously been the attention of trials which illustrate their practicality among other specialties.[9,11]

In this study, we utilize our novel departmental mobile application (Wayne State University Surgery Department application) to grade intraoperative competence of plastic surgery residents. The WSU Surgery application has been customized to allow evaluations through the SIMPL application, based upon the Zwisch scale. Frequent feedback is important within the context of intraoperative competency, and here, we will prove feasibility of a novel smartphone evaluation for plastic surgery assessment. To our knowledge, this is the first investigation into the use of smartphone evaluations among plastic surgeons.

METHODS

This study was a prospective pilot study that was designed to assess the feasibility of obtaining real-time evaluations of plastic surgery residents in a single plastic surgery program, using a customized innovative smartphone application. It occurred over a 6-month period (November 2018 through May 2019). Residents of all training levels were evaluated in this study. Evaluations were collected at three teaching hospitals with different faculty members. Many common plastic surgery procedures of varying complexity were included in the study. Self-evaluation and self-reflection were promoted as an important part of the process.

The Zwisch scale was chosen for this study because it has previously been evaluated as both a feasible and reliable tool for intraoperative assessment.[9,11] At the start of the study, a 1-hour frame of reference training was provided to application users on how to use the smartphone application and the Zwisch grading system. This training session provides education about the application but also has been illustrated to ensure rater reliability.[13] Faculty members were not given any further incentive or reminders about evaluations, except when notified by the operating resident. The Zwisch evaluation is completed in a timely manner and correlates strongly with intraoperative performance.[9]

The study began with a survey of faculty opinions before the launch of the smartphone-based evaluation system. It assessed the perceived usefulness, ease of use, weaknesses, and barriers of the smartphone evaluations. A poststudy survey of the faculty was similarly conducted and included the same parameters. Data were collected and interpreted from the pre and postsurveys. Additionally, an unstructured interview of the Clinical Competency Committee was performed to compare the effectiveness of the feedback provided from the new evaluation tool against the previous evaluation forms in assessing clinical competency.

RESULTS

A 10-question prestudy survey with answers ranging from 1 (very unfavorable and strongly disagree) to 5 (very favorable and strongly agree) was conducted. The Clinical Competency Committee faculty ranked the prestudy resident assessment tool just above average with an average score of 2.82 of 5. On the poststudy survey, with the addition of information from the Zwisch scoring mobile application, the same faculty ranked the evaluation tool very favorably with an average score of 4.64 of 5. (See figure, Supplemental Digital Content 2, which displays a visual representation of clinical competency committee survey scores, before and after use of the application, http://links.lww.com/PRSGO/B909.) On interview, the faculty described the addition of Zwisch scoring as an “excellent adjunct of objective data to assess resident surgical skill/progress” but encourage evaluators to “add more comments on each case” if possible.

DISCUSSION

The traditional form of feedback within surgery residency relies upon verbal, end-of-rotation feedback. This is less accurate, less specific, and is infrequently performed. With residencies moving toward more individualized assessments of their residents, and increased sociopolitical pressures within our surgical training environment, programs require a set of tools to evaluate residents that are accurate and valid. Additionally, these tools should be easy to use, widely accessible, and rapid.

Our department has completed a study to evaluate the feasibility of real-time, smartphone-based, operative evaluations within plastic surgery residency. The SIMPL application has already been evaluated within the context of general surgery; however, this is the first study to be deployed within a plastic surgery setting. Data from the general surgery literature illustrated that the average time to complete a study was approximately 27 seconds, and that the majority of evaluations were completed within the first 24 hours following the operation.[11] Our faculty averaged less than 1 minute for completing evaluations, suggesting that these can be performed without significant disruption to daily workflow.

The SIMPL application has also been deployed in specialties aside from general surgery with good results. Among oral and maxillofacial residents, 71% of procedures performed over the study time period were completed with an associated SIMPL assessment.[12] Our results again echo those previously published, as 184 procedures were performed during our time period of data collection (Table 1). One hundred twenty-six SIMPL evaluations were associated with these procedures, representing a 68.4% SIMPL completion rate. These numbers indicate that operative evaluations can be easily performed for the majority of cases; however, it begs the question of how participation can be increased in the future. This could occur for several reasons, including technology limitations and user limitations. For example, hospitals are often large buildings with operating rooms occasionally located within basement or ground levels. Phones may have limited access to cellular data or Wifi connectivity in these locations. If poor service is encountered, it could affect participation. Additionally, although unlikely in today’s culture, all participants may not possess a smartphone capable of installing applications.

Table 1.

Tabulated Procedures

Procedure	Freq
Abdominoplasty	8
Brachioplasty	2
Breast augmentation	1
Breast reconstruction with implant/tissue expander	16
Breast reduction	19
Burn reconstruction—extremity including hand	10
Burn reconstruction—face, eyelid, lips, nose, and ears	1
Craniomaxillofacial reconstruction	2
Fat grafting—breast	2
Fat grafting (absent breast)	6
Lasers	2
Liposuction-trunk	4
Mastopexy	2
Nerve decompression—digital	4
Nerve decompression—median at wrist (carpal tunnel)	5
Nerve decompression—other	2
Nerve decompression—ulnar at elbow	2
Open treatment of Dupuytren contracture	2
Operative repair of fracture or dislocation—CRPP	2
Operative repair of fracture or dislocation—ORIF	1
Other hand procedures (other deformity/disease process including trigger finger, foreign bodies, biopsies, and abscesses)	14
Other head and neck reconstructive procedures (including bone graft harvest)	1
Other procedures (head and neck neoplasms)	2
Other procedures for head and neck aesthetic deformity	1
Other procedures for head and neck congenital defects (including alveolar grafting, distraction, and orthognathic)	2
Recon after head/neck neoplasm resection w/local flap	1
Reconstruction hand/upper extremity—skin graft	5
Reconstruction hand/upper extremity—primary closure	1
Release of joint contracture	4
Repair/reconstruct extensor tendon with or w/o graft	1
Repair/reconstruct nerve with or w/o graft	1
Rhinoplasty	1
Secondary cleft lip or palate repair	1
Tenolysis/tendon lengthening	5
Tissue expansion (other than breast)	1
Treat benign lesions of skin	7
Treat congenital deformity of hand (including syndactyly)	6
Treat malignant lesions of skin (including Moh’s reconstruction)	4
Treat nasal fracture	1
Treat neoplasm (benign or malignant including ganglion cysts, giant cell tumors, and mucous cysts)	3
Treat other deformities of breast (reconstructive incl nipple reconstruction, aberrant breast, Poland syndrome, and transgender)	4
Treat other deformities of lower extremity	2
Treat other trunk deformities (including debridement, skin graft/substitute, excision soft-tissue tumors, and perineal/genital reconstruction)	3
Treat pressure ulcer with flap	1
Treat pressure ulcer without flap (primary closure/graft)	1
Treat upper midface fracture—frontal sinus	1
Treat upper midface fracture—Orbit ORIF	1
Treat upper midface fracture—Le Fort 2/3	2
Treat upper midface fracture—ZMC	1
Treatment lower extremity wound with graft	8
Treatment lower extremity wound with local flap	4
Vascular malformation	1

CRPP, closed reduction percutaneous pinning; ORIF, open reduction internal fixation

The availability of electronic assessments provides trainees with constructive feedback that can be used to improve future performances. The majority of trainees (92.9%) feel that general intraoperative feedback is helpful; however, 77% of trainees indicate that they infrequently receive feedback in the current training environment.[14] One could imagine compiling these assessments into a database to correlate resident performance overtime, which could help assess intraoperative progress on an individualized case basis.

Potential user limitations can include knowledge of how to use the application and knowledge of the application’s existence. In reviewing a multi-institutional trial evaluating SIMPL within the general surgery residencies, it is interesting to note that one program vastly outperformed other surgery programs. The most productive program was able to complete 2006 evaluations, whereas the second most productive program provided 729 evaluations. Therefore, the most engaged program was able to more than double all the other programs involved in the study.[11] All institutions participating in this trial were relatively large centers, and if the variation in case evaluations was not due to variation in case load itself, perhaps individual behavioral characteristics play a role in increased use of the application.

In our study, faculty members were not given reminders to fill out SIMPL evaluations. They were not additionally incentivized in any manner as this would bias interpretation of feasibility. Perhaps, in a future study examining the validity and reliability of intraoperative evaluations, programs could increase their rates of participation with faculty goal setting and “push notifications” from the application to increase response rates.

If the SIMPL application is to be used to evaluate residents regularly, faculty incentivization or goal setting may become increasingly important because of their function as evaluators. Described elsewhere, residents initiate assessments more frequently, have higher response rates, and perform more evaluations on average, compared to staff physicians.[11] Although residents complete evaluations of themselves frequently, multiple viewpoints from several evaluators would give additional data points for competency tracking. Separate assessment from multiple evaluators will also increase the statistical power of the assessment and is more likely to avoid the “hawk-dove effect,” in which residents are more likely to avoid tough evaluations and seek out those faculty members who are more benign.[14] One could then imagine a certain database of surgical operations and evaluations which the department leadership could access while reviewing residents on the routine semiannual basis.

With the evolution of technology, we now have means to obtain real-time surgical evaluations. Here, we illustrated the feasibility of a novel smartphone application combined with intraoperative evaluations within the context of plastic surgery residency. Feasibility was illustrated by active engagement on a majority of operations, completion of evaluations in a timely manner, and an increase in user desire for the application between pre and poststudy questionnaires. Regardless of practice pressures and changes to resident training, a smartphone application is a great adjunct in the role of intraoperative performance. Evaluations on a smart phone, similar to those done in the present study may potentially be used in the future to monitor residents’ progress.

CONCLUSION

Our study demonstrates the feasibility of smartphone assessments within plastic surgery using a novel smartphone application. Additionally, our results indicate a drastic increase in user desire, as well as ease of use of a smartphone application, with an increase from our prestudy score of 2.82 to our postsurvey score of 4.64 of 5 with the smartphone application.