In-person versus virtual suturing and knot-tying curricula: Skills training during the COVID-19 era.

BACKGROUND: The coronavirus disease 2019 restrictions on in-person simulation activities necessitated modifying the traditional boot camp skills curriculum for matriculating PGY1 general surgery residents to a virtual format. This study investigated the relative effectiveness of in-person versus virtual instruction on the development of suturing and knot-tying skills.
METHODS: In all, 55 residents participated in a validated, proficiency-based, preinternship skills curriculum, 26 in 2019 and 29 in 2020. Both groups received an introduction to the curriculum, were given time for practice and offered one-on-one tutoring by faculty by request, and completed a filmed posttest. The 2019 class received in-person instruction during a boot camp at the end of June, while the 2020 class was provided with suture kits and received instruction via Zoom throughout June. The 2 groups were compared by post-test performance, date of task proficiency, and additional coaching required.
RESULTS: In 2019, 5.7% of the posttest tasks were graded as proficient versus 87% in 2020. The 2020 class outperformed the 2019 class on every metric in every task (P < .001). In 2019, faculty spent 55.5 hours with residents in one-on-one remediation, resulting in proficiency in 64% of the tasks by November. In 2020, 18 hours of one-on-one remediation resulted in proficiency in 92% of the tasks by September.
CONCLUSION: Learners who received virtual instruction with access to materials at home gained proficiency in suturing and knot-tying skills earlier and with less coaching. These data demonstrate that the virtual curriculum is effective and an improvement on the previous in-person curriculum.

Background

Recent efforts such as the development of the Resident Prep Curriculum by the American College of Surgeons, Association of Program Directors in Surgery, and Association for Surgical Education have pushed to standardize and improve the preparation that fourth-year medical students receive for surgical residencies. However, there remains large variability in the structure and assessment of skills training, and evidence for the persistence of skills is lacking.1, 2, 3 Accordingly, many general surgery residency programs hold intern boot camps for incoming PGY1 residents to review skills and verify basic suturing and knot-tying proficiency.

Many activities in medical education have been affected by the coronavirus disease 2019 (COVID-19) pandemic, but those that require in-person and hands-on instruction, such as intern boot camps, have been especially impacted. In July 2020, social distancing guidelines at the University of Texas (UT) Southwestern Medical Center precluded holding an in-person intern boot camp, necessitating the transition of the existing program to a virtual format. The objective of this study was to assess the efficacy of surgical training using self-directed study with faculty-facilitated instruction via Zoom video conferencing service (Zoom Video Communications, Inc, San Jose, CA) versus traditional in-person boot camp instruction by comparing the performance of the 2020 PGY1 matriculating class to the 2019 class on the boot camp posttests.

Methods

Participants

This study was approved by the institutional review board at UT Southwestern Medical Center, and 55 general surgery interns participated as part of their curriculum. The intern class of 2019 contained 13 categorical residents and 13 preliminary surgery residents. The intern class of 2020 contained 13 categorical residents, 15 preliminary surgery residents, and 1 integrated vascular resident. Both resident classes had comparable recruitment processes, scores, and experience. The Institutional Review Board at UT Southwestern determined this to be an exempt study because the data were collected for program quality assessment and improvement purposes (STU-2020-0212).

Curriculum and performance metrics

The open surgery skills curriculum was previously developed by Scott et al and was determined to demonstrate the ability to differentiate trainee and expert performance (construct validity). The curriculum is composed of 12 tasks that assess needle driver handling, 1- and 2-handed knot-tying skills, and interrupted and running suturing skills (Table I ).

Table I

Task breakdown of the open skills curriculum

Task	Description
1	Palm Needle Driver
2	Knot Tying, No Tension, 2-Handed
3	Knot Tying, No Tension, 1-Handed
4	Knot Tying, Under Tension, 2-Handed, Surgeon’s Knot
5	Knot Tying, Under Tension, 2-Handed, Slip Knot
6	Knot Tying, Under Tension, 1-Handed, Slip Knot
7	Suturing, Interrupted, Simple
8	Suturing, Interrupted, Horizontal Mattress
9	Suturing, Interrupted, Vertical Mattress
10	Suturing, Running, Simple
11	Suturing, Running, Subcuticular
12	Suturing, Interrupted, Subcuticular

Based on methods from Scott et al, each task was graded by time to completion and by specific errors made, both of which were used to calculate a composite score. Trainees were considered proficient in a task when their composite score exceeded a threshold specific to each task, also determined by Scott et al. Historically, trainees have had the most difficulty with the running suturing tasks, so these tasks were also scored according to a modified Objective Structured Assessment of Technical Skills (OSATS) scale to guide feedback during the remediation sessions. The scale involved a combination of 5-point Likert scales in 4 categories for a maximum of 20 points: respect for tissue, time and motion, instrument handling, and knowledge of procedure. There was no passing threshold for the modified OSATS score. The time to completion, errors, and modified OSATS were assessed by a blinded, non-MD grader trained in video-based assessment of surgical tasks.

Materials

In addition to needle drivers, forceps, scissors, and suturing materials, the curriculum requires a knot-tying board, a Fundamentals of Laparoscopic Surgery (FLS) Penrose drain, and a suturing model. Tasks 1 to 6 were performed on the knot-tying board, tasks 7 to 9 were performed on the FLS Penrose drain, and tasks 10 to 12 were performed on the suturing model. A Dog Abdominal Surrogate for Instructional Exercise model (DASIE; DASIE Surgical Training Tools; Guelph, ON, CA) was used in 2019, and the VATA Suture Skills Trainer (VATA, Inc, Canby, OR) was used in 2020. The VATA model was chosen in 2020 because it was found to more closely simulate live tissue. Each of these models is shown in Figure 1 .

Fig 1

Required models for the open skills curriculum. Clockwise from top left: knot-tying board, FLS Penrose drain, VATA Suture Skills Trainer (used in 2020), and DASIE model (used in 2019).

Training and conditions

In 2019, 40 learners from the General Surgery; Oral and Maxillofacial Surgery; Urology; and Ear, Nose, and Throat residency programs participated in the intern boot camp, which paralleled the described American College of Surgeons, Association of Program Directors in Surgery, and Association for Surgical Education Resident Prep Curriculum with additional sessions tailored to our institution. The open skills curriculum was introduced during a 1-hour session on June 24 in the UT Southwestern Simulation Center. The faculty simulation director led the session with the assistance of instructional videos previously recorded by Scott, et al. Links to these videos were provided to the learners at this time. The residents completed an additional 4 hours of practice with the faculty simulation director with assistance from simulation staff during the week of boot camp. Three of the tasks were also practiced and tested during “Top Knot,” a simulation knot-tying competition conducted at the end of the boot camp. The residents were offered voluntary 1-hour 1:1 tutoring sessions with the faculty simulation director before the posttest, but zero residents accepted this offer. The posttest for this curriculum was conducted on July 17 with only the 26 General Surgery residents. The UT Southwestern Simulation Center was open for voluntary practice throughout the entire period between curriculum introduction and post-test.

In 2020, owing to campus-wide restrictions on in-person activities in response to the COVID-19 pandemic, the boot camp was reduced to half of the previous year’s sessions and conducted over Zoom throughout June. Only the 29 General Surgery residents participated in the virtual program. Personal VATA Suture Skills Trainers were sent to each resident by May 30, along with FLS vessel loops, Velcro strips for fixation of the FLS vessel loops, and extra silk ties and sutures. Additionally, phone stands were sent to each resident to facilitate filming their attempts on each task for feedback from faculty.

The curriculum was divided into three 1-hour sessions and introduced via Zoom starting on June 1. New instructional videos of the tasks were distributed to the residents after each session. These new videos were filmed with over-the-shoulder and frontal toward-the-performer perspectives and demonstrated each task performed within passing criteria without interruption. The original instructional videos paused at various points throughout each task to convey technical instruction, and it was believed that watching a continuous passing attempt would facilitate complete understanding of each task given the virtual learning environment. Additionally, 3 virtual tutoring sessions with a 1:4 faculty-to-resident ratio were held before the start of boot camp.

In-person boot camp with proper social distancing measures began on June 22. As in the previous year, the residents were offered voluntary 1-hour 1:1 tutoring sessions with the faculty simulation director. Nine residents completed Zoom tutoring sessions and 9 residents completed in-person tutoring sessions at the UT Southwestern Simulation Center. The in-person sessions were held with social distancing and sterilization precautions and proper personal protective equipment was worn at all times. The posttests were held in-person on July 26 and 29. A comparison of the training conditions and curriculum structure in 2019 and 2020 are shown in Figure 2 .

Fig 2

Comparison of administration of the open skills curriculum in 2019 and 2020.

In both 2019 and 2020, the posttests were filmed in a manner that deidentified the performer except for a preassigned resident identification number (Fig 1, bottom left). To help those residents who did not pass each task to reach proficiency, weekly protected time for skills practice was provided. In addition, 1:1 tutoring sessions with the faculty simulation director were made available by appointment. Further posttests were held with individual residents once improved performance had been demonstrated to the simulation director.

Statistical analysis

The post-test video was graded retrospectively by video review. The time to completion, errors, and composite score were assessed for all 12 tasks, and modified OSATS were assessed for tasks 10 and 11. Mann-Whitney U tests employing 2 tails were used to analyze differences in time, composite score, and modified OSATS between the 2 cohorts. χ2 test was used to determine if the virtual curriculum was associated with more trainees achieving proficiency on all 12 tasks.

Results

More residents achieved proficiency in all 12 tasks following the virtual curriculum in 2020 (P = .008). In 2019, 0 residents (0.0%) achieved proficiency in all tasks. When considering the tasks in aggregate, 18 out of 312 tasks (5.7%) were marked as proficient. In 2020, 8 residents (27.6%) achieved proficiency in all tasks. In aggregate, 302 out of 348 tasks (86.8%) were marked as proficient. In 2020, decreased time to completion and increased score were observed for all tasks (P < .001), and increased modified OSATS scores were observed for the running suturing tasks (P < .001). A comparison of the median time to completion, score, and modified OSATS for each task are shown in Table II .

Table II

Comparison of posttest performance for the 2019 vs 2020 cohorts

Task	Median time			Median score			Median OSATS
	2019	2020	P value	2019	2020	P value	2019	2020	P value
1	8	6	<.001	52	54	<.001
2	20	9	<.001	37	51	<.001
3	19.5	8	<.001	38.5	52	<.001
4	28	11	<.001	26	48	<.001
5	26	13	<.001	24	47	<.001
6	22	12	<.001	31	48	<.001
7	42	17	<.001	75	103	<.001
8	70.5	28	<.001	45	92	<.001
9	69.5	29	<.001	50	91	<.001
10	270	149	<.001	316	451	<.001	14	16	<.001
11	394	189.5	<.001	206	410.5	<.001	13.5	16.5	<.001
12	72	30	<.001	48	90	<.001

OSAT, Objective Structured Assessment of Technical Skills.

In 2019, the faculty simulation director spent 55.5 total hours in 1:1 remediation with the residents after the posttest, resulting in 64% of the tasks marked as proficient by November. In 2020, 18 hours were spent in remediation with the residents after the posttest, resulting in 92% of the tasks marked as proficient by September.

Discussion

The principal difference between the curriculum each class experienced was the timing of the instruction and the teaching modality. While the time that the 2020 cohort spent training before matriculation is technically voluntary time, attendance was excellent. Only 2 sessions were missed by 2 separate residents owing to travel. In aggregate, 80% more tasks were successfully completed in 2020 than in 2019, and the 2020 cohort outperformed the 2019 cohort in every metric on every task. Furthermore, the simulation director spent 37.5 more hours in remediation with the 2019 cohort through November, and yet 28% fewer tasks were successfully completed at this point compared with the 2020 cohort. Additionally, at the time of the posttest, staff perceived that the 2020 cohort demonstrated improved performance.

It is clear from these findings that the virtual curriculum was more effective at achieving proficiency in open suturing and knot-tying skills than the previous in-person curriculum. Based on feedback from the residents, we believe the improved performance is not necessarily owing to the virtual format but rather to having access to materials and instruction before matriculation when clinical duties compete for practice time. Additionally, residents indicated that the new instructional videos demonstrating real-time passing attempts from 2 points of view increased their understanding of the level of performance required to pass. Owing to the favorable findings and feedback, this virtual curriculum will be used in 2021 and is strongly being considered for use in the post–COVID-19 era. We believe that holding the skills training before matriculation will allow residents to accumulate more practice time and better prepare them for the surgical duties of internship.

This study has some limitations. First, it is possible that not all of the improved performance observed in the 2020 cohort can be attributed to the virtual curriculum, and some improvement is owing to what some have called “the COVID effect.” Owing to the restrictions on travel and in-person gatherings in June 2020, the residents may have devoted more time to training than they otherwise would have in their last few weeks before matriculation. Although we did not measure residents’ practice time, and therefore cannot assess the effect of this factor in this study, we plan to observe what effect, if any, relaxed social distancing guidelines have on performance when this curriculum is repeated in 2021. Second, it is possible that the residents in 2020 received more 1:1 instruction than those in 2019 because the virtual program only included General Surgery residents. The total time spent in sessions, excluding the voluntary 1:1 tutoring sessions in 2020, was roughly equivalent between the 2 years, as was the time between curriculum introduction and posttest. Therefore, the effect of any small increase in 1:1 instruction is considered to be minimal. Third, the suturing model for tasks 10, 11, and 12 was changed for the 2020 cohort. Although this change may have affected performance in a positive or negative fashion, improved performance was consistently seen across every task.

As stated above, the 2020 curriculum will be used in 2021 and is being considered for use in the years afterward. There are no plans to alter the structure of the curriculum or the format of the individual sessions. The primary change needed is to transition to another teleconferencing application with more advanced and user-friendly annotation features. This would facilitate teaching in the virtual setting, where it is impossible to physically alter the trajectory of the trainees’ hands or adjust their grip on an instrument. In the distant future, we hope to develop a platform that allows trainees to upload videos of their performance and receive immediate, automated feedback regarding whether or not they have demonstrated proficiency on a task. We believe these improvements will improve both the teaching experience for the faculty and the training experience for the residents.

Funding/Support

The Department of Surgery at UT Southwestern Medical Center paid for all of the equipment used in this study.

Conflicts of interest/Disclosure

The authors have no conflicts of interest or disclosures to declare.