Learning Fundamentals of Laparoscopic Surgery Manual Skills: An Institutional Experience With Remote Coaching and Assessment.

INTRODUCTION: COVID-19 has altered the landscape of traditional surgical education. This is a pilot study of remote coaching and assessment of Fundamentals of Laparoscopic Surgery (FLS) manual skills in obstetrics and gynecology residents.
MATERIALS AND METHODS: PGY-3 obstetrics and gynecology residents participated in remote assessment of FLS manual skills using a live streaming platform. Learners who showed deficiencies in proficiency participated in live-streamed coaching sessions. The coaching sessions continued until the learner and coach mutually agreed that the learner was prepared for the skills portion of the FLS certification exam. The primary outcome was learner performance on skills assessment with external validation through the FLS manual skills exam pass rate.
RESULTS: One learner demonstrated proficiency at baseline and the remaining nine underwent remote coaching sessions. Learners received a mean of two coaching sessions to reach mutually agreed readiness for the FLS exam as per learner and coach. All residents performed significantly better at the conclusion of the remote coaching series (11.3 ± 0.82) as compared to their baseline scores (8.8 ± 2.82) (P < .03; 95% CI, 0.31-4.69). Proficiency was externally validated through the FLS exam taken by each resident 1-2 weeks following their final assessment, with all learners passing the manual portion. The average satisfaction of learners with remote coaching was 77/100 (range 50-100). 100% of learners felt prepared for their FLS certification exam and 100% would recommend this remote training program to a colleague.
CONCLUSION: Remote coaching and assessment of FLS skills yields similar results to traditional face-to-face instruction. Published by Oxford University Press on behalf of the Association of Military Surgeons of the United States 2021. This work is written by (a) US Government employee(s) and is in the public domain in the US.

INTRODUCTION

As of May 2020, graduating obstetrics and gynecology residents are required to pass Fundamentals of Laparoscopic Surgery (FLS) before board certification.[1] This mandate required many programs to develop an FLS curriculum de novo, with little guidance on how to adapt to the bstetrics and gynecology resident population. Residency programs were confronted by an additional hurdle with the COVID-19 pandemic, with restrictions imposed for many forms of traditional face-to-face learning.

Before the pandemic, remote learning was being explored because of concerns for patient safety, cost effectiveness of training, and limitations of work hour restrictions. Cost and time are often significant barriers to development and implementation of new platforms.[2] Remote learning has been evaluated in surgical education for both didactic and psychomotor skills, with a systematic review finding that e-learning platforms are at least as effective as traditional methodology.[3] Web-based courses have been previously developed to assist with cognitive[4] and psychomotor[5] development of gynecology residents in laparoscopic skills. However, psychomotor skills in these studies were further refined by traditional face-to-face coaching after completion of these modules. These blended courses and simulation curriculums were growing in popularity, when the safety of continuing the “live sessions” were called into question by the COVID-19 pandemic.

With a longstanding FLS training program and passing rates averaging over 97% since implementation in 2010, our program did not have to alter surgical curriculum following the American Board of Obstetrics and Gynecology (ABOG) mandate. However, with traditional face-to-face instruction of routine educational activities curtailed because of COVID-19, we moved to a live streaming platform for FLS preparation for our PGY-3 residents. The objective of our study was to evaluate whether remote coaching and assessment of FLS manual skills achieves comparable outcomes to traditional face-to-face coaching in preparing PGY-3 residents for their FLS exam. We hypothesize that remote coaching and assessment will be similar to the traditional model in PGY-3 residents preparing for their FLS manual skills exam at our institution.

METHODS

In this prospective, pilot study conducted from June 30, 2020 to September 15, 2020, all 10 PGY-3 residents at the UPMC Obstetrics and Gynecology residency training program participated in remote assessment and, if needed, remote coaching in preparation for their FLS manual skills exam. This study was approved by the University of Pittsburgh School of Medicine Institutional Review Board on May 12, 2020 (IRB STUDY20040191).

In 2010, the Obstetrics and Gynecology residency program at UPMC Magee-Womens Hospital internally initiated a requirement that all graduating residents must pass FLS. Surgical education leadership identified the FLS product as a well-developed program to serve as the backbone of a longitudinal laparoscopic simulation curriculum. Fundamentals of Laparoscopic Surgery is comprised of both didactic and manual skills portions. The manual skills are comprised of a series of five laparoscopic tasks including peg transfer, precision cutting of a gauze circle, placement of an interrupted suture with extracorporeal and intracorporeal knot tying, and use of an endoloop to simulate a traditional approach to appendectomy. In a common space within the resident lounge, the residents have access to two FLS trainer boxes, FLS approved instruments, and a supply of the consumable materials including (Fig. 1A). Each resident is also given a home trainer box with supplies at the beginning of PGY-1.

FIGURE 1.

FLS simulation lab space for OB/GYN residents with remote coaching modification. (A) FLS simulation lab space for OB/GYN residents. (B) Remote coaching and assessment modification to simulation lab space. Phone stand was used to support learners’ phones to project the screen while completing tasks.

The longitudinal curriculum begins with PGY-1 residents who receive training under the guidance of minimally invasive gynecologic surgery (MIGS) fellows for 30-60 minutes a week during a 5-to-6 week rotation. Fundamentals of Laparoscopic Surgery training is typically done in couplets, with PGY-1 and PGY-2 residents working in tandem with the fellow on both FLS-specific tasks and other laparoscopic skills. Two months before taking the FLS exam, in the fall of their third year, PGY-3 residents participate in an FLS refresher course culminating in a competitive assessment (FLS Olympics). Each resident receives two face-to-face coaching and assessment sessions with the senior MIGS fellows for up to 60 minutes each. If a resident is flagged as being unprepared for their FLS manual skills exam or self-identifies as requesting additional help, they receive additional training during this time with the MIGS fellow.

Because of local COVID-19 restrictions, traditional face-to-face laparoscopic simulation educational sessions were put on hold in late March 2020. A pilot program with the PGY-1 residents was started for ad hoc FLS coaching using a live streaming platform with positive feedback from both learners and coach. These remote learning sessions required the addition of a phone holder to live stream the monitor to the coach via the learners’ smart phones (Fig. 1B). The decision was made to proceed with the PGY-3 refresher course and FLS Olympics utilizing remote learning given the uncertainty of COVID-19 restrictions and the need to ensure our residents were adequately prepared for the FLS exam.

Before starting the virtual training program, all learners were provided packets with supplies and confirmed to have trainer boxes in their possession. The PGY-3 residents underwent remote assessment on a live streaming platform using the learners’ cell phones to mirror the screen. These baseline assessment sessions were recorded for scoring. Scoring was completed using a scoring rubric by two independent graders (Appendix 1). The FLS scoring guide is not publicly available, and it is unknown how precision, time, and errors are accounted for in the development of the score. The scoring rubric utilized in this study was developed to allow objective grading of tasks and attempts to account for demerits during scoring, all in accordance with descriptions of successful task completion on the FLS website. To assist with validation of this scoring rubric, graders ranked videos by the level of proficiency. Maximum summative scores correlated with “expert” and “novice” videos. Feedback was given to learners at the conclusion of scoring. Learners who achieved a maximum skill score of 12 were exempted from coaching sessions.

Coaching sessions were performed via a live streaming platform with the senior MIGS fellow. During these sessions, the coach had access to a trainer box and supplies to demonstrate refinement in technique that may be difficult to verbalize. Coaching sessions continued until the learner and coach mutually agreed that the learner was prepared for her FLS manual skills exam. Each resident completed a final assessment ∼2 weeks before the FLS exam. This assessment was conducted via a live streaming platform and was recorded and graded using the same rubric as the baseline assessment described above. Residency leadership provided FLS test scores in order for study investigators to determine external validation. Surveys were sent to learners 2 weeks following their FLS exams to assess learner satisfaction and to receive feedback for future implementation of remote surgical education.

Continuous variables were reported as mean (SD) for normally distributed variables. Paired t-test was performed to evaluate change in learner performance. Kappa was used to assess interrater reliability. For comparative purposes, learners who were unable to perform all tasks at baseline were given an arbitrary score of 1,801 seconds given that 1,800 seconds is the maximum time allotted for completion of all FLS tasks in the skill manual. A sample size calculation was not performed; rather all participants were included. All statistical analyses were performed using STATA/SE v.15.0 software.

RESULTS

All 10 UPMC Magee obstetrics and gynecology PGY-3 residents completed baseline and final FLS assessments. Interrater reliability was 1 for all peg transfer, precision cut, and ligating loop metrics (Table I). Variation was observed on the intracorporeal and extracorporeal tasks with the black dot entry (κ = 0.8) and exit (κ = 0.75) and avulsion of the drain (κ = 0.95). One learner was assessed to be proficient at baseline, achieving a maximum skill score of 12, and therefore did not participate in remote coaching sessions. Of the nine residents that participated in the remote coaching sessions, a mean of two coaching sessions were completed before learner and coach mutually agreed that the resident was proficient in all FLS manual skills. These remote learning sessions ranged in length from 5 to 24 minutes with a mean of 14 minutes.

TABLE I.

Interrater Assessment of FLS Proficiency

Peg transfer (n = 20)	Κ
Task completed	1
Errors during task	1
Precision cut (n = 20)
Task completed	1
Reasons for not completing	1
Ligating loop (n = 20)
Task completed	1
Reasons for not completing	1
Suture with knot (n = 40)
Enter black dota	0.8
Exit black dota	0.75
Penrose toucha	1
Drain avulseda	0.9
…extracorporeal knot (n = 20)
Three extracorporeal knots	1
…intracorporeal knot (n = 20)
Surgeon’s knot first tie	1
Alternate hands second throw	1
Alternate hands third throw	1

Proficiency metrics that were the same across tasks were combined for interrater reliability assessment (i.e., all drain avulsion metrics from both intra- and extra-corporeal tasks were combined for assessment).

At baseline, four learners were unable to complete all five of the FLS tasks, while at their final assessment all learners were able to successfully complete all of the tasks. Residents performed significantly better at the conclusion of the remote coaching series (11.3 ± 0.82) as compared to their baseline assessment (8.8 ± 2.82, P < .03) (Fig. 2). Time to complete tasks was also assessed through summation of time to complete all tasks (Fig. 3). Learners’ times improved from an average of 1,202.6 seconds (SD 567.9) at baseline to an average of 605.9 seconds (SD 119.5) during their final assessment. Proficiency was externally validated through the FLS exam proctored 1-2 weeks following their final assessment, with all learners passing the manual skills portion on their first attempt.

FIGURE 2.

Composite FLS scores during baseline and final assessment. Bold line denotes average score with brackets depicting standard deviation.

FIGURE 3.

Summative time to complete FLS tasks during baseline and final assessment. Bold line denotes average time to complete all FLS tasks with brackets depicting standard deviation. Learners who were unable to complete all tasks were given a time of 1,801 seconds (1,800 seconds is the maximum time allotted in the FLS manual skills guide).

Learner satisfaction and feedback was elicited at the completion of her FLS exam. Eight of the ten learners completed the survey. The average satisfaction with remote coaching was 77/100 on a sliding scale with 0 being completely unsatisfied and 100 being completely satisfied. Learners rated remote coaching as highly convenient with a mean score of 86.8/100 on the same sliding scale. Learner self-assessment in skills proficiency was a mean of 79.3% improvement. 100% of learners felt prepared for their FLS certification exam and 100% would recommend this remote training program to a colleague.

DISCUSSION

We found that the FLS manual skills pass rate (100%) of PGY-3 residents following a remote coaching and assessment program was consistent with the performance of past years’ PGY-3 residents (97.3% pass rate). Remote teaching and assessment of FLS manual skills appears to be similar, in this cohort, to in-person coaching and assessment that has historically been utilized in our residency program.

Learners were highly satisfied with the flexibility of the remote learning sessions. We were surprised that none of the PGY-3 learners opted to have their sessions utilizing their home trainer boxes, instead preferring to utilize the materials in the simulation lab. The location of the lab within the resident lounge may have made these sessions less onerous during the course of their workday and encouraged practice during “down time”. Although we relied on resident cell phones to broadcast the image on their screen, alternate set-ups could include screen sharing a laptop or tablet screen receiving live feed from the trainer box or from the camera directly. General surgery has required FLS since 2008 and their residents likely have access to the trainer boxes and supplies. Opening dialogue to share space and materials with other trainees is one avenue for programs to explore in order to reduce initial costs of curriculum implementation.

Limitations of this study include the small sample size and the use of PGY-3 residents. We recognize that our overall pass rate for the manual portion of FLS is 97.3%. Other studies have shown that basic laparoscopic experience shortens the learning curve of FLS manual skills.[6] However, finding that 4/10 of our learners were unable to complete the tasks at baseline reinforces the need at our program for concentrated training before the exam. This baseline performance may have been affected by the disruption of the traditional FLS training because of COVID-19 during the learners’ PGY-2 year. This concentrated curriculum was able to overcome these baseline shortcomings and enable the learners to pass the manual skills portion of the exam.

Another major limitation is the scoring rubric utilized for this study. Although on the FLS exam each task is weighted equally for the composite score, the rubric for this study weighted certain tasks such as the suturing tasks more heavily because of the additional metrics that were observed and graded. In addition, efficiency of movement is not captured by our scoring rubric. Without knowing the exact details of how deficits are weighted in the formal exam or how to objectively score efficiency of movement, scoring learners ability to perform each aspect of the tasks mentioned in the training manual was deemed the best surrogate for exam readiness.

The potential of remote coaching and assessment for surgical education is seemingly boundless. Learners are no longer tied to the faculty at their institution. Utilizing proven coaches and curriculum from outside institutions or providing additional coaching for struggling learners is an untapped resource for remote surgical education.

Remote surgical coaching should also be explored for use in surgeons in low-volume or resource settings. High-volume surgeons are associated with improved patient outcomes when compared to their low or very-low-volume surgical colleagues across a range of disciplines.[7-14] Implementing coaching sessions for surgeons geographically separated from their coaches in simulation, “live” surgery, or recorded portions of surgeries is an exciting use of this teaching modality. Through both surgical mentoring and coaching, we are excited about the potential of this low cost, remote coaching to address skill decay in low-volume settings. Currently, we are exploring the use of such coaching sessions for maintenance and refinement of laparoscopic skills with a subset of our military colleagues in low-volume settings both stateside and overseas.

In conclusion, all of our residents passed their FLS manual skills exams demonstrating remote coaching, and assessment of FLS manual skills appears to be an inexpensive, convenient, and viable alternative to traditional face-to-face teaching for surgical education in obstetrics and gynecology residents.

Click here for additional data file.