An online feedback system for laparoscopic training during the COVID-19 pandemic: evaluation from the trainer perspective.

Objective: A system to provide feedback for laparoscopic training using an online conferencing system during the COVID-19 pandemic was developed. The purpose of this study is to evaluate this system from the trainer perspective. Design: A procedural feedback system using an online conferencing system was devised. Setting: Surgical training was observed using an online conferencing system (Zoom). Feedback was provided while viewing suture videos which are, as a feature of this system, pre-recorded. Feedback was then recorded. Trainer comments were then converted into text, summarized as feedback items, and sorted by suture phase which facilitates reflection. Trainers completed a questionnaire concerning the usability of the online feedback session.
Results: Eleven trainers were selected. Physicians had an average experience of 21.9 ± 5.9 years (mean ± standard deviation). The total number of feedback items obtained by classifying each phase was 32. Based on questionnaire results, 91% of trainers were accustomed to the use of Zoom, and 100% felt that online procedural education was useful. In questions regarding system effectiveness, more than 70% of trainers answered positively to all questions, and in questions about efficiency, more than 70% of trainers answered positively. Only 55% of the trainers felt that this system was easy to use, but 91% were satisfied as trainers. Conclusions: The results of the questionnaire suggest that this system has high usability for training. This online system could be a useful tool for providing feedback in situations where face-to-face education is difficult.

Introduction

The COVID-19 pandemic makes it difficult to provide routine medical care. To prevent nosocomial infections, many hospitals have been forced to restrict elective surgical operations, and conferences and medical communication have been curtailed to reduce contact between people [1]. Resident education programs are also affected, and in some areas it is difficult to carry out regular face-to-face education due to infection control [2, 3]. There is a hindrance to the acquisition of surgical techniques by young surgeons due to the many restrictions in place. Laloo et al. reported that in a survey of 743 physicians of various grades and disciplines, 69% felt that the COVID-19 pandemic had a negative impact on training [4].

For laparoscopic surgery, training is often given outside of the operating room, such as suturing practice using a training dry-box in simulation sessions [5, 6]. However, during the COVID-19 pandemic where contact with people is restricted, it is difficult to receive face-to-face advice from trainers during hands-on seminars. As a result, surgical residents are less likely to receive feedback and evaluation from trainers, and less likely to conduct their own critical reflections. Even with limited face-to-face feedback, it is natural for surgical residents to want direct feedback to improve their skills. Therefore, even during the COVID-19 pandemic, a system is needed for trainers to evaluate and provide feedback to residents.

An online conferencing system has made it possible to hold academic meetings in the COVID-19 era. Most academic meetings are conducted online, using data viewing systems, allowing surgeons to share, discuss, and learn the latest techniques as before [7]. In medical education, remote lectures are given using technologies such as video conferencing systems and e-learning platforms [8, 9], and the role of social media in surgical education is increasing [10].

Education in surgical techniques is also shifting to online platforms. From March 2021, the suturing course of the Japan Society of Endoscopic Surgery (JSES) was changed from a conventional face-to-face format to online [11]. In this format, a laparoscopic training box is distributed in advance, the screen is shared in an online conferencing system while performing a procedure, and remote feedback is provided in real-time. However, the method by which a trainee receives feedback while performing a procedure in real-time results in the workload of the trainee increasing. Since it is not possible to pause or repeat a procedure, it is difficult for the trainer to comprehensively extract feedback items and point out accurately by observing the procedure only once.

Therefore, in this study, we devised a system by which a trainer reviews procedure videos recorded in advance by a trainee and the trainee is given feedback by the trainer using an online conferencing system in real-time. Using this system, trainees can reflect on the feedback on demand by recording the feedback. In addition, this procedure feedback system was evaluated from the viewpoint of ease of feedback and usability for the trainer. To realize appropriate feedback and facilitation, it is necessary to evaluate the education system from the perspective of the trainer. Recommendations for building an online education system have been reported [12]. To the best of our knowledge, there are no existing reports that utilized this type of educational system with evaluation from the perspective of the trainer.

Materials and methods

This system has two major features. One is the use of pre-recorded procedure videos in a real-time feedback session with an online conferencing system. The other feature is to record the feedback, convert it to text, then classify the feedback items, which makes it easier for the trainee to reflect on the procedure and associated feedback from the trainer. In devising a procedure feedback system, we focused on improving the effectiveness, efficiency, and satisfaction of guidance for trainers. These features are based on the definition of usability by ISO 9241–11: 2018, “the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” [13] This system had the following three aims: Effectiveness: The system was designed to enable the trainer to provide sufficient feedback to the trainee, and ensure that the trainer's remarks are clearly integrated in the trainee's reflection. Efficiency: The trainer should be able to think of and convey smoothly what the trainer wants to provide as feedback to the trainees using this system. Satisfaction: This is an integrated evaluation of the effectiveness and efficiency of the system. Overall, the system should be satisfactory as a feedback system for the trainer.

System flow

Table 1 shows the details of the devised system. We made it as simple and efficient as possible for the trainer to give online feedback for a procedure. We thought that familiar software was appropriate for selecting an online conferencing system, so we adopted Zoom (Zoom Video Communications, San Jose, CA, USA), which is widely used in Japan. The trainer and the trainee were connected to Zoom, and procedure videos made in advance were shown using the screen sharing function. A video of the trainee's procedure was prepared in advance so that the trainer could pause or repeat it, and the trainee could concentrate on feedback. The video was not provided to the trainer in advance. During the feedback phase of the interaction, the video and audio were recorded using standard Zoom functions, and the trainees were able to reflect the trainer's comments later. Reynolds et al. state that reflection and critical reflection are important in learning [14]. In addition, Kolb an organizational behaviorist, advocates four stages of the experiential learning model including (1) experience, (2) reflection, (3) standardization, and (4) practice [15]. In this system, the above-mentioned features make it easier to recognize the content of critical and objective feedback items from experts, and to reflect on what was done well and what needs to be improved, thereby facilitating accurate reflection and critical reflection regarding one's own behavior. By practicing, all steps from (1) to (4) of the experiential learning model are covered, and by repeating this process, improvement can be expected.

Table 1

System flow.

Trainee	Trainer
Record the procedure video in advance- Laparoscopic suturing task in a training box - 3-0 silk, three knots - Repeat three times
Make an appointment to meet online
Real-time feedback session (Zoom)- Show the video of the procedure with pause/repeat as needed - Provide feedback - Record the feedback session - Speech-to-text conversion - Sorting instruction items
Reflect at anytime- Feedback session video - Text data

Procedure video

The trainee who performed the procedure in the video is a surgeon ten years after medical school graduation, at the intermediate level. He specializes in gastrointestinal surgery and is board certified by the Japan Surgical Society, but is not yet qualified as a surgeon by the Endoscopic Surgical Skill Qualification System of the JSES, which was established to improve the quality of laparoscopic surgery in Japan, and is proof of being an expert laparoscopic surgeon. A qualified surgeon's supervision of laparoscopic surgery has been shown to improve the proficiency and safety of laparoscopic surgery performed by novices [16]. We recorded video of a laparoscopic suturing task using a laparoscopic training box (Lapatre-K, KOTOBUKI Medical Inc., Saitama, Japan), and silicon pad (Custom-made products, KOTOBUKI Medical Inc., Saitama, Japan) (Figure 1 ). Laparoscopic instruments, including a needle holder (EYP-2009S–CNK, HEIWA MEDICAL INSTRUMENTS Co., Ltd., Yamaguchi, Japan), Maryland dissector (RMK-BL1, KOTOBUKI Medical Inc., Saitama, Japan) were used. The task included making three knots with a 3-0 silk suture, which was repeated three times. A home-use video camera (HDR-CX670, Sony Marketing Inc., Tokyo, Japan) was used for video recording. Image resolution was set at 960 × 540 (width × height) and video recorded at 30 frames per second. The times required were 66, 52, and 57 s, for a total of 175 s. All trainers used the same procedure video for feedback.

Figure 1

The procedure video.

Trainers

Eleven trainers were included in the study. The number of years of experience for trainers was 21.9 ± 5.9 years (mean ± standard deviation). Since sufficient experience was required for trainers, 10 were selected from the qualified surgeons of the endoscopic surgical skill qualification system of the JSES, and 1 was selected from those who had been working as a trainer. Nine of the 11 trainers were acquainted with the trainee.

Suture phase classification

To ensure that feedback from the trainer is effectively utilized, a system was devised to record the feedback and allow the trainee to reflect on the feedback as needed. In addition, we extracted feedback items from trainers' comments and categorized them based on the concept of the suture phase. By categorizing items according to the four phases of suturing shown in Table 2 , we were able to organize the feedback items and make it easier for the trainees to reflect on the results. To complete three knots, phases were repeated in the following order: Phase 1→2→3→4→3→4→3→4. This arrangement was based on the study by Cuschieri et al [17]. The specific steps in processing the video recording of feedback are shown below. First, IBM Watson Speech to Text (International Business Machines Corporation, USA) was used as a speech recognition system to automatically convert the trainer's speech into text, which was later modified by manual input. Feedback items were extracted from the obtained text data and sorted. For example, “driving the needle is curved” and “the short-tail is long” were described in short, straightforward sentences. To encourage reflection, we described not only points that should be improved but also points that were accomplished well. In addition, they were categorized by the phases of suturing as described above. Extraction, sorting, and categorization of feedback items were performed manually.

Table 2

Suture phase classification.

Start
Phase 1”Grasping the needle”	Insert the forceps into the dry-box	Image 1
	Grasp the needle with the needle holder in the right direction	Image 2
Phase 2”Driving the needle”	Entrance biteExit bite	Image 3
	Remove the needle from the pad surface	Image 4
Phase 3”Preparation for knot tying”	Pull the suture and make an adequate short tail	Image 5
	Establish the C-loop	Image 6
Phase 4”Knot tying”	Take one or two throws around the forcepsGrab the short tail	Image 7
	Tie the knot	Image 8
Return to Phase 3 until three knots are completed.
End

Usability of the feedback system for trainers

We conducted a heuristic evaluation from the trainer's point of view by experts who are familiar with surgical education methods and effectiveness. Since trainees are not experts in educational methods and effectiveness, a subjective evaluation was not performed for them. In general product and system usability evaluation, it is common for experts to subjectively evaluate heuristics [18]. We surveyed trainers about effectiveness, efficiency and satisfaction, and usability of feedback using this system. As shown in Tables 3 and 4 , questionnaire items were largely divided into (1) experience with use of the online conferencing system and general interest in online procedural education, and (2) evaluation of the system. Questions were created using Google Form (Google, USA) and sent to the trainers via e-mail to be answered online. Questions about effectiveness, efficiency, satisfaction were included in the questionnaire (2) respectively to enable usability evaluation. At the end of the questionnaire, free-form text was permitted.

Table 3

Questionnaire (1) Usual use of online conferencing system and general interest in online procedural education.

Question	Every dayn (%)	Once every 2–3 daysn (%)	Once/weekn (%)	Once or twice per monthn (%)
Frequency of use of online conferencing system	0 (0 %)	2 (18 %)	7 (64 %)	2 (18 %)
	Very familiar/familiarn (%)		Not very familiar/not familiarn (%)
Are you familiar with the use of Zoom?	10 (91 %)		1 (9 %)
	Yesn (%)		Non (%)
Do you have experience with online procedural education?	2 (18 %)		9 (82 %)
Are you interested in online procedural education?	11 (100 %)		0 (0 %)
Do you feel that online procedural education is useful?	11 (100 %)		0 (0 %)

Table 4

Questionnaire (2) Evaluation of this system by trainers.

Question	Strongly Agree/Agreen (%)	Strongly Disagree/Disagreen (%)
Effectiveness
<Feedback session>
Is it good to use pre-recorded procedure videos?	10 (91 %)	1 (9 %)
Were the videos used sufficient for feedback?	11 (100 %)	0 (0 %)
<For trainee's easy reflection>
Do you feel it is good to record feedback session?	10 (91 %)	1 (9 %)
Do you feel it is good to make the recorded video of feedback available for trainees to review later?	11 (100 %)	0 (0 %)
Do you feel it is good to transcribe the recorded video of feedback for detailed analysis of the feedback content?	8 (73 %)	3 (27 %)
Do you feel it is good to make a recorded video of feedback widely available to a certain range of people, including students?	9 (82 %)	2 (18 %)
Do you feel that you would like to view a video of another doctor's feedback?	11 (100 %)	0 (0 %)
Efficiency
Was the process for starting feedback smooth?	9 (82 %)	2 (18 %)
Did you immediately think of what you wanted to give as feedback?	9 (82 %)	2 (18 %)
Were you able to convey what you wanted to convey well?	8 (73 %)	3 (27 %)
Satisfaction
Overall, was the system easy to use?	6 (55 %)	5 (45 %)
Are you satisfied with your work as an instructor?	10 (91 %)	1 (9 %)

Ethical considerations

This study was approved by the Kitaibaraki City Hospital Ethics Review Committee (No.0302), In this study, we explained to the trainers that their names would not be released, that participation was voluntary and there would be no disadvantage for refusal, and the purpose and content of the study. Data will be carefully stored and deleted, with due consideration given to the protection of personal information.

Results

Eleven trainers were selected. Duration of the real-time feedback session was 930.3 ± 442.8 s (mean ± standard deviation). IBM Watson Speech to Text was used to convert trainers’ comments into text. Thirty-two feedback items were extracted from text data, and based on the concept of suture phase, with 5 in phase 1, 6 in phase 2, 8 in phase 3, and 13 in phase 4.

The results of the questionnaire were as follows.

The usual use of the online conferencing system and general interest in online procedural education are shown in Table 3. Eighty-two percent of the trainers used an online conferencing system at least once weekly and 91% of the trainers responded that they were familiar with the use of Zoom. They had no previous experience with online procedural education such as this system, except for two trainers who participated as instructors in the suture ligation course of the JSES. The background of the trainers was that they were familiar with the use of Zoom and were in favor of online procedural education.

Evaluation results for the system are shown in Table 4. More than 70% of trainers gave positive evaluations for all questionnaire items except “Overall, was the system easy to use?” Only about half of the trainers felt that this system was easy to provide feedback, but more than 90% of the trainers were more satisfied with their work as instructors using this system.

Discussion

While face-to-face feedback is limited due to the COVID-19 pandemic, we have devised a procedural feedback system using an online conferencing system. This system has two features including: (1) the use of pre-recorded videos of suturing for real-time feedback using Zoom, and (2) the feedback content is recorded, converted to text, and categorized by suture phase for easy reflection by the trainee. In addition, we conducted a survey of trainers to evaluate the system from the trainer's point of view. The trainers in this study are strongly interested in online procedural education and gave high evaluations of the ease of feedback and usability of this system.

First of all, the evaluation items in the questionnaire concerning the effectiveness of feedback correspond to “is it good to use pre-recorded procedure videos?” and “were the videos used sufficient for feedback?” The responses to these questions were positive for the majority of trainers. The use of pre-recorded videos allows the trainer to pause and re-show the videos during feedback. This has the advantage of making it easier to list items for feedback. In face-to-face online real-time feedback, such as the suture training course organized by the JSES, “pause” and “repeat” are not possible. Some of the trainers commented on this point, “I feel that this is a new and prospective educational method,” and “Onsite training was a one-time manual procedure, but I feel that the use of recorded data is superior in that it allows repeated review and sharing of problems through video images.” Video-based feedback could be developed into an online feedback system based on actual surgical videos in the future.

The next question, “was the process for starting feedback smooth?” corresponds to efficiency in terms of operation. This was also highly evaluated. Based on the results of the questionnaire (1), Zoom is a familiar tool among trainers, and its use is appropriate in terms of efficiency of operation. If trainers and trainees perform a procedure online in real-time, it would be somewhat cumbersome to prepare the laparoscopic dry-box training system and set it up so that it could be shared during the video call. The use of pre-recorded videos may have addressed this problem and contribute to efficiency of feedback. It is convenient to be able to give feedback at any time based on a video prepared in advance. This could be a useful system for trainers with busy schedules and the ongoing efforts to reform the way they work. Grant et al. describe the advantages of distance education as the ability to provide quality-assured content to all busy learners, cost-effectiveness, and effective use of faculty time [19]. In the questionnaire, “Did you immediately think of what you wanted to give as feedback?” and “were you able to convey what you wanted to convey well?” were also evaluated positively. Some trainers commented that “it would be good to be able to provide instruction remotely” and that “such a system would be essential in the future.”

Since the effectiveness and efficiency of this system were positively evaluated, the level of satisfaction as a trainer after feedback also increased.

All three items in the definition of usability were evaluated highly, suggesting that the usability of this system for trainers is high and favorable.

As another feature of this study, to ensure that the trainer's feedback is effectively utilized, a system was devised that allows the trainee to easily reflect on it. In past surveys, trainees have mentioned that one of the advantages of on-demand online education is the ability to reflect at their own pace [20]. In this system, the feedback given while watching a video of a procedure is recorded, making it possible to review it at any time in an on-demand manner. In addition, feedback items were easily extracted by converting them into text, and suturing was classified by phase to make it easier to generalize and organize.

In the questionnaire, almost all participants were positive about having their feedback recorded and reviewed later by trainers, and more than 70% felt that converting their comments to text was acceptable. In addition, 80% were willing to make a recorded video of feedback widely available to students and others. All trainers felt that they would like to view the videos of other trainers giving feedback to trainees, indicating the generous and strong interest of surgical trainers in procedural education. This is a very valuable opinion for the future development of procedural education using this system.

In the future, we would like to automate the entire process of creating text data and extracting feedback items, but this is the first report and this was done manually. We believe that the organization and utilization of feedback items may be further developed by incorporating automated quantitative analysis and analysis by natural language processing.

This paper has acknowledged limitations. This paper investigates usability by trainers. A video of one trainee was used for verification and reporting. However, in usability reports using newly developed medical systems or devices, it is common to have a single system or device as the target [21, 22]. This system encourages the trainee's reflection, and in the future, evaluation of the effects of this system from the trainee's point of view will be studied. With full use of this system, it is intended that trainees pre-record their own procedures to receive feedback, so that is not designed for complete novice users.

Conclusion

We devised an online feedback system for laparoscopic training and evaluated its usability for trainers. The system provides sufficient items for feedback, and survey results show that the system is highly useable by trainers. We consider that this online education system could be a useful tool to give feedback for procedures in situations where face-to-face education is difficult. We hope to use this system as a basis for improving the quality of online procedural education in the future. While this pilot study was limited to giving feedback for a training task, a similar system might be useful for providing feedback for clinical procedures as well.

Declarations

Author contribution statement

Daigo Kuboki, Hiroshi Kawahira, Yoshitaka Maeda, Kosuke Oiwa, Teruhiko Unoki:

Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Alan Kawarai Lefor: Analyzed and interpreted the data; Wrote the paper.

Naohiro Sata: Conceived and designed the experiments;Contributed reagents, materials, analysis tools or data; Wrote the paper.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Data availability statement

Data will be made available on request.

Declaration of interest's statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.