Development and clinical implementation of a structured, simulation-based training programme in laparoscopic appendectomy: description, validation and evaluation.

Background: Laparoscopic appendectomy is a common procedure in general surgery but is likely underused in structured and real-life teaching. This study describes the development, validation and evaluation of implementing a structured training programme for laparoscopic appendectomy. Study design: A structured curriculum and simulation-based programme for trainees and trainers was developed. All general surgery trainees and trainers were involved in laparoscopic appendectomies. All trainees and trainers underwent the structured preprocedure training programme before real-life surgery evaluation. A standardised form evaluated eight technical steps (skills) of the procedure as well as an overall assessment, and nine elements of communication (feedback), and was used for bilateral evaluation by each trainee and trainer. A consecutive, observational cohort over a 12-month period was used to gauge real-life implementation.
Results: During 277 eligible real-life appendectomies, structured evaluation was performed in 173 (62%) laparoscopic appendectomies, for which 165 forms were completed by 19 trainees. Construct validity was found satisfactory. Inter-rater reliability demonstrated good correlation between trainee and trainer. The trainees' and trainers' stepwise and overall assessments of technical skills had an overall good reliability (intraclass correlation coefficient of 0.88). The vast majority (92.2%) of the trainees either agreed or strongly agreed that the training met their expectations.
Conclusion: Structured training for general surgery residents can be implemented for laparoscopic appendectomy. Skills assessment by trainees and trainers indicated reliable self-assessment. Overall, the trainees were satisfied with the training, including the feedback from the trainers. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Laparoscopic appendectomy is considered safe to be performed by surgical residents.

Laparoscopic appendectomy is often performed by novice surgical trainees.

A structured trainee–trainer programme led to a high satisfaction rate of trainees in achieving learning objectives.

Reliable self-assessment of real-life surgical technical skills between trainees and trainers was demonstrated and may be used for evaluation.

A structured approach to laparoscopic appendectomy is an opportunity to facilitate systematic feedback during surgical training.

Introduction

Acute appendicitis is a global disease with incidences varying between 105 and 151 per 100 000 person years and an estimated lifetime risk of 1 in 15.1 Globally, appendectomy is one of the most commonly performed procedures in general surgery, with approximately 280 000 appendectomies being performed each year in the USA alone. Appendectomies are among the first procedures done during surgical training and are considered safe when done by surgeons in training.2–4 Despite this, laparoscopic appendectomy is likely an underused procedure for structured surgical training of junior surgical residents. Further, studies evaluating the learning situation between the trainee and the trainer are lacking. Also, whether objectives are met during training are scarcely reported. Previous attempts to standardise laparoscopic appendectomy5 have not been adopted as a universal approach. Furthermore, a virtual care pathway approach for acute appendicitis6 and a proficiency-based virtual reality simulation training curriculum for laparoscopic appendectomy7 have been developed. However, these lack clinical transition with systematic implementation in real-life practice.

Consequently, the objectives of this study were to develop, validate and implement a structured training programme involving both trainer and trainee in a structured presurgical curriculum and simulation before evaluating stepwise and overall performance in real-life surgeries. Evaluation of technical skills and perception of the guidance provided during operations were done between trainees and trainers.

Methods

Study design

A structured and standardised simulation-based training programme was developed along similar lines as the LapCo8 programme for laparoscopic colectomy developed in the UK and used in several European countries.

Educational background theory

From learning theory, Kolb defines experiential learning as a process by which knowledge is created through the transformation of experience, with individuals learning through a cycle of concrete experience, reflection, conceptualisation and experimentation.9 Surgical training—whether it is simulation-based or situated in real-life operations—should acknowledge this cycle and facilitate the trainee’s process through the cycle. This cycle was acknowledged in the training programme when it was introduced as a three-step process study (figure 1) based on

Figure 1

Design of the laparoscopic appendectomy training programme.

The development of an agreed-upon core curriculum and structured template for performing a laparoscopic appendectomy.

A structured simulation-based training programme for trainees and trainers implemented during educational sessions.

A score template for structured training feedback during real-life laparoscopic appendectomies as learning situations in which the performing surgeon (the trainee) was junior to the trainer.

Settings

The curriculum and simulation-based training methodology were developed, validated and implemented at Stavanger University Hospital, one of six university hospitals in Norway.

Participants

Eligible for inclusion were all surgical trainees involved in all consecutive on-call or daytime laparoscopic appendectomies. Appendectomies performed during night-time (between 23:00 and 07:00) were not considered learning situation and hence were excluded. The on-call team consists of surgical trainees in general surgery with general and gastrointestinal surgery consultant on-call cover. During laparoscopic appendectomies, the junior residents were trainees, while the senior residents (>4 years of experience) or consultant surgeons served as trainers.

Intervention

The LapApp design consists of standardisation of a procedure curriculum, simulation in a dry-lab environment, and a structured training programme and teaching structure during real-life procedures. Subsequently, each trainee and trainer evaluate both the trainee’s performance (skills) and the quality of communication by the trainer during the training (feedback).

Standardisation of real-life laparoscopic appendectomies

The development of a standardised and stepwise approach to the procedure (table 1) was based on the techniques currently being used at the hospital. Meetings were held to achieve a consensus among all consultant surgeons (n=13) in the department of gastrointestinal surgery. Every part of the procedure was defined in detail to ensure that every learning situation (in real-life surgery) was executed in the same manner.

Table 1

Stepwise and standardised approach to the procedure

Steps (1–8)	Description
Abdominal access	Umbilical incision; sharp dissection until fascia is visualised; lift fascia between Kocker’s clamps and divide the fascia before 12 mm trocar is inserted.
Trocar placement	12 mm trocar in left iliac fossa, 5 mm trocar approximately two finger widths cranial to the symphysis pubis
Appendix identification	Inspecting all four quadrants and identifying appendix using atraumatic graspers
Handling the bowel	Making sure the small bowel is handled in an atraumatic manner
Dividing mesoappendix	Alternately using bipolar diathermy and cold scissors to ensure haemostasis
Dividing appendix	Placement of two Endoloops and transection using cold scissor
Extracting appendix	Using an Endobag and extracting the appendix through the umbilical (12 mm) trocar, control for any leak and/or bleeding from the caecum/appendiceal stump
Closure	Cross suture in fascia, intracutaneous suture in skin

Simulation training

Surgical trainees at Stavanger University Hospital are obliged to complete at least three sessions of 45 min self-training per week in a minimum of 3 weeks on a standard box trainer (SimSurgery D-box) before being allowed to perform any procedures in the operating theatre.

The LapApp training programme targets the inexperienced surgical residents, with very limited or even no experience in minimally invasive surgery. The simulation training intends to prepare the trainees for the live procedures by allowing repeated practice in a safe environment. Minimally invasive surgery introduces several challenges for the inexperienced trainee, among them the loss of finger dexterity, and the counterintuitive fulcrum effect (when the surgeon move his/her hand to the right, the instrument moves to the left and vice versa). These fundamental procedural skills are central to practice, especially in this early stage of training—and box training has proven effective in obtaining them.10 11The aim for the simulation training is not to practice the appendectomy procedure itself, but rather to train on fundamental skills by using prefabricated trays to practice slalomer exercise, moving objects through various obstacles and similar exercises.

Structured training programme

The structured training programme consists of two parts: an introductory course for all trainees and a train-the-trainer course for all trainers.

The introductory course targeting the trainees is a theoretical course divided into two parts and recognises that surgical expertise is not confined to procedural dexterity. The first part instructs the trainees on ‘how to be a surgeon’ and emphasises the importance of introducing oneself to the patient, being informed of all essential information in the patient’s history and the importance of the WHO Safe Surgery checklist.12

The second part focuses on the disease (acute appendicitis) and the procedure (laparoscopic appendectomy), detailing symptomatology, clinical findings, indications for radiology, and the different steps in the standardised approach. Potential hazards are covered in the course. The residents are encouraged to use simulation-based tools and multimedia tools proven to be effective in surgical education.13–15

The LapApp Train-the-Trainer course is a highly interactive 3-hour pedagogics course training six participants at a time. The course starts with the participants defining their learning objectives and formulating their focus for the course. The next session is about ‘the difficult trainee’ in which each of the participants would share a story with a difficult learning situation before the group reflects on the stories. Following this, a practical training exercise was commenced in which the participants instruct each other on a box trainer (Simulation Dbox), but with the participant performing the task while blindfolded. This demonstrates the importance of concise communication during surgery.

The next session covers the three major steps of the pedagogical teaching structure: the set-up before an operation/teaching session, the dialogue during the operation and the evaluation after the learning situation. The last session is a practical exercise in which two junior residents come to the course to do a task in the box trainer. The six course participants are divided into two groups and take turns in instructing the resident. Their instruction is afterwards evaluated by both the junior resident and the course instructors.

The set-up includes preparation, assessment of the trainee, establishment of a common goal, setting specific goals for learning, and establishing clear rules for the intraoperative communication between trainee and trainer.

The dialogue focuses on communication, how to avoid cognitive overload, how to instruct clearly and how to give feedback. An important tool is the ‘stop command’. A common way of instructing is to give advice consecutively during the procedure. In response to the stop command, the trainee should freeze all instruments and try to identify the problem(s) that led the trainer to stop the operation. The goal is that the trainee both identifies the problem and comes up with a possible solution. The trainer then decides whether it is safe to let the trainee proceed. The goal is more effective training through increased trainee participation.

The evaluation deals with performance improvement, feedback, summing up and formulating a ‘take-home message’ to promote self-reflection and give the trainee a specific goal to work toward until the next real-life learning situation and, more specifically, when doing a laparoscopic appendectomy.

Evaluation form

To study different effects, an evaluation form based on the validated LapcoNor form16 17 was completed after every laparoscopic appendectomy that represented a learning situation. The evaluation of the technical skills uses a scoring system that is based on how much of the procedure the trainee is actually able to perform independently and safely. Possible scores range from 1 to 6 (table 2).17

Table 2

Scoring system used in the evaluation forms

1	Not performed by trainee, step had to be done by trainer
2	Partly performed by trainee, step had to be partly done by trainer
3	Performed by trainee with substantial verbal support
4	Performed by trainee with minor verbal support
5	Competent performance, safe (without guidance)
6	Proficient performance, ‘could not be better’

The evaluation form consists of three parts. First, the trainee scores his/her own technical skills in each of the steps defined in the standardised approach to the procedure (tables 1 and 2). A total of eight steps are evaluated for the technical skill execution (table 1). In addition, an overall assessment of technical skills is done by both trainee and trainer. The overall assessment does not emphasise any of the steps as more or less important. Second, the trainer evaluates the trainee for each of the technical steps in the same manner as the trainee evaluates her/himself (tables 1 and 2). Third, and lastly, the trainee evaluates the trainer on the perioperative communication. This evaluates the overall trainee satisfaction with the learning situation, addressing points such as if the training was adjusted to the trainee competence, if the trainer took over at an appropriate time (if at all), if the trainer gave too much or too little verbal instruction, if the trainer did too much or not enough of the procedure, if there were derived learning objectives, and if the training overall met the trainee’s expectations. This evaluation was scored by categories as ‘totally agree/agree/uncertain/disagree/totally disagree’ with predefined statements.

Implementation period

Clinical enrolment during laparoscopic appendectomy procedures extended from 1 January to 31 December 2018, covering a 12-month period.

Outcomes evaluated

The primary outcomes were the degree of implementation of the structured training programme, the evaluation of trainees’ self-assessment compared with the trainers’ assessment of technical skills and the trainees’ perception of the training (the quality of feedback from the trainers). Implementation of the programme was defined by the number of completed evaluation forms. The position of the instructor (either a senior resident or consultant surgeon) was recorded, as were the gender of each trainee and trainer.

For the trainee’s perception of the training, some questions were a mixture of negative and positive statements. To facilitate the description of the results easier, we defined a very positive (‘strongly disagree’ for the negative questions and ‘strongly agree’ for the positive questions) and positive feedback (‘disagree’ for the negative questions and ‘agree’ for the positive questions). Very negative and negative feedback are defined the opposite way, while neutral feedback remained as such.

Validation

Construct validity was tested by splitting procedure volumes into four groups of different experience levels. Due to the lack of a ‘gold standard’ method, it was not possible to establish criterion-related validity. Both trainees and trainers had to fill in the same scoring sheet, which facilitated the assessment of inter-rater reliability. This was done for the overall score of the procedure and for the overall average score across all steps of the procedure.

Statistical analysis

The computation was done in R V.3.6 (www.r-project.org) and MATLAB V.9 (MathWorks, Natick, Massachusetts, USA). The graphics was derived using MATLAB V.9. Statistical analyses were done by Social Package for Social Sciences for Mac V.26.

Descriptive methods were used to characterise the sample (trainees and trainers). The reliability of the evaluations was assessed using intraclass correlation coefficients (ICCs, consistency, two-way mixed-effects). Construct validity was analysed by comparing four experience groups. Inter-rater reliability between the two was measured using the ICC and demonstrated on a Bland-Altman plot.18

ICC values less than 0.5 suggest poor reliability; values between 0.5 and 0.75 indicate moderate reliability; values between 0.75 and 0.9 indicate good reliability; and values greater than 0.90 indicate excellent reliability. 19 Trainees perception of the training was assessed graphically.

Results

During the study period, 173 evaluation forms were returned, of which 165 (95%) were complete and available for data analysis. Returned evaluation forms indicated that structured training had been executed (figure 2) for an implementation rate of 62.4%.

Figure 2

Flowchart showing the implementation of the training programme legend. ‘Night-time procedures’ denote any surgery between 23:00 and 07:00. SUH, Stavanger University Hospital.

A total of 19 trainees (43% women) and 26 trainers (42% women) were involved. The trainers were senior residents (n=14, 54%) or consultant surgeons (n=12, 46%). Among the senior residents, there was an even gender distribution with 50% (n=7) female trainers. Among the consultant surgeons, 33% (n=4) were female trainers.

The trainer was a senior resident in 80% of the cases (n=133 of 165) and a consultant surgeon in 20% of the cases (n=32). No formal rating of complexity of the procedure was done, but based on trainees’ judgements of the procedures, some 80% of the appendices were deemed to be inflamed (phlegmonous or gangrenous appendicitis), while 12% had a macroscopic perforation and some 6% had negative gross inspection for inflammation.

Assessment of the technical steps of the procedure

There was good agreement between trainees’ self-assessment and the trainers’ assessment on all technical skills (figure 3). The ICC of 0.88 (95% CI 0.84 to 0.91) on the summed scores of each element of the technical steps indicated overall good reliability. None of the ICC values for each of the technical steps were below 0.80

Figure 3

ICC of self-assessment. Legend: the correlation between trainees’ self-assessment and trainees’ assessment for each of the eight steps of the procedure. ICC, intraclass correlation coefficient.

Construct validity was considered by analysing the variance of four experience groups, and the difference between the groups was confirmed (figure 4).

Figure 4

Construct validity of the laparoscopic appendectomy evaluation. Box plots for four different experience groups with overall assessment scores by trainer for each groups. The box line is the median; boxes represent 25th and 75th percentiles; and whiskers are ranges for each group.

Inter-rater reliability was analysed by comparing the trainee’s and the trainer’s score on overall assessment and is depicted in a Bland-Altman plots that showed good correlation between trainer and trainee scores (figure 5).

Figure 5

Bland-Altman plots for the inter-rater difference. Legend: the plots compare the trainee self-assessment to trainer assessment for (A) the overall procedure score by stepwise 1–6 evaluation and (B) as an average of all steps for each procedure, as scored by trainer and trainee overall performance.

Trainees’ perception of the training

The trainees’ evaluation of the trainers is depicted in figure 6, with feedback scores observed being mainly ‘positive’ and ‘very positive’. Preoperative agreement and/or postoperative discussion showed the lowest positive feedback around 60% as well as noteworthy negative/very negative evaluations (around 10% or more). A ’very negative’ feedback was most frequently observed for the instructors taking over at appropriate time (around 15%). The remaining questions had mostly ‘positive/very positive’, and 5% or less ‘negative/very negative’ feedback.

Figure 6

Perceived agreement of the training goals in the laparoscopic appendectomy programme. Legend: evaluation in categories as reported by the trainees.

About 60% (n=99) of the trainees felt that the preoperative agreement and/or postoperative discussion were satisfactory.

A total of 97% of the trainees felt that the trainer adjusted the level of training to their competence and 2% were neutral, leaving 1% of the trainees not satisfied with how the training met their level of competence.

Regarding whether the trainer took over the operation at the appropriate time, 75% of the trainees either agreed or strongly agreed with this.

For intraoperative instructions, the trainees were asked about their trainer’s verbal guidance, and 1% and 2% of the trainees, respectively, responded that the trainers gave them either too much or too little verbal guidance.

The trainees were also asked to consider whether the trainer did either too much or too little of the procedure him/herself. In response, 4% of the trainees felt that the trainer did too much, while 3% felt that the trainer did too little.

Overall, 92% of the trainees either agreed or strongly agreed that the training was meeting their expectations.

Discussion

After validating the programme, this study found an overall high implementation rate of a structured training and evaluation programme for surgical trainees doing laparoscopic appendectomies in a university hospital. Hence, implementing LapApp was feasible and the structured training was used in almost two-thirds (62%) of all eligible, daytime laparoscopic appendectomies performed during the study period. The data indicate reliable self-assessment of real-life surgical technical skills among trainees. Overall, the trainees’ and trainers’ evaluations of performance indicated good reliability. The vast majority felt that training was adjusted to their abilities, while more than half (60%) of trainees felt the preoperative goals/conversation was satisfactory.

The result is consistent with other studies20 and is important because reliable self-assessment is considered beneficial for surgical training.21 Approximately 60% of the trainees felt that the preoperative agreement and postoperative discussion were satisfactory. Thus, we believe the preoperative conversation and goals to be an essential area for further improvement, as opportunities for briefing, feedback and reflection are considered highly important parts of a learning experience.21 Despite the fact that only 60% felt the preoperative conversation to be satisfactory, almost all trainees agreed that trainers adjusted the training to their competence level and gave appropriate amount of verbal guidance. However, 15% either disagreed or strongly disagreed that the trainer took over at the appropriate time. The data do not allow us to distinguish whether they thought the trainer took over too soon or too late, or the reason as to why the trainer took over. This should be further explored and refined in the design of the LapApp training programme to allow for mutual feedback and to explore areas of improvement in the educational process.

Trainees in this programme were obliged to self-train in a simulation dry lab. Simulation-based training has been proven effective for multiple outcomes,22 23 even in the form of self-training without feedback. Notably, specific and validated simulation tools for training in laparoscopic appendectomy exist, extending from porcine models24 to virtual reality simulators,25 and these would give more specific simulation training. While such high-end simulations could be implemented in both the training and the evaluation of trainees, they are not universally available and are more costly. A structured approach to learning for operations already taking place is thus an excellent opportunity for enhanced learning in clinical practice.

Demonstrating the potential effects of an educational programme in laparoscopic appendectomy on patient outcome would be desirable but very challenging in a typical high-resource setting where quality of care and patient outcomes are good. Furthermore, even though it has been demonstrated that superior technical performance positively affects patient outcomes,26 this is not necessarily applicable in laparoscopic appendectomies where complications may more often be related to the appendicitis than to the appendectomy,27 and using laparoscopic appendectomies as an area of learning has been proven safe.2–4

Limitations

Some limitations warrant further discussion. For one, the trainees and the trainers filled out their scores on the same evaluation form, possibly leading to bias in reporting as this was not anonymous between trainer and trainee. This bias could possibly effect both the evaluation of the trainees’ technical skills, as well as the trainees’ review of their trainers. Evaluation forms from training situations in one hospital during a given time period may reflect the spread in age, experience and gender diversity available at the time, and this may have affected the study results. The case-specific complexity of each procedure was not considered, and this may have affected the scores given for each step (eg, trocar placement in a very obese patient may be different from that in a young slender patient; a very inflamed or retrocecally located appendix may be different from a slightly inflamed and readily mobile appendix). Lastly, the agreed-upon standardised protocol for laparoscopic appendectomy may differ from other institutional protocols. Notably, no uniform consensus agreement on how to teach or best perform laparoscopic appendectomy exists today. Despite these shortcomings, the structured programme using a standardised evaluation demonstrated good implementation and good reliability among trainer and trainees in real-life laparoscopic appendectomies. Refinement of steps and subsequent identification on points for enhanced learning may further improve the learning situation, identify steps that are deemed technically more difficult and, thus, enhance focus on steps during teaching which may potentially reduce the learning curve towards proficiency for junior surgical trainees.

Conclusion

LapApp was implemented as an educational programme in a busy surgical unit. An easy, basic educational programme may enhance the learning of a basic procedure such as laparoscopic appendectomy and may facilitate trainee–trainer interaction with focused aims of steps of technical skills with constructive feedback and evaluation. The study found reliable self-assessment by trainees comparing to the skills assessment by trainers. Trainees were overall satisfied with the training. The high number of appendectomies performed worldwide provide an opportunity for using and maximising training for surgical trainees during laparoscopic appendectomy.