Surgical performance of large loop excision of the transformation zone in a training model: A prospective cohort study.

Large loop excision of the transformation zone (LLETZ) is one of the most common procedures in operative gynecology and it is a routine part of the surgical training program of residents. There is, however, no established and standardized method of teaching residents how to perform LLETZ. Here, we present a surgical training model and assessed the improvement of surgical skills during repeated hands-on trainings of LLETZ in this model.Surgical novices and experts were recruited and were shown a LLETZ training video and then performed 3 LLETZ training sessions on consecutive days. Surgical skills were assessed by Objective Structured Assessment of Technical Skills (OSATS). Global rating scale (GRS), confidence (CON), fragmentation rate (FR), performance time (PT), and OSATS scores were calculated. Intra- and interobserver variabilities were determined. The construct validity of OSATS was assessed comparing metric scores of novices with those of experts.Sixty-eight probands (58 novices, 10 experts) were recruited. GRS, 2.3 ± 1.3 (median ± SD) versus 1.4 ± 0.6, P < .001; CON, 2.7 ± 0.9 versus 1.6 ± 0.6, P < .001; FR, 81% versus 100%, P < .001; PT, 152 ± 33 versus 120 ± 27 seconds, P = .006; and OSATS scores, 18.8 ± 1.3 versus 19.1 ± 1.1, P = .16 of novices improved from session 1 to session 3. OSATS showed construct validity with metric scores (GRS, 1.1 ± 0.3 vs 2.3 ± 0.8, P < .001; CON, 1.0 ± 0.0 vs 2.7 ± 0.9, P < .001; PT 125 ± 30 vs 152 ± 33 seconds, P = .02; OSATS scores, 19.6 ± 0.7 vs. 18.8 ± 1.3, P = .02) reliably discriminating between experts and novices. Intra- and interobserver variabilities across probands were 0.99 ± 0.03 and 0.64 ± 0.10, respectively. OSATS scores were independent of handedness, sex, and regular sports activity in univariate and multivariate analyses.Repeated hands-on trainings improve surgical performance of LLETZ in a surgical training model with construct validity.

Introduction

Large loop excision of the transformation zone (LLETZ) is a standard technique for the surgical treatment of high-grade dysplasia of the cervix.[ LLETZ is one of the most common procedures in operative gynecology and it is a routine part of the surgical training program of residents.[ There is, however, no established and standardized method of teaching residents how to perform LLETZ. Typically, residents gain experience in a learning-by-doing approach with experts demonstrating and assisting procedures in a master and apprentice setting. This method is subjective and has been criticized for methodological and ethical reasons.[ Developing standardized, reliable, and efficient methods of teaching gynecology residents how to perform surgical procedures such as LLETZ is an important and challenging task. Solid evidence regarding the teaching and training of LLETZ based on prospective trials is lacking (PubMed search, January 22, 2016; search terms: conization, LLETZ/LEEP, training, model, teaching, and dummy). Therefore, we have constructed and tested a simple and inexpensive LLETZ training model using 2 porcine sausages in an easy to construct, yet rigid and maintainable plastic assembly. The feasibility and validity of this training model has been tested and previously published.[ In the current study, we aimed to establish the construct validity of this model. Furthermore, we wanted to test the hypothesis that repeated hands-on trainings of LLETZ using this surgical training model will significantly improve the surgical skills of novices.

In previous studies, we used the Objective Structured Assessment of Technical Skills (OSATS) method to evaluate and score the technical performance of hysteroscopy,[ vaginal operative delivery by vacuum extraction,[ and resolving a shoulder dystocia.[ We and others found that OSATS is a reliable and reproducible method to objectively assess the technical skills of both experts and novices.[ For example, Martinez et al[ used OSATS to study the long-term effects of a training workshop regarding the theoretical knowledge and the practical skills of gynecology residents to repair fourth-degree lacerations. They found that residents improved on theoretical knowledge and OSATS scores after the training workshop and maintained this improvement for 6 months. Antomarchi et al[ assessed the reliability of OSATS for evaluating a vaginal delivery simulation device. In their study, OSATS was a reliable means to assess medical students’ competence in procedural skills using a simulator for vertex presentation delivery. Based on this evidence supporting OSATS to measure surgical skills,[ we used a 20-item OSATS checklist to assess and quantify the technical skills of surgical novices performing a LLETZ procedure.

Although OSATS seems to be a reliable means for structured analysis of technical skills, this does not necessarily mean that it also has construct validity. Construct validity is defined as the degree to which a test measures what it claims, or purports, to be measuring.[ To establish the construct validity of our study setting, we compared metric parameters such as performance time (PT), global rating scale (GRS), confidence (CON), and OSATS scores between surgical novices and surgical experts. We defined construct validity of our porcine LLETZ training model as given if all metric parameters significantly differed between experts and novices favoring the expert group. In this respect, we wanted to make sure that the results of the performance assessments of our study are representative and clinically meaningful. Moreover, we were interested in identifying the specific steps of the LLETZ procedure best defining who is an expert and who is not.

We hypothesized that our training model has construct validity and will lead to a measurable improvement in surgical skills. To test this hypothesis, we designed a prospective cohort study measuring OSATS scores of surgical novices repeatedly training a defined LLETZ algorithm after an expert demonstration session using a training video. In addition, we compared the performances of novices and surgical experts to establish construct validity.

Methods

This prospective cohort study was carried out at the Department of Obstetrics and Gynecology, Ruhr-Universität Bochum, Bochum, Germany, in a population of consecutive medical students, who took part in a gynecology rotation. In addition, expert surgeons (consultants or residents with at least 4 years of surgical experience) were recruited among the clinical staff of the Department. Approval for this study was obtained by the institutional review board of the Ruhr-Universität Bochum Medical Faculty (registration number 5200–15). Informed consent was obtained from all study probands. Figure 1 shows a diagram of the study probands’ flow through the study. Inclusion criteria were informed consent and being a medical student without prior surgical training. Exclusion criteria were presence of a language barrier, previous exposure to LLETZ, and previous exposure to LLETZ training. Probands were shown a LLETZ training video (available as Supplemental Video 1). This video was produced by our Department and is in use for the purpose of resident training. The video was originally recorded in German, but is available in a dubbed English version as an online supplementary file. In the video, one of the authors (CBT) acts as an expert instructor demonstrating all steps of performing a LLETZ according to the OSATS checklist used in this study. Table 1 shows all 20 OSATS items in detail. In the training video, we used the same LLETZ training model as the one used for the study. Figure 2 shows the model in detail. The feasibility and validity of a simpler, less robust version this sausage-based LLETZ training model has been previously published.[ The key maneuvers of LLETZ were as follows: application of a 5% acetic acid solution to the cervix; identification of the acetowhite lesion; choice of the appropriate loop according to the size of the cervix, removal of the cone using adequate speed, and exploration of the cervical canal with a Hegar dilator. Additional tissue (“cowboy hat”) was excised from the endocervix using a rectangular loop with a smaller diameter. Endocervical curettage was performed and hemostasis was obtained with a ball electrode sparing the cervical canal. We used high-frequency wire loops and ball electrodes (Erbe, Tübingen, Germany) for the procedures.

Figure 1

Diagram of the proband flow through the study.

Table 1

Objective structures assessment of technical skills checklist.

Figure 2

Training model. A, The model in use during a training session with gynecology residents. B, Close-up view of the model with an approx. 3-cm-diameter sausage representing the cervix with an “acetowhite lesion” (painted with white correction fluid), accessible through a cup representing the vagina held open by a speculum, and grasped with forceps. C, Schematic drawings of the model. The enclosure is made from a rigid plastic container. A hole is cut into the bottom and fitted with an inwall socket whose bottom was also cut out. Velcro strip are fastened to the sides of the enclosure and a hole is drilled in the side to mount a fitting for a vacuum line to suck out fumes. The “vagina” is detachable to facilitate access and cleaning (it will become quite greasy), and is made from 2 plastic cups (bottoms cut out) with Velcro strips glued in between the cups (aligned to fit the strips on the outer enclosure) and a matching hole for the fume exhaust. Two types of sausages are used: a large (approx. 9–10 cm diameter) support sausage, to which the return electrode is attached, and a smaller type of sausage with approx. 3 cm diameter that represents the cervix and is tightly inserted into a cavity scooped from the support sausage (conductivity between the sausages must be ensured). The sausage assembly is inserted from the back end of the rigid enclosure and both are fastened to a support stand using Velcro strips. After each procedure, the “vagina” is detached and the “cervix” can easily be replaced with a new one. The support sausage can be used for many procedures over several days when kept in a refrigerator in between sessions.

Probands were not allowed to film the procedures, to take photos, or to write notes. Immediately after the video instruction (session 1) and on consecutive days (sessions 2 and 3), probands were tested by 1 assessor (SS). All sessions were recorded on video. The outcomes GRS and CON were graded using a 5-item scale with lower values denominating better performance. Both GRS and CON were rated by the probands themselves and by the assessor directly after the procedure. PT was measured in seconds and any fragmentation of the cone was noted (fragmentation rate [FR], ie, the proportion of procedures where fragmentation occurred). In addition, the weights and heights of the excised cones were measured. To grade each proband's performance, the recorded videos were evaluated at a later time with the evaluator blinded to session number or group assignment. OSATS scores were calculated by adding points given for each of the 20 items on a task-specific check list (Table 1) with one point for correctly performing each item and zero points for not performing or not correctly performing the item. Thus, a higher score indicates greater proficiency.

We tested the intra- and interobserver variability of the OSATS score assessment by having 25 randomly chosen videos scored by four different raters with one of them rating all videos 3 times. Fleiss’ kappa[ was calculated based on the assumption that the mean proportion of agreement after random assignments by the raters was fixed at 0.5 in case of yes/no items and 0.2 in case of 5-point scales. In addition, the percentage of rater-agreement was calculated by pairwise comparisons of the ratings: for each disagreement, a disagreement score was increased by 1; agreement was then expressed as 1 − (disagreement score/maximally possible disagreement score).

The construct validity of the OSATS setting used in this study was assessed by comparing predefined metric scores between 10 experts and the 58 surgical novices recruited for this study. The metric scores were PT, GRS, CON, and OSATS scores. We defined construct validity as given, when all assessed metric scores were significantly different between the 2 groups favoring the experts.

We chose to recruit medical students, because they were naïve to any form of previous LLETZ training or practical experience with LLETZ. Thus, in our view, in contrast to residents, medical students were ideal subjects in the sense that they did not introduce bias in terms of different years of residency experience, past practical LLETZ experience, or previous LLETZ training experience.

When comparing different groups, categorical variables were analyzed by χ test and continuous variables were compared using the Mann–Whitney U test with a significance level of .05. Wilcoxon signed rank or paired t tests were used for the analysis of repeated measures data. We performed a multiple linear regression model to test whether the training effect, as measured by OSATS scores, was independent of potential confounders such as the probands’ gender (male vs female) or handedness. Values are given as means. As this was not a comparative study, power and sample size calculations were not performed. We used the statistical software SigmaPlot 12.5 (Systat Software Inc., San Jose, CA) for statistical analysis.

Results

We recruited 68 of 75 screened probands for this study (7 declined to participate); 58 were novices and 10 were experts. Characteristics of the study population are given in Table 2. To assess whether repeated LLETZ trainings improved surgical performance of novices, we compared GRS, CON, FR, PT, and OSATS scores in 3 consecutive training sessions. We found that the surgical performance continuously improved over time and with increasing number of trainings. Table 3 shows all performance scores of novices in sessions 1 to 3. Specifically, rater-assessed GRS (rater-assessed: 2.3 ± 1.3 vs 1.4 ± 0.6, P < .001; self-assessed: 2.4 ± 0.8 vs 2.1 ± 0.7, P = .001), CON (rater-assessed: 2.7 ± 0.9 vs 1.6 ± 0.6, P < .001; self-assessed: 2.6 ± 0.8 vs 2.1 ± 0.9, P < .001), PT (152 ± 33 vs 120 ± 27 seconds; P = .006), and OSATS scores (18.8 ± 1.3 vs 19.1 ± 1.1; P = .16) of novices improved from session 1 to session 3. Figure 3 demonstrates that OSATS scores, PT, self-, and rater-assessed CON and GRS scores of novices continuously improved during all 3 training sessions, but never reached the level of experts. Of note, both experts and novices scored better in rater-assessed GRS and CON scores than in self-assessed GRS and CON scores, indicating that both novices and experts tend to underrate their own performance.

Table 2

Characteristics of study probands.

Table 3

Comparison of performance scores in training sessions 1–3 in surgical novices.

Figure 3

Main results. Total Objective Structured Assessment of Surgical Skills (OSATS) score, performance time (PT; in seconds), global rating scale (GRS) and confidence (CON) of the expert group (E) and the novices at training sessions 1, 2, and 3 (N1–3) are shown; numbers in parentheses indicate the number of probands. Box plots: thick lines indicate medians, boundaries, whiskers, and filled circles the 25th/10th/5th and the 75th/90th/95th percentiles, respectively; in case of the experts (N = 10), the medians and ranges are indicated instead. White circles are means, lines between N1-N3 show linear regressions through the means. Statistical significant differences between groups (Mann–Whitney U test; PT: t test) or over time (Wilcoxon signed rank test; PT: paired t test) are indicated by brackets within panels (in case of GRS/CON also vertical lines connecting panels); levels of significance are as follows: ∗, P < .05; ∗∗, P < .01; ∗∗∗, P ≤ .001.

We tested the construct validity of the OSATS model used in this study by comparing metric and nonmetric scores of experts and novices. We found that our model reliably discriminated between experts and novices regarding GRS, CON, PT, and OSATS scores. Specifically, GRS (1.1 ± 0.3 vs 2.3 ± 0.8; P < .001), CON (1.0 ± 0.0 vs 2.7 ± 0.9; P < .001), PT (125 ± 30 vs 152 ± 33 seconds; P = .02), and OSATS scores (19.6 ± 0.7 vs 18.8 ± 1.3; P = .02) were significantly different between experts and novices, all favoring the expert group. Table 4 shows GRS, CON, PT, and OSATS scores and other scores broken down by surgical expertise group. In addition, we tested the intra- and interobserver variabilities of the OSATS score assessment by having a subset of 25 randomly chosen videos scored by 4 different raters with one of them rating all videos 3 times. We found that the intra- and interobserver variabilities were low with agreement scores of 0.99 ± 0.03 and 0.64 ± 0.10 across subjects, and 0.99 ± 0.02 and 0.63 ± 0.32 across items (OSATS including GRS and CON). Corresponding Fleiss’ kappa values were 0.86 ± 0.07 and 0.49 ± 0.13, and 0.97 ± 0.07 and 0.49 ± 0.44, respectively. Figure 4 shows a graphical depiction of the agreement scores and Fleiss’ kappa values broken down by each of the OSATS score items and GRS and CON, and broken down by study proband.

Table 4

Construct validity of the training model.

Figure 4

Intra- and interobserver variabilities. Agreement score (solid lines; as described in the methods section) and Fleiss’ kappa (dotted lines) are shown broken down by each of the 20 Objective Structured Assessment of Surgical Skills (OSATS) score items, global rating scale (GRS), and confidence (CON) (panel A) and by proband (panel B).

In surgical novices, we assessed sex (female; n = 42 vs male; n = 16), handedness (right-handed; n = 55 vs left-handed; n = 3), regular sports activities (defined as at least 3 times/week; yes; n = 31 vs no; n = 27), and the type of Medical School Curriculum (regular curriculum; n = 54 vs an experimental ‘model’ curriculum; n = 4). Assessing these potential confounders, we found that OSATS scores were independent of handedness, sex, regular sports activity, or type of curriculum in univariate and multivariate analyses (Table 5).

Table 5

Dependence of scores, global rating scale, and confidence on proband characteristics.

Discussion

In this prospective trial, we found that repeated hands-on trainings improved the surgical skills of LLETZ in a surgical training model. The OSATS used for assessing surgical performance showed construct validity. This study adds to the literature supporting the use of training models for improving the surgical performance of residents. In addition to other procedures in the field of Obstetrics and Gynecology such as hysteroscopy,[ vaginal operative delivery by vacuum extraction,[ or resolving a shoulder dystocia,[ the current study suggests that LLETZ can also be effectively trained by using a dummy model. Based on the results of our study, we suggest that implementing this tool into clinical practice might improve the technical skills of residents, shorten their learning curve in the operating room, and might increase their motivation assuming that they feel more confident and have more satisfactory experiences during their first steps in the surgical theater. Furthermore, it may also improve patient safety.

Simulator trainings are time- and cost-intensive procedures. Raque et al,[ for example, analyzed the cost-effectiveness of a virtual reality simulator program. They found that the use of an endoscopic simulator increased the costs of training without accelerating the learning curve or decreasing faculty time spent for instruction. Therefore, inexpensive and effective simulators should be developed and rigorously tested in clinical trials, if simulator trainings are to be implemented into the clinical routine. Therefore, in the present trial, we tested a low-tech, inexpensive, and easy to use LLETZ simulator, based on previous experience of our own group and of others.[ We found that this low-tech tool is an effective means to train surgical novices in the technique of LLETZ. In contrast to other training gadgets such as virtual reality programs, the LLETZ training tool used in our study has a cost level < 10 USD to construct, can be easily assembled, and does not infer significant running costs.

One of the aims of our study was to establish the construct validity of the LLETZ training model used in the study. Therefore, we compared metric and nonmetric parameters between surgical novices and surgical experts. We found that construct validity of our porcine LLETZ training model was established, because parameters such as PT, GRS, CON, OSATS scores, and cutting time significantly differed between experts and novices favoring the expert group. Based on these results, we assume that this training model is representative of the real performance capacity of probands and is able to produce clinically meaningful results. In addition, we were interested in identifying the specific steps of the LLETZ procedure best defining who is an expert and who is not. GRS, CON, and cutting time were the items most significantly differing between experts and novices. Thus, one of the most significant characteristics of experts was that they used a well-dosed and slow cutting speed, whereas novices acted much more hastily. Thus, using an appropriate cutting speed should be emphasized when teaching novices how to perform LLETZ.

Based on a PubMed literature search (January 22, 2016; search terms: conization, LLETZ/LEEP, training, model, teaching, dummy), no data on the practical usefulness or the procedural efficacy of LLETZ training models have been published. Thus, our study is the first report indicating that LLETZ can be effectively trained before residents are allowed to perform this procedure in real patients in the operating room. Our study, however, has limitations. First, we have tested medical students, because they were naïve to LLETZ and other surgical procedures. On the other hand, obstetrics and gynecology residents might be different from this study population regarding motivation, self-selection, professional attitude, and procedure-specific theoretical knowledge. We have not assessed prior theoretical knowledge, a potential confounder, in our novice probands (eg, through a quiz taken before discussing any procedure-specific items or showing the training video), but it is unlikely that this plays a significant role as practical skills and (theoretical) knowledge do not necessarily correlate with each other and the effectiveness of the training model for increasing the practical skills of the probands was clearly demonstrated, not so much through an increase in the OSATS score, but by the significantly decreased procedure time, quality of conization (removal of the lesion), and rater and self-assessed global rating and confidence. Furthermore, our study does not guarantee that an improvement in LLETZ performance on a training model does also translate into an improved surgical performance in the operating room. Lastly, an improvement over 3 days might not be representative of a long-term improvement. Repeated LLETZ trainings over a long period might be necessary to reach the ultimate flattening of the learning curve.

Loss to follow up in our study was considerable with 24%. This might be because of the fact that medical students are less motivated to learn LLETZ compared with obstetrics and gynecology residents. Moreover, we have not used financial or other compensations for study participation for ethical reasons. We have not used systematic measures of evaluation to find out the reasons for nonadherence. A likely reason, however, was that because of a restriction imposed by our institutional review board we could not test students during the hours allotted to their rotation, but in their free time, and some decided to have better things to do. However, as participation was strictly voluntary, we believe that there was no motivation-related selection bias.

Conclusion

In summary, the results of our study indicate that repeated hands-on trainings using a LLETZ simulation model help to achieve a significant improvement of technical performance of surgical novices, and that the LLETZ training tool used in this study has construct validity. Therefore, this study supports the incorporation of LLETZ training models and OSATS into educational curriculums.

Acknowledgments

The authors would like to acknowledge the support by the Open Access Publication Funds of the Ruhr-Universität Bochum, Germany.