Intense Simulation-Based Surgical Education for Manual Small-Incision Cataract Surgery: The Ophthalmic Learning and Improvement Initiative in Cataract Surgery Randomized Clinical Trial in Kenya, Tanzania, Uganda, and Zimbabwe.

Importance: Cataracts account for 40% of cases of blindness globally, with surgery the only treatment. Objective: To determine whether adding simulation-based cataract surgical training to conventional training results in improved acquisition of surgical skills among trainees. Design, Setting, and Participants: A multicenter, investigator-masked, parallel-group, randomized clinical educational-intervention trial was conducted at 5 university hospital training institutions in Kenya, Tanzania, Uganda, and Zimbabwe from October 1, 2017, to September 30, 2019, with a follow-up of 15 months. Fifty-two trainee ophthalmologists were assessed for eligibility (required no prior cataract surgery as primary surgeon); 50 were recruited and randomized. Those assessing outcomes of surgical competency were masked to group assignment. Analysis was performed on an intention-to-treat basis. Interventions: The intervention group received a 5-day simulation-based cataract surgical training course, in addition to standard surgical training. The control group received standard training only, without a placebo intervention; however, those in the control group received the intervention training after the initial 12-month follow-up period. Main Outcomes and Measures: The primary outcome measure was overall surgical competency at 3 months, which was assessed with a validated competency assessment rubric. Secondary outcomes included surgical competence at 1 year and quantity and outcomes (including visual acuity and posterior capsule rupture) of cataract surgical procedures performed during a 1-year period.
Results: Among the 50 participants (26 women [52.0%]; mean [SD] age, 32.3 [4.6] years), 25 were randomized to the intervention group, and 25 were randomized to the control group, with 1 dropout. Forty-nine participants were included in the final intention-to-treat analysis. Baseline characteristics were balanced. The participants in the intervention group had higher scores at 3 months compared with the participants in the control group, after adjusting for baseline assessment rubric score. The participants in the intervention group were estimated to have scores 16.6 points (out of 40) higher (95% CI, 14.4-18.7; P < .001) at 3 months than the participants in the control group. The participants in the intervention group performed a mean of 21.5 cataract surgical procedures in the year after the training, while the participants in the control group performed a mean of 8.5 cataract surgical procedures (mean difference, 13.0; 95% CI, 3.9-22.2; P < .001). Posterior capsule rupture rates (an important complication) were 7.8% (42 of 537) for the intervention group and 26.6% (54 of 203) for the control group (difference, 18.8%; 95% CI, 12.3%-25.3%; P < .001). Conclusions and Relevance: This randomized clinical trial provides evidence that intense simulation-based cataract surgical education facilitates the rapid acquisition of surgical competence and maximizes patient safety. Trial Registration: Pan-African Clinical Trial Registry, number PACTR201803002159198.

RCT Entities:   Population Interventions Outcomes

Introduction

Of the 36 million people globally who are blind, more than one-third have blindness due to cataracts.[1] Surgery remains the only treatment option for cataracts. An estimated 14 million cataract operations are performed globally annually.[2,3] Cataract surgery can effectively restore vision, is one of the safest and most cost-effective of all health care interventions, and confers a large financial return on investment.[4,5] However, in many regions, the rate of cataract surgery is insufficient to address the burden of avoidable blindness.

Of the more than 230 000 ophthalmologists worldwide, the lowest mean number of ophthalmologists per million population is found in sub-Saharan Africa, at 2.5.[6] The global estimated mean number of ophthalmologists per million population is 31.7; however, less than half perform cataract surgery (mean number, 14.1 ophthalmologists per million population).[6] There is an urgent need to train and equip more ophthalmic surgeons to address the burden of surgically treatable blindness.

In sub-Saharan Africa, the median number of cataract surgical procedures performed by trainee ophthalmologists in the first 2 years of training was zero.[7] In mainland China, the median number of cataract surgical procedures performed by senior trainees by the end of 3 years of training was zero.[8] Traditional surgical education is resource intensive. Slowly building surgical competence through trial and error by practicing solely on patients is unethical, and maximizing patient safety and reducing surgical errors must be priorities. Simulation-based education can help address this training need, especially in low-income settings where the disease magnitude is greatest.[9]

Intensive simulation-based surgical education has been shown to increase surgical skills and decrease complication rates.[10] During the past 10 years, randomized clinical trials (RCTs) have been conducted for surgical education, predominantly in laparoscopic surgery.[11] The literature on simulation-based surgical education in eye care, however, is inadequate, despite widespread adoption and large expenditure.[12]

Many animal, cadaver, artificial, and virtual reality models have been used in ophthalmic surgical education, including for cataracts.[12,13,14,15] Retrospective studies have shown a reduction in complication rates with access to, and mandatory training using, a virtual reality simulator for cataract surgery training.[16,17] Recent systematic reviews of trials involving simulation-based training or assessment of ophthalmic surgical skills concluded that studies are heterogeneous and that methodological rigor is inadequate.[12,18]

We therefore designed and conducted the Ophthalmic Learning and Improvement Initiative in Cataract Surgery (OLIMPICS) Trial. The aim of the trial was to evaluate the effect of intense simulation-based surgical education in cataract surgery on surgical competence, as well as subsequent live surgery outputs and outcomes compared with conventional training alone.

Methods

Study Design

We designed a multicenter, multicountry, investigator-masked, parallel-group RCT conducted from October 1, 2017, to September 30, 2019. Competency was assessed at baseline and in follow-up assessments over the course of 15 months. Trainee ophthalmologists from 5 ophthalmology training program institutions in Nairobi, Kenya; Moshi, Tanzania; Kampala and Mbarara, Uganda; and Harare, Zimbabwe were assessed for eligibility. Written informed consent was obtained from all participants. Participants were given no incentives or compensation. No changes to methods were made after trial commencement (trial protocol in Supplement 1). Ethical approval was attained from 10 separate research ethics committees. Full details are in Supplement 1.

Participants and Prerandomization Baseline Assessment

Inclusion criteria included having performed zero complete manual small-incision cataract surgery (SICS) procedures as primary surgeon and having performed parts of (or assisted in) fewer than 10 separate SICS procedures. After consent, participant trainees were evaluated in country. Baseline assessment included recorded performance of 3 surgical simulation procedures each. These assessments were anonymized and remotely graded in a masked fashion using the Ophthalmic Simulation Surgical Competency Assessment Rubric (Sim-OSSCAR).[19] A standardized knowledge assessment was also administered, providing further baseline data. Participants were assured of confidentiality and anonymity of individual outcome assessments.

Randomization

The randomization sequences were computer generated centrally by a statistician (M.J.K.) based at the London School of Hygiene & Tropical Medicine who was independent of all other aspects of the trial. We randomly allocated candidates at the site level into batches of 2 or 4 trainees, with equal numbers of intervention and control allocations in each batch. Preprinted allocation cards that specified the center, batch group, unique identifier, and allocation (intervention or control) were concealed inside opaque sealed envelopes. This ensured that the principal investigator, coinvestigator, and participants had no prior knowledge of the allocation until the envelopes were opened. All the envelopes in the batch had an identical external appearance and batch label code. All trainees in the batch were each invited to simultaneously select and open one of the envelopes and to reveal their allocation card. If an odd number of participants were identified in a center, the final participant was invited to select 1 of 2 identical envelopes in a batch of 2. This ensured randomization, as all candidates had an equal chance of being in either group.

Intervention

The simulation-based training was conducted at the purpose-built Surgery Training Unit, University of Cape Town, South Africa. The SICS procedure was deconstructed and instruction on individual steps was achieved using the Peyton 4-stage approach to teaching a practical skill.[20] Feedback was given to participants while they engaged in sustained deliberate practice of a particular step.[21] Once all parts of the SICS procedure were covered, the full procedure was performed on high-fidelity synthetic simulation eyes,[22] after a round of mental rehearsal (the cognitive rehearsal of a task before practice).[23] Participants were able to record their surgical performance and engage in reflective learning by watching their performance on an iPad.[24] This was enhanced by formative assessment and outcome measurement as they graded their performance against the Sim-OSSCAR.[19] All training was conducted by one of us (W.H.D.). The study protocol and standard operating procedures, including a detailed description of the intervention, are available in Supplement 1.

Control participants were offered the same training in Cape Town, South Africa, after 1 year. Both the intervention and control groups continued to undergo conventional postgraduate ophthalmology training.

Outcomes

Participants were followed up at 3 months after the intervention and at 1 year. Assessments included 3 sequential simulation SICS procedures recorded in the same manner as the baseline assessment. There was no time limit on the surgical procedure recordings. Further assessments included a supervised live SICS procedure at 12 months and a summary report of cataract surgery numbers and outcomes over 1 year. No changes to study outcomes were made after trial commencement.

The primary outcome measure was the difference in Sim-OSSCAR[19] scores between groups at 3 months. Each of the 20 items in the matrix was graded on a modified Dreyfus score (novice, advanced beginner, and competent). The minimum score was 0 points and the total possible score was 40 for each procedure. Masked assessments were performed remotely by 2 independent expert SICS surgeons (S.M. and L.H.-W.).

Secondary outcome measures included assessment of surgical competence at 12 months (live and simulation), number of live SICS procedures performed, and surgery outcomes for a period of 12 months. Number and outcomes of live SICS procedures performed were self-reported retrospectively in a summary report after 12 months.

Statistical Analysis

Based on data from a pilot study, we anticipated a difference in Sim-OSSCAR scores between groups of 9 of 40 points, and an estimated variability of 0.9 SD. We therefore calculated that a sample of 23 individuals in each group would have 80% power and 95% confidence to detect a significant difference in scores. We aimed to recruit 25 individuals per group, to provide 2 extra participants per group for any loss to follow-up.

The distributions of baseline variables by treatment group were compared. The primary outcome measure was the mean score of 3 masked assessments at 3 months of simulation surgical performance using the Sim-OSSCAR.

Intention-to-treat analysis was used for all outcome measures. Primary analysis included a linear regression model with mean Sim-OSSCAR scores at 3 months as the outcome and trial group as the exposure, adjusting for baseline mean Sim-OSSCAR score taking training center as a random effect. A similar approach was used for secondary outcome measures of competence. Mean live SICS procedure ICO (International Council of Ophthalmology) Ophthalmology Surgical Competency Assessment Rubric (ICO-OSCAR)[25] score at 1 year was analyzed by a t test. The number of surgical procedures performed in 1 year was analyzed using a Poisson regression, with trial group as the exposure of interest, adjusting for training center. Patient-specific outcomes for all surgical procedures performed during the 12-month period included the number of patients with poor postoperative visual acuity per surgeon, analyzed using the Wilcoxon rank-sum test. Further assessment included percentage rates of operative complications of posterior capsule rupture (PCR), analyzed using linear regression.

An α level of P < .05 was considered statistically significant for the primary outcome. P values were 2-sided. A κ coefficient of 0.75 or more for interassessor agreement of video grading scores was considered to be excellent.[26]

Data were initially entered into Microsoft Excel, version 15.31 (Microsoft Corp). Statistical analysis was performed using Stata, version 15.1 (StataCorp). A data monitoring and trial advisory committee oversaw the study.

Prevention of Bias

It is accepted that there will be variability in individual participants’ inherent or natural surgical aptitude. All efforts were made to standardize the training offered to the intervention participants (as well as to the control participants after the 1-year period). The intense simulation course was held in the same standardized surgical training unit, and all training was conducted by one of us (W.H.D.). Recordings of live and simulation surgical procedures were anonymized. Every effort was made to reduce contamination bias. Numerous standard risk-of-bias criteria may be used to evaluate RCTs. These criteria are further illustrated in the trial protocol (Supplement 1).

Results

A total of 52 potential participants were assessed for eligibility between October 1, 2017, and May 21, 2018. Fifty participants were recruited, and 49 participants were included in the final intention-to-treat analysis (Figure 1). Two potential participants were excluded before randomization owing to prior surgical experience. One trainee in the control group completed baseline assessments but suddenly left the training program, and was not contactable. All 5 ophthalmology training programs contributed participants (4 from the Kilimanjaro Christian Medical Centre, 8 from Mbarara University, 10 from Makerere University, 17 from the University of Nairobi, and 10 from the University of Zimbabwe). There were no unintended effects in each group.

Figure 1.

Trial Flowchart

The control group received the allocated intervention after an initial follow-up period of 1 year.

Table 1 shows participants’ baseline characteristics. There was good balance between groups. A total of 757 videos from across the different time points were independently graded in a masked fashion, each by 2 graders, of which 297 baseline and 3-month recordings contributed to the primary outcome measure. Interobserver reliability correlation showed a κ coefficient of 0.86 for total scores. Intraobserver agreement was 0.87.

Table 1.

Baseline Characteristics of the Intention-to-Treat Population

Characteristic	Intervention group (n = 25)	Control group (n = 24)
Age, mean (SD), y	32.4 (5.0)	32.2 (4.3)
Sex, No. (%)
Female	16 (64.0)	10 (41.7)
Male	9 (36.0)	14 (58.3)
Year of training, mean (median)	1.4 (1)	1.5 (1)
MCQ score, mean (SD), %	60.2 (4.7)	65.8 (3.3)
SICS procedures assisted or partially performed, mean (median)	0.6 (0)	0.6 (0)

Abbreviations: MCQ, multiple-choice question; SICS, small-incision cataract surgery.

The mean (SD) Sim-OSSCAR scores at 3 months were 33.7 (3.0) (84.3% of points) for the intervention group and 17.9 (5.9) (44.8% of points) for the control group (P < .001) (Table 2). Linear regression analysis of Sim-OSSCAR scores at 3 months, taking into account center clustering, illustrated a large effect of the intervention. Those who received the training were estimated to have unadjusted scores 15.8 points higher (95% CI, 13.2-18.5) (P < .001) than those who did not receive the training. The difference in Sim-OSSCAR scores was 16.6 points higher (95% CI, 14.4-18.7) with adjustment for baseline scores (P < .001) (Table 2 and Figure 2).

Table 2.

Objective Evaluation of SICS Sim-OSSCAR Scores at Baseline and 3 Months

SICS simulation competency	Mean (SD) [%]a		Difference score, %	95% CIa	P value
	Intervention group score	Control group score
Baseline	10.8 (6.7) [27.0]	12.8 (6.9) [32.0]	2.0 (5.0)	−1.9 to 5.8	.32
At 3 mo	33.7 (3.0) [84.3]	17.9 (5.9) [44.8]	15.8 (39.5)	13.2 to 18.5	<.001
At 12 mo	32.9 (3.6) [82.3]	24.4 (5.5) [61.0]	8.2 (15.5)	5.5 to 11.0	<.001
At 15 mo	33.5 (1.7) [83.8]	35.4 (2.2) [88.5]	1.9 (4.8)	−1.5 to 5.3	.26

Abbreviations: SICS, small-incision cataract surgery; Sim-OSSCAR, Ophthalmic Simulation Surgical Competency Assessment Rubric.

Scores are out of a possible maximum score of 40.

Figure 2.

Surgical Competency Scores by Group

Arrows indicate training intervention for intervention and control groups. Sample sizes for groups at baseline are intervention, 25 and control, 24; at 3 months are intervention, 25 and control, 24; at 1 year are intervention, 22 and control, 23; and at 15 months are intervention, 10 and control, 16. Scores are of a possible total of 40. Sim-OSSCAR indicates Ophthalmic Simulation Surgical Competency Assessment Rubric.

The mean (SD) Sim-OSSCAR scores at 1 year were 32.9 (3.6) (82.3%) for the intervention group and 24.2 (5.5) (61.1%) for the control group (Table 2). Scores at 1 year were 8.5 points (95% CI, 6.7-10.9; P < .001) higher in the intervention group compared with the controls, adjusting for baseline scores, supporting a continued benefit from the training intervention.

Live surgical performance on patients was recorded anonymously at the 1-year mark for both groups, before the training course intervention began for the control participants. The mean surgical competency score using the ICO-OSCAR was 62.3 of 95 (65.6%) for the intervention group and 45.0 of 95 (47.4%) for the control group (difference, 17.3 points; 95% CI, 5.2-29.3; P = .006).

The total number of live SICS procedures performed in 1 year (from 0 to 12 months) was recorded for each participant. Intervention group participants performed a mean of 21.5 surgical procedures as the primary surgeon and assisted in 24.6 cataract surgical procedures. Control group participants performed 8.5 surgical procedures as the primary surgeon and assisted in 10.9 cataract surgical procedures during the same period. The mean difference was 13.0 surgical procedures (95% CI, 3.9-22.2; P < .001). Poisson regression analysis, with trial group as the exposure of interest, adjusting for training center, showed strong evidence that those who received the intervention training performed more live surgical procedures (as primary surgeon or assistant) than did those in the control group; those receiving the intervention performed 2.5 times (95% CI, 2.2-3.0) as many surgical procedures as those who did not.

The proportion of good outcomes (day 1 presenting visual acuity, ≥6/18) was 36.8% (138 of 375) and of poor outcomes (presenting visual acuity, <6/60) was 10.1% (38 of 375) for the intervention group; for the control participants, the proportion of good outcomes was 25.6% (30 of 117) and of poor outcomes was 12.8% (15/117). There was no significant difference in the proportion of good or poor outcomes between groups (Wilcoxon rank-sum P = .90 for the intervention group and P = .95 for the control group).

The mean PCR proportion during the 1 year after the training intervention was 70.7% lower at 7.8% (42 of 537) for intervention trainees, compared with 26.6% (54 of 203) for the control participants for the same 12-month period (difference, 18.8%; 95% CI, 12.3%-25.3%; P < .001). Figure 3 illustrates the regression plot of the number of cataract surgical procedures performed and number of PCRs by group. For those who had performed surgery, logistic regression (where the unit is surgery, outcome is PCR, and intervention is the only difference) illustrated a strong effect of the intervention. Intervention participants had a higher chance of having no PCR (odds ratio, 4.27; 95% CI, 2.74-6.65; P < .001).

Figure 3.

Linear Regression of Number of Posterior Capsule Ruptures and Number of Cataract Surgical Procedures by Group

Data from 21 intervention and 16 control trainees. Shaded areas indicate 95% CIs.

Discussion

The OLIMPICS trial has demonstrated that an intense 5-day simulation-based cataract surgical education course successfully improved the main outcome of cataract surgical competence at 3 months, and that the benefits persist over 1 year. There is evidence from secondary outcomes that live surgical performance was improved and patient safety benefited from reduced surgical complication rates. This multicenter RCT supports the use of intense simulation training for cataract surgery.

Although the trainees in the intervention group performed and assisted in more live cataract surgical procedures in the year after the intervention training, it is unlikely that the better competency scores in the intervention group at 3 months are a result of having performed more SICS procedures. This is because most cataract surgery cases performed as primary surgeon were after the first 3 months of the study.

The implications for real-world training programs are compelling. Trainee eye surgeons should be afforded the opportunity to participate in focused, intense simulation training courses. We believe that supervised live surgical training on patients should begin only after engaging in adequate deliberate practice with feedback, reflective learning, and a competency outcome assessment benchmarked to appropriate standards. The International Council of Ophthalmology has developed a comprehensive residency curriculum and standards for graduates to have basic competence before performing cataract surgery.[27]

The implications for patient safety are ethically imperative. We illustrated a dramatic 70.7% reduction in surgical complication rates in the cases performed as primary surgeon in the first year of conventional training. Retrospective studies have shown that access to a virtual reality simulator for cataract surgery training (Eyesi; VR Magic) resulted in a 38.1% reduction in PCR rates for cataract surgical procedures performed by junior trainees in the UK, from 4.2% to 2.6%.[16] Mandatory simulator training for novice residents in the US showed a retrospective comparative reduction in PCR rates from 4.8% to 2.2%.[17] A retrospective study in India of wet-laboratory cataract surgery training using goat eyes showed PCR rates of 14.3% vs 6.9%.[28]

Limitations and Strengths

This study has some limitations. A potential limitation of the OLIMPICS trial is the use of the Sim-OSSCAR[19] rather than live surgical competency assessment with the ICO-OSCAR[25] as the primary outcome measure. We argue, however, that this is a strength. The simulation environment and use of the validated Sim-OSSCAR affords participants the chance to complete as much of the cataract surgery procedure as they can without potential harm to patients, whereas live surgery is prone to greater variation that impairs its use for comparative purpose with small samples. All live surgery performed at the 12-month assessment was supervised by a local senior surgeon. At their professional discretion, they could take over surgery at any time, and for that part of the procedure the trainee would score zero on the live ICO-OSCAR rubric. The live surgical competency scores are therefore more complex to interpret. They are based on the variable takeover threshold of different senior surgeons; the comorbidity, risk-stratification, and complexity of a particular case; the confidence level of an individual trainee; and other factors. The use of the simulation artificial eye afforded a standardization that would not have otherwise been achievable in the live surgical setting. Furthermore, it would have been unsafe and unethical for untrained surgeons to be evaluated on surgical procedures performed on patients at a very early stage. Limitations of the study also include variability in training opportunities and training environments. To mitigate against this variability, the randomization was stratified by institution, resulting in equal numbers of intervention and control participants within an institution. This may, to a large extent, compensate for the inter-institutional variability, leading to balance between trial groups in factors such as cataract case mix (number and complexity). Another potential limitation, which is impossible to quantify, is the Hawthorne effect, whereby the behavior of participants of a study is altered owing to their awareness of being observed.[29]

This study also has some strengths. The strengths of the OLIMPICS trial are its RCT methodology, standardized intervention training for all participants, investigator masking, and double marking of all 757 surgical videos (each video was marked by 2 independent graders).

A critical review of simulation-based medical education suggested 12 areas or features of best practice,[30] many of which had been identified by other educational theorists. Of these, skill acquisition and maintenance, feedback, sustained deliberate practice, curriculum integration, outcome measurement, and simulation fidelity are key.[10] These findings suggest that simulation-based surgical education should not be perceived as merely having access to a wet laboratory, dry laboratory, or computerized or full-immersion virtual reality simulator. For greatest impact, simulation-based surgical education should be seen and used as a comprehensive educational package. Part of this included the digital classroom, where procedures are recordable so that the trainee gets feedback on the whole process and can also review it themselves, engaging in critical reflective learning.

Conclusions

The OLIMPICS trial illustrated a positive effect on patient safety. Not only are trainees and trainers afforded a safe, calm, and effective environment to teach and learn away from patients but the result appears to be a substantial reduction in the rates of surgical complications. With RCT-level evidence of the utility of intense simulation-based surgical education for cataract surgery, the opportunity is presented for us to protect the patients we and our trainees serve, to collectively and collaboratively work together to have this approach to surgical education implemented and mandated.