Can simulators be applied to improve cataract surgery training: a systematic review.

OBJECTIVE: The purpose of this paper was to conduct a systematic review of existing literature on simulation-based training of cataract surgery. Available literature was evaluated and projections on how current findings could be applied to cataract surgery training were summarised. The quality of included literature was also assessed. METHODS AND ANALYSIS: The PubMed, Embase and Cochrane Library databases were searched for articles pertaining to simulation training in cataract surgery on 18 November 2019. Selected articles were qualitatively analysed.
RESULTS: A total of 165 articles were identified out of which 10 met inclusion criteria. Four studies reported construct validity of the EyeSi simulator. Six studies demonstrated improved surgical outcomes corresponding to training on the simulator. Quality assessment of included studies was satisfactory.
CONCLUSION: Current studies on simulation training in cataract surgery all point towards it being an effective training tool with low risk of study biases confounding this conclusion. As technology improves, surgical training must embrace and incorporate simulation technology in training. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Individual studies have reported varying results ranging from no improvement to significant improvement in surgical performance of residents trained with simulators.

Simulator training is effective at both assessing simulated surgical competency and at reducing the complication rates of surgeries performed by residents who trained on them.

Future studies need to look at simulator-only regimes against non-simulator-only regimes to truly gauge how much of the improvement is due to simulator training.

Introduction

Surgery is a field that was quick to realise the potential of simulation training and was drawn to the possibility of practising complex procedures without risk to patients.1–4 In the past two decades, there has been an emergence of simulation training in many specialties including cardiothoracic,5 orthopaedic,6 laparoscopic7 and ophthalmic surgery.3

Surgical education has traditionally been conducted under the master-apprentice model with the trainee graduating through the model of ‘see one, do one, teach one’.8 This model, however, is limited through its dependence on patients,9 and high rates of complications among patients used as training cases for residents.10 Both of these are drawbacks that the use of simulators in training could negate.2 In recent years, many studies on the use of virtual reality simulators in ophthalmic surgery have been undertaken.11 Studies on simulated cataract surgeries have primarily attempted to either demonstrate construct validity of the EyeSi ophthalmic surgical simulator or attempted to demonstrate meaningful skill transfer from the EyeSi to real-life cataract surgery.11 While studies generally report positive patient-related outcomes and good efficacy of simulated cataract surgery,12 a systematic review of current literature is necessary to assess the state of current technology and to evaluate the nuances of which surgeons and residents stand to benefit most from simulation-based cataract surgery training.

This review assesses studies currently available that have evaluated the use of simulators in cataract surgery training.

Methodology

Literature search

The PubMed, Embase and Cochrane Library were searched using the following keywords: ‘cataract surgery’, ‘phacoemulsification’, ‘virtual reality’, ‘EyeSi and ‘training’. The searches were conducted on 18 November 2019. References of included studies were evaluated to find potential manuscripts.

Study selection

Selected studies were abstract reviewed by two authors (MARS and TMA). Both authors then reviewed the entire texts. Articles that were found to be mutually eligible by both authors were included.

Data extraction

Data from the articles was extracted into a spreadsheet. The design, skills trained, number and type of participants and outcome measures of each study were extracted. Skills trained on the EyeSi simulator were classified as navigation training, forceps training, bimanual training, antitremor training, capsulorhexis and phacoemulsification. Outcomes were classified as skill assessment, complication rate, skill acquisition and operating time. Data extraction was done by two authors (MARS and TMA).

Data terms

‘Navigation training’ indicates training modules on the EyeSi that involve the trainee moving a probe within the anterior chamber. ‘Skill acquisition’ refers to outcomes in which the impact of EyeSi training interventions is externally validated against different criteria than ones practised under, for example, Objective Structured Assessment of Cataract Surgical Skill (OSACSS) ‘skill assessment’ outcomes refer to the ability of the EyeSi to accurately gauge the skill level of users.

Risk of bias

The risk of bias of all included studies was gauged in accordance with the Cochrane Handbook.13 All articles were reviewed by both authors independently and assessed as unclear, low or high risk of bias.

Patient and public involvement

Given the nature of this study, it was not possible to involve the public or patients in its development.

Results

The initial search identified 165 articles. After abstract screening, 136 articles were removed for being non-ophthalmic. A total of 13 remaining articles were evaluated and 3 excluded. The studies that provided qualitative results evaluating the simulator and its effect on training were included. The process is depicted in figure 1.

Figure 1

Flow chart of study selection process.

Only the study from Staropoli et al14 had all bias items identified as low risk. The study by Staropoli et al was also the only study that had a protocol which enabled appraisal of its reporting bias. There were three single group studies for which assessment of allocation bias and performance bias was not feasible.15–17 The summary of bias assessment is illustrated in figure 2.

Figure 2

Risk of bias summary. Green circle=low risk of bias; blank=unclear risk of bias/not applicable to the study design; red circle=high risk of bias. Software used: Review Manager V.5.3 (The Nordic Cochrane Centre, The Cochrane Collaboration, Copenhagen, Denmark).

Study characteristics

All included studies were published between 2012 and 2019. All studies were conducted using the EyeSi simulation program. McConnel et al18 used the capsulorhexis intensive training curriculum add-on of EyeSi to specifically train residents for capsulorhexis. The study characteristics are summarised in table 1.

Table 1

Group attributes of included studies

Study attributes	Studies (number)	Participants (number)
All studies	10	453
Study design
Construct validity	4	76
Surgical outcomes	6	377
Participants*
Residents	6	137
Surgeons	5	316
Skills trained
Navigation training	6	116
Forceps training	5	94
Bimanual training	6	116
Antitremor training	6	116
Capsulorhexis	8	168
Phacoemulsification	5	84
Years of publication
2012–2016	2	58
2016–2019	8	395

*One study included both residents and surgeons resulting in the given overlap in study numbers.

Summary of results

Ten studies were deemed pertinent in answering the query of this literature search on how simulators can be used in the training of cataract surgery. Results show that the EyeSi Surgical Simulator can be used to train and to assess cataract surgery performance. Details of each study are summarised in table 2.

Table 2

Overview of individual study attributes

Study	Skills trained	Study type	Participants	Measured outcomes	Summary of effect
Jacobsen et al15	NavigationAntitremorForcepsBimanual trainingCapsulorhexisPhacoemulsification	Single group	19 surgeons	Correlation between OSACSS scores and EyeSi scores(Validity)	OSACSS scores correlated to EyeSi scores (p=0.003)
Ferris et al21	N/A	Retrospective cohort study	265 surgeons	PCR rates pre and post EyeSi(Surgical outcomes)	38% reduction in PCR rates following EyeSi implementation (p=0.003)
Bozkurt et al19	Navigation training,Forceps training,Bimanual trainingAntitremor trainingCapsulorhexis	Non-randomised group comparison	7 PGY16 PGY23 surgeons	Mean scores in the capsulorhexis module Non-dominant hand Mature cataract Performance improvement (Validity)	PGY1 <PGY2<surgeons PGY1 <PGY2<surgeons PGY1 <PGY2<surgeons Surgeons<PGY2<PGY1
Staropoli et al14	Navigation, Bimanual trainingAntitremor trainingCapsulorhexis, Phacoemulsification	Non-randomised group comparison	11 PGY3 residents trained prior to EyeSi implementation11 PGY3 residents trained after the instalment of the EyeSi	Posterior capsule ruptureVitreous prolapseRetained lens fragmentZonular dehiscenceEndophthalmitisIOL dislocationReturn to OR(Surgical outcomes)	Complication rate in the simulator trained was 2.1% vs 5.1% in the simulator-naïve (p=0.037)
Lucas et al22	Capsulorhexis, Phacoemulsification	Non-randomised group comparison	7 PGY2 pre EyeSi7 PGY2 post EyeSi	Posterior capsule rupture rateAphakia rateNucleus fragment dislocation rateExtracapsular conversion rate(Surgical outcomes)	Complication rate among the EyeSi trained residents was lower (p=0.031)
Rohipoor et al20	Navigation training,Forceps training,Bimanual trainingAntitremor trainingCapsulorhexis	Retrospective cohort study	30 residents	Relationship between EyeSi scores in early residency and surgical performance measures in the final year of residency(Validity)	Correlation between forceps training and navigation training scores on the simulator with total GRASIS scores
Thomsen et al17	Navigation training,Forceps training,Bimanual trainingAntitremor trainingCapsulorhexisPhacoemulsification (divide and conquer)	Single group	11 surgeons	EyeSi simulator scoresMotion tracking scores(Validity)	The two scores were strongest correlated (p=0.017)
Thomsen et al16	Navigation trainingForceps trainingBimanual training,Antitremor trainingCapsulorhexis Phacoemulsification (divide and conquer)	Single group, pre-post test	18 surgeons	Change in OSACSS scores after training intervention(Surgical outcomes)	Novice and Intermediate surgeons showed significant improvement after training (p=0.008, p=0.018) Experienced cataract surgeons did not benefit
Pokroy et al23	N/A	Non-randomised group comparison	20 residents	1 Incidence of posterior capsule tear with or without vitreous loss, and2 Operating time(Surgical outcomes)	Complication rate was not significantly different Residents trained on the EyeSi had quicker surgical times
McCannel et al18	Capsulorhexis	Single group, pre-post test	38 residents	The rate of errant CCC(Surgical outcomes)	There was a 68% reduction in the rate of errant CCCs in the postintervention cohort (p=0.0001)

CCC, continuous curvilinear capsulorhexis; GRASIS, Global Rating Assessment of Skills in Intraocular Surgery; N/A, not available; OSACSS, Objective Structured Assessment of Cataract Surgical Skill; PCR, posterior capsular rupture.

PGY1

PGY1

PGY1

Surgeons

Of the 10 studies, 4 evaluated the ability of the simulator to accurately discriminate between novice and experienced surgeons and the ability of the simulator to correlate real-life performance with simulation performance.15 17 19 20 These studies correlated participant scores on the EyeSi with parameters gauging real-life surgical experience of the participants. These studies established construct validity. The navigation training, forceps training, bimanual training and capsulorhexis models were the most extensively tested modules of the construct validity studies.

The remaining six studies assessed the impact that training on the EyeSi had on surgeon performance during real life surgery. These studies gauged the impact by the number of complications incurred in real-life cataract operations before and after formal training of participants on the EyeSi. Results showed a decrease in the complication rate conducted by surgeons with prior training on the EyeSi. Jacobsen et al looked at posterior capsular rupture (PCR) rates and McConnel et al looked at errant continuous curvilinear capsulorhexis (CCC) rates. Pokroy et al did not find a significant difference in complication rates of EyeSi-naïve versus EyeSi-trained residents.

Description of individual studies

Validity studies

Jacobsen et al15 investigated how performance on the EyeSi simulator correlates with real-life cataract surgery performance. The study consisted of 19 surgeons of varying surgical experience. The participating surgeons performed three real-life phacoemulsification procedures and were graded on them according to the OSACSS criteria. The participants were then immediately evaluated on a prevalidated simulation test on the EyeSi. Results showed a statistically significant correlation between the simulator performance score and the mean OSACSS score across all experience levels with a Pearson’s correlation of 0.65 (p=0.003, R2=0.42).

Bozkurt et al19 conducted a study evaluating if real-life surgical experience correlates with scores in the EyeSi cataract surgery simulator. A total of 16 participants were divided into three groups based on the number of cataract surgeries they had performed in the past. All participants performed and were graded on multiple simulated cataract surgeries on the EyeSi. Their scores on the simulator were then correlated with the experience group they fell under. Results showed that the more experienced group was found to have significantly different scores than the less experienced groups (p=0.009). Notably, the groups of physicians with less experience exhibited greater improvement over the course of their trial on the EyeSi indicating shorter learning curves for novices.

Rohipoor et al20 correlated the performance of 30 residents on the EyeSi during early residency with surgical experience and scores in their final year of residency. Surgical experience in their final year was gauged by the total number of phacoemulsification surgeries performed as the primary surgeon along with their scores on prevalidated Global Rating Assessment of Skills in Intraocular Surgery (GRASIS) evaluation forms during their final year. There was a significant correlation between the capsulorhexis task score on the simulator in early residency and the total number of cataract surgeries performed in their final year of residency (r=0.74, p=0.008).

A study conducted by Thomsen et al17 showed a correlation between a proficiency-based test on the EyeSi to real-life performance measured by motion-tracking software of cataract surgical videos. Eleven surgeons were recorded performing three standard cataract surgeries and then graded by validated motion tracking software. The motion tracking score was calculated by multiplying path length with the number of movements. A lower score indicated better surgical prowess. Results showed a strong inverse correlation (p=0.013) between scores on the EyeSi proficiency test and the motion-tracking score.

Surgical outcomes studies

Six studies assessed the impact training on the EyeSi had on surgeon performance in the operating room.14 16 18 21–23 These studies gauged the impact by the number of complications incurred in real-life cataract surgery before and after formal training of participants on the EyeSi.

Ferris et al21 conducted a retrospective audit of first and second-year trainees’ PCR rates over 7 years across 29 National Health Service ophthalmology units. Participating centres were contacted to ascertain the date when their surgeons had access to an EyeSi machine. All 16 871 operations by 265 surgeons were classified as before, after or no access to EyeSi. There was a 38% reduction in the first and second-year trainees’ unadjusted PCR rates from 4.2% in 2009 prior to EyeSi to 2.6% in 2015 for surgeons with access to an EyeSi.

Staropoli et al14 conducted a study evaluating cataract surgery complication rates of 11 EyeSi-trained residents vs 11 EyeSi-naïve residents at the same institute from the prior year. A sample size of 501 surgeries for the simulator trained and 454 surgeries for the simulator naive residents were analysed. The complication rate in the simulator-trained group was 2.1% compared with 5.1% for the simulator naïve. This was a statistically significant difference (p=0.037).

Lucas et al22 conducted a study investigating the complication rates of the first 10 cataract surgeries by two groups of second-year residents. Group 1 had no experience with EyeSi and group 2 had intermediate-level experience with EyeSi. Both groups consisted of seven residents each. A total of 140 surgeries were analysed, 70 by each group. The total number of complications was 19 (27.14%) in group 1 and 9 (12.86%) in group 2. This was a statistically significant reduction (p=0.031).

A study conducted by Pokroy et al23 found that training residents on the EyeSi simulator shortened their learning curve when exposed to real surgeries. The study analysed the posterior capsular rupture rate and operation duration of the first 50 cataract surgeries by two groups of residents at a single residency programme. The first group consisted of residents trained before the introduction of the EyeSi and the second group consisted of residents trained with the EyeSi following its introduction to the programme. Results showed that out of the 500 surgeries conducted by each group there were 40 PCR in group 1 and 35 in group 2. Notably, however, the simulator trained residents had higher PCR rates during their first 25 operations (10%) than the simulator-naïve group (8.8%). Through cases 26–50 however, these numbers dropped to 7.2% and 3.6%, respectively. While the overall complication rate was not significantly different between the two, simulator-trained residents showed shorter learning curves than their simulator-naïve counterparts.

McCannel et al18 compared the rate of errant CCCs during a simulator-trained and simulator-naïve group of residents. Errant CCCs were defined as attending physician take-over, radialisation of the CCC or conversion to a can opener capsulorhexis. A total of 1037 consecutive cataract surgeries performed at Harbor-UCLA Medical Center. The baseline cohort consists of 434 cataract surgeries performed prior to EyeSi intervention. The postintervention cohort consisted of 603 cataract surgeries performed during the following 2 years. There were 68 (15.7%) errant CCCs in the baseline cohort and 30 (5.0%) errant CCCs (p<0.0001) in the postintervention cohort. This corresponded to a 3.2-fold or 68% reduction in the rate of CCCs following the induction of EyeSi training into the residency programme.

Thomsen et al16 conducted a study in which 18 cataract surgeons of varying experience (novice: 0 surgeries, intermediate 0–75, experienced 75–999, expert >1000) were graded according to the OSCASS criteria during three cataract surgeries before and three cataract surgeries after a training intervention on the EyeSi. The simulation intervention consisted of achieving a passing score on the EyeSi simulator. The three surgeries before and after the intervention were graded on the OSACSS scale and results were compared. A comparison of results showed novice and intermediate surgeons showing improvements of 32% and 38%, respectively, after virtual reality training (p=0.008 and p=0.018). Experienced and expert cataract surgeons did not see a statistically significant improvement from simulator training.

Discussion

In the adoption of any new instrument, its validity must be proven. Gallagher et al24 define six criteria to gauge validity:

Face validity: will the instrument measure what it is supposed to measure? A subjective validation.

Content validity: an estimate of the validity of a testing instrument based on detailed examination of test contents. A rigorous subjective validation.

Construct validity: an evaluation of the degree of a testing instrument to identify the quality it was designed to measure. Often gauged by the ability of an instrument to differentiate novices from experts.

Concurrent validity: an evaluation in which the relationship between the test scores and the scores on another instrument purporting to measure construct are compared.

Discriminate validity: an evaluation that reflects the extent to which the scores generated by the assessment tool correlate with factors with which they should correlate.

Predictive validity: the extent to which the scores on a test are predictive of actual performance.

Face and content validity are proven in the early stages of instrument development and are not significantly consequential.24 Establishing construct and concurrent validity is the first step in the adoption of simulators to current surgical training programmes and assessment criteria. Four studies in current literature showed that the EyeSi has concurrent validity.15 17 19 20

Jacobsen et al15 correlated scores from the OSACSS criteria and found that participants’ OSACSS scores correlated with EyeSi scores across all experience levels. Rohipoor et al20 correlated GRASIS scores with EyeSi proficiency scores. Motion tracking-based grading of cataract surgeries was also found to be correlated with EyeSi scores by Thomsen et al.17 Lastly, the total number of surgeries performed by surgeons was also correlated with their EyeSi scores by Bozkurt et al.19 Through this the concurrent validity of the EyeSi cataract simulator was validated across four independent indices: OSCASS, GRASIS, motion-tracking grading and surgical experience.

Specific modules of the EyeSi were also construct validated. The capsulorhexis module, in particular, was validated by both Rohipoor et al20 and Bozkurt et al19 which showed that EyeSi was able to accurately differentiate between novice and experienced surgeons. Capsulorhexis is generally believed to be the most accurately simulated step currently available in the EyeSi cataract simulator18 and these findings corroborate that belief. Antitremor and forceps training on the EyeSi were also individually construct validated against GRASIS scores by Rohipoor et al.20

Of all included studies, only Rohipoor et al15 17 19 20 established predictive validity through looking at how EyeSi scores of residents in early residency positively correlated with performance in their final year of residency.

These findings demonstrate that individual models of the EyeSi are independently correlated to established indices. Further analysis of which specific models are most strongly correlated with real-life performance could yield valuable information on how to further optimise both EyeSi grading software.

Building on the growing body of evidence supporting construct and concurrent validity of the EyeSi, numerous ophthalmic residency programmes have recently incorporated the EyeSi into their training programmes.14 18 21–23 This has led to studies that compare the outcomes of EyeSi-integrated training programmes with the outcomes of traditionally structured programmes. These studies all report positive outcomes associated with supplementing training programmes with the EyeSi.14 18 21–23 All but one study reported a statistically significant reduction in the complication rates of surgeons trained with EyeSi compared with their EyeSi-naïve counterparts.23

The risk of bias assessment shows that only one study was fully bias free. The majority of the bias in the studies arose under ‘other bias’ and was due to studies comparing complication rates between EyeSi-naïve and EyeSi-trained batches of residents. With each successive year, training protocols, knowledge and resources available to residents improve. As such, a natural improvement over the years is expected, which cannot be controlled for when comparing two cohorts of residents from different years, as acknowledged by Ferris et al.21 Moreover, there exists an inherent variation in residents’ skill within each year that is not neutralised due to the small sample size of trainees. Despite this bias however, all studies generally ranked low in bias risk and adhere to acceptable reporting standards.

Moving forward, the projected impact that adoption of the EyeSi can have on training programmes lies in its potential for improving patient safety and the earlier competency gains of surgical trainees. Under traditional training models, patients’ eyes serve as a training ground for residents in training.9 Cataract surgery conducted by trainee surgeons is associated with increased complication rates.25 With EyeSi it may no longer be necessary to compromise on patient safety in order to facilitate resident training. The EyeSi allows for residents to gain hands-on equipment and situational experience. This simultaneously shifts the learning curve of phacoemulsification towards the safe and controlled environment of the simulator where complications do not result in harm to patients. Of the studies evaluating complication rates, four of five also showed significantly decreased complication rates by residents who trained on the EyeSi prior to conducting real-life operations.14 18 21 22 This shows how the EyeSi objectively improves patient health outcomes by foregoing both the learning curve in a controlled environment and by training surgeons who go on to have lower complication rates. Meanwhile, the four construct validity studies unanimously showed that the EyeSi simulator realistically simulated many aspects of cataract surgery for the trainee.15 17 19 20 This proves that the EyeSi also does not compromise on the quality of resident training.

A big hurdle in wider adoption of high fidelity simulation training, particularly in low-income and middle-income countries, is the significant initial cost of purchasing a simulator as well as the ongoing maintenance and upgrade costs: the purchase cost of an EyeSi is ~£100 000–£150 000, with ongoing maintenance costs in the region of £5000–£10 000 per year for the cataract module only. There may also be additional costs of upgrading modules that undergo iterative improvements.

These costs may be mitigated in a number of ways: share with other teaching programmes within the same institution where additional EyeSi modules can be acquired, for example, the vitreo-retinal module, in effect sharing the acquisition cost. The cost of acquisition for an individual centre could further be diluted by sharing the simulator with other institutions in designated regional training centres. Pooling resources allows a number of trainees to undergo simulation training as an introductory step prior to intraocular surgery. This has the additional benefit of aligning divergent training programmes in a region to a similarly high level of standardised basic training.

The cost of simulation also needs to be balanced against the long-term cost benefits of dealing with fewer per-operative complications, and the sequelae of complications. The undoubted safety benefit to patient not being on a trainee’s learning curve may not be something that can be easily costed in a tangible fashion.

Discussions among surgical trainers in the UK in recent years report fewer trainee complications which is resulting in a greater prominence of simulation training rather than wet-lab training prior to intraocular surgery within many training programmes in the UK.21 Because of this, many regions in the UK are adopting the above model and it will be interesting to note whether these anecdotal reports are reflected in the upcoming Royal College’s National Ophthalmology Database audit, and whether the complication rates differ between simulation-induced trainees and the rates of trainees prior to simulation training.

Strengths and limitations

A strength of this study is that in our understanding it is the first systematic review conducted on simulator training in cataract surgery within the past 6 years.12 26 Given the novelty of this technology and the volume of new studies published on it, this represents a significant interval which requires review. Second, the study is conducted as per the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.27 A limitation of this study is that unpublished literature was not searched for which may have led to only studies reporting positive results being selected due to selective reporting bias. The second limitation is the lack of current studies comparing EyeSi-only training regimes against traditional-only training regimes to truly gauge how much of the improvement is due to EyeSi training. This is crucial as current studies subject their intervention groups of trainees to EyeSi training in addition to the same traditional-only training their controls also undergo. This results in the intervention cohorts having greater net training hours than the control cohorts. It stands to reason that the addition of any supplementary training programme will bring about some improvement in trainee performance. This confounds the degree to which the noted improvement in the intervention cohorts may be credited to EyeSi.

Conclusion

In this systematic review of the available literature, the EyeSi simulator has shown to be a tool that can accurately assess surgeon proficiency. Risk of bias analysis of included studies also ruled out any significant bias that may have influenced the conclusions made in this review. Moreover EyeSi has also shown to augment current resident training programmes with evident improvements in patient outcomes in cataract surgery. It may be argued that simulated surgery moves the learning curve of manipulating instruments within the anterior chamber to a simulated environment away from patients, reducing the risk of harm. As technology continues to improve further, surgical training must embrace and incorporate simulation technology in training.