The development of laparoscopic skills using virtual reality simulations: A systematic review.

BACKGROUND: Teaching based on virtual reality simulators in medicine has expanded in recent years due to the limitations of more traditional methods, especially for surgical procedures such as laparoscopy. PURPOSE OF REVIEW: To analyze the effects of using virtual reality simulations on the development of laparoscopic skills in medical students and physicians. DATA SOURCES: The literature screening was done in April 2020 through Medline (PubMed), EMBASE and Database of the National Institute of Health. ELIGIBILITY CRITERIA: Randomized clinical trials that subjected medical students and physicians to training in laparoscopic skills in virtual reality simulators. STUDY APPRAISAL: Paired reviewers independently identified 1529 articles and included 7 trials that met the eligibility criteria.
FINDINGS: In all studies, participants that trained in virtual simulators showed improvements in laparoscopic skills, although the articles that also had a physical model training group did not show better performance of one model compared to the other. LIMITATIONS: No article beyond 2015 met the eligibility criteria, and the analyzed simulators have different versions and models, which might impact the results.
CONCLUSION: Virtual reality simulators are useful educational tools, but do not show proven significant advantages over traditional models. The lack of standardization and a scarcity of articles makes comparative analysis between simulators difficult, requiring more research in the area, according to the model suggested in this review. SYSTEMATIC REVIEW REGISTRATION NUMBER: Registered by the Prospective Register of Systematic Reviews (PROSPERO), identification code CRD42020176479.

Introduction

In 1950 the time it took for medical knowledge to double was estimated to be 50 years, whilst in 2020 that time would be 73 days [1]. To keep up with this growth and adapt to the challenges that healthcare presents, new technologies involving both the social role of the profession and changes in the healthcare environment are considered promising complementary tools [2]. They can be used both to treat diseases and promote health (e.g. smokers [3] and the chronically ill [4]), as well as to help professionals with their practice and training (e.g. recognition of sepsis [5] and trauma screening [6]).

Among these new technologies, virtual reality (VR)—a computer simulation where the physical presence of the user is projected in a virtual environment [7]—is gaining in popularity and becoming more accessible [8-10]. Its development has been rapid, showing great application to education and training [11, 12]. Within the medical field, where its use is already widespread in procedures, diagnoses and professional training, virtual reality has great potential for expansion [13].

In the surgical context, the use of VR is highlighted in training for minimally invasive procedures, such as laparoscopy, through the use of different simulators on the market—MIST-VR, LapSim, laparoscopy VR and SINERGIA [14]. Specifically for this type of procedure, learning occurs empirically through trial and error until the technique is perfected, with the learning curve revolving around 65 procedures for laparoscopists [15]. In this way, the use of virtual reality simulators (VRS) provides safe, controlled environments, with reusable resources and with techniques that can be more easily measured when compared to practice in real models, reducing the learning curve [16].

Over the past decade, although some literature reviews have sought to analyze the use of VRS for the education and training of health professionals in a general surgical context, most do not address its use for laparoscopy specifically. In 2004, Aggarwal R. et al. [17] suggested the need to validate curricula to improve medical teaching in surgery. Willis R.E. et al. [18] noted that this technology would be more similar to a video game than to a training method.

On the other hand, most reviews that sought to analyze the VRS within the laparoscopy training field faced significant limitations [19-22]. According to Nagendran M. et al. [19], the use of VRS could decrease the surgical time of surgical trainees with laparoscopic experience—the articles analyzed showed approximate reductions of between 30–58%. Despite the promising results, this review faced many difficulties, as all trials analysed were classified as high risk of bias, only two compared VRS with a different training method, and one of them did not fully disclose the magnitude of results. Another review by Alaker M et al. [20] also studied the effects of VRS and showed encouraging results, though the bulk of its articles focused on comparisons between VRS and no training-groups, and it did not make considerations for the costs of modern VRS and how they compare to other training methods.

For these reasons, a new systematic literature review is needed, analysing the effects of the use of virtual reality simulations on the development of laparoscopic skills in medical students and physicians.

Objectives

Develop a systematic review of the literature to analyse the effects of using virtual reality simulations on the development of laparoscopic skills in medical students and physicians.

Methodology

This systematic review was carried out in accordance with the items of Preferred Reports for Systematic Reviews and Protocol Meta-Analysis (PRISMA-P) [23]. This study was registered by the Prospective Register of Systematic Reviews (PROSPERO, identification code CRD42020176479) before the research was carried out.

The elaboration of the scientific question was based on the PICO strategy [24], considering: medical professionals or students (patient or problem); use of virtual reality and physical model simulations (Intervention); there is no standard comparator to be considered in this study (Control or Comparison); all outcomes available in the literature were considered in the analysis (outcome).

Eligibility criteria

Types of studies

The articles were selected based on their titles and abstracts according to the relevance of their data regardless of their publication status. Only clinical trials were considered.

Types of participant

Study participants were medical students and physicians who underwent VRS training aimed at developing laparoscopic skills.

Types of evaluation

Only physical models simulators (PMS) were considered as evaluation models, which were defined as simulations with the use of cadavers, anatomical parts, animals or a black box (trainer-boxes).

Types of variables / Parameters analysed

Data regarding the authors, date and location (country) of the publication were collected and arranged in tables. Data were also collected regarding the number of participants analyzed in the study, sex, age, training, predominance of right handedness, previous surgical experience, training time and simulation methods used for evaluation.

Exclusion criteria

Studies will be excluded if: (1) they have heterogeneous populations in terms of academic degree; (2) do not use a standard assessment method for the entire duration of the study, or do not have pre-assessment; (3) use VRS or augmented reality simulators as the single evaluation method or in a control group; (4) are not related to the question in the review; (5) are in a language other than english, portuguese or spanish; (6) are incomplete, unpublished or inaccessible articles to authors.

Literature review

The survey was conducted on April 20, 2020, without language restrictions, in the Medline database (via PubMed) - www.pubmed.com, EMBASE - www.embase.com and Database of the National Institute of Health

Using the search tool, we selected MeSH terms from the most relevant publications to conduct a new search, in order to obtain articles that could be included in this systematic review.

In addition, a manual search of theses, meetings, references, study records and contact with experts in the field was carried out.

Search strategy

The keywords were equally used in all databases, respecting their heterogeneities (for example, terms “Emtree” and terms “MeSH” were mapped in Embase and Medline, respectively).

The keywords were: “data display”, “computer simulation”, “learning”, “education”, “students, medical”, “education, professional”, “education, medical, continuing”, “education, medical, graduate”,”Education, medical”,”internship and residency”,”laparoscopy”.

The search strategy was: ((data display) OR (computer simulation)) AND ((learning) OR (education)) AND ((students, medical) OR (Education, Professional) OR (Education, Medical, Continuing) OR (Education, Medical, Graduate) OR (education, medical) OR (internship and residency)) AND (laparoscopy).

Data extraction

The data for each study were extracted independently by three authors (JVT, VC and WAM). Disagreements were resolved by consensus. If no consensus was reached, a fourth author (AM) would be consulted. Data extraction was carried out using the Rayyan tool - https://rayyan.qcri.org/ [25].

All studies were analyzed based on their titles and abstracts, according to inclusion and exclusion criteria. If the eligibility criteria were met, the full text would be extracted. All studies that were eligible for qualitative analysis were described in the “Results” section.

Missing data were clarified by contacting the authors directly.

Data validation

Three authors (JVT, VC and WAM) carried out the data validation through the discussion of the selected works. If no consensus was reached, a fourth author (AM) would be consulted.

The risk of bias for intervention-type studies was analyzed using the guidelines of the Cochrane Back Review Group (CBRG) [26].

All selected studies were considered.

Statistical analysis

A descriptive synthesis will be produced with tables and figures and, if a number of studies with sufficient quality are available, a meta-analysis will be carried out with measures of heterogeneity and publication bias. The data will also be presented through forest-plots, according to their statistical relevance.

Responsibility/Author contributions

All the authors involved participated in the drafting of the systematic review project, by identifying key articles, selecting keywords and writing the review project. The first author (JVT) was responsible for coordinating the group, guiding the organization of the review project, searching for articles to be reviewed and writing the text. The second (VSC) and the third author (WAM) were also responsible for the review and writing of the project, as well as for the search for articles to be reviewed. The remaining authors, in turn, were responsible for guiding and evaluating the final text.

Results

Research flow

The electronic search found 1904 results for the keywords used. After removing 375 duplicates, we considered 1529 potentially eligible studies. Of these, 1476 did not respect the inclusion criteria. After accessing the full text, three were excluded because they had a heterogeneous population, 24 who did not use a standard evaluation method or did not have a pre-evaluation, four because they used VRS or augmented reality simulator as a single evaluation method or in a control group, six for presenting inaccessible full text and nine for not complying with the inclusion criteria. Only seven studies were considered eligible for qualitative analysis and only one article was eligible for meta-analysis [Fig 1].

Fig 1

Quality of evidence

After reading the articles included in the systematic review, the following factors were analysed to determine the level of evidence: study design and selection, detection, loss, reporting and information bias. The summary of the risk of bias analysis for each of the included articles was shown in Figs 2 and 3.

Fig 2

Fig 3

Only 2 of 7 articles had a low risk of bias by the randomization process: Ahlberg G et al. and Palter VN et al. reported clearly that the allocation process was carried out randomly and blindly [27, 28]. The other 5 were classified as having some concerns for not reporting this information.

Regarding the bias due to intended intervention (effect of adhering), 6 of the 7 articles presented low risk due to the non-applicability of this criterion to most signaling questions in this domain. Van Bruwaene S et al. was the only study that presented a high risk of bias. He reported preliminary considerations on the procedures in case of any technical failures in the implementation of the experiment. However, it cannot be said that the researchers took appropriate compensatory measures, since the establishment of a minimum training time alone does not guarantee the adequate implementation of such training for the groups [29].

In the context of intended intervention (effect of assignment) bias, no article has reached the low level of bias because the nature of a training study requires that participants and supervisors are aware of their interventions. In this context, four articles presented a high risk of bias, three due to the non-completion of all assignments by participants; and one for having an ineligible participant. The other articles were classified as some concerns because they did not show any flaws in the analysis of the participants.

All the articles analysed showed a low missing outcome data bias. It is worth mentioning that Diesen DL et al., despite being classified under low bias, had a significant loss of data, which is more likely related to the study methodology, and not to an exclusion of unfavorable results [30]. The other authors presented the data in full of almost all study participants, without compromising the quality of the information.

For measurement of outcome bias, four of the seven articles obtained low risk of bias because they had blinded observers for the intervention and use validated measurement methods that are identical between groups and with high interrater reliability. Diesen DL et al. and Munz Y et al. were classified as some concerns because they do not measure the reliability of the observers [30, 31]. Van Bruwaene S et al. had a high risk of bias due to its low interrater reliability, between 47 and 65% [29].

As for selection by the reported result bias, all studies had a low risk of bias with the exception of Ahlberg G et al., who had a high risk. This study differed from the others in that it presented different parameters in the post-intervention assessment compared to the pre-intervention. The data reported in the first evaluation included time, economy of movement and precision, whereas those in the second included only precision variables [27].

None of the seven articles analyzed presented a high risk of general bias, four of which were classified as some concerns, mainly due to the influence of the intended intervention (effect of assignment) bias and the lack of information about the randomization process.

Study characteristics

All included studies are complete, published and have no conflict of interest. Any doubts about the available data were supplemented by contacting the respective authors. The demographic profiles are shown in Table 1; the characteristics of the methodology of the experiments are shown in Table 2; the main changes, conclusions and results are available in Tables 3–6.

Table 1

Demographic characteristics of the studies.

Author, publication date and country	Number of participants	Academic status	Past surgical experience
Van Bruwaene S et al. (2015), Belgium	Total: 30CG: 10IG1 (VR): 10IG2 (Organ training): 10	Medical Students	None or barely any
Palter VN et al. (2014), Canada	Total: 16CG: 8IG: 8	Residents	<10 laparoscopic cholecystectomies
Diesen DL et al. (2011), USA	Total:18IG1 (VR): 10IG2 (BT): 8Medical interns (first year residents):11Medical students:12	Medical Students and Residents	-
Ganai S et al. (2007), USA	Total:20CG:10IG:10	Medical Students	-
Ahlberg G et al. (2007), Sweden	Total: 13CG: 6IG: 7	Residents	Only assisted laparoscopic cholecystectomyCG: median 18 (range 10–30)IG: median 15 (range 10–25)
Munz Y et al. (2004), United Kingdom	Total: 24CG:8IG1 (VR): 8IG2 (BT): 8	Medical Students	-
Torkington J. et al. (2001), United Kingdom	Total: 30CG: 10IG1 (VR): 10IG2 (Surgical Trainer): 10	Medical Students	None or barely any

CG = Control Group. IG = Intervention Group. VR = Virtual Reality. BT = Box-trainer.

Table 2

Study methods.

Author, publication date and country	VRS utilized by intervention group	Instruments utilized by other groups	Method of assessment before and after training	Tasks applied in training	Main parameters	Time in training	Time between assessments
Van Bruwaene S et al. (2015), Belgium	LapMentor VR Trainer (Simbionix USA Corp)	CG: No trainingIG2: Organs (dissections)	Laparoscopic cholecystectomy on a live porcine model	Complete Cholecystectomy skills:1. Bean Drop2. Rope Pass3. Checkerboard4. Laparoscopic skills testing and training (LASTT)5. Suturing	A. Time:Time for each taskD. Others:Overall expert’s grade	IG1 (VR): 5h sessions (10 sessions)IG2 (Organ): 1h daily (10 days)	1 week
Palter VN et al. (2014), Canada	LapSim VR Simulator (Gothenberg,Sweden, 2008 version)	CG: No training	Laparoscopic cholecystectomy on a patient	1. Instrument navigation2. Grasping3. Cutting4. Clipping5. Lifting and Grasping	B. Economy of movement:Economy of movement (time and motion)C. Precision:1. Respect for tissue2. Precision of operative techniqueD. Others:1. Instrument handling2. Knowledge of instruments3. Use of assistants4. Flow of operation and forward planning5. Knowledge of specific procedure	1h sessions	Median of 18 days (range 14–36)
Diesen DL et al. (2011), USA	Computer simulation	IG2: BT	Laparoscopy on a porcine model	1. 30° camera navigation2. Clipping and electrocautery3. Knot-tying4. 2-eye-hand coordination exercises	B. Economy of movement:Economy of movementsD. Others:1. Perceptual ability2. Scope orientation3. Appropriate control of needles and instrument handling4. Appropriate clipping and eletrocautery	6 months	6 months
Ganai S et al. (2007), USA	EndoTower (Verefi Technologies, Inc, Elizabethtown, PA)	CG: No training	Telescope navigational assessment on a porcine model	Angled-telescope navigation	A. Time:Total timeC. Precision:1. Horizon and total errors2. Instrument collisions and scope smudgesD. Others:1. Visualization of the object of interest centered on the monitor and proper scope orientation	1 hour sessions	3 to 4 weeks
Ahlberg G et al. (2007), Sweden	LapSim VR Simulator (Surgical Science Inc., Gothenburg, Sweden)	CG: No training	Laparoscopic cholecystectomy on a patient	1. Suturing with and without easy grip function2. Lifting3. Grasping4. Clipping5. Ultrasonographic dissection with both hands	A. Time:Total timeB. Economy of movement:1. Instrument path length2. Instrument angular lengthC. Precision:1. Tissue damage2. Maximum and minimum damage3. Errors	1 week	Within 6 months
Munz Y et al. (2004), United Kingdom	LapSim VR Simulator (Surgical Science Inc., Gothenburg, Sweden)	CG: No trainingIG2: BT	BT cutting and clipping task	1. Grasping2. Cutting	A. Time:Total time for each taskB. Economy of movement:2. Total number of hand movements made3. Total distance traveled for each hand4. Economy of hand movementC. Precision:5. Number of errors	30 min sessions per week (3 sessions)	-
Torkington J. et al. (2001), United Kingdom	MIST-VR (Virtual Presence, London, SE1 2NL)	CG: No trainingIG2: Surgical Trainer	BT grasping and cutting suture task	1. Instrument Navigation2. Coordination3. Grasping4. Cutting5. Precision and Speed	A. Time:Total timeB. Economy of movement:1. Number of movements made2. Distance traveled by each instrument3. Speed of travel	1 hour sessions	-

CG = Control Group. IG = Intervention Group. VR = Virtual Reality. BT = Box-trainer. MIST-VR = Minimally Invasive Surgery Trainer-Virtual Reality.

Table 3

Main changes and conclusions of the studies.

Author, publication date and country	Pre-intervention assessment main results	Post-intervention assessment main results	Study conclusions
Van Bruwaene S et al. (2015), Belgium	A. Intergroup:No significant differences were found between groups	A. Intergroup:1. IG2 (Organ) had the shortest time and IG1 (VR) improved compared to CG2. IG2 (Organ) had better overall quality score and IG1 (VR) improved compared to CGB. Intragroup:Both IGs showed improvement between assessments	For trainees who are proficient in basic laparoscopic skills, the efficacy of the VRS training model remains to be proven
Palter VN et al. (2014), Canada	A. Intergroup:Both groups presented similar OSATS scores in OR	A. Intergroup:IG exhibited superior OSATS scores than the CG in OR	Deliberate individualized practice on VRS could improve technical performance in the operating room. This could mean that implementing a simulation-based curricula in residency training programs could lead to positive results
Diesen DL et al. (2011), USA	A. Intergroup:No significant differences were found between groups	A. Intergroup:No significant difference between training groupsB. Intragroup:Overall improvement in scores after training	BT and VRS are equally effective means of teaching laparoscopic skills to novice learners
Ganai S et al. (2007), USA	A. Intergroup:No significant differences were found between groups	A. Intergroup:1. IG had a more notable improvement in object visualization, scope orientation, horizon scores, number of smudges and collisions and errorsB. Intragroup:1. Both groups showed improvement in time2. Both groups showed improvement in scope orientation scores between tests	VRS can be used to improve operative surgical skill
Ahlberg G et al. (2007), Sweden	A. Intergroup:No significant differences were found between groups	A. Intergroup:CG presented a variation in total errors 8 times higher than IG	VRS could improve the initial learning curve in laparoscopic procedures, such as laparoscopic cholecystectomy. The LapSim Simulator should be used to train new laparoscopists until they reach a proficiency level.
Munz Y et al. (2004), United Kingdom	A. Intergroup:No significant differences were found between groups	A. Intergroup:IG1 (VR) and IG2 (BT) performed better than CG but there were no significant differences between one anotherB. Intragroup:Both IGs showed overall improvement in scores after training	LapSim can be used to teach skills that are transferable to real laparoscopic tasks, but it appears that there are no advantages to using VR over BT and vice versa
Torkington J. et al. (2001), United Kingdom	A. Intergroup:No significant differences were found between groups	A. Intergroup:1. Left-hand performance: both IGs were less economical in the number of movements and tended to move at greater speeds compared with CG2. Right-hand performance: both IGs were more economical in the number of movements; and there were no significant change in time completion compared with the CGB. Intragroup:1. Both IGs showed worse left-hand performance2. Both IGs showed better right-hand performance	MIST-VR can be used by novices to transfer skills to simple real tasks and its results are similar when compared with conventional training

CG = Control Group. IG = Intervention Group. VR = Virtual Reality. BT = Box-trainer. OR = Operation room. VRS = Virtual reality simulator. MIST-VR = Minimally Invasive Surgery Trainer-Virtual Reality. OSATS = Objective Structured Assessment of Technical Skills.

Table 6

Main changes in study accuracy.

Measured precision parameters		Ganai S et al. (2007), USA			Ahlberg G et al. (2007), Sweden
		Mean value	SD	P value	Mean value	SD	P value
Pre-assessment	Horizon error (#)	CG: 4.4IG: 3.8	95% CI:CG:(2.4–6.4)IG:(2.0–5.6)	-	-
	Instrument collisions (#)	CG: 7.1’VR: 5.3’	CG:(2.5–11.7)VR: (2.3–8.3)	-
	Scope smudges (#)	CG:2.1’VR: 2.2’	CG:(0.3–3.9)VR: (0.4–4.0)	-
	Total errors (#)	CG:13,6VR:11.3’	CG:(7.4–19.8)VR: (6.9–15.7)	-
	Tissue damage (#)	-			CG: 4IG: 2	Range:CG: 2–9IG: 0–3	-
	Maximum damage (mm)				CG: 5.2IG: 4.7	Range:CG: 0.7–15.9IG: 0–7.4	-
Post-assessment	Horizon error (#)	CG:IG: 1.3’	CG: (0.9–4.9)IG:(0.7–1.9)	-	-
	Instrument collisions (#)	CG:3.5’IG: 2.3	CG:(1.4–5.7)IG:(0.1–4.5)	-
	Scope smudges (#)	CG:2.4’IG: 0.4’	CG:(0.4–4.2)IG:(0–1.0)	-
	Total errors (#)	CG:8.8’IG: 3.9’	CG:(1.3–6.5)IG:(3.8–13.8)	-	CG: 86.2IG:28.4	95% CI:CG: 58.18–114.12IG: 23.51–33.32Variance:CG: 916.68IG: 118.69	0.0037’
	Exposure errors (#)				CG: 53.4IG:15.0	95% CI:CG: 16.7–90.13IG: 11.16–18.79Variance:CG: 623.31IG: 68.44	0.0402’
	Clipping and tissue division errors (#)				CG: 7.1IG:1.9	95% CI:CG: 3.95–10.25IG: 0.93–2.87Variance:CG: 41.11IG: 5.57	0.008’
	Dissection errors (#)				CG: 29.5IG: 11.5	95% CI:CG: 13.99–45.01IG: 8.82–14.08Variance:CG: 61.5IG: 28.77	0.031’
Comparison between assessments	Horizon error (#)	-		< 0.05	-
	Instrument collisions (#)			0.06
	Scope smudges (#)			< 0.051’
	Total error score (#)			< 0.05

CG = Control Group. IG = Intervention Group. CI = Confidence Interval.

Collectively the studies elected a total of 156 participants, with 40 residents and 116 medical students. It is worth considering that Diesen DL et al. partially analysed its sample, reporting only a distribution of 18 of its 23 total participants [28]. Only Ahlberg G et al., Palter VN et al. and Munz Y et al. reported a gender distribution of the sample [27, 28, 31].

Diesen DL et al., Ganai S et al. and Munz Y et al. did not perform a previous assessment of the laparoscopic skills of medical students or residents [30-32]. The remaining studies reported little or no experience in laparoscopic cholecystectomy through self-reported questionnaires or practical tests.

The training sessions in the studies lasted about 1 hour; and the time between assessments ranged from 1 week to 6 months, with Munz Y et al. and Torkington J et al. not reporting the period between assessments [31, 33].

The most prevalent VRS was the LapSim-VR-Simulator (Gothenberg, Sweden, 2008v), used in 3 of the 7 studies. Van Bruwaene S et al. used the LapMentor VR Trainer (Simbionix USA Corp) [29]; Torkington J et al. used MIST-VR (Virtual Presence, London, SE1 2NL) [33]; Ganai S et al. used Endotower (Verefi Technologies, Inc, Elizabethtown, PA) [32]; and Disen DL et al. used simulators from Medical Education Technologies, Inc. (Sarasota, Florida) and Immersion (San Jose, California) [30].

Of the 7 articles, 5 performed their assessments on PMS using laparoscopic cholecystectomy with in vivo models (3 swine and 2 humans), and Ganai S et al. only performed a telescope navigation assessment during the porcine procedure [32]. Only 2 studies carried out their assessments on PMS with non-living models, with Munz Y et al. using exercises in a water-filled glove that mimicked a gallbladder; and Torkington J et al. specific exercises in Box-trainer (BT) [33].

Regarding the different parameters analyzed, 4 of the 7 articles measured the total time of the procedures during the assessments, with an improvement in the average time of the VRS groups in relation to the control groups. The study with the most expressive results was that of Ahlberg G et al., who reported that the intervention group performed the surgery 58% faster when compared to the control group [27]. On the other hand, that of Van Bruwaene S et al. showed a difference of just over 5% between these groups, with a reduction of 21.3% within the VRS group and 17% for the control group [29].

The economy of movement was explored by 2 articles, which analyzed different variables. Ahlberg G et al. considered the distance covered in meters and angular movement of the instruments, whereas Torkington et al. assessed the number of movements per hand, hand speed and variation in hand distance during the experiment. In both studies, improvements in these criteria do not appear to be clear or significant [27, 33].

In the precision parameters, the total number of errors was analyzed by 2 articles, both of which showed a significant improvement in the average of errors, especially in Ganai S et al., whose number of errors in the VRS group was less than half of the control group [32]. On the other hand, although Ahlberg G et al. searched for total errors only in the post-assessment, the drop rates for this parameter are similar [27].

Palter VN et al. and Diesen DL et al. evaluated their assessments through scores. Palter VN et al. used the OSATS scale, obtaining significant improvement in its indexes for the VRS group in relation to the control (p <0.03). Diesen DL et al. developed its own scale for the study, obtaining better results in general for the BT group in relation to VRS, with emphasis on the instrument handling exercises, especially in needle transfer (p <0.0002) and in camera navigation (p <0.006) [28, 30].

No study reported significant differences between groups in the pre-intervention assessment. In the post-intervention assessment, All seven of the assessments showed that VRS produced an improvement in the laparoscopic skills of the participants, although Munz Y et al. and Torkington J et al. have not found differences in performance between VRS or PMS training that would justify the superiority of one method over the other [31, 33]. Van Bruwaene S et al. was the only one whose group with training in PMS obtained better results in all parameters in relation to the VRS group, also expressing the need for further studies to define the quality of teaching by VRS [29].

Discussion

VRS-based teaching is expanding in medicine due to the limitations of more traditional methods, especially for surgical procedures [34, 35]. There are already several areas, such as neurosurgery [36], ophthalmic surgery [37] and digestive endoscopy [38], considering the implementation of this technology in their training curricula. In this scenario, systematic reviews were produced to analyse the relation between the use of VRS and the learning of surgical techniques, considering the expansion of this technology in medical teaching curricula [39-43].

The field of laparoscopic surgery also follows this trend, with conclusions that vary depending on the studies. In general terms, reviews conclude that VRS is a method with the potential to develop varied surgical skills. However, most articles only compare VRS with untrained groups; whilst articles that use comparative groups with other methods show mixed results [19, 20].

Despite recommending the incorporation of VRS in laparoscopic surgical training curricula, Alaker M et al. did not observe any statistically significant difference between VRS and BT groups regarding time and score. In addition, their comparison of ‘virtual reality vs box and video trainers combined’ did also fail to find statistically significant differences between VRS and these more traditional methods [20]. Another review, by Nagendram et al. found that operative performance was significantly better in the VRS group in comparison to the BT group, but this was limited to only two articles, of which, one did not fully disclose the magnitude of the difference or other quantitative results [19]. Thus, there is not enough evidence to justify the use of this new technology in place of more traditional training in PMS.

In this systematic review, 6 of the 7 articles analysed compared the performance between a VRS group and a control group, 3 of which also used a third group in PMS. All observed an improvement of the new technology in relation to the control, evidencing the VRS as a viable alternative for the teaching [27–29, 31–33]. However, part of our results were similar to those of Alaker M et al. [20]: in all the articles we analysed that compare VRS with PMS, the use of technology has not shown significant gain [29–31, 33].

One explanation for this outcome lies in the sample limitations of these studies, whose population includes medical students and residents. There is a significant difference between the level of surgical and clinical experience between these two population types and, in this sense, medical students may not fully benefit from the training they have undergone [29–31, 33]. In addition, the sample size of these studies, between 18 to 30 participants [29, 30, 33], may not have been enough to achieve an adequate population representation. Thus, the results obtained in VRS may have been underestimated.

Another factor that can contribute to better performance results in PMS when compared to VRS is the tool used to carry out the assessments. Of the 4 articles, 2 (Munz Y et al. And Torkington J et al.) performed their pre- and post-training measurements on box-trainers [31, 33]. This can give an advantage to groups that trained in PMS over VRS groups due to the similar nature in training and assessment.

Due to the very nature of digital simulations, with software and engines that are constantly updating, it is difficult for the literature to follow and be able to produce studies that are consistent with the current situation of VRS. The studies analysed in this systematic review are between 5 and 19 years old and do not use equipment corresponding to the technological advances of their respective times. Palter VN et al., For example, published their work in 2014, but used a 2008 model (LapSim VR Simulator—Gothenberg, Sweden) [28]. Ahlberg G et al. (2007) and Munz Y et al. (2004) also used this same simulator, but in even older and different versions [27, 31]. Thus, there is a great temporal difference between the development of a VRS, its validation and the review of its results in the literature.

Due to this constant updating of technology, the need to repeatedly carry out new studies to keep up to date with newer versions of VRS can bring high costs to the research team, since the equipment is usually expensive and the production of RCTs is time consuming. Speich B et al. compared the average production cost of RCTs between 2012 and 2016, concluding that, although the value may fluctuate depending on the scope of the project and study design, there was an average cost of USD 72,000.00 in preparing a survey in both times [44].

In this review, of the 7 articles analysed, 4 use VRS LapSim or LapMentor in their methodologies. Currently, the purchase price of this equipment is approximately USD 70,000.00 and USD 84,000.00, respectively, excluding USD 15,000.00 for additional modules (2018 versions) [45]. Therefore, the acquisition of only a single piece of equipment is enough to more than double the average costs of research [44], contributing to the scarcity of literature to compare this technology with PMS methods, since it is not attractive for a research group to constantly produce such expensive update articles.

In contrast to the high prices of VRS, traditional PMS, especially box-trainers, are much more accessible for acquisition and use in training healthcare professionals. A complete box-trainer, including camera and equipment, costs between USD 1,000 and USD 6,000.00 [46], up to 84 times cheaper than current VRS. It should be noted that the VRS could save significant costs for the institution by reducing the need for proctors and replenishment of animal and synthetic material. Though they have other maintenance needs typical of any software/hardware, and some demand subscriptions or the continued purchase of new modules, effectively nullifying this effect. Nevertheless, there is still a huge disparity in costs that may justify the fear of investing in VRS for medical curricula. Especially considering the absence of solid evidence about the advantages of the new technology in relation to regular methods, which was verified by 4 of our 7 articles [29–31, 33].

Study limitations and methodologies

The most common limitation reported in the studies analyzed was the small sample size [Table 7], with an average of approximately 22.3 participants per study and 9.2 per group. Ergo, it may have decreased the statistical power of the studies and the results obtained may not be representative enough of the population. In addition, of the 7 studies, 4 used medical students and 2 used residents. Diesen DL et al. was the only one that used a mixed training sample [30]. Although not reported by the author, this population heterogeneity is a potential confounder factor in the study.

Table 7

Study limitations reported.

Author, publication date and country	Reported limitations
Van Bruwaene S et al. (2015), Belgium	1. Small sample size
	2. Medical students without clinical or surgical experience compared with the residents, might have had insufficient knowledge to fully profit from the training and its effects might have been underestimated
	3. Hard to verify equal amount of training in the organ and VR training group as the first one was restricted by time and the second by proficiency parameters
	4. Interrater (between raters) reliability was low
Palter VN et al. (2014), Canada	Using different supervisors in the OR
Diesen DL et al. (2011), USA	1. Small sample size
	2. Did not use the most recently released software
Ganai S et al. (2007), USA	1. Small sample size
	2. Absence of specific training for the residents (CG)
Ahlberg G et al. (2007), Sweden	1. Small sample size
	2. Using different supervisors in the OR
Munz Y et al. (2004), United Kingdom	IG2 (BT) could have advantage over IG1 (VR) by training with real laparoscopic instruments
Torkington J. et al. (2001), United Kingdom	Short training time to achieve both hands proficiency

CG = Control Group. IG = Intervention Group. OR = Operation room. VR = Virtual reality. BT = Box-trainer.

According to Van Bruwaene et al., The use of medical students in this type of training may not represent reliable results, given the lack of experience in relation to residents [29]. In this sense, residents would be a more suitable population for studies on surgical training. On the other hand, we could consider that students represent a more adequate sample due to their lesser degree of contact with laparoscopy and, therefore, would be less biased than residents, who are more exposed to practices outside the study. However, the current literature does not present significant information about the advantages of one group in relation to another.

We grouped the performance parameters of the participants into three main categories: time, economy of movement and precision. The time item was exposed quantitatively by 4 of the 7 articles [27, 29, 32, 33], while the other categories were only addressed by 2 articles each [27, 32, 33]. Thus, despite comparing the performance of VRS groups with PMS groups, the analysed articles show great methodological differences between them. This made it difficult to compare the results for this review, making it impossible to develop a meta-analysis based on the articles.

Some studies have presented different methodological elements. Torkington et al. presented its data through the variation between the initial and final results, which would be more appropriate for a statistical analysis [33]. In addition, the use of validated and standardized scores could be a way to circumvent the heterogeneity of parameters, as was the case with Palter VN et al. when using OSATS [28]. Although Diesen DL et al. adopted a score in his study, it was personalized and has no widespread use in the literature [30].

Among the limitations of our systematic review, our results might have been influenced due to the use of different models and versions of VRS between studies, since the analysed articles date from 2001 to 2015. No more recent articles were found that were eligible during the screening of literature.

To better assess the use of VRS in the development of laparoscopic skills, we propose the use of RCTs that use homogeneous samples, submitted to equal assessments before and after training, which do not favour one group over the other, either by the nature of the simulator or the type of task. We suggest that they report in more detail the training process and time, as well as the participants’ history of laparoscopic experience. The parameters evaluated in the assessments must be standardized so that it is possible to build a comparative analysis between the different studies. This can be done with scores validated by the literature, containing variables such as total time, number of errors and distances covered by the instruments.

Conclusions

Concomitant to recent advances in the area, VRS was described by the 7 articles analysed in this review as an effective teaching method. However, no study has been able to point out a significant advantage of VRS when compared to PMS, although the price difference between simulators can be as high as 84 times. Nonetheless, there is a lack of standardization and a scarcity of articles in the literature, hindering the comparative analysis of the performance in VRS in relation to PMS, therefore, more research in this area is necessary.

PRISMA 2009 checklist.

(DOC)

Click here for additional data file.

1 Mar 2021

PONE-D-20-37286

The development of laparoscopic skills using virtual reality simulations: a systematic review

PLOS ONE

Dear Dr. Meyer,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please address the concerns of the expert reviewers, especially the comparison with the previous surveys and meta-analyses and show the differences and discrepancies.

Please submit your revised manuscript by Apr 15 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Mohammed Saqr, Ph.D

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please describe the data extraction methods in more detail. We would expect to see reporting of the specific information extracted from the manuscripts.

3. Please include your tables as part of your main manuscript and remove the individual files. Please note that supplementary tables (should remain/ be uploaded) as separate "supporting information" files

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: N/A

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This systematic review article summarizes the results on the use of virtual reality simulator for the development of laparoscopic skills. A total of 1529 articles were collected and out of which 7 were selected after the exclusion criteria. The authors conclude that based on the seven articles, that virtual reality simulator doesn’t provide significant advantage over traditional training .

Training in laparoscopic surgery can vary from basic to advanced procedural skills. Moreover, virtual reality simulators are not uniform with some are designed to teach basic skills and some has both basic as well as advanced skills. The metrics used for assessing can also vary significantly between the simulators. Not all simulators have force feedback. Moreover, fidelity of these simulators is also very important (ie., whether they have undergone construct validity). These issues greatly limit the usefulness of metal analysis since the number of articles were going to be always very few.

The authors have summarized the results from the 7 papers but my concern is that the conclusion is severely limited by the factors mentioned above. Unlike the authors mentioned in the paper, there has been a review published on the virtual reality training for laparoscopic surgery (Alaker M, Wynn GR, Arulampalam T. Virtual reality training in laparoscopic surgery: A systematic review & meta-analysis. Int J Surg. 2016 May;29:85-94.). Also, the authors have cited the paper by Nagendran et al., which does a meta analysis of virtual reality training in laparoscopic surgery with focus on randomized clinical trials with primary and secondary outcomes but failed to elaborate on it. In view of the two review papers that has been published and the current review paper which by exclusion criteria doesn’t include papers beyond 2015, the real impact of the metal analysis is minimal. At the least, the authors should compare the conclusions from these two meta analysis to their conclusions and explain clearly any discrepancies.

The authors mention that in the article by Diesen et al., they didn’t report the VRS used in the study, but they did mention it in the discussion section (3rd paragraph). They have used both the Medical Education Technologies Inc. laparoscopic simulator as well as the Immersion virtual reality laparoscopic simulator. One would expect a thorough review of the paper especially for meta analysis. This needs to be fixed if the authors decided to revise the paper.

Reference 13 is incomplete and is also listed as part of reference 12.

My overall recommendation is reject. If authors substantially revise and compare to existing literature reviews on VRS for laparoscopic surgery, it may be a useful contribution.

Reviewer #2: I believe that this is an important piece. It is absolutely shocking that after 15 years of marketing and pushing for laparoscopic simulation the most we can find in the literature at this point to support its use is 7 papers. The number of poorly constructed C and D level publications in this space is simply incredible. The sections of the paper breaking down these shortcomings is a call-to-arms for those in this space to structure their research correctly and to power the studies appropriately. Thank you. There are a few grammatical issues in the paper that should be reviewed; a number of past/present tense inconsistencies as well as some simple capitalizations in the middle of sentences, etc.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Juan C Cendan, MD, FACS

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

4 Mar 2021

Dear reviewers,

Thank you for inviting us to submit a revised draft of our manuscript.

We also appreciate the time and effort you and each of the reviewers have dedicated to providing insightful feedback on ways to strengthen our paper. Thus, it is with great pleasure that we resubmit our article for further consideration. We have incorporated changes that reflect the detailed suggestions you have graciously provided. We also hope that our edits and the responses we provide below satisfactorily address all of the issues and concerns you and the reviewers have noted.

Reviewer #1: This systematic review article summarizes the results on the use of virtual reality simulator for the development of laparoscopic skills. A total of 1529 articles were collected and out of which 7 were selected after the exclusion criteria. The authors conclude that based on the seven articles, that virtual reality simulator doesn’t provide significant advantage over traditional training .

Training in laparoscopic surgery can vary from basic to advanced procedural skills. Moreover, virtual reality simulators are not uniform with some are designed to teach basic skills and some has both basic as well as advanced skills. The metrics used for assessing can also vary significantly between the simulators. Not all simulators have force feedback. Moreover, fidelity of these simulators is also very important (ie., whether they have undergone construct validity). These issues greatly limit the usefulness of metal analysis since the number of articles were going to be always very few.

The authors have summarized the results from the 7 papers but my concern is that the conclusion is severely limited by the factors mentioned above. Unlike the authors mentioned in the paper, there has been a review published on the virtual reality training for laparoscopic surgery (Alaker M, Wynn GR, Arulampalam T. Virtual reality training in laparoscopic surgery: A systematic review & meta-analysis. Int J Surg. 2016 May;29:85-94.). Also, the authors have cited the paper by Nagendran et al., which does a meta analysis of virtual reality training in laparoscopic surgery with focus on randomized clinical trials with primary and secondary outcomes but failed to elaborate on it. In view of the two review papers that has been published and the current review paper which by exclusion criteria doesn’t include papers beyond 2015, the real impact of the metal analysis is minimal. At the least, the authors should compare the conclusions from these two meta analysis to their conclusions and explain clearly any discrepancies.

Thank you for pointing out this review from Alaker M et al. (2016). Indeed, we agree that there was a need to go more in depth in the existing literature. In order to address this issue, we have expanded the section of our introduction in which we discuss the standing of our review in relation to the ones done prior. Furthermore, as you suggested, other changes have been made in our discussion, considering the results of our review and how they compare to the previous ones. We believe that our paper has significantly improved with these adjustments, your feedback was much appreciated.

The authors mention that in the article by Diesen et al., they didn’t report the VRS used in the study, but they did mention it in the discussion section (3rd paragraph). They have used both the Medical Education Technologies Inc. laparoscopic simulator as well as the Immersion virtual reality laparoscopic simulator. One would expect a thorough review of the paper especially for meta analysis. This needs to be fixed if the authors decided to revise the paper.

You are right, we have updated this in our review and made a thorough revision of our paper for any other misplaced information.

Reference 13 is incomplete and is also listed as part of reference 12.

This has been fixed and now both references 12 and 13 are complete.

Reviewer #2: I believe that this is an important piece. It is absolutely shocking that after 15 years of marketing and pushing for laparoscopic simulation the most we can find in the literature at this point to support its use is 7 papers. The number of poorly constructed C and D level publications in this space is simply incredible. The sections of the paper breaking down these shortcomings is a call-to-arms for those in this space to structure their research correctly and to power the studies appropriately. Thank you. There are a few grammatical issues in the paper that should be reviewed; a number of past/present tense inconsistencies as well as some simple capitalizations in the middle of sentences, etc.

We agree with you and hope that our research can play its role in helping the scientific community. Additionally, our paper was sent for grammatical correction to a british translator.

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

26 Apr 2021

PONE-D-20-37286R1

The development of laparoscopic skills using virtual reality simulations: a systematic review

PLOS ONE

Dear Dr. Meyer,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. As you can see, the article needs improvement, especially, in the light of discussion of how the findings compare to earlier research,

Please submit your revised manuscript by Jun 10 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Mohammed Saqr, Ph.D

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: N/A

Reviewer #3: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors have addressed some of the issues that were noted in their initial submission. Specifically, added reference to a relevant meta-analysis paper by Alaket et al. With respect to new information that can be obtained from this meta analysis, compared to the other paper even though there is a clear four years difference, is very minimal. Even more, in this paper there is no meta-analysis in the end which is a big let down and results were more a qualitative summary. Discussion of cost of simulators is perfectly fine but over time, VR simulator can save cost for the hospital by reducing or eliminating the need for proctors, no need to replenish materials such has synthetic, cadaveric or animal tissues or organs etc. (advantages of VRS). I think comparing to the existing meta-analysis, this review paper doesn't add more information and may be premature at this point for publication unless criteria of selection is changed to be able to perform an actual meta-analysis.

Reviewer #3: This systematic review article summarizes the results of using virtual reality simulators for developing laparoscopic skills from1529, of which, only 7 were selected. The authors concluded that virtual reality simulator has no significant edge over traditional training. I believe that this topic is important, and such review is needed.

Despite not including studies beyond 2015, the authors stated that no eligible studies were found. I belive this point should be highlighted in the abstract.

I also advise the authors to update their manuscript with the following references :

1- Gurusamy K, Aggarwal R, Palanivelu L, Davidson BR. Systematic review of randomized controlled trials on the effectiveness of virtual reality training for laparoscopic surgery. British Journal of Surgery. 2008 Sep;95(9):1088-97.

2- Jin C, Dai L, Wang T. The application of virtual reality in the training of laparoscopic surgery: a systematic review and meta-analysis. International Journal of Surgery. 2020 Dec 9.

Finally, I would also suggest that the paper be reviewed by a native English speaker to adjust for some grammatical errors.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

7 May 2021

Dear reviewers,

Thank you for inviting us to submit a revised draft of our manuscript.

We also appreciate the time and effort you have dedicated to providing insightful feedback on ways to strengthen our paper. Thus, it is with great pleasure that we resubmit our article for further consideration. We have incorporated changes that reflect the detailed suggestions you have provided. We also hope that our edits and the responses we provide below satisfactorily address all of the issues and concerns you have previously noted.

Reviewer #3: This systematic review article summarizes the results of using virtual reality simulators for developing laparoscopic skills from1529, of which, only 7 were selected. The authors concluded that virtual reality simulator has no significant edge over traditional training. I believe that this topic is important, and such review is needed.

Despite not including studies beyond 2015, the authors stated that no eligible studies were found. I belive this point should be highlighted in the abstract.

I also advise the authors to update their manuscript with the following references:

1- Gurusamy K, Aggarwal R, Palanivelu L, Davidson BR. Systematic review of randomized controlled trials on the effectiveness of virtual reality training for laparoscopic surgery. British Journal of Surgery. 2008 Sep;95(9):1088-97.

2- Jin C, Dai L, Wang T. The application of virtual reality in the training of laparoscopic surgery: a systematic review and meta-analysis. International Journal of Surgery. 2020 Dec 9.

Finally, I would also suggest that the paper be reviewed by a native English speaker to adjust for some grammatical errors.

Indeed, we do believe the abstract should inform our readers about the lack of eligible studies beyond 2015, as this data is relevant and they could benefit from it beforehand. Hence we have incorporated your suggestion within the abstract limitations section.

Also, thank you for pointing out these two references, as they significantly improved our research and have been included within our text. Finally, our paper was once again revised by another native english-speaking translator and we hope all grammatical issues have now been corrected.

Reviewer #1: The authors have addressed some of the issues that were noted in their initial submission. Specifically, added reference to a relevant meta-analysis paper by Alaket et al. With respect to new information that can be obtained from this meta analysis, compared to the other paper even though there is a clear four years difference, is very minimal. Even more, in this paper there is no meta-analysis in the end which is a big let down and results were more a qualitative summary.

We have considered all your suggestions and really appreciate the consideration for our research, thank you for providing us with another significant point of view, which inspired us to seek improvements for elevating the quality of our research. We hope now that our paper meets your expectations with the newest changes and improvements, just as we expect that some of our decisions regarding other changes you proposed have been properly explained

Discussion of cost of simulators is perfectly fine but over time, VR simulator can save cost for the hospital by reducing or eliminating the need for proctors, no need to replenish materials such has synthetic, cadaveric or animal tissues or organs etc. (advantages of VRS).

Although the main goal of our review was not to analyse the cost-effectiveness of VRS, we believe that such considerations are of great importance and, as such, agree on the necessity to explore this subject. As you suggested, there was a need to expand our discussion on the matter of advantages of VRS and we, accordingly, addressed it. We now hope this segment of our review pleases and offers a more appropriate understanding of the question at hand.

I think comparing to the existing meta-analysis, this review paper doesn't add more information and may be premature at this point for publication unless criteria of selection is changed to be able to perform an actual meta-analysis.

Regarding the changes in eligibility criteria you suggested, we believe that, given the current stage of review, such changes would be unviable, as they would either demand a complete do over of the entire project or be included in such a late stage, as to compromise the impartiality of the paper. It should also be noted that unlike Alaker et al., the other review you presented, we opted for a more restrictive eligibility criteria that explicitly required both pre and post-assessment for every study we included, in order to favor quality over quantity.

In the near future, we hope to once more submit other projects and studies to this Journal, making sure to take into account all suggestions you made early on and, certainly, your remarks will inspire us to continue seeking the betterment of any paper we may produce.

Submitted filename: Response to Reviewers 2.docx

Click here for additional data file.

19 May 2021

The development of laparoscopic skills using virtual reality simulations: a systematic review

PONE-D-20-37286R2

Dear Dr. Meyer,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Mohammed Saqr, Ph.D

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #3: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #3: All of my comments have been addressed by the authors. No further revisions are needed from my side.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #3: Yes: Ismail Ibrahim Ismail

24 May 2021

PONE-D-20-37286R2

The development of laparoscopic skills using virtual reality simulations: a systematic review

Dear Dr. Meyer:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Mohammed Saqr

Academic Editor

PLOS ONE

Table 4

Main study time changes.

Measured time parameters		Van Bruwaene S et al. (2015), Belgium			Ganai S et al. (2007), USA			Ahlberg G et al. (2007), Sweden			Torkington J. et al. (2001), United Kingdom
		Mean value	SD	P value	Mean value	SD	P value	Mean value	SD	P value	Mean value	SD	P value
Pre-assessment	Total time (s)	CG: 2820IG1(VR): 2820IG2(Organ): 2760	IQR:CG: 1260IG1(VR): 600IG2(Organ): 660	0.642’	CG: 426IG: 388	95% CI:CG: 360–492IG: 321–456	-	CG: 114IG: 110	Range:CG: 73.4–182.4IG: 61.9–150.9	-	-
Post-assessment	Total time (s)	CG: 2340IG1(VR): 2220IG2(Organ): 1800	IQR:CG: 720IG1(VR): 1320IG2(Organ): 120	0.046’	CG: 296IG: 260	95% CI:CG: 238–354IG: 194–326	< 0.05	58% longer in CG compared with IG	-	0.586’	-
Comparison between assessments	Total time variation (s)	-			-			-			CG: -9.4IG1(VR): -23.7IG2(Surgical Trainer): -7.0	CG: 12IG1(VR): 7.8IG2(Surgical Trainer): 8.8	One-way ANOVA analysis of all groups:0.43

CG = Control Group. IG = Intervention Group. VR = Virtual Reality. IQR = Interquartile range. CI = Confidence Interval. ANOVA = Analysis of Variance.

Table 5

Main changes in movement economics of the studies.

Measured economy of movement parameters		Ahlberg G et al. (2007), Sweden			Torkington J. et al. (2001), United Kingdom
		Mean value	SD	P value	Mean value	SD	P value
Pre-assessment	Left instrument path length (m)	CG: 1.4IG: 1.3	Range:CG: 1.1–1.7IG: 1.2–1.6	-	-
	Left instrument angular path (°)	CG: 314.7IG: 317.8	Range:CG: 276.4–470.2IG: 253.2–397.9	-
	Right instrument path length (m)	CG: 1.2IG: 1.2	Range:CG: 1.0–1.4IG: 1.0–1.4	-
	Right instrument angular path (°)	CG: 274.9IG: 290.8	Range:CG: 238.5–358.9IG: 240–365.8	-
Comparison between assessments	Number of movements variation for left hand (#)	-			CG: -11.0IG1(VR): 0.9IG2(Surgical Trainer): 10.0	CG: 5.6IG1(VR): 2.9IG2(Surgical Trainer): 6.5	One-way ANOVA analysis of all groups:0.03 (0.02a)
	Speed of travel variation for left hand(cm/sec)				CG: -0.2IG1(VR): 0.5IG2(Surgical Trainer): 0.4	CG: 0.3IG1(VR):0.2IG2(Surgical Trainer): 0.2	One-way ANOVA analysis of all groups:0.04 (0.01a)
	Distance variation for left hand (cm)				CG: -34.8IG1(VR): 9.8IG2(Surgical Trainer): 31.8	CG: 23.7IG1(VR): 11.6IG2(Surgical Trainer): 2.8	One-way ANOVA analysis of all groups:0.11
	Number of movements variation for right hand (#)				CG: 3.7IG1(VR): -17.9IG2(Surgical Trainer): -7.0	CG: 7.1IG1(VR): 6.4IG2(Surgical Trainer): 3.5	One-way ANOVA analysis of all groups:0.05 (0.03a)
	Speed of travel variation for right hand(cm/sec)				CG: 0.1IG1(VR): 0.2IG2(Surgical Trainer): 0.0	CG: 0.1IG1(VR): 0.2IG2(Surgical Trainer): 0.3	One-way ANOVA analysis of all groups:0.82
	Distance variation for right hand (cm)				CG: 5.2IG1(VR): -45.6IG2(Surgical Trainer): -7.3	CG: 25.8IG1(VR): 25.6IG2(Surgical Trainer): 16.9	One-way ANOVA analysis of all groups:0.29

CG = Control Group. IG = Intervention Group. VR = Virtual Reality. ANOVA = Analysis of Variance.