Transfer of learning from simulated setting to the clinical setting: identifying instructional design features.

Background: Transfer of learning (ToL) is the endpoint of simulation-based training (SBT). It is affected by numerous factors, which can be classified into 3 categories: learner characteristics, work environment, and training design. The first 2 have been identified to some extent in previous research. In this study, the aim was to identify the instructional design (ID) features affecting the ToL in SBT.
Methods: This qualitative study was conducted in 2 phases. Phase 1 covers thematic analysis of comparative studies in the field of SBT. A systematic search was performed on 6 databases of Ovid MEDLINE, EMBASE, PsycINFO, CENTRAL, Scopus, and Web of Science, and the references of related systematic reviews were also checked. In phase 2, semi-structured interviews were conducted with key informants (instructors and learners) and analyzed using directed content analysis. The results of the 2 phases were combined, and finally ID features of SBT were identified and categorized.
Results: In the first phase, 121 comparative studies were reviewed and in the second phase, 17 key informants were interviewed. After combining the results of the phases, the ID features affecting the ToL in SBT were classified into 3 broad categories and 15 subcategories as follows: (1) presimulation: preparation, briefing, and teaching cognitive base; (2) underlying theories: deliberate practice, mastery learning, and proficiency-based training; (3) and methods and techniques: distributed practice, variability, increasing complexity, opportunity for practice, repetitive practice, active learning, feedback/debriefing, simulator type, and simulator fidelity.
Conclusion: Although learning is transferred from the simulated setting to the clinical setting, this process is not automatic and straightforward. Numerous factors affect this transfer. The results of this research can be used in designing and evaluating the SBT programs.

↑ What is “already known” in this topic:

Transfer of learning is referred to as the endpoint of SBT. According to previous studies, learning is transferred from the simulated setting to the clinical setting. However, this process is not automatic and straightforward, because many factors affect it. Some of these factors, including feedback, repetitive practice, and variability, have already been identified.

→ What this article adds:

In this study, by extensively searching databases and conducting interviews with key informants, 15 features of instructional design affecting transfer of learning were extracted. These features fall into 3 broad categories: presimulation; underlying theories; and methods and techniques. The results of this research can be used to design and evaluate simulation sessions.

Introduction

Simulation is “a technique that creates a situation or environment to allow persons to experience a representation of a real event for the purpose of practice, learning, evaluation, testing, or to gain understanding of systems or human actions.” (1). It facilitates any kind of learning, whether in the domai n of cognitive, affective, and psychomotor, and allows learners to practice principles and skills in a controlled environment and learn to prepare themselves for safer patient care (2). Nowadays, instead of learning the skills on a patient in the clinical environment, students initially learn them on the simulator (3).

There are different methods and types of simulation, including full-body manikins, part-task trainers, screen based simulators, virtual reality, and simulated patients (2). Learning objectives, level of fidelity needed, and learning level of trainees are 3 influential factors in choosing the method and type of simulation (4).

The efficacy of simulation-based training (SBT) has been reported in many published systematic reviews and meta-analyses (5-8). Kirkpatrick's 4-level model is frequently used in the evaluation of educational programs and includes reaction, learning, behavior, and results (9). In SBT, the trainees’ level of learning (a skill or knowledge) is evaluated in a simulated setting and on a simulator. However, SBT is effective when the learner is prepared to apply what s/he has learned in the simulated setting to real patients in the clinical setting. This is the third level of Kirkpatrick’s model, that is, "behavior change," which is more specifically known as transfer of learning (10). As Norman et al expressed, 1 of the assumptions of SBT is that the skills learned through the simulator could be applied to the real patients (11). Nonetheless, the results of more than 30 years of research show that transfer of learning to the clinical setting is not an easy task (12).

Transfer of learning has a broad meaning and it has been supported by research for more than 120 years, especially in the literature of applied psychology and organizational learning (13). Contrary to the popular belief, transfer of learning has a complex and dynamic process, and it is affected by a set of factors (14). According to studies, the factors affecting transfer of learning are classified into 3 categories: learner characteristics, training design, and work environment (15). In the past, there was not much evidence regarding the transfer of learning; however, nowadays, it is strongly claimed that learning from the simulated setting can be transferred to the clinical setting (16-22). In fact, there have been debates over the utility of SBT for decades, that is, to understand whether or not simulation works. However, nowadays, the main question is how SBT works, and how we can design and implement it to maximize learning and facilitate transfer of learning (23).To this end, identifying the factors affecting transfer of learning is more important. As mentioned earlier, 3 categories of factors affect the transfer of learning; however, the purpose of this study is to find factors related to instructional design (ID), because the characteristics of the learners and factors related to the work environment have been identified and explained clearly in several previous studies (24-26).

Methods

This qualitative study was performed in 2 phases. The first phase included thematic analysis of comparative studies related to SBT, and the second was the directed content analysis of qualitative interviews with learners and instructors of SBT.

Phase 1: Thematic Analysis of Comparative Studies Related to SBT

Review Question: Based on comparative studies, which features of instructional design related to SBT can affect the transfer of learning in undergraduate and postgraduate medical trainees?

Information Sources and Search Strategy:A systematic search was performed on 6 databases, including Ovid MEDLINE, EMBASE, PsycINFO, Cochrane Central Register of Controlled Trials (CENTRAL), Scopus, and Web of Science. Searching included free keywords and controlled terms. Terms and their derivatives were combined with appropriate Boolean operators. Wildcard and truncation operators were also used to increase search sensitivity. The search was conducted on August 12, 2019. Table 1 shows the search strategy developed for the Ovid MEDLINE database. This search strategy was adapted to other databases and modified as needed. In addition to searching the databases, the references of related systematic reviews were also examined. The Full search strategy for all databases is given in the Appendix S1.

Table 1

Search Strategy for Ovid MEDLINE

Row	Syntax	N
1	simulation.ab,ti.	175122
2	simulator?.ab,ti.	18054
3	manikin?.ab,ti.	2711
4	mannikin?.ab,ti.	62
5	mannequin?.ab,ti.	1610
6	exp Simulation Training/	7701
7	Patient Simulation/	4738
8	High Fidelity Simulation Training/	185
9	exp Computer Simulation/	223787
10	virtual reality.ab,ti.	8250
11	Virtual Reality/	1044
12	augmented reality.ab,ti.	1529
13	exp Education, Medical/	158221
14	educat$.ti,ab.	546268
15	train$.ti,ab.	493886
16	learn$.ti,ab.	359073
17	instruct$.ti,ab.	89094
18	teach$.ti,ab.	178995
19	curricul$.ti,ab.	51461
20	1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10 or 11	367730
21	13 or 14 or 15 or 16 or 17 or 18 or 19	1436455
22	20 and 21	38476
23	exp Clinical Study/ or comparative study/	2527306
24	22 and 23	5046
25	limit 24 to english language	4941

Inclusion Criteria: All comparative studies (RCT, quasi-experimental, cohort, 1-group pretest-posttest studies) that met the following criteria were included in the study:

Using SBT as the main intervention;

Investigating undergraduate and postgraduate medical trainees;

Assessing transfer of learning on the patients and in real clinical setting;

Evaluating only technical skills and procedures;

Comparative studies;

Published only in English language.

Exclusion Criteria:

Other health profession trainees;

Qualitative, review, descriptive, and editorial studies;

Nontechnical skills (such as leadership, teamwork, communication skills);

Assessing learners' skills on a simulator or an animal or a human cadaver;

Lack of full text of articles; and

Published in a language other than English.

Selection of Studies:All retrieved articles were imported into EndNote X9 software. After removing duplicate records, the studies were selected through 3 screening stages. In title screening, clearly irrelevant articles were excluded from the review. Then, title and abstract screening was performed according to the inclusion and exclusion criteria. When there was no agreement on the abstracts or there was insufficient information, the full-texts of the articles were reviewed. Conflicts were resolved through discussion.

Data Extraction and Analysis: Thematic analysis was used to find themes (instructional design features). The 5 stages of thematic analysis are familiarization with data, assigning the initial codes, searching for themes, reviewing the themes, and charting themes according to the objectives of the study (27). For this purpose, the Introduction and Methods sections of each article were carefully studied and the features related to the SBT instructional design were highlighted in the PDF file. Then, all the articles were imported into MAXDA 2018 software. Previously identified instructional design features, such as feedback, repetitive practice, and fidelity, were used as a starting point of theme classification, and new themes were added. Finally, the obtained themes (factors) were reviewed several times and categorized based on similarities.

In addition, basic study characteristics, such as source, year, study design, topic, learners, and sample size were extracted for each article.

Phase 2: Directed Content Analysis of Qualitative Interviews

In order to emphasize the context and increase the strength of the study, in-person interviews were conducted with key informants. By “key informants” we mean all individuals who participated in the simulation sessions as learners or instructors. Instructors should have at least 2 years of teaching experience in SBT, and learners should have participated in at least 10 simulation sessions. Sampling was done completely purposefully. Participants of both genders, various disciplines, positions, and hospitals were selected for interview to maximize the variation of sampling. Individuals were called and invited to interview. If they accepted the invitation, the time and place of the interview were set. In the interview session, first, the purpose of the study was explained and a brief description was given about the concept of transfer of learning and related notions, and the interview was audio recorded with the consent of participants. The overall structure of the questions was clear because directed content analysis had been used for data analysis. Since numerous factors had been obtained from the previous stage, the questions were in line with those questions. Meanwhile, some open-ended questions were asked to identify other factors, especially with regard to the context. In the interview session, first, an open-ended question was asked. For example, the following questions were asked from the instructors and learners, respectively:

- “What factors affect the process of transferring technical skills learned in the simulated environment to the real patient?”

- “Have you experienced transfer of learning? If yes, what were the factors behind this transfer?”

This was followed by more detailed and exploratory questions using predefined codes and levels. After each interview, the data were implemented, and directed content analysis was performed on them, and subsequent interviews were conducted based on them. Interviews continued until data saturation.

The set of codes (factors) identified from the first phase of the research was used as a guide for coding the text of the interviews. Therefore, whenever a text was related to the previously identified factors, the same code was assigned to it. New codes were also emerged and stored separately in the MAXQDA software. The codes were reviewed several times, and after being summarized, they were placed in the preexisting categories or in the new category based on similarity and appropriateness.

Results

Phase 1

The process of study selection is presented in Figure 1. After removing duplicates, 14,620 records remained. Then, 10,773 articles in the title screening and 3,445 articles in the abstract screening were excluded. Next, 295 studies were excluded by full-text screening, leaving 107 studies for final review. After reviewing the reference list of systematic reviews, 14 new articles were obtained. Therefore, a total of 121 articles were reviewed.

Fig. 1

Flow diagram illustrating the study selection process.

The characteristics of the articles and their full reference lists have been presented in a table in the Appendix S2 and S3. The studies had been published between 1987 and 2019 in 74 different journals. In addition, journals of surgery and anesthesia played a major role in publishing these articles.

In total, 10 one-group pretest-posttest studies, 10 cohort studies, 10 quasi-experimental studies, and 91 (75%) true experimental (RCT) studies were included in the review. Most of the studies (46%) were 2-group pretest-posttest and 2-group posttest only (37%). Also, 81% (91) of study participants were postgraduate medical trainees (PGMT), 21% (18) undergraduate medical trainees (UGMT), and 3.3% (4) a mix of PGMT and UGMT. Minimally invasive surgeries/procedures, Central Venous Catheter (CVC) insertion, and intubation were the most commonly taught clinical topics using simulation.

Table 1 (in the Appendix S4) shows the ID features that lead to the transfer of learning in SBT. In total, 3 broad categories of factors affecting learning transfer were identified: presimulation, underlying theories, and methods and techniques. Each of these categories have their own subcategories that are described below.

Presimulation

Briefing: In this session, the instructor explains the objectives of the session, the duration of the training, the role of the learners, and the teaching method. Defining the roles and tasks is especially important for group and scenario-based simulation sessions. Also, if there is a simulator or medical device that students are not familiar with, it should be introduced and described. The briefing takes place just before the simulation session (1, 28). Briefing sessions were reported in 23 (19%) studies.

Teaching Cognitive Base: Before starting the simulation, learners should be familiar with the theoretical and cognitive base of the procedure. Items, such as the importance and necessity of the procedure, anatomy, indications, contraindications, and steps to perform the procedure should be described. This section can be presented in the form of text, videos, e-learning, lectures, et cetera (29). In 64 (52.8%) interventions, teaching the theoretical dimension of procedures was reported. In 42 (34.7%) interventions, the procedure technique was shown as a preprepared film before starting the simulation operation. The cognitive dimension could be presented several days before the simulation or in the simulation session or before the start of the simulation operation. No specific information was extracted from the reports.

Underlying Theories

Deliberate Practice (DP): This theory was developed by Ericsson et al. According to them, significant improvements in performance occur when the following cases are met: The objectives of the task are well defined, individuals are motivated to progress, they receive feedback and adequate opportunities for repetition, and gradual modification of their performance is provided (30).

Nine features or requirements of DP in simulation-based medical education are as follows: (1) learners with high motivation and good concentration; (2) engaging in a well-defined goal or task; (3) appropriate difficulty level of the tasks; (4) frequent and focused training; (5) accurate assessment; (6) informative feedback from educational resources (eg, simulator and teacher); (7) learners monitoring their own learning experiences and review and correct strategies, errors, and levels of learning, and reengage in DP if necessary; (8) learners are evaluated for competency; (9) learners are allowed to go to the next level/unit.

The effect of DP on learning and its transfer in simulation has been well documented. However, not all the above 9 features have been necessarily observed in all articles. Mostly, they included 3 phases of practice, informative feedback, and reinforcement. In total, this method was used in 11 (9%) studies to learn the skills. It is noteworthy that most of these articles merged mastery learning (ML) and DP.

Mastery Learning (ML): ML is not a new approach and it dates back to the 1950s and 1960s; it is the brainchild of John Carroll and Benjamin Bloom. According to this approach, if all learners are given ample opportunity, all of them can achieve all or most of the learning outcomes at the mastery level (31). ML is a rigorous type of competency-based education. In ML, the educational outcomes are the same (with little or no difference), but the training time may be different among learners (32).

ML is a set of 7 complementary features as follows:

Initial or diagnostic test;

Clearly defined learning activities, arranged in units from easy to difficult;

Educational activities;

Determining the minimum accepted standard;

Formative evaluation with practical feedback to ensure the achievement of the minimum acceptance standard (acceptable level of mastery);

Proceed to the next training unit based on the evaluation result; otherwise,

Continue training until the desired level of mastery is achieved (33).

In 11 (9%) cases of the interventions, simulations were performed using the ML approach.

Proficiency-based Training (PBT): PBT is conceptually very similar to ML, although there are some differences. First, it is used to help learning certain technical skills. Second, it is the most common term and paradigm in surgical simulation, and is therefore primarily used in the surgical education literature (34). In the PBT approach, the end point of teaching a technical skill is when the individual has a performance similar to that of an expert. Learning experiences of trainees are tailored to their individual needs. This means that, like ML, the time to complete the training and the number of practices and repetitions are different for each person (35).The important point in this approach is to determine the level of proficiency. This level is determined by the experts. In our research, the simulator used in this approach was mostly virtual reality. Therefore, these levels have already been embedded as a program by simulator developers. In 49 (40.49%) studies, simulation interventions were performed with PBT approach. Considering the topics covered, we find that almost all of the interventions were about 1 of the minimally invasive surgeries/ procedures (MIS). Therefore, we can conclude that MIS (skill), PBT (training approach), and virtual reality (simulator type) are completely interrelated. This means that combining them increases learning and transfer of learning.

Methods and Techniques

Feedback and Debriefing: In simulation, feedback refers to information given or dialogue between participants, facilitators (instructors), simulator, or peers to improve understanding of concepts or aspects of performance (28). Feedback is very important in simulation, and some scholars believe that if we remove feedback from the simulation, almost no learning will happen (36). Feedback was reported in 86 (71%) studies, but the sources of feedback were as follows:

Instructor or facilitator (46; 38%);

Simulator (66; 54.5%);

Haptic feedback (40; 33%);

Audiovisual feedback (23; 19%);

Unknown (3; 2.47%).

Haptic feedback is particularly relevant to virtual reality simulators. This type of feedback can be used to simulate contact, touching a limb or part of the body, and cutting (1). A number of studies have reported more than 1 source of feedback, so the sum of the percentages of feedback sources is greater than the feedback itself. In simulation, there are generally 2 types of feedback: one is during-session feedback, which can occur immediately, and the other is end-of-session feedback (including debriefing). Immediate feedback is most effective in teaching individual procedural skills, and if final feedback is provided, it can help increase learning and transfer of learning (37). The term debriefing is specific to team simulation training, which is based on guidelines, such as ACLS, ATLS, PALS, et cetera. Some consider feedback and debriefing as completely separate concepts, but some simulation articles refer to group feedback at the end of a session as debriefing. Debriefing is essentially a reflection and 1 of the most important factors in the transfer of learning in studies related to resuscitation training. Feedback was provided in 68 (56.19%) papers during the simulation session and in 23 (19%) papers at the end of the simulation session. The procedures taught in these 121 articles were mostly of individual type; hence, the number of feedbacks during the sessions was higher than end-of-session ones.

Distributed Practice:This method is in contrast to massed practice. According to this method, to teach a skill, it is better to divide the practice into shorter sessions over a longer period of time (38).For example, if we have 8 hours to teach a skill, instead of an 8-hour session, we can hold it in the form of four 2-hour sessions. This method is widely used in the approaches of DB, ML, and PBT. In this study, we considered simulation interventions as a distributed practice when they were held in more than 1 session/day. Thus, the distributed practice method was used in most (71%) of the interventions.

Repetitive Practice:Repetitive practice is a basic principle in learning any technical skill. In fact, repetitive practice quickly automates skills, and the key is to transfer skills from the simulator to the real patient (39). It should also be noted that, if it is embedded in DP, it would be more effective than unstructured and thoughtless practice. Nevertheless, repetitive practice helps learn the skills, especially in novices. Once the learning curve reaches the plateau level, the routine repetition of a skill no longer improves one's performance, in which case the principles of DP should be used (40, 41). Repetitive practice was reported in 72 (56.19%) papers. Our criterion for the repetitive practice was performing a task on the simulator for more than 1 time.

Increasing Complexity: If we break down the steps of a procedure or task from simple to complex and gradually increase its difficulty, the learning of skills will be easier for learners, especially novices (39). Although this approach could be implemented in the scenario-based simulations, it was observed that, in the reviewed studies, MIS procedures had been mainly taught with virtual reality simulators. In total, 42 (34.7%) cases of the interventions used the technique of gradually increasing complexity to teach skills.

Variability:Variability simply refers to the use of a variety of practices to teach a particular concept or skill. For example, instead of executing and practicing a specific scenario for 5 times, it is better to have a different scenario each time, but the task is the same. In other words, this diversity is about the surface features of a task. For example, when describing a particular clinical symptoms, we can give examples of patients of different ages, genders, races, and medical histories (42). In scenario-based simulation training, this variety of practices can be considered in the scenario itself. Moreover, in advanced simulators, such as full-body manikins and virtual reality, this practice may be embedded as a program in the simulator itself. In the present study, most of the studies that received the variability code used virtual reality simulators for training. Overall, 26 (21.48%) studies used the variability technique.

Simulator Fidelity:Simulator fidelity is one of the challenging topics. It is usually divided into 3 categories: low, medium, and high fidelity. It simply means the realism level of the simulator, and how similar it is to the real world (1). High-fidelity simulators have long been thought to improve transfer of learning, but various studies have shown that transfer of learning does not depend solely on the level of fidelity. In general, educational goals, type of procedure, and level of learners determine the fidelity of a simulator (11). By simulator fidelity, we actually mean physical fidelity. Nowadays, this term is used to mean simulation itself, in which case the simulator is only one of its components. In this view, in addition to the simulator fidelity, there are at least 2 other kinds of fidelity: environmental fidelity and psychological fidelity.

Environmental Fidelity: To what extent does the simulated environment (simulator, room, tools, equipment, moulage, and sensory prop) represent the reality and appearance of the real environment?

Psychological Fidelity: To what extent does the simulated environment stimulate the underlying psychological processes required in the real environment? In fact, it is the level of realism perceived by learners (11).

No specific information was extracted from environmental and psychological fidelity of the experimental studies. Regarding the physical fidelity of the simulator, although most of the educational interventions (68 cases; 56.19%) used high-fidelity simulators, in 26 (21.48%) cases they used low-fidelity simulators, such as task trainers; they also showed transfer of learning. A combination of them was used in 5 (4.13%) studies. The remaining studies did not report the type of fidelity.

Simulator Type:Virtual reality simulators were used in 65 (53.71%) interventions; task trainer simulators in 23 (19%); full-body manikins in 6 (4.95%), and box trainers in 6 (4.95%). Finally, in 8 (6.61%) interventions, a combination of simulators was used. Also, some studies did not report the type of simulator.

Phase 2

A total of 17 participants were interviewed. The minimum and maximum interview durations were 25 minutes and 65 minutes (average duration = 41 minutes). Demographic information of participants is presented in the Appendix S5. Through directed content analysis, a total of 98 initial and open codes (including codes obtained from phase 1) were identified. One of the aims of qualitative directed content analysis is to see whether the findings show evidence of support or nonsupport for a phenomenon. Thus, the following question was raised:

- From the interviewees’ point of view, do the existing factors influence the transfer of learning meaningful to them?

In phase 1 of the study, a total of 12 factors related to instructional design in transfer of learning were extracted. Two of these factors were also confirmed in interviews (feedback and teaching cognitive base). Since there was no experience for some factors, practically no specific information was obtained. For example, there was no teaching or learning experience by ML, PBT, or DP. However, 3 categories and several subcategories were extracted from the interviews in addition to the previous 12 factors.

Feedback Source

In the interviews, in addition to the instructor and the simulator, peer feedback was extracted as one of the main sources of feedback in the simulation sessions. In this regard, 1 student said:

"If I know, for example, that my peer knows the content very well, and the teacher is very busy, I may ask my peer to help me learn a skill. Then, I can practice it in front of the instructor and get his/her feedback. Overall, I personally get a lot of feedback from my peers."

Preparation

It includes items such as choosing the appropriate time and location for training, scenario design, et cetera.

Appropriate Time

"The time of the class is also important. For example, when our simulation sessions are scheduled at 4 PM or 5 PM, then it seems as if it is not part of our curriculum….”

Appropriate Location

“... I do not think it is a good idea to hold our classes in the stressful situation of the hospital.”

Predesigned Scenario

"Depending on the year of the residency, we run 3 different scenarios. For example, if we are going to teach resuscitation. There is a simple resuscitation practice for PGY1 residents and a more complicated one for PGY3 ones (like infants’ resuscitation).

Opportunity for Practice

"But at the end, I came to the conclusion that a simulation workshop would not be complete unless every single person sitting there would eventually run the procedure once."

Active Learning

"The teacher should involve the learners and constantly ask them questions. Sometimes, our simulation classes are not different from the theory classes; they are spiritless. I attended a simulation class at a hospital. It was very good; there were few learners and the teacher gave us a complete feedback."

Similarities Between Learning Environment and Transfer Environment

"The more the environment or simulator is similar to the clinical environment, the better. Of course, the instructor can also play a major role. That means s/he can help get closer to the real environment."

Integration of the Results of the 2 Phases of the Study

As mentioned earlier, fidelity is divided into 3 categories: physical, environmental, and psychological. Thus, the category "similarity between learning environment and transfer environment" was merged with environmental fidelity. In Phase 1, the instructor and the simulator were identified as 2 sources of feedback, and in the interviews, a peer was added. Preparation was placed under the general category of presimulation, and the opportunity for practice and active learning was included in the category of methods and techniques. Therefore, in 2 phases of the study, 15 features of instructional design that are effective in transferring learning from the simulated setting to the clinical setting were extracted. These 15 features were classified into 3 general categories (Table 2).

Table 2

ID Features That Lead to the Transfer of Learning in SBT

Presimulation	Methods & Techniques
Preparation	Distributed practice
Briefing	Variability
Teaching cognitive base	Increasing complexity
Underlying theories	Opportunity for practice
Deliberate practice	Repetitive practice
Mastery learning	Active learning
Proficiency-based training	Feedback/debriefing
	Simulator type
	Simulator fidelity

Discussion

Many factors are influencing the transfer of learning from the simulated environment to the clinical environment. In previous studies, these factors were not clearly and comprehensively identified. As a result, in this study, we identified these factors using a 2-phase study. These factors, which are the characteristics of ID, were classified into 3 general categories: presimulation, underlying theories, and methods and techniques. Some of the above factors overlap, which means that if we use one 1 approach, it automatically covers some other factors as well. For example, if we use the ML approach, factors such as feedback, repetitive practice, and increasing complexity are embedded in it. However, this does not mean that these factors can be used only with these approaches. There are many studies that have used only repetitive practice or feedback without performing all the steps of ML. In general, the above factors have a positive effect on learning and transfer of learning. However, if they are utilized in the form of a coherent approach, such as ML and PBT, they can be more effective. If we consider the transfer of learning as the end point and measure of the effectiveness of any educational intervention, SBT meets this criterion; and the review of 121 comparative studies (75% were RCTs) confirmed this issue.

It was reconfirmed that a wide range of skills and procedures can be taught using simulation techniques. This can include complex procedures like laparoscopy cholecystectomy (43) and simple procedures, such as injections. The important point is to identify and use the factors affecting learning and transfer of learning in SBT(23).

Although the transfer of learning has been extensively studied by cognitive psychologists and educational specialists for more than a century, it has rarely been considered by medical educators (44). According to Dyre and Tolsgaard, there are many articles in medical education confirming that transfer of learning can occur, but few studies explore when, why, and how the transfer can be optimized theoretically and conceptually (45). In other words, the transfer of learning in medical education is taken for granted, while various studies in the field of medical education have repeatedly shown that this field is a challenging task for both students and teachers (46). When most lessons are taught outside the clinical setting, instructors need to devise a method to design the curriculum so that the learning can be transferred to the real environment (47).

A number of factors affecting the transfer of learning that were obtained in this study were similar to those reported in previous studies (39, 48, 49). However, the present study was different from them in 4 major aspects:

Only studies that really measured the transfer of learning were examined;

The strength of evidence was relatively high (75% were RCTs[randomized controlled trials]);

The number of factors extracted was higher than the previous studies and were classified into 3 categories;

By conducting a qualitative interview, issues related to context were also considered.

In this study, all simulation modalities (patient simulations, VR, etc.) and all medical trainees’ levels (undergraduate and postgraduate) were included, which may have ignored their particular conditions and requirements.

Conclusion

Although transfer of learning from a simulated environment to a clinical setting is done regularly, this process does not occur automatically and directly. In general, 3 categories of factors, including learner-related factors, factors related to educational design, and factors related to clinical environment have an impact on transfer of learning. In this study, only factors related to educational design were extracted from comparative studies and qualitative interviews. Each of these 16 factors can be explored in more depth in future studies. Experimental studies as well as systematic reviews are recommended. The results of this research can be used in designing simulation-based medical education programs as well as evaluating these programs.

Acknowledgment

Special thanks to all the faculty members and learners who participated in the research.

Conflict of Interests

The authors declare that they have no competing interests.

Appendix S1

Full Search Strategy for Six Databases Search date: 12 August 2019

1- Ovid MEDLINE(R) and Epub Ahead of Print, In-Process & Other Non-Indexed Citations, Daily and Versions (R) 1946 to August 09, 2019
Row	Syntax	N
1	simulation.ab,ti.	175122
2	simulator?.ab,ti.	18054
3	manikin?.ab,ti.	2711
4	mannikin?.ab,ti.	62
5	mannequin?.ab,ti.	1610
6	exp Simulation Training/	7701
7	Patient Simulation/	4738
8	High Fidelity Simulation Training/	185
9	exp Computer Simulation/	223787
10	virtual reality.ab,ti.	8250
11	Virtual Reality/	1044
12	augmented reality.ab,ti.	1529
13	exp Education, Medical/	158221
14	educat$.ti,ab.	546268
15	train$.ti,ab.	493886
16	learn$.ti,ab.	359073
17	instruct$.ti,ab.	89094
18	teach$.ti,ab.	178995
19	curricul$.ti,ab.	51461
20	1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10 or 11	367730
21	13 or 14 or 15 or 16 or 17 or 18 or 19	1436455
22	20 and 21	38476
23	exp Clinical Study/ or comparative study/	2527306
24	22 and 23	5046
25	limit 24 to english language	4941
2- Embase 1974 to 2019 Week 32 (OvidSP) Search date: 12 August 2019
Row	Syntax	N
1	simulation.ab,ti.	187436
2	simulator?.ab,ti.	23800
3	manikin?.ab,ti.	3949
4	mannikin?.ab,ti.	83
5	mannequin?.ab,ti.	2491
6	exp Simulation Training/	3916
7	Patient Simulation/	756
8	High Fidelity Simulation Training/	265
9	exp simulation/	278975
10	exp Computer Simulation/	112145
11	virtual reality.ab,ti.	10979
12	Virtual Reality/	15315
13	augmented reality.ab,ti.	1852
14	1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10 or 11 or 12 or 13	395646
15	exp medical education/	300492
16	educat$.ti,ab.	716976
17	train$.ti,ab.	654469
18	learn$.ti,ab.	467062
19	instruct$.ti,ab.	121097
20	teach$.ti,ab.	229201
21	curricul$.ti,ab.	65579
22	15 or 16 or 17 or 18 or 19 or 20 or 21	1914304
23	14 and 22	55340
24	exp comparative study/ or exp controlled study/	7869395
25	23 and 24	13595
26	limit 25 to (english language and embase and journal)	7849
3- PsycINFO 1967 to August Week 1 2019 (OvidSP) Search date: 12 August 2019
Row	Syntax	N
1	simulation.ab,ti.	27376
2	simulator?.ab,ti.	4290
3	manikin?.ab,ti.	354
4	mannikin?.ab,ti.	22
5	mannequin?.ab,ti.	174
6	exp Simulation/	59868
7	exp Computer Simulation/	15149
8	exp Virtual Reality/	7920
9	exp Augmented Reality/	398
10	virtual reality.ab,ti.	4648
11	augmented reality.ab,ti.	587
12	1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10 or 11	78768
13	exp Medical Education/	23000
14	educat$.ti,ab.	431270
15	train$.ti,ab.	282152
16	learn$.ti,ab.	413163
17	instruct$.ti,ab.	124043
18	teach$.ti,ab.	246927
19	curricul$.ti,ab.	56215
20	13 or 14 or 15 or 16 or 17 or 18 or 19	1062791
21	12 and 20	22287
22	medical.ab,ti.	177550
23	medicine.ab,ti.	51166
24	healthcare.ab,ti.	38430
25	physician?.ab,ti.	57399
26	health profession?.ab,ti.	3042
27	22 or 23 or 24 or 25 or 26	271975
28	21 and 27	1626
29	limit 28 to ("0300 clinical trial" or "0400 empirical study")	1035
30	limit 29 to (peer reviewed journal and english language)	765
4- EBM Reviews - Cochrane Central Register of Controlled Trials July 2019 (OvidSP) Search date: 12 August 2019
Row	Syntax	N
1	simulation.ab,ti.	6052
2	simulator?.ab,ti.	2315
3	manikin?.ab,ti.	1178
4	mannikin?.ab,ti.	11
5	mannequin?.ab,ti.	366
6	Patient Simulation/	450
7	exp Computer Simulation/	1511
8	virtual reality.ab,ti.	2315
9	augmented reality.ab,ti.	128
10	1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9	11404
11	exp Education, Medical/	3074
12	educat$.ti,ab.	54046
13	train$.ti,ab.	86835
14	learn$.ti,ab.	25476
15	instruct$.ti,ab.	21877
16	teach$.ti,ab.	17313
17	curricul$.ti,ab.	3893
18	11 or 12 or 13 or 14 or 15 or 16 or 17	164977
19	10 and 18	5258
20	limit 19 to english language	4250

Appendix S2

- Study Characteristics of Included Papers

Source	Journal	Study Design	Is RCT?	Topics	Trainees	N
1- (Ewy GA, et al.,1987)	Academic Medicine	2PP	No	Physical examination	UGME	116,92
2- (Ovassapian A, et al.,1988)	British journal of anaesthesia	2PP	Yes	Intubation	PGME	16,16
3- (From RP, et al.,1994)	Anesthesia and analgesia	2PP	Yes	Airway Management	UGME	49,48
4- (Peugnet F, et al.,1998)	Computer Aided Surgery	2PP	Yes	laser coagulation	PGME	5,3
5- (Tuggy ML,1998)	The journal of the American board of family practice	2PO	Yes	Flexible sigmoidoscopy	PGME	5,5
6- (Scott DJ, et al.,2000)	Journal of the American College of Surgeons	2PP	Yes	laparoscopic cholecystectomy	PGME	9,13
7- (Hamilton EC, et al.,2001)	American journal of surgery	2PP	Yes	laparoscopic hernia repair	PGME	10,11
8- (Naik VN, et al.,2001)	Anesthesiology	2PP	Yes	Intubation	PGME	12,12
9- (Ost D, et al.,2001)	American journal of respiratory and critical care medicine	2PO	Yes	bronchoscopy	PGME	3,3
10- (Edmond CV, Jr.,2002)	Laryngoscope	2PO	NO	endoscopic sinus surgery	PGME	2,2
11- (Hamilton EC, et al.,2002)	Surgical Endoscopy and Other Interventional Techniques	1PP	NO	laparoscopic cholecystectomy	PGME	19
12- (Rowe R, et al.,2002)	Anesthesia & Analgesia	2PP	Yes	Intubation	PGME	12,8
13- (Seymour NE, et al.,2002)	Annals of surgery	2PO	Yes	laparoscopic cholecystectomy	PGME	8,8
14- (Gerson LB, et al.,2003)	Endoscopy	2PP	Yes	Sigmoidoscopy	PGME	9,7
15- (Gormley GJ, et al.,2003)	Annals of the rheumatic diseases	2PP	Yes	shoulder injection	UGME	20,20
16- (Lee SK, et al.,2003)	Journal of trauma	2PP	Yes	trauma assessment	PGME	30,30
17- (Abrahamson S, et al.,2004)	Quality & Safety in Health Care	2PO	Yes	Intubation	PGME	5,5
18- (Blum MG, et al.,2004)	Annals of thoracic surgery	2PO	Yes	Bronchoscopy	PGME	5,5
19- (Di Giulio E, et al.,2004)	Gastrointestinal endoscopy	2PO	Yes	upper gastrointestinal endoscopy	PGME	11,11
20- (Grantcharov TP, et al.,2004)	British journal of surgery	2PP	Yes	Laparoscopic cholecystectomy	PGME	8,8
21- (Sedlack RE, et al.,2004)	American journal of gastroenterology	2PO	Yes	colonoscopy	PGME	4,4
22- (Sedlack RE, et al.,2004)	Clinical Gastroenterology and Hepatology	2PO	Yes	flexible sigmoidoscopy	PGME	19,19
23- (Ahlberg G, et al.,2005)	Endoscopy	2PO	Yes	Colonoscopy	PGME	6,6
24- (Hochberger J, et al.,2005)	Gastrointestinal endoscopy	2PP	Yes	upper gastrointestinal endoscopy	PGME	9,5
25- (Schijven MP, et al.,2005)	Surgical Endoscopy	CO	-	laparoscopic cholecystectomy	PGME	12,12
26- (Stitik TP, et al.,2005)	American journal of physical medicine & rehabilitation	2PP	Yes	injection skills	PGME	15,15
27- (Banks E, et al.,2006)	American Journal of Obstetrics and Gynecology	2PP	Yes	episiotomy repair	PGME	12,12
28- (Chaer RA, et al.,2006)	Annals of Surgery	2PO	Yes	Peripheral Catheterization	PGME	10,10
29- (Cohen J, et al.,2006)	Gastrointestinal Endoscopy	2PO	Yes	Colonoscopy	PGME	23,22
30- (Scerbo MW, et al.,2006)	Journal of Infusion Nursing	2PP	Yes	intravenous (IV) procedures	UGME	12,14
31- (Ahlberg G, et al.,2007)	American Journal of Surgery	2PP	Yes	laparoscopic cholecystectomy	PGME	7,6
32- (Banks EH, et al.,2007)	American Journal of Obstetrics and Gynecology	2PP	Yes	laparoscopic tubal ligation	PGME	10,10
33- (Cosman PH, et al.,2007)	Studies in Health Technology and Informatics	2PO	Yes	Laparoscopic skills	PGME	5,5
34- (Miranda JA, et al.,2007)	Journal of Hospital Medicine	2PO	No	central venous catheter (CVC) insertion	PGME	16,38
35- (Park J, et al.,2007)	American Journal of Surgery	2PP	Yes	colonoscopy	PGME	12,12
36- (Shavit I, et al.,2007)	Archives of Pediatrics and Adolescent Medicine	2PO	No	Procedural sedation	PGME	16,16
37- (Thomas-Gibson S, et al.,2007)	Endoscopy	1PP	No	colonoscopy	PGME	21
38- (Draycott TJ, et al.,2008)	Obstetrics and Gynecology	1PP	NO	Shoulder dystocia	Both	254
39- (Howells NR, et al.,2008)	Journal of bone and joint surgery	2PO	Yes	arthroscopic skills	PGME	10,10
40- (Ossowski KL, et al.,2008)	Laryngoscope	2PO	Yes	Laryngoscopy	UGME	10,10
41- (Van Sickle KR, et al.,2008)	Journal of the American College of Surgeons	2PP	Yes	Laparoscopic skills	PGME	11,11
42- (Wayne DB, et al.,2008)	Chest	2PO	No	Advanced Cardiac Life Support	PGME	38,40
43- (Yi SY, et al.,2008)	Studies in Health Technology and Informatics	2PO	No	colonoscopy	PGME	5,6
44- (Barsuk JH, et al.,2009)	Archives of Internal Medicine	CO	No	central venous catheter (CVC) insertion	PGME	92,92
45- (Barsuk JH, et al.,2009)	Journal of Hospital Medicine	CO	No	central venous catheter (CVC) insertion	PGME	28,13
46- (Barsuk JH, et al.,2009)	Critical Care Medicine	CO	No	central venous catheter (CVC) insertion	PGME	76,27
47- (Britt RC, et al.,2009)	American Journal of Surgery	2PO	Yes	central venous catheter (CVC) insertion	PGME	13,21
48- (Domuracki KJ, et al.,2009)	Resuscitation	2PO	Yes	cricoid pressure	UGME	53,48
49- (Friedman Z, et al.,2009)	Regional Anesthesia and Pain Medicine	2PP	Yes	Epidural anesthesia	PGME	12,12
50- (Gaies MG, et al.,2009)	Pediatrics	2PP	Yes	BMV, CVC, LP	PGME	18,18
51- (Hogle NJ, et al.,2009)	Surgical Endoscopy	2PO	Yes	laparoscopic cholecystectomy	PGME	6,6
52- (Larsen CR, et al.,2009)	BMJ	2PO	Yes	laparoscopic skills	PGME	11,10
53- (Mohan PVR, et al.,2009)	Medical Journal Armed Forces India	2PP	Yes	laparoscopic cholecystectomy	PGME	12,12
54- (Sotto JA, et al.,2009)	Studies in Health Technology and Informatics	2PO	Yes	Peripheral venous cannulation	UGME	20,20
55- (Bruppacher HR, et al.,2010)	Anesthesiology	2PP	Yes	weaning from bypass	PGME	10,10
56- (Butter J, et al.,2010)	Journal of General Internal Medicine	2PO	No	Cardiac auscultation (Physical exam)	UGME	77,31
57- (Evans LV, et al.,2010)	Academic Medicine	2PO	Yes	central venous catheter (CVC) insertion	PGME	90,95
58- (Ferlitsch A, et al.,2010)	Endoscopy	2PO	Yes	upper gastrointestinal endoscopy	PGME	14,14
59- (Fried MP, et al.,2010)	Otolaryngology--head and neck surgery	2PO	Yes	Endoscopic sinus surgery	PGME	12,13
60- (Gauger PG, et al.,2010)	American Journal of Surgery	2PO	Yes	laparoscopic skills	PGME	7,7
61- (Haycock A, et al.,2010)	Gastrointestinal Endoscopy	2PO	Yes	colonoscopy	Both	18,18
62- (Kallstrom R, et al.,2010)	Journal of Endourology	1PP	No	Transurethral Resection of Prostate	PGME	24
63- (Lenchus JD,2010)	Journal of the American Osteopathic Association	1PP	No	CVC, LP, paracentesis, thoracentesis	both	60
64- (Schout BM, et al.,2010)	BJU International	2PO	Yes	cystourethroscopy	UGME	50,50
65- (Sroka G, et al.,2010)	American Journal of Surgery	2PP	Yes	Laparoscopic Cholecystectomy	PGME	8,8
66- (Tongprasert F, et al.,2010)	Prenatal Diagnosis	CO	No	Cordocentesis	UGME	5,5
67- (Wahidi MM, et al.,2010)	Chest	CO	No	bronchoscopy	PGME	22,22
68- (De Ponti R, et al.,2011)	Journal of the American College of Cardiology	2PO	Yes	transseptal catheterization	PGME	7,7
69- (Ghaderi I, et al.,2011)	American Journal of Surgery	1PP	No	laparoscopic incisional hernia repair (LIHR)	PGME	14
70- (Johnson SJ, et al.,2011)	Human Factors	2PO	Yes	Interventional Radiology procedures	PGME	7,7
71- (Khouli H, et al.,2011)	Chest	2PP	Yes	Central venous catheter (CVC) insertion	PGME	24,23
72- (Palter VN, et al.,2011)	Annals of Surgery	2PP	Yes	Abdominal fascial closure	PGME	9,9
73- (Zendejas B, et al.,2011)	Annals of Surgery	2PP	Yes	laparoscopic inguinal hernia repair	PGME	26,24
74- (Ahya SN, et al.,2012)	Seminars in Dialysis	1PP	No	Hemodialysis catheter insertion	PGME	12
75- (Bagai A, et al.,2012)	Circulation Cardiovascular interventions	2PP	Yes	Cardiac Catheterization	PGME	11,15
76- (Ende A, et al.,2012)	Gastrointestinal Endoscopy	2PP	Yes	diagnostic upper endoscopy	PGME	10,9,9
77- (Franzeck FM, et al.,2012)	Surgical Endoscopy	2PP	Yes	laparoscopic camera navigation	UGME	12,12
78- (Fried MP, et al.,2012)	Archives of otolaryngology--head & neck surgery	2PP	No	Endoscopic sinus surgery	PGME	8,6
79- (Hseino H, et al.,2012)	Simulation in healthcare	2PO	Yes	endovascular skills	PGME	5,5
80- (Orzech N, et al.,2012)	Annals of Surgery	2PP	Yes	laparoscopic suturing skills	PGME	10,10
81- (Palter VN, et al.,2012)	Annals of Surgery	2PO	Yes	laparoscopic colorectal surgery	PGME	9,9
82- (Stather DR, et al.,2012)	Respirology	CO	No	Bronchoscopy	PGME	4,4
83- (White ML, et al.,2012)	Pediatric Emergency Care	1PP	No	lumbar puncture	PGME	21
84- (Daly MK, et al.,2013)	Journal of Cataract and Refractive Surgery	2PO	Yes	Cataract Extraction	PGME	11,10
85- (Gala R, et al.,2013)	Obstetrics and Gynecology	2PP	Yes	laparoscopic skills	PGME	48,54
86- (Palter VN, et al.,2013)	Annals of Surgery	2PP	Yes	Laparoscopic Cholecystectomy	PGME	10,10
87- (Pokroy R, et al.,2013)	Graefes Archive for Clinical and Experimental Ophthalmology	CO	No	cataract surgery	PGME	10,10
88- (Todsen T, et al.,2013)	BMC Medical Education	2PO	Yes	Urethral catheterization	UGME	17,14
89- (Balci MBC, et al.,2014)	Nobel medicus	2PO	Yes	Laparoscopic Skills	PGME	8,8
90- (Bansal VK, et al.,2014)	Journal of Surgical Education	2PP	Yes	laparoscopic cholecystectomy	PGME	9,8
91- (Cannon WD, et al.,2014)	Journal of Bone and Joint Surgery	2PP	Yes	arthroscopic knee surgery	PGME	27,21
92- (Edrich T, et al.,2014)	Journal of Cardiothoracic and Vascular Anesthesia	2PP	Yes	echocardiography	PGME	23,23
93- (Ferrero NA, et al.,2014)	Anesthesiology	2PP	Yes	Transesophageal Echocardiography	PGME	21,21
94- (Hong P, et al.,2014)	International Journal of Pediatric Otorhinolaryngology	2PP	Yes	Myringotomy and tympanostomy tube insertion (MT)	UGME	13,11
95- (McIntosh KS, et al.,2014)	Canadian Journal of Gastroenterology & Hepatology	2PP	No	colonoscopy	PGME	10,8
96- (Minai F, et al.,2014)	Journal of Anaesthesiology, Clinical Pharmacology	2PO	Yes	intubation	UGME	28,29
97- (Palter VN, et al.,2014)	Annals of Surgery	2PP	Yes	Laparoscopic Cholecystectomy	PGME	8,8
98- (Udani AD, et al.,2014)	Anesthesiology Research and Practice	2PP	Yes	subarachnoid blocks (SAB)	PGME	11,10
99- (Grover SC, et al.,2015)	Gastrointestinal Endoscopy	2PP	Yes	colonoscopy	PGME	16,17
100- (Koch AD, et al.,2015)	Gastrointestinal Endoscopy	2PP	Yes	colonoscopy	PGME	8,10
101- (Peltan ID, et al.,2015)	Simulation in healthcare	2PP	Yes	Central Venous Catheter (CVC) Insertion	PGME	36,37
102- (Tolsgaard MG, et al.,2015)	Medical Education	2PP	Yes	Ultrasonography	UGME	16,14
103- (Aloosh M, et al.,2016)	Journal of Endourology	1PP	No	Ureteroscopy	PGME	5
104- (Arias T, et al.,2016)	International Journal of Gynaecology and Obstetrics	2PO	Yes	vaginal examination	UGME	66,21
105- (Asoglu MR, et al.,2016)	Journal of the Turkish-German Gynecological Association	CO	No	hysterectomy	PGME	75,98
106- (Jaffer U, et al.,2016)	Journal of Surgical Education	1PP	No	Ultrasonography	UGME	24
107- (Thawani JP, et al.,2016)	Journal of Clinical Neuroscience	2PO	Yes	Endoscopy	PGME	3,3
108- (Waterman BR, et al.,2016)	Orthopedics	2PP	Yes	Diagnostic Shoulder Arthroscopy	PGME	12,10
109- (Bloch A, et al.,2017)	Anesthesia and Analgesia	2PP	Yes	Echocardiography	PGME	22,21
110- (Boza C, et al.,2017)	Surgical Endoscopy	2PP	No	advanced laparoscopy	Both	10,12,5
111- (Crochet P, et al.,2017)	Journal of Surgical Education	CO	No	Laparoscopic Suturing	PGME	12,6
112- (Dyre L, et al.,2017)	Medical Education	2PP	Yes	Ultrasonics	UGME	30,26
113- (Lotfy M, et al.,2017)	Egyptian Journal of Surgery	2PP	Yes	laparoscopic appendectomy	PGME	15,15
114- (Rosen H, et al.,2017)	Journal of Obstetrics and Gynaecology Canada	2PP	Yes	Ultrasonography	PGME	9,9
115- (Tolsgaard MG, et al.,2017)	Annals of Surgery	2PO	Yes	Ultrasonography	PGME	26,26
116- (Maertens H, et al.,2017)	European Journal of Vascular and Endovascular Surgery	2PP	Yes	Endovascular skills	PGME	9,10,10
117- (Kallidaikurichi Srinivasan K, et al.,2018)	BMJ open	2PO	Yes	Epidural Analgesia	PGME	13,9
118- (Garfjeld Roberts P, et al.,2019)	Arthroscopy - journal of arthroscopic and related surgery	2PP	Yes	diagnostic knee arthroscopy	PGME	15,13
119- (Ostergaard ML, et al.,2019)	European Radiology	2PO	Yes	Ultrasonography	PGME	11,9
120- (Popovic B, et al.,2019)	American Journal of Cardiology	2PO	Yes	Cardiac Catheterization	PGME	10,10
121- (Wong DT, et al.,2019)	European Journal of Anaesthesiology	2PP	Yes	bronchoscopic-guided intubation	Both	16,15

Appendix S4

Effective ID factors influencing transfer of learning in SBT (Study ID refers to references listed in appendix)

ID Features		Study ID	N
Feedback		1,3,4,5,6,7,8,9,10,11,12,13,14,15,16,18,19,21,22,23,24,25,26,27,28,29,30,31,34,35,37,39,41,42,43,44,45,46,48,51,52,54,55,56,57,58,60,61,62,63,65,68,70,72,73,74,75,76,77,78,79,80,81,84,86,88,89,91,95,96,97,98,99,101,102,106,107,108,109,110,111,112,115,116,117,120	86
Feedback timing	During	1,4,5,6,7,9,10,11,12,13,14,15,18,19,21,22,23,25,26,28,29,30,31,35,37,41,42,43,44,45,46,48,51,52,54,55,56,58,62,63,65,70,72,73,74,75,77,78,79,80,81,84,86,88,89,91,95,96,97,98,99,101,102,107,108,109,110,112,115	68
	After (with debriefing)	14,23,24,27,30,31,42,44,52,55,56,60,63,68,73,77,97,99,102,111,115,116,120	23
Feedback Source	Instructor	4,7,8,13,14,15,16,23,24,27,28,31,34,39,41,42,44,45,46,52,55,56,57,60,63,65,70,72,73,74,76,77,88,96,97,98,99,101,102,106,110,111,115,116,117,120	46
	Simulator (force/haptic)	6,9,10,11,12,13,14,19,21,22,25,28,29,30,35,37,41,43,48,51,52,54,58,62,68,70,75,77,78,79,80,86,89,91,99,102,107,108,112,115	40
Simulator (Audio visual)	1,10,18,23,26,35,41,42,43,54,55,56,75,78,80,81,84,95,97,99,109,112,115	23
Simulator (no type stated)	4,5,31	3
Mastery Learning		33,41,43,44,45,46,56,57,74,98,101	11
Proficiency based training		3,4,11,12,13,14,17,19,23,24,25,28,31,33,41,47,48,51,52,54,58,59,60,62,65,68,72,75,78,79,80,81,84,85,86,89,91,95,97,98,105,107,108,110,111,115,116,117,119	49
Deliberate practice		44,45,46,56,63,74,82,97,98,101,118	11
Increasing complexity		5,10,11,15,19,20,21,22,25,28,29,30,31,33,35,37,41,43,51,53,54,58,64,67,70,78,80,81,82,84,85,86,89,97,99,100,110,113,116,117,118,119	42
Repetitive Practice		2,4,6,7,9,11,12,13,14,17,19,20,23,24,25,28,29,31,33,41,43,44,45,46,47,48,51,52,53,54,56,57,58,59,60,62,65,67,68,72,73,74,75,77,78,79,80,8,1,84,85,86,89,90,91,95,97,98,100,103,101,104,105,107,108,109,110,11,115,116,117,119	72
Variability		5,9,12,15,18,19,21,22,28,29,30,37,42,49,54,55,56,61,62,82,93,95,99,100,118,121	26
Distributed Practice		1,2,3,5,6,7,9,10,11,12,14,17,18,19,20,21,23,24,25,26,28,29,30,31,33,34,37,39,41,42,43,44,45,46,49,50,51,52,53,54,55,57,58,59,60,61,62,63,64,65,66,67,71,72,73,75,76,77,79,80,81,82,84,85,86,87,89,90,91,93,95,96,97,98,99,100,101,105,107,108,110,111,113,116,117,118,119	86
Teaching cognitive base		2,3,8,9,10,11,15,16,19,23,24,25,27,28,29,32,33,34,37,40,42,43,44,45,47,49,50,53,57,58,59,61,62,65,67,73,75,76,77,79,81,83,86,88,89,91,92,93,94,96,97,99,100,101,104,106,108,109,111,114,116,117,120,121	64
Demonstration of procedures (Film)		7,8,9,14,16,18,21,22,23,25,26,30,35,37,40,42,44,46,54,57,58,59,60,63,64,65,69,71,73,76,79,80,81,83,91,92,93,95,101,106,108,110,111	42
briefing		3,19,27,29,30,52,55,69,79,80,81,84,91,94,95,97,98,102,103,112,115,117,120	23
Simulator Fidelity	High Fidelity	1,4,5,9,10,11,12,13,14,16,18,19,20,21,22,23,24,25,28,29,30,31,33,35,37,42,43,49,51,52,53,55,56,58,60,61,62,64,68,70,75,77,78,79,80,81,82,84,87,89,91,92,93,95,97,99,100,102,103,105,106,107,108,109,112,114,120,121	68
	Low Fidelity	7,8,15,27,34,38,40,45,46,47,57,63,66,69,72,74,83,88,94,96,98,101,104,111,113,117	26
	Mixed	41,50,76,86,115	5
Simulator Type	Virtual Reality	4,5,9,10,11,12,13,14,19,20,21,22,23,25,28,29,30,31,33,35,37,43,49,51,52,53,54,58,59,60,61,62,64,67,68,70,75,77,78,79,80,81,82,84,87,89,91,92,93,95,97,99,100,102,103,106,107,108,109,112,114,116,119,120,121	65
	Part-task trainer	7,8,15,18,27,34,38,40,45,46,47,57,63,66,73,74,83,88,96,98,101,104,117	23
	Full body Manikin	1,16,42,55,56,71	6
	Box Trainer	6,65,69,90,11,113	6
	Mixed	3,41,50,76,86,105,115,118	8

Appendix S5

Demographic information of participants

Demographic variables	N
Level	Faculty Member	9
	Medical Student	3
	Resident	5
Age	Min	22
	Max	54
	Average	37.4
Sex	Female	7
	Male	10
Specialty	Emergency Medicine	4
	General Surgery	2
	Orthopedics	2
	Obstetrics and Gynecology	2
	Anesthesiology	1
	Internal Medicine	2
	Pediatrics	1
	Medical Students	3