The feasibility of simulation-based high-stakes assessment in emergency medicine settings: A scoping review.

Using simulation in high-stakes assessments has been evolving as a method to improve the assessment process. There is a concurrent need to address challenges and establish best practices to ensure the best quality when implementing high-stakes evaluations. The aim of this study is to provide an insight for stakeholders about using multiple modalities of simulation in high-stakes evaluations by presenting challenges, best practices, and future directions described in the relevant literature. A scoping review of original studies (from the year 1994-2021) including the use of common modalities (standardized patients, high-fidelity mannequins, part-task trainers, virtual simulation, and hybrid simulation) was conducted. The search covered the common databases: PubMed, Education Resource Information Center, Cumulative Index to Nursing and Allied Health Literature, and the Cochrane library. Initial screening in the databases resulted in 111,253 articles. After the application of a refining search strategy, 47 articles were included for comprehensive evaluation. Most articles were about credentialing/licensure purposes of assessment. Many articles were specialty-specific, especially focusing on anesthesia. Most challenges described were related to the validity of the assessment that should be considered in the planning phase. Best practices were mostly related to planning for measures to assure the validity of the assessment tools and process. The use of multimodality simulation for high-stakes assessment is growing despite challenges; this growth is associated with the accumulation of experience shared in literature. This growth will help to improve planning, practices, and goals achievement from such an application. Copyright:

Introduction

Assuring the competency of health care providers (HCPs) is an expected outcome of training desired by accrediting institutions. Stakeholders (e.g., the learners, teachers and educational institutions, and the wider society) deliver competency assessment of grading and practicing healthcare provider's paramount attention based on its impact on the quality of health care.[1] Moreover, the process of assessment is a key driver for the development of educational systems. Competency assessment of healthcare providers has been growing concurrently with the development of the healthcare system, yet it is still challenging to provide a high-quality evaluation. This challenge is further magnified when dealing with high-stakes assessments which could be defined as, “an evaluation process that has a major academic, educational, or employment consequence....”[2] In this case, the sensitivity and the demands for quality assurance are much higher than the demands for low-stakes (or formative) assessments.

Simulation has been introduced to the field of training and education successfully.[34] Recently, using simulations in high-stakes assessments have been evolving as a method to enhance and improve the assessment process.[5] There are many examples of successful implementation of high-stakes assessment integrating simulation. In 1998, the Educational Commission for Foreign Medical Graduates introduced the Clinical Skills Assessment, which uses standardized patient (SPs) as the modality of simulation, for certification of foreign medical school graduates.[6] In Israel, simulation has been implemented in many high-stakes assessment projects for more than 10 years. These projects include medical schools’ candidate selection, national board examination in anesthesiology, and national accreditation for paramedics.[7] In 2003, the Royal College of Physicians and Surgeons of Canada introduced a hybrid simulation in the Comprehensive Examination in Internal Medicine.[8]

Although medical literature shows a positive attitude toward such an implementation, articles from nursing literature demonstrated a more conservative attitude.[29] The limited experience with such use of simulations and the lack of relevant expertise in the field of nursing can make the process of applying quality assurance measures more challenging. Without the ability to assure the quality, the validity of the assessment would be questionable. After a meeting discussion at the International Nursing Association for Clinical Simulation and Learning, a conclusion came to show that students should not be judged by one modality until further experience grows to demonstrate that this assessment method provides a quality assessment.[2]

In relation to the specialty of emergency medicine (EM), the Australasian College for Emergency Medicine planned to introduce high-stakes simulation-based summative assessments in the form of Objective Structured Clinical Examinations into the Fellowship examination starting from 2015.[5] Regarding pediatric EM, there is an increasing use of simulations in the assessment of candidates for the programs.[1] However, the existing literature about the experience around using simulation-based assessments in high-stakes EM assessment is quite limited. This article intended to shed a light on the status of simulation-based assessments in high-stakes settings and to drive best practices that should be applied in high-stakes EM assessments. Specific attention is paid here toward using simulations for credentialing and licensure purposes that lead to the decision of pass/fail that permits HCPs to practice on real patients since those settings are believed to provide a more authentic reflection of high-stakes assessments.[10]

The aim of this study is to provide a scoping review of the currently existing literature about the utilization of different simulation modalities in high-stakes evaluations to describe its challenges, best practices, and future directions. Since the aim of this research is not to get a summarized answer, and the research topic is broad, the scoping review approach was used to review the exciting literature instead of a systematic review or meta-analysis.

Materials and Methods

Scoping review is an approach to synthesize evidence by providing an overview of the existing research on a, usually, broad research question with limited information in the literature.[1112] This study follows steps of conducting a scoping review which include: defining research question, searching relevant studies, selecting relevant studies, charting data, and describing results.

First, the research question was formulated as the following: What are the best practices, challenges, and future directives of using multiple modalities of simulation in high-stakes clinical assessments, and how do these relate to the specialty of EM? Second, relevant studies were initially screened using terms were: “simulation,” “high-stakes,” and “assessments.” The term “summative” was also added after the initial screening of titles. Furthermore, the search term “board-certification” was added later to further focus on the high-stakes aspect of the aim.

Third, regarding selecting relevant studies, the search covered the common well-known databases in medical education: PubMed, Education Resource Information Center, Cumulative Index to Nursing and Allied Health Literature, and the Cochrane library. The intention was to include all studies discussing the application of any modality of simulation in high-stakes clinical assessment. The exclusion criteria were:

Studies which are not for credentialing/licensure purposes, reentry, and selection processes

Studies which are mainly intended for patient safety and quality improvement

Studies with a focus on the formative/feedback aspects of assessment

Studies about computer-based simulated cases since these are not considered a high-fidelity simulation and it can be considered with the computer-delivered MCQs examination

Studies without any of the well-known modalities of clinical simulation

Studies from specialties other than medicine or nursing

Abstract or full-text article is not available for analysis

Remotely conducted (virtual) examination.

Fourth, regarding data charting, the search was conducted on July 15, 2021, from 8 am to 4 pm, and included studies published from 1994 to July 2021, since the modern practice of simulation had started in the early 90s. The electronic searches in many databases start with that year, and it is thought that searching for articles older than this year would not reflect the current understanding of educational best practices.[7] Articles were fully read searching for data that fits under ant of the three key themes of the study (challenges, best practices, and future directives); extracted data were categorized accordingly. Finally, results were summarized and reported in tables in concordance with the three key themes.

A certain strategy was used to perform this scoping review, [Figure 1] for the outlines of search strategy. The initial screening in databases was using the phrase “simulation assessment” to get a general impression about exciting data. All types of publications were included initially (including editorials, reviews, etc.). The search needed to be refined using the Booleans and specific key terms. After consulting with the librarian, the decision was to use the following search strategy: “simulation” AND “assessment” AND “summative” OR “high-stakes” OR “board certification.” All studies written in English, with or without another language, were included.

Figure 1

Search strategy

Then, the titles were screened using all the exclusion criteria. Second, using the same exclusion criteria, the abstracts of these articles were screened to yield articles to be read in full. Then, the references were manually reviewed for possible articles to be included. Specific attention was paid to articles directly related to EM to be comprehensively analyzed.

Results

The initial screening for simulation assessment in the databases resulted in 1,11,253 articles using the following search strategy: Simulation AND assessment AND “summative” OR “high-stakes” OR “board certification;” this resulted in a total of 3711 articles included for title check.

After applying the exclusion criteria, 91 articles remained after checking the titles. Then, after screening the abstracts of these articles, only 44 articles remained to be read in full. When references were manually reviewed further 3 articles were added. All studies found were written in English, although three articles were multilingual. Specific attention was paid to 4 articles found to be directly related to EM.

Finally, this search yielded 47 articles, which were included for analysis [Tables 1 and 2]. The full text of those articles was retrieved, and they were read in full. All articles discussed at least one aspect of at least one modality of simulation applicable to high-stakes clinical assessments. Most articles were about credentialing/licensure purposes of assessment; few articles were related to reentry or selection. Most of the articles which discuss simulation-based assessments were too general. However, many were specialty-specific, with articles on the topic of anesthesia being the most abundant among those. There were variations in different variables between all articles as presented in Tables 1 and 2.

Table 1

Details of articles involved in the study

Article (number according to reference)	General versus specific for high-stakes	Modalities of simulation	Specialty	Type of high-stakes mentioned	Themes discussed	Targeted group by the assessment
O’Leary[5]	High-stakes	Part-task, mannequins	EM	Credentialing	Challenges	Postgraduates
					Best practices
Furman et al.[13]	General	SP	General	Credentialing	Best practices	Postgraduates
Boulet[14]	High-stakes	SPs, mannequins, part-task and hybrid	General	Selection and credentialing	Best practices	Postgraduates
Rizzolo et al.[2]	High-stakes	SPs	Nursing	Credentialing	Best practices	Undergraduates
Kardong-Edgren et al.[9]	High-stakes	SPs, mannequins, part-task and virtual simulation	Nursing	Credentialing	Challenges	Undergraduates
Calhoun et al.[1]	High-stakes	Mannequins, part-task and hybrid	Pediatric EM	Credentialing	Best practices, future directions	Postgraduates
Petrusa[15]	High-stakes	SPs, part-task, hybrid	General	Credentialing	Challenges, future directions	Postgraduates
Hatala et al.[8]	High-stakes	SPs, part-task, hybrid	Internal medicine	Credentialing	Best practices (sharing experience)	Postgraduates
Leblanc[10]	High-stakes	Mannequins	Anesthesia	Credentialing	Challenges	Postgraduates
					Best practices
					Future directives
Bensfield et al.[16]	High-stakes	Mannequins	Nursing	Credentialing	Best practices (experience)	Undergraduates
Ziv et al.[7]	High-stakes	Mannequins	Anesthesia	Credentialing	Best practices	Postgraduates
Adamo[17]	General	SP	General	Credentialing	Challenges	Postgraduates
					Best practices
					Future directives
Chambers et al.[6]	High-stakes	SP	General	Credentialing	Best practices	Postgraduates
Weinger et al.[18]	High-stakes	Mannequins	Anesthesia	Renewal	Best practices	Postgraduates
John et al.[19]	General	SPs, mannequins	General	Credentialing	Best practices	Postgraduates
Cannon et al.[20]	High-stakes	Mannequins	IM	Selection	Future directions	Postgraduates
Collins and Harden[21]	General	SPs	Medical school	Credentialing	Challenges, best practices	Undergraduates
Boulet and Murray[22]	General	Part-task, mannequins	Anesthesia	credentialing	Best practices	Postgraduates
Tavakol et al.[23]	General	Part-task, virtual simulation, SP	Surgery	Credentialing	Best practices	Postgraduates
Swing[24]	General	Part-task, mannequins, SP	General	Credentialing	Best practices	Postgraduates
Srinivasan et al.[25]	General	Mannequins, virtual simulation, SP	General	Credentialing	Best practices	Postgraduates
Boulet et al.[26]	General	SP	General	Credentialing	Best practices	Postgraduates
Glavin and Gaba[27]	General	SP, part-task	General	Credentialing	Challenges	Postgraduates

EM=Emergency medicine, SP=Standardized patient, IM=Internal medicine

Table 2

Details of articles involved in the study

Article (number according to reference)	General versus specific for high-stakes	Modalities of simulation	Specialty	Type of high-stakes mentioned	Themes discussed	Targeted group by the assessment
Sadeghi et al.[28]	High-stakes	SP	Psychiatry	Credentialing	Future directions	Postgraduates
Leo et al.[29]	High-stakes	SP, mannequins	Critical care	Credentialing	Best practice	Postgraduates
Harvey and Radomski[30]	High-stakes	SP, mannequins	Medical school	Credentialing	Challenges	Undergraduates
Berkenstadt et al.[31]	High-stakes	SP, mannequins	Anesthesia	Credentialing	Challenges	Postgraduates
					Best practices
Turner and Dankoski[32]	General	SP	General	Credentialing	Best practices	Postgraduates
McNaughton et al.[33]	General	SP	Psychiatry	Credentialing	Challenges	Undergraduates and postgraduates
					Best practices Future directions
Feldman et al.[34]	General	SP	General	Credentialing	Best practices	Postgraduates
Levine et al.[35]	High-stakes	Part-task, virtual simulation, SP	General	Credentialing	Best practices	Postgraduates
					Future directions
Schuwirth and van der Vleuten[36]	General	SP	General	Credentialing	Best practices	Undergraduates and postgraduates
Yudkowsky[37]	High-stakes	SP	Psychiatry	Credentialing	Challenges	Postgraduates
					Best practices
Oermann et al.[38]	High-stakes	SP	Nursing	Credentialing	Best practices	Undergraduates
Goldberg et al.[39]	High-stakes	SP, mannequins	General	Credentialing	Best practices	Postgraduates
Holmboe et al.[40]	General	SP, mannequins, virtual reality	General	Credentialing	Challenges	Credentialing
					Best practices
					Future directions
Banerjee[41]	High-stakes	SP, part-task mannequins, virtual reality	General	Credentialing	Challenges	Postgraduates
DeMaria et al.[42]	High-stakes	Mannequins	Anesthesia	Reentry	Best practices	Postgraduates
Orledge et al.[43]	High-stakes	Mannequins, part-task, virtual reality/haptic	Anesthesia	Reentry	Best practices	Postgraduates
McGrath et al.[44]	General	Virtual reality	EM	Credentialing	Challenges	Undergraduates and postgraduates
					Best practices
Warner et al.[45]	High-stakes	SP	Anesthesia	Credentialing	Best practices	Postgraduates
Bauer et al.[46]	General	SP	General	Credentialing	Best practices	Undergraduates
Nadir et al.[47]	High-stakes	SP, part-task mannequins, virtual reality	EM	Reentry	Best practices	Postgraduates
					Future directions
Monti et al.[48]	High-stakes	SP	Medical school	Credentialing	Challenges	Undergraduates
					Best practices
					Future directions
Arrogante et al.[49]	General	SP	Nursing	Credentialing	Challenges	Undergraduates
					Best practices
Isaak et al.[50]	General	SP, mannequins part-task	General	Credentialing	Challenges	Postgraduates
					Best practices
					Future directions
Sabzwari et al.[51]	High-stakes	SP	Medical school	Credentialing	Best practices	Undergraduates
					Future directions

EM=Emergency medicine, SP=Standardized patient

Most publications gave a general overview and were not specific to any specialty. This could be due to the scarcity of publications describing practical experiences coming from clinical specialty; this is more obvious when talking about EM. It was not surprising to find out that most of the specialty-specific publications discuss the topic of anesthesia, as it is the leading specialty in the field of clinical simulation. In addition, it was not unexpected to find that SP was the most common modality of simulation to be applied in high-stakes assessments with well-established standards and relatively long experience of the application. Most of the studies were about credentialing/licensure purposes (which aim to assess the competency of healthcare providers to practice on real patients).

Many modalities of simulation have been used in training and assessments. The most commonly used modality in high-stakes assessments is the SP.[6] Other modalities include high-fidelity mannequins, part-task trainers,[13] virtual simulators, and hybrid simulation (a combination of any two or more of these modalities).[1415] Only few studies describe the use of more than one simulation modality in high-stakes assessments, and almost all of them are from fields outside of EM.

While significant variability in articles’ types, methodology, and the manner of presentation was found, the following themes emerged: Challenges, best practices, and future directions. I decided to correlate the findings to these themes; when in doubt, the opinion of a 2nd reviewer expert in the field was taken (acknowledged). The best practices were elaborated on extensively, especially the psychometric measures, across the publications. Hence, it was decided to subdivide this theme into subthemes. For the sake of practicality, this theme got divided into the following subthemes: Preassessment, during the assessment, and postassessment. A summary of best practices is provided in Table 3. Further discussion of the findings is provided in the discussion section.

Table 3

Classification of best practices based on the stage of assessment implementation

Examples of best practices

Planning phase

Right modality of simulation for the tasks to be evaluated

Real practice design to reflect the highest possible fidelity

Appropriate structure and resource standards of the assessment center

Take proper security measures to prevent breach of the exams’ content

Clear and user-friendly scenarios for SPs, examiners and technical support staff

Determine the metrics of the assessment

Carefully chosen/designed assessment tools (checklists, global ratings) to facilitate high standard metrics

Raters should be trained and qualified based on a preset protocol

Measures to assure high reliability and validity e.g., increasing the number of scenarios, designing task-specific stations and standardizing the exams administration

Develop targeted metrics, e.g., Kappa >0.75, reliability, validity, accessibility, feasibility

During the assessment phase

Timing of each station should be 5-10 min; in acute care medicine this can be shortened to 5 min

Video and audio recording, as a quality measure, to be considered

Synchronizing and standardizing timing of the exam conducted in multiple centers

Availability of real-time technical support in all centers

Reinforcing measures to prevent cheating

Room designation during the examination, i.e., examiner, standardized patient and/or technician

Post assessment phase

Evaluation of the assessment process

Psychometric analysis of the scores to support the validity of the assessment

Exploring for confounding factors that influence the outcome

Review the issues that appeared during the exam and actions taken accordingly

Develop continuous improvement plans

Update manuals and/or protocols

Demonstrate results and appeal process to the learners

Conduct remediation process as planned before

SP=Standardized patient

Planning

Because the purpose of high-stakes assessment affects the ways in which the assessment should be designed, the assessment should be planned carefully to reflect a true evaluation of the desired tasks from the examinee.

Implementation

Much logistics should be considered during the lifetime of the assessment. Usually, each station will take 5–10 min; in acute care medicine, timing can be shortened to 5 min.[14] Video and audio recording is a quality measure to be considered during the examinations.[913] If the examination is conducted in multiple centers, the timing and operations should be synchronized and standardized; real-time technical support should be available in all centers.[13] Measures to prevent cheating should be reinforced. The responsibility for each room during the examination should be designated, i.e., the examiner, the SP, or the technician.

Postimplementation

Scores collected should facilitate psychometric analysis to support the validity of the assessment. Scores post hoc analysis should be conducted; driven data should reflect the desired outcomes; exploring for confounding factors that influence the outcome. Issues that appeared during the exams should be reviewed and actions should be taken based on that.[2]

Challenges

Although simulation technology has shown tremendous development, there are still limitations in displaying certain manifestations. The high cost is a major challenge. Rater recruitment and training are time- and effort-consuming processes.[2] The variability between the training centers imposes a big challenge to be addressed when preparing for the examinations.[9] Another challenge is the presence of a wide variety of assessment tools with low validity.[1]

Future directions

With the expansion of aspects of clinical competencies to be assessed, there will be a growing need for validated assessment tools in the field of EM. The interest in applying entrustable professional activities would reflect on the design of the assessment.[1] With the increasing popularity of simulation, mobile units of simulation assessment tools may get integrated into the assessment system soon.[15]

Discussion

The increasing utilization of high-stakes assessment is reflected in this review by the increasing number of the found related studies in the past 5 years. This increasing utilization is associated with growing experience which is presented by the diversity of findings related to the three main themes of this review. Overall, most of the studies were about licensure/credentialing; which is a finding consistent with what has been described in the literature before.[14] On the other hand, this review is pointing to a limited experience in using simulation for the selection process or reentry/renewal purposes; however, it is expected to be see more implementation of simulation for those purposes as the scope of utilization of simulation is increasing. Most of the experience with using simulation in these settings was for postgraduates; this can be explained by the need to get a higher level of assessments as the level of the targeted group is getting higher.

It is important for stakeholders to be familiar upfront with the anticipated obstacles that may hinder the application of a high-quality simulation-based high-stakes assessment [Table 3]. Regarding the first theme about challenges to implement simulation in high-stakes examinations, related articles found were prevalent in this review. This is an expected finding as using simulation is logistically demanding, especially in high-stakes situations. The application of quality assessment using multiple modalities of simulation faces many challenges.[14] These challenges can delay the translation of the rapid growth of simulation-based examinations (SBEs) into the area of high-stakes assessments. Some of the challenges discovered by this review are common to most types of assessments, whereas others are unique to simulation-based assessments. In addition, it was found that the higher the stakes of the assessment the greater the challenges will get, and some are easier than others to overcome.[5]

As expected, most of the articles gave recommendations regarding best practices of implementing simulation in high-stakes assessments; however, seldom it is to find a comprehensive list of best practices in a single article. This reflects the width and complexity of this subject and further justifies the need for this review and similar studies in this area. As this review demonstrates, to optimally assess learners, it is recommended to utilize multiple modalities of assessment to evaluate different aspects of the examinee's competency. Hence, a simulation-based assessment is not recommended to be used as the sole method to judge the learner's competency.[2]

It is paramount to give special efforts for the planning phase when designing the scenarios.[11314] For example, designing an assessment for board certification would be different from an assessment meant for a selection for residents.[214] Furthermore, planners should choose the right modality of simulation for the tasks to be evaluated. For example, high-fidelity mannequins and part-task trainers can assess the skills of airway management. The design should reflect the real practices; designers should aim for the highest possible fidelity. The assessment center should have the structure and sources to promote the highest standards.[13] Cautious security measures should be taken to prevent a breach of the exams’ content, especially after piloting the examination.[213] The portrayal of scenarios should be clear and user-friendly for SPs, examiners, and technical support staff.[13]

A very important part of the planning is to determine the metrics of the assessment.[14] Assessment tools, wither checklists or global ratings, should be carefully chosen/designed to facilitate the production of high standards metrics.[14] This decision should be done by the examination committee. Raters should be trained and qualified based on a preset protocol. Measures to assure high reliability and validity (e.g., increasing the number of scenarios, designing task-specific stations, and standardizing the examinations administration) should be given a special dedication.[1415] For high-stakes assessments, there should be targeted metrics; for example, Kappa should be >0.75.[5] Metrics such as reliability, validity, accessibility, and feasibility are important parameters to ensure the quality of the assessment process.

The real-time implementation of the modalities at the time of the exam gets the most attention; however, this step does not come in a solo. Although it has a strong impact on then impression about the assessment process, it does not reflect the quality of the whole process. Some specifics in relation to this phase were mentioned in detail in the studies involved in this review. Those parameters can help in benchmarking, however, there is room for development and improvements to suite different needs.

Test's quality assurance should be followed to go beyond the time of the assessment itself. There should be a system to evaluate the assessment process. A continuous improvement plan should be in place. Furthermore, manuals of protocols should be updated. Results and the appeal process should be demonstrated to the learners as per the plan. The process of remediation should be conducted according to the plan and events raised during the assessment.[13] All previously mentioned points should be worked on.

Since simulation-based assessment is an evolving area, it is important to know the future direction of the field. As demonstrated in this review, the application of simulation in different disciplines is growing. The increasing attention toward competency-based assessments will increase the call for using simulation in high-stakes assessments. There are many opportunities for growth of this modality of assessment with the rapid growth of the applications of simulation. For EM specialists, this should open a growing opportunity for adapting experiences of other clinical disciplines.[1]

Limitations

This study is not without limitations. As this field is relatively new, the number of relevant studies was limited. Because of the newness of this field, there was still a significant amount of differing perspectives, and general best practices were yet to consistently emerge in the literature. The heterogeneity and complexity of the articles included also limited the ability to collect all the detailed information presented. To overcome this limitation, I tried to follow a chronological approach and to include the most prevalent topics presented in the literature. There could be valuable information about challenges, best practices, and future directions that could be derived from formative assessment literature which was excluded in our search. Furthermore, half of the articles reviewed here included some discussion about formative assessments. However, the novelty in this study is providing a layout for guidance to use all modalities of simulation to assess the competency of healthcare providers. There is no other study in the literature collecting the exact same data.

Conclusions

The use of simulation for high-stakes assessment is promising. There should be a balance between the need to measure specific competencies with the impeded limitations of simulators. Using a simulation-based assessment should not be the sole method to assess learners’ performance; it should be one part of many for a holistic approach. Because of the impeded obstacles and challenges, the growth and spread of such an experience are relatively slow when compared to SBE. To get stakeholders’ approval, this method of assessment should demonstrate reliability, validity, accessibility, cost, and feasibility. Stakeholders should aim to establish an accrediting process to train and credential all simulation-based high-stakes assessment programs. Hence, the simulation has a great potential to promote safe clinical practice on patients by assuring the competency of healthcare providers serving the community.51

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.