Assessing the performance and satisfaction of medical residents utilizing standardized patient versus mannequin-simulated training.

BACKGROUND: Conducting simulations of rapidly decompensating patients are a key part of internal medicine (IM) residency training. Traditionally, mannequins have been the simulation tool used in these scenarios.
OBJECTIVE: To compare IM residents' performance and assess realism in specific-simulated decompensating patient scenarios using standardized patients (SPs) as compared to mannequin.
METHODS: Nineteen IM residents were randomized to undergo simulations using either a mannequin or an SP. Each resident in the two groups underwent four different simulation scenarios (calcium channel blocker overdose, severe sepsis, severe asthma exacerbation, and acute bacterial meningitis). Residents completed pretest and post-test evaluations as well as a questionnaire to assess the reality perception (realism score).
RESULTS: Nine residents completed mannequin-based scenarios, whereas 10 completed SP-based scenarios. Improvement in the post-test scores was seen in both groups. However, there were significantly higher post-test scores achieved with SP simulations in three out of the four scenarios (P=0.01). When compared with the mannequin group, the SP simulation group showed a significantly higher average realism score (P=0.002).
CONCLUSIONS: Applying SP-based specific-simulation scenarios in IM residency training may result in better performance and a higher sense of a realistic experience by medical residents.

RCT Entities:   Population Interventions Outcomes

Introduction

Simulation-based training (SBT) in medical education has grown tremendously over the last 5 years. It has been reported to be an effective method of teaching internal medicine (IM) residents both procedural and nonprocedural skills without practicing on actual patients.1–3 Historically, SBT has utilized standardized patients (SPs), mannequins, or web-based virtual scenarios.4,5 Traditionally, simulation mannequins are used for teaching procedural skills, surgical skills, and team training.6 Mannequins can also be used to simulate a patient with clinical deterioration. In contrast, SPs – typically medically trained actors or medical personnel – are often used to teach both physical examination and patient communication skills.7,8 SPs have also been employed to teach residents in high-stakes scenarios such as delivering difficult news and patient behavioral emergencies.9,10

Early recognition of patients with clinical deterioration is an essential skill for IM residents. Medical learners must be able to identify severe illness and initiate appropriate lifesaving treatment in a timely manner. Such early recognition requires the integration and synthesis of multiple patient factors, including vital signs, symptoms, comorbid conditions, and response to treatment. Moreover, IM residents must be ready to treat catastrophic medical events, some of which they read about, but may never physically encounter during the course of their residency training. To achieve this goal, SBT is a safe and effective method for teaching crisis resource management for a variety of medical and surgical learners, as well as practicing physicians and other medical personnel.11–13 Regular participation in simulated deteriorating patients’ scenarios translates into earlier recognition and treatment of such emergencies in the hospital setting.14 Despite most residency programs utilizing both mannequins and SPs in training, studies in surgical and trauma situations have demonstrated significant differences in performance, competency, and realism perception when mannequins are compared with SPs in teaching crisis resource management.15,16 However, no differences between the two modalities were found when comparing communication, leadership, and cooperation outcomes.17 Most of these comparison studies focus on either procedural skills or communication and teamwork skills; there is a lack of material comparing simulation mannequins to SPs in teaching medical learners clinical reasoning skills in scenarios involving decompensating patients.

At our institution, the traditional IM curriculum in clinical reasoning and critical thinking skills focused on the learner interacting with a simulation mannequin. Feedback from this experience was mixed, with the learners often pointing out that the interaction with the mannequin was unrealistic. We hypothesized that using an SP instead of a mannequin would not only give a more realistic experience but would also improve medical knowledge acquisition. Therefore, the aim of this study was to compare residents’ performance in management of four scenarios depicting patient clinical deterioration utilizing either a high-fidelity simulation mannequin or SP.

Methods

Study design and participants

Our prospective, randomized cohort study was approved by the Mayo Clinic Institutional Review Board. The study took place at the J. Wayne and Delores Barr Weaver Simulation Center at Mayo Clinic in Jacksonville, FL, USA, between January 1, 2014 and June 30, 2014. A total of 19 IM residents in their second and third year of training participated in four decompensating patient simulation scenarios: calcium channel blocker overdose, severe sepsis, severe asthma exacerbation, and acute bacterial meningitis.

The residents were advised that their involvement in the study was optional; all 19 residents agreed to participate. All participating residents signed the Mayo Clinic Simulation Center informed consent form which gave consent both to be involved in simulation research and to be recorded on film if necessary. The residents were randomly assigned to either the mannequin group or the SP group. Residents in each group underwent the four simulation scenarios sequentially. All of the scenarios occurred within the same day or two concurrent days, and they were supervised by one attending physician experienced in SBT.

The deteriorating patient scenarios were developed using evidence-based practice and guidelines for simulation scenarios. Between January and March 2014, 9 of the 19 total residents participated in the four simulated scenarios utilizing a SimMan 3G mannequin (Laerdal Medical Ltd, Orpington, UK), as shown in Figure 1. Between April and June 2014, the other 10 residents participated in the same simulated scenarios under the supervision of the same attending physician utilizing SPs along with computer-simulated vital signs on a monitor. A total of three SPs were used for the four scenarios, with each scenario utilizing the same SP. Each SP received detailed training in the specific deteriorating patient scenario, pertinent physical examination findings, and answers to the potential residents’ questions.

Figure 1

A SimMan 3G Mannequin (Laerdal Medical Ltd, Orpington, UK) at the J. Wayne and Delores Barr Weaver Simulation Center at the Mayo Clinic in Jacksonville, FL, USA. The room is equipped with all the necessary supplies to simulate real-life scenarios to ensure optimum training.

Each resident completed the same five-question medical knowledge test immediately before and after each scenario. Supplementary material shows the questionnaire used for the calcium channel blocker toxicity scenario. In addition, each resident completed a form to rate realism of the simulation using a 5-point Likert scale questionnaire.

Outcome measures

Outcome measures for this study were the differences in pretest, post-test, and realism scores between the SP and mannequin groups.

Statistical analysis

A Mann–Whitney U-test was used to compare the two unevenly distributed groups. A standard t-test was performed on collected data of the different scenarios and on the realism score. A P-value of <0.05 was considered statistically significant.

Results

Eight (42%) of the 19 residents were female, 10 (54%) were in their second year of IM training, and 9 (46%) were in their third year. The mean (SD) age of the participants was 29 (4) years.

Residents in both groups (with all scenarios combined) had equivalent scores on the pretests with an average of 2.56/5 points (51%) in the mannequin group and 2.53/5 points (50%) in the SP group. Residents in the SP group scored higher on the post-test than those in the mannequin group (4.53/5 points [90%] vs 3.69/5 points [73%], P=0.01). The improvement of the SP group scores compared to the mannequin group reached statistical significance in all scenarios except for the acute bacterial meningitis scenario (Table 1). The highest score improvement in the SP group was in the calcium channel blocker overdose simulation with 2.4/5 points (48%) improvement from the original score while the mannequin group in the same scenario had 1.3/5 points (26%) improvement from the original score (P=0.03). Overall score improvement was 1.2/5 points (24%) in the mannequin group and 1.8/5 points (36%) in the SP group (P=0.01).

Table 1

Comparison of the test scores of the two simulation groups among the four clinical scenarios

Clinical scenario	Pretest scorea		Post-test scorea		Score improvementa		P-value
	Mannequin group	Standardized patient group	Mannequin group	Standardized patient group	Mannequin group	Standardized patient group
CCB overdose	2.22	2.2	3.56	4.6	1.3	2.4	0.03
Severe sepsis	2.11	2.4	3.33	4.4	1.2	2.0	0.04
Status asthmaticus	2.78	2.6	4.11	4.6	1.3	2.0	0.02
Acute bacterial meningitis	3.11	2.9	3.78	4.5	0.7	1.6	0.17
All scenarios combined	2.56	2.53	3.69	4.53	1.2	1.8	0.01

Notes:

Scores reported as average for 9 residents in the mannequin group and 10 residents in the standardized patient group. Full score (100%) is 5.

Abbreviation: CCB, calcium channel blocker.

The residents who experienced the deteriorating patient scenarios utilizing SPs also gave a higher realism score than those who underwent the mannequin scenarios (4.73/5 vs 2.00/5, P=0.002; Table 2 and Figure 2). The difference in the realism score was ranked highest in the severe sepsis scenario with 2.56/5 points in the mannequin group and 4.8/5 in the SP group (P=0.009).

Table 2

Realism score (reported by residents) among the four different clinical scenarios

Clinical scenario	Realism scorea		P-value
	Mannequin group	Standardized patient group
CCB overdose	2.11	4.70	0.001
Severe sepsis	2.56	4.80	0.009
Status asthmaticus	1.78	4.70	0.001
Acute bacterial meningitis	1.56	4.70	0.002
All scenarios combined	2.00	4.73	0.002

Notes:

Numbers reported as average for 9 residents in the mannequin group and 10 residents in the standardized patient group. Highest score is 5.

Abbreviation: CCB, calcium channel blocker.

Figure 2

Comparing pretest, posttest, and realism score in the four clinical simulation scenarios.

Notes: The simulation scenarios are (A) calcium channel blocker overdose; (B) severe sepsis; (C) severe asthma exacerbation; and (D) acute bacterial meningitis.

Abbreviations: SP, standardized patient.

Discussion

Our study results suggest that the employment of SPs is associated with better post-test scores and higher realism perception by IM residents. IM trainees had higher post-test scores when the simulation of decompensating patient scenarios utilized SPs rather than a mannequin. Our results are consistent with previously conducted studies on surgical SBT15,16,18 and highlight the role of SP in providing a more real-life experience within simulation training.

Although both the mannequin and SP groups demonstrated the improved understanding of decompensating patient management after the scenarios, residents who underwent SP-based simulation exhibited better knowledge acquisition reflected by their higher post-test score, in addition to receiving a more realistic experience than residents who underwent the mannequin simulation.

The development of contemporary teaching methods has mandated integration between real-time and simulation, or virtual patient training. With the progression of simulation techniques, the expectations of simulation training have increased to include almost all the residency training programs in the USA.1,17 Although most of the studies on simulation training come from trauma or surgical education programs, a few studies have included medicine trainees and nursing students.16,19 These studies provided contradicting results between SP and mannequin simulation. In one study of nursing students assessing clinically decompensating patients, SP simulation was not superior to the mannequin simulation in terms of clinical performance.20 However, both simulation strategies were equally effective in enhancing the clinical experience for the trainees.

An important merit of our study is that it was conducted on medical residents. To our knowledge, this is one of the first studies to compare SP and mannequin simulation for medical trainees, particularly IM residents. In a previous study, trainees rated a simulation mannequin as being very realistic to a patient scenario.21 However, the realism of a mannequin was not compared to that of SP. It is logical that IM residents responded more effectively to live patients rather than to a mannequin because of the interactive nature of human beings. Such interaction mimics the real-life physician–patient experience in a more realistic manner. On the basis of these results, the use of SPs in our residency program was determined to be a more effective and realistic teaching tool than the use of a mannequin in simulated decompensating patient scenarios. We initiated a focused simulation curriculum for IM residents where SPs replaced mannequin use as the main training resource.

Another important factor to consider when discussing the use of a high-fidelity simulation mannequin versus an SP is the monetary advantage. High-fidelity simulators such as our SimMan 3G, along with all the support software and optional packages, can cost over $70,000 per device. Using professional SPs can be much more cost effective and medical and academic staff can serve as SPs at no cost to the institution. We encourage institutions to review their curriculum first before purchasing any simulation equipment. If the goals of the curriculum can be achieved through the use of inexpensive equipment or staff on hand, then using those means to achieve curricular goals and using simulation funds elsewhere is recommended. In our study, our IM residents appeared to have a better-learning experience with the less-expensive resource.

Our study has several limitations. The study was conducted at a single academic medical center and may not be applicable to different simulation settings. In addition, the sample size is relatively small (19 residents) due to the small capacity of our residency program. Another limitation was that the study groups were assessed in series, the SP group 3 months after the mannequin group, instead of having both groups participating in parallel. This limitation would add the risk of the mannequin group passing on the content of the scenarios to the SP group who participated later in the year. We tried to eliminate this bias by asking all residents participating to keep the scenario content to themselves until the data collection was complete. Residents were assessed using an objective post-test rather than a checklist of actions that showed mastery in recognizing deteriorating patients and initiating appropriate clinical interventions. Also, the scope of this study did not extend to the evaluation of potential cognitive skills transference gained by the SP group to actual patient care. Moreover, when residents took a pretest, they became aware of the clinical scenario rather than independently determining the decompensating patient’s diagnosis. The scenarios used in this study are critical to IM training and may not apply to procedural or surgical specialties. However, the results of this study may be applicable to nonprocedural crisis resource management scenarios in other specialties. This study has similar results to the suggested guidelines for writing technical reports for simulation in education for health professionals that supported the use of SPs as an effective teaching tool in medical education.21

Conclusion

Our study results indicate that utilizing SPs was more effective and better mimicked realistic patient care than the use of a mannequin in acutely decompensating patient scenarios for IM residents. Our study only focused on level 1 of the Kirkpatrick model of training (reaction) and only looked at immediate retention of information, not long-term learning and retention. Future research in this field is needed as there are only a few studies comparing the use of mannequin to SPs in graduate medical education. These studies could evaluate both long-term knowledge retention and patient quality outcomes to see if simulation training with SPs enhances health care education. Such studies could shift the focus of medical education in the future.

Supplementary material

A: Pretest/post-tests

Date _________________    PGY Year_________________

Presimulation or postsimulation? _________________

What is the initial antidote for calcium channel blocker overdose?

Atropine

IV steroids

IV calcium

IV metoprolol

Which of the following medications overdoses would NOT present with bradycardia and hypotension?

Clonidine

Digoxin

Amlodipine

Verapamil

Calcium channel blocker overdose results in what changes to serum blood glucose?

Hyperglycemia

Hypoglycemia

Euglycemia

No change

Which of the following is NOT a current recommended therapy for calcium channel blocker overdose?

High-dose insulin therapy

Lipid emulsion therapy

Aggressive IVFs to initially correct hypotension

Activated charcoal administration

Can a calcium channel blocker overdose be corrected with hemodialysis?

Yes, all drugs can be removed with hemodialysis

Yes, because the drug is not lipid bound

No, because of the activated coagulation cascade

No, because the drug is lipid bound

Date _________________    PGY Year_________________

Realism score

How realistic did this simulation feel to you (how likely were you to suspend disbelief?

1	2	3	4	5
Very fake (Did not seem real at all)	Not realistic (Minimal parts felt real)	Realistic (> 50% of the simulation felt real)	Very realistic (Almost all the simulation felt real)	Real (I could not tell the difference from reallife)