Impact of standardized patients on the training of medical students to manage emergencies.

BACKGROUND: Teaching emergency management should educate medical students not only for facts and treatment algorithms but also for time effective physical examination, technical skills, and team interaction. We tested the hypothesis, that using standardized emergency patients would be more effective in transmitting knowledge and skills compared with a more traditional teaching approach.
METHODS: Medical students (n = 242) in their fourth (second clinical) year were randomized to receive either training on standardized patients simulating 3 emergency settings ("acute chest pain," "stroke," and "acute dyspnea/asthma") or traditional small group seminars. Before and after the respective training pathways, the students' knowledge base (multiple-choice examination) and practical performance (objective structured clinical examination using 3 different emergency scenarios) were assessed.
RESULTS: Teaching using standardized patients resulted in a significant albeit small improvement in objective structured clinical examination scores (61.2 ± 3 for the standardized patient trained group vs 60.3 ± 3.5 for the traditional seminar group; P = 0.017, maximum achievable score: 66), but no difference in the written examination scores (27.4 ± 2.4 vs 27.0 ± 4.4; P = 0.341; maximum achievable score: 30).
CONCLUSION: Teaching management of emergencies using standardized patients can improve medical students' performance in clinical tests, and a change from traditional seminars in favor of practice sessions with standardized patients does not compromise the learning of medical facts.

RCT Entities:   Population Interventions Outcomes

Introduction

Teaching management of medical emergencies to medical students and residents alike should consist of improving factual knowledge, and also practical skills such as quick patient assessment, manual skills, and overall care.

Many techniques can be taught on simulators, including airway management and endotracheal intubation,[ cardiopulmonary resuscitation,[ or vascular access. However, even high-fidelity simulators, although useful to teach procedural skills[ such as endotracheal intubation and more complex scenarios,[ do not provide physician–patient interaction and probably do not improve communication skills. Accordingly, standardized patients, sometimes referred to as simulated patients, are increasingly used in the education of students and residents,[ and, albeit very costly compared with traditional seminars, this may improve trainee communication skills[ and teaching in various medical fields.[

However, the effect on clinical performance when using standardized patients has not been studied to a great extent, mostly using observational studies or self-report satisfaction questionnaires.[ Few controlled trials assess the impact of simulation in teaching of medical emergencies,[ and randomized controlled trials are lacking.

With tightening budgets of medical schools and limited physician time, money should only be spent on teaching techniques that have proven to be effective.

Thus, our study was designed to assess whether medical students’ performance is altered after being trained either on standardized patients or using traditional seminars. Two outcomes were analyzed:

Performance in a written multiple choice test to assess factual knowledge

Performance in an objective structured clinical examination (OSCE) to assess technical and communicative skills.

Additionally, students were asked after the course how well they felt prepared to handle a similar emergency in the future (perceived preparedness).

Material and methods

Study cohort and interventions

After approval by the local ethics committee of the medical faculty (Ethikkommission der Medizinischen Fakultät, Robert Koch-Str. 9–11; 45147 Essen, Germany), its faculty board committee on teaching, and the students‘ informed written consent, 2 consecutive classes of medical students were included. The participants were fourth (second clinical) year medical students (n = 274) and scheduled to attend a 2-week course in emergency medicine. Three emergency scenarios were assessed: acute chest pain, stroke, and acute dyspnea/asthma. The students completed a questionnaire regarding potential confounders. Biographic data, previous medical training, and, in particular, the amount of previous exposure/training to treat medical emergencies (ie, as an EMT or paramedic) were recorded for all participants.

Participants were randomized via a computer-generated random list to receive either three training sessions (90 minutes each) on standardized patients, or 3 traditional seminars of equal duration, each covering the 3 scenarios. For the purposes of randomization, each participant was assigned a number. The computer randomly assigned a form of training (simulation or seminar) to each participant. The order of cases was also assigned at random using computer-generated random numbers (www.randomizer.org).

Standardized patients were professional actors recruited and trained by medical faculty for each scenario to represent a standardized patient. The traditional seminars were taught using standardized educational materials (presentation, script for the lecturer).

The 2 emergency medical technicians (EMTs) assisting the student team leader being examined were also standardized and portrayed by trained employees from our Department of Anesthesiology and Intensive Care Medicine. The assistants were unaware of the students’ prior study allocation and history, and they were instructed both to be cooperative and to perform the routine tasks of a nurse or EMT in an emergency setting.

Staff members scoring the students‘ written knowledgebase tests, the standardized patients in the OSCE, and examiners scoring the OSCE were all unaware of the students‘ allocation to the traditional or the standardized patients learning pathways. All OSCE examinations were scored by the same staff members.

Measurements

The effects of the training method were assessed by comparing between the 2 groups the performance in a written examination and a clinical examination at the completion of the course. After the course, students were examined regarding their factual knowledge base using a written examination composed of 30 multiple-choice questions. The multiple-choice questions had been used in previous other cohorts and have successfully been validated. Various techniques had been used for validation. For instance, inverse receiver operator characteristics were calculated for every question analyzing the ability of a specific question to discriminate students with a better overall performance from those with worse results. Clinical and communication skills had been evaluated in 3 previous OSCEs[ for each of the 3 respective scenarios taught.

The 3 OSCE cases were scored using case-specific score sheets, and a maximum of 22 points was achievable for each case. Thus, the maximum score was 66 for the 3 OSCE stations and 30 for the written test. The OSCE score sheet included clinical tasks, and also communicative aspects, and the performance of each student functioning as the leader of a medical emergency team consisting of 3 providers was examined. As a performance measure, unrelated to the 3 scenarios being taught, the participants’ performance in a resuscitation scenario (ACLS) using a high-fidelity simulator (METIman, CAE, St. Laurent, Quebec, Canada) was also studied. All students received identical training in ACLS before this examination. There was 1 evaluator for each OSCE case; thus the same raters scored each student. A measurement of inter-rater reliability was unnecessary.

The study design is depicted in Fig. 1 and an exemplary score sheet for 1 of the scenarios is shown in Fig. 2.

Figure 1

Flow chart of study.

Figure 2

Sample OSCE-scoring sheet for chest pain scenario (English translation). OSCE = objective structured clinical examination.

For both the traditional teaching and the standardized patients’ pathways, students were also asked how well they felt prepared to handle such a patient after the lesson (perceived competence). A numerical rating scale was used and graded from 1 (“I feel much better prepared”) to 5 (“I feel much worse”).

Statistical analyses

Results are depicted as means ± standard deviation (SD) unless otherwise indicated. SPSS statistical software (IBM, Armonk, NY), Stata (StataCorpLP, College Station, TX), and R (R Core Team, www.r-project.org) were used for statistical computations.

Since we are interested in a change evoked by training, a strong effect of the training on the results in the written examination was looked for. Therefore, an a priori power analysis was performed assuming a Cohen effect size d[ of 0.5, an alpha error of 0.05, and a power of 0.95. One hundred five participants were required in each study arm. Since a single semester cohort would not have yielded an adequate sample size, the study was conducted over a period of 2 semesters.

A Shapiro-Wilk test was applied to test for normal distribution, and a Student t test for unpaired samples was used to compare mean values of normally distributed variables between study cohorts. The results of the questionnaire testing the preparedness of the students were compared between the conventional group and the group being trained on simulated patients using the Kruskal-Wallis test.

An a priori alpha error P of less than 0.05 was considered to be statistically significant.

Results

In all, 274 students were initially enrolled and randomly allocated to either the traditional seminar or standardized patient teaching pathway, and 242 students completed the course and their examination scores, and data were subsequently analyzed (Fig. 1). Thirty-two students were not included in the data analysis because they did not participate fully in the course and missed training sessions due to sickness (30) or dropout.[

The incidence of the students‘ previous medical experiences, for example, training as a nurse or paramedic, internships in emergency medicine, or dedicated courses in emergency medicine, were not different among cohorts and neither were the students’ age, sex, or years of enrolment in medical school. There were no differences between groups in these aspects and thus they cannot be regarded as potential confounders.

OSCE

For the 3 OSCE scenarios with standardized patients, the students taught with traditional seminars scored an average of 60.3 points ± 3.5 (±SD), whereas students of the standardized patient cohort scored 61.2 points ± 3 (t = −2.140 [Student t test; 1-tailed, unequal variances], d.f. = 221, difference = −0.913; P = 0.017, Cohen d: −0.279; Fig. 3).

Figure 3

Results for OSCE and written examination. Data derived from the assessment of 242 medical students randomly allocated to receive training by either traditional seminars (conventional teaching) (n = 113) or simulated patients (n = 129). (∗) P = 0.017; NS P = 0.341 (Student t test for unpaired samples). OSCE = objective structured clinical examination.

In contrast, the students’ performance in the ACLS scenario, that is, a scenario unrelated to the test scenarios, did not differ between cohorts (standardized patient cohort: points 15 ± 1.3 vs traditional cohort: 15.2 points ± 1.1; P = 0.253).

Knowledge (written examination scores)

The traditional seminar cohort's average was 27.0 points ± 4.4 and the students taught on standardized patients scored an average of 27.4 ± 2.4 points (t = −0.955 [Student t test; 2-tailed, equal variances], d.f. = 240, difference = −0.427; P = 0.341, Cohen d: −0.123; Fig. 3).

Students’ perceived competence (self-assessment)

When asked (numerical rating scale) how well the students felt to be prepared for handling a particular emergency after having received instructions, those taught using standardized patients for the stroke scenario felt better prepared (Kruskal-Wallis; P < 0.0001). However, there was no significant difference between subcohorts for chest pain or acute dyspnea/asthma scenarios (Kruskal-Wallis; P = 0.067 and P = 0.899, respectively).

Discussion

Students being taught on standardized patients demonstrated a small but statistically significant benefit in clinical testing (OSCE) while not showing a disadvantage in medical knowledge when compared with their fellow students taught with traditional seminars. Whereas statistically significant, the better testing results probably do not reflect a relevantly better performance. The effect size is small.

Whereas this difference is small in absolute size, it still seems remarkable in different ways.

First, the standardized patient group lacked a traditional seminar without an inferior performance in the written examination testing factual knowledge. Thus, at least in our setting, replacing traditional teaching with standardized patients does not compromise the acquisition of factual knowledge. However, teaching using standardized patients requires additional resources consuming more human resources and time, for example, instructors, elaborate preparations including recruitment of actors, training the actors, preparing the scenario for each participant including applying moulage to the actors and preparation of medical equipment, and providing instruction and feedback to the participants. The apparently small benefit resulting from teaching using standardized patients may or may not justify these investments.

Students were asked after each course about their perceived preparedness in handling a patient showing the respective emergency syndromes. Whereas there were no differences between cohorts for the acute chest pain or acute dyspnea scenarios, the group being taught on the stroke scenario using standardized patients felt significantly better prepared. The reason for this difference cannot be pinpointed by our study. However, one may speculate that recognition and assessment of stroke involves more patient interaction and physical examination, whereas assessment of acute chest pain or dyspnea scenarios follows a rather rigid algorithm requiring execution of predetermined tasks. Thus, teaching using standardized patients may be more effective in some scenarios than in others. No studies have addressed these issues of medical education so far.

Limitations

Some limitations of this study should be discussed. Our faculty has strict requirements on the specifics and design of OSCE, and case scenarios must not be longer than 6 minutes. Accordingly, a longer physician–patient interaction might be required to demonstrate even better behavioral skills. Three defined scenarios with the trained standardized patients presenting very specific symptoms were used. Thus, only little patient history taking was required to arrive at a diagnosis. Accordingly, the selected scenarios and the assessment of teaching results might not fully explore the skills acquired by the students by teaching with standardized patients.

We did not perform any testing before the course. Thus, we are unable to report a gain in factual knowledge or skills. Our written examinations are not validated to being repeated with different questions for comparison of results, and repeating the OSCE examination, for the purpose of this study, would have created an undesirable training effect.

Furthermore, we tested a specific cohort of medical students in an advanced stage of medical training. Potentially, teaching using simulated patients may yield different results in other cohorts, that is, younger medical students, interns, and residents. However, to address effects in all these groups was beyond the scope of our study.

In conclusion, teaching handling medical emergencies using standardized patients slightly but significantly improved medical students’ performance in a structured clinical test compared with a traditional seminar cohort, without compromising factual knowledge. Whether this small improvement in student performance is quantitatively meaningful, given tight budgets and a lot of additional resources required must be decided individually.