High-fidelity simulation versus case-based discussion for teaching medical students in Brazil about pediatric emergencies.

OBJECTIVE: To compare high-fidelity simulation with case-based discussion for teaching medical students about pediatric emergencies, as assessed by a knowledge post-test, a knowledge retention test and a survey of satisfaction with the method.
METHODS: This was a non-randomized controlled study using a crossover design for the methods, as well as multiple-choice questionnaire tests and a satisfaction survey. Final-year medical students were allocated into two groups: group 1 participated in an anaphylaxis simulation and a discussion of a supraventricular tachycardia case, and conversely, group 2 participated in a discussion of an anaphylaxis case and a supraventricular tachycardia simulation. Students were tested on each theme at the end of their rotation (post-test) and 4-6 months later (retention test).
RESULTS: Most students (108, or 66.3%) completed all of the tests. The mean scores for simulation versus case-based discussion were respectively 43.6% versus 46.6% for the anaphylaxis pre-test (p=0.42), 63.5% versus 67.8% for the post-test (p=0.13) and 61.5% versus 65.5% for the retention test (p=0.19). Additionally, the mean scores were respectively 33.9% versus 31.6% for the supraventricular tachycardia pre-test (p=0.44), 42.5% versus 47.7% for the post-test (p=0.09) and 41.5% versus 39.5% for the retention test (p=0.47). For both themes, there was improvement between the pre-test and the post-test (p<0.05), and no significant difference was observed between the post-test and the retention test (p>0.05). Moreover, the satisfaction survey revealed a preference for simulation (p<0.001).
CONCLUSION: As a single intervention, simulation is not significantly different from case-based discussion in terms of acquisition and retention of knowledge but is superior in terms of student satisfaction.

INTRODUCTION

Medical education has traditionally relied on apprenticeship, in which students train with real patients in actual clinical settings. This strategy incorporates patient-level discussions at the bedside, often referred to as “rounds,” which are a form of case-based learning that is vital for patient care. Case-based learning is a long-established pedagogic method that usually occurs via small group discussions of patient cases in healthcare. As an educational tool, case-based learning is highly appreciated by learners and instructors but is associated with inconclusive results regarding its effects on learning compared with other strategies 1. Still, case-based learning offers certain advantages over didactic learning, including reflective observation and the opportunity for active experimentation, both of which are key aspects of experiential learning 2.

Simulation is a technique used to replace or amplify real experiences using guided experiences that evoke or replicate substantial aspects of the real world in a fully interactive manner 3. Recent movement toward decreased orientation times, work-hour restrictions for residents, and an increased commitment to patient safety have limited the opportunities for patient encounters and thus constitute a strong argument for simulation 4–6. In fact, simulation has been increasingly used to boost the growth of learner knowledge, to provide controlled and safe practice opportunities and to shape the acquisition of skills 7. There are several types of simulators, ranging from very simple ones to highly advanced computer-controlled simulators, and the fidelity of the simulation depends on how close the simulation is to reality 8. Simulation is especially of use for pediatric emergencies because critical events are infrequent, yet patients have good outcomes if correctly managed 8.

Simulation has been shown to be an effective training method, with large effects on knowledge, skills and behavioral outcomes 9. Multiple studies have also compared non-simulation-based teaching methods with simulation-based medical education 10–14, revealing small to moderate positive effects for simulation 14. However, to date, there is insufficient evidence from prospective studies comparing simulation with case-based discussion, and whether simulation yields higher knowledge attainment and retention than case-based discussion does is unclear.

A vast proportion of the knowledge acquired during medical school is not retained for a long period of time, especially when this knowledge is not used in daily practice 15. For example, Yang et al. assessed knowledge retention after an advanced cardiac life support course (ACLS) and found a decay in skills and knowledge 6 months after the course. However, clinical experience had a positive impact on knowledge retention 16. Thus, an important strategy to increase knowledge retention is to increase clinical practice. However, the reduction of hospital hours for students, the rarity of certain events and the ethical imperative to preserve patient safety conflict with this need. Thus, active learning methods such as case-based discussion and high-fidelity simulation, used in parallel to patient rounds and emergencies, may complement clinical practice, leading to higher retention of knowledge.

The objective of the present study was to compare high-fidelity simulation with case-based discussion for teaching medical students about pediatric emergencies, as assessed by a knowledge post-test, a knowledge retention test and a survey of satisfaction with the method. Our hypothesis was that the same case of an infrequent type of high-acuity pediatric emergency presented as a case-based discussion or as a simulation would yield similarly immediate knowledge gain but that higher retention of knowledge and greater student satisfaction would be attained with the simulation-based learning method.

MATERIALS AND METHODS

We conducted a non-randomized controlled trial of simulation-based teaching compared with case-based discussion among medical students using a pre-test, a post-test and a retention test (4–6 months later) via multiple-choice questionnaires (MCQs). Two themes (anaphylaxis in a teenager and supraventricular tachycardia (SVT) in an infant) were crossed over in the intervention, producing two experimental cohorts: group 1 - students experiencing a high-fidelity simulation of anaphylaxis (SIM-ANA) followed by an SVT case-based discussion (CD-SVT) and group 2 - students experiencing a high-fidelity simulation of SVT (SIM-SVT) followed by an anaphylaxis case-based discussion (CD-ANA). After each activity, the students anonymously filled out a satisfaction questionnaire composed of 8 statements in 3 domains (level of realism, quality of instruction and overall satisfaction). The statements were specifically rated using a 5-point Likert scale, with scores ranging from 1–5, adapted from the scale described by Cheng et al. 17.

The voluntary intervention was offered to all 174 sixth-year (the final year in Brazilian medical education) medical students of the Faculdade de Medicina da Universidade de São Paulo (FMUSP-University of São Paulo Medical School) during their first-semester rotation clerkship in pediatrics (emergency department (ED) or pediatric ward) in 2012. Generally, six groups of 13–16 students each rotate every semester in each specialty. In the current study, as the groups were already pre-assigned by convenience prior to their rotation, they were not randomized. Student demographics were obtained, including information that the investigating team thought could alter performance.

Anaphylaxis and SVT were chosen as themes because they are typical of but not exclusive to pediatric emergency, rare, and potentially fatal, as well as because they require specific knowledge and swift action to be correctly managed 18–21. During their pediatric emergency rotation, students are assigned shifts in which they care for pediatric patients in the ED while directly supervised by attending physicians and faculty. The students also attend multiple lectures on diverse emergency themes and receive training in medical procedures, basic life support and pediatric advanced life support in a skills laboratory supervised by faculty. Neither theme used in the present project had been addressed in a specific lecture during the regular clerkship.

Learning objectives were defined for anaphylaxis and SVT, and one scenario for each theme was constructed. Table 1 summarizes the instructional design for both interventions, which was based on Chiniara's framework for instructional design in simulation 22 and Cook's levels of instructional design 23.

Table 1

Comparison of the instructional design for the two methods.

	Simulation	Case-based discussion
Level 1: Instructional medium	Simulation	Face-to-face
Level 2: Configuration/Simulation modality	Simulated clinical immersion	Small group (13–16 medical students) discussion
Level 3: Instructional method	Instructor-based learning simulation (led by a pediatric emergency physician)	Case discussion of anaphylaxis in a teenager or SVT in an infant* (led by a pediatric emergency physician)
Level 4: Presentation	Location: FMUSP skills laboratory, simulating the ED resuscitation area (equipment and medications provided), and debriefing room Fidelity: High-fidelity Type of simulator: SimMan® or SimBaby® (Laerdal Medical) Scenario: Anaphylaxis in a teenager or SVT in an infant* (10–15 min) Feedback: Video-assisted immediate debriefing (20 min) Team composition: 4–8 medical students assigned to different roles Images and ancillary exam results provided upon request*	Location: FMUSP skills laboratory debriefing room Duration: 45–60 min PowerPoint® with history, physical exam results, images and ancillary exam results, as needed during the discussion*

*Same scenario/case history and physical exam findings presented, same images and ancillary exam results provided

The pre-test was composed of 14 MCQs, with 7 for each theme. The post-test was composed of 20 MCQs, including the original 14 pre-test questions and 3 additional questions for each theme. Finally, the retention test was composed of 26 MCQs, including the 20 post-test questions and 3 additional questions for each theme. Based on the learning objectives, 30 questions were developed by pediatric emergency physicians who were blinded to the scenarios. These questions were then piloted by medical students in a pre-study phase, with assessment of face, content and construct validity, and the questions were finalized according to an item response analysis for reliability. The order of the questions was changed between tests, and the additional questions mentioned above were added to control for learning with the tests themselves 24,25. The participants received feedback on the test results only after the retention test.

Our study was conducted on a convenience sample of 174 sixth-year medical students. Preliminary data collected during instrument development with 20 students from the previous year's class showed a mean score of 7.7 on the retention test, with a standard deviation (SD) of 1.8. We calculated that if 50 students (approximately 60% of the students) participated in each arm, assuming a two-sided significance level of 0.05 and an SD of 2, we would have 90% power to detect a difference of 0.65 in our retention mean (a 5% improvement in scores).

Test results were compared using t-tests for equality of means of independent samples. The mean results were specifically compared between the groups for each test. The satisfaction survey results were analyzed for each statement using the paired-samples test. In addition, Pearson correlation analysis was performed on the demographic and educational characteristics and test results to assess possible differences between students. The models used for analysis were those present in Statistical Package for the Social Sciences (SPSS) v18.

Ethics

The institutional review board of Faculdade de Medicina da Universidade de São Paulo approved this study.

RESULTS

Of the 174 students, 163 (93.6%) agreed to participate, and 124 (71.2%) filled in the satisfaction survey for case-based discussion and high-fidelity simulation. In total, 108 (66.3%) students completed all steps of the study (high-fidelity simulation, case-based discussion, pre-test, post-test and retention test), including 43 in group 1 (SIM-ANA/CD-SVT) and 65 in group 2 (CD-ANA/SIM-SVT), as shown in Figure 1.

Figure 1

Student participation.

Table 2 shows the comparison of demographic and educational characteristics between the groups; the only educational differences were the mean performance in ED clerkship and a previous adult ED rotation.

Table 2

Demographic and educational characteristics.

Characteristic	Group 1 (n=43)	Group 2 (n=65)
Intended medical specialty
Internal medicine or pediatrics	18.6%		12.3%
Surgery	20.9%		12.3%
Other	27.9%		43.1%
Did not know/Did not answer	32.6%		32.3%
Gender
Male	51.2%		67.7%
Age
≤25 years	76.7%		70.8%
26–29 years	20.9%		24.6%
≥30 years	2.4%		4.6%
Advanced life support courses completed
None	88.4%		78.5%
1 or more	11.6%		21.5%
Grade - ED clerkship rotation
Mean (0–10)	8.5		8.8*
Grade - Ward clerkship rotation
Mean (0–10)	8.7		8.7
Previous rotation - Adult ED clerkship	0	28*

*paired t-test, p<0.05, 95% CI

Knowledge improvement from the pre-test to the post-test was observed for both teaching methods across the two themes (p<0.05). However, there were no significant differences in knowledge at the post-test or retention test between the teaching methods for either theme (Figure 2).

Figure 2

Pretest, post-test and retention test results (mean scores) for each theme among the medical students, stratified by the teaching method.

Pearson correlation analysis found no correlation between baseline demographics or educational characteristics and test results for most characteristics; the only characteristic that influenced the results was the order of clerkship rotations. Group 2 included 28 students who had completed their adult ED clerkship rotation (comprising internal medicine, trauma, cardiology and neurology) prior to their pediatric rotation. This subgroup had higher mean pre-test scores for SVT (38.7%) than group 1 (31.6%; p=0.06) and the rest of group 2 (30.1%; p=0.02) did. This difference was not demonstrated at the post-test, with a score of 41.8% for this subgroup compared with 47.7% for group 1 (p=0.16) and 43% for the rest of group 2 (p=0.72), but was again present at the retention test for SVT, with a score of 46.5% for this subgroup compared with 39.5% for group 1 (p=0.04) and 37.8% for the rest of group 2 (p=0.01). In contrast, no significant differences were found between this subgroup and the other students for the anaphylaxis theme.

A high degree of satisfaction was reached with both methods (Figure 3). However, simulation was perceived as more realistic, a more positive experience and more useful to the profession. The vast majority believed that simulation should be mandatory, with a mean score of 4.9 (out of 5). Although the same person facilitated both activities, instructor knowledge and enthusiasm were rated higher for simulation. The greatest mean difference in ratings was for the statement “this activity got my heart pumping.”

Figure 3

Satisfaction survey for each teaching method (*p<0.001).

Two statements were not significantly different between the methods: one was related to ancillary exams and images, which participants found to be equally helpful in both methods (p=0.11), and the other was related to the learning environment, which was considered to be non-threatening for both methods (p=0.47).

DISCUSSION

In this population of medical students, we found no significant differences in immediate knowledge gain or retention when comparing high-fidelity simulation with case-based discussion. In particular, both methods showed improved post-test scores for both themes (anaphylaxis and SVT), and these scores were maintained over time. However, satisfaction was higher with the simulation method.

In a systematic review and meta-analysis of simulation studies, Cook et al. reported a small to moderate effect of simulation compared with other instruction methods 14. In our study, each group experienced one high-fidelity simulation and one case-based discussion, and as a single intervention, case-based discussion did not appear to be more effective. Similar results were shown when a one-time simulation-based workshop was compared with a case-based workshop for continuing medical education for internal medicine faculty and resident physicians 26. However, integrating multiple simulations into a curriculum appears to have a measurable effect on knowledge gain 10. Due to the nature of active learning, in the current study, the scenarios were slightly more demanding of the students' previous knowledge than most of the activities in the regular curriculum, which could have altered retention. The highest scores in our study were in the 60% range, so there was definite room for improvement among the students on the topics of both SVT and anaphylaxis. If multiple, similar simulations and/or case-based discussions had been utilized, the students' knowledge might have increased, and at a certain point, a difference might have been shown between the methods. This concept was demonstrated in a cohort of third-year medical students allocated to simulation of septic shock and case-based discussion of cardiogenic shock or vice versa, in which the students demonstrated superior understanding of shock after a simulation experience, regardless of the type of shock discussed 27. Additionally, in a week-long acute-care course, fourth-year medical students randomized to simulation performed better than students randomized to problem-based learning did, as measured by performance checklists 28.

Satisfaction was high for both methods but was significantly higher for simulation. These results were not unexpected, as satisfaction with simulation has been consistently shown 29. In the present study, simulation was rated as being closer to reality and as causing an increased heart rate while remaining a non-threatening learning experience. This is exactly the response that one would hope for within simulation-based training. Adult learning theory states that effective practice-based learning occurs when individuals relate their learning experience to their actual practice and when the experience causes a significant change in their body state to promote reflection, without being so stressful as to impede learning 4. Similar results were shown in a month-long crossover study of curriculum format, in which fourth-year medical students felt that simulation was more stressful but also more enjoyable, more stimulating and closer to the actual setting than group discussions were 10. As the average student entering medical school becomes more tech savvy, it will be imperative to align educational formats with students' learning styles and preferences.

Although not formally measured by this study, the faculty were clearly enthusiastic about the active learning methods used in this project, resulting in plans to incorporate both simulations and case-based discussions into the regular curriculum. However, as our results show similar effectiveness for both methods in terms of knowledge acquisition and retention, one must consider the significantly higher cost and work associated with simulation when planning a learning activity. A reasonable approach would be to use simulation only when the learning objectives include technical and non-technical skills and behaviors, whereas case-based discussion should be prioritized when the objectives are mainly knowledge based.

In this study, in the subgroup with a previous clerkship rotation in the adult ED, higher pre-test scores and knowledge retention were found for SVT. However, it is unclear whether this result was an effect of non-randomization or if the previous exposure to emergencies allowed the group to relate the simulation to their experience to better retain knowledge, as clinical experience has been linked to better knowledge retention and improved skills after simulation 16.

We used the same instructor for simulation and case-based discussion, which can be viewed as a limitation. We addressed this limitation by having the exact same scenario presented. We also added one item to the satisfaction survey to evaluate the instructor's enthusiasm and knowledge. Although there was a difference in this item, with a better rating for simulation (4.86) than for case-based discussions (4.72), the rating was high in both instances. Furthermore, having the same instructor makes comparing interventions, rather than facilitators, possible.

This study has certain limitations. First, we were not able to randomize our students. However, as the baseline demographics were similar between the groups, we do not feel that significant bias was introduced by the group allocation. Second, due to the voluntary nature of this study and the need to measure retention after 4–6 months, we had a high (33.7%) loss to follow-up, which potentially introduced bias and might have influenced the findings. This possible issue was minimized by the fact that we still analyzed a large number of subjects that was above the calculated sample size. Third, the results are representative of a small cohort within a single medical school and thus cannot be safely validated externally. Fourth, there is a limitation regarding the strength of the intervention. The number of hours of simulation has been specifically linked to learning gains 30, so as discussed above, a one-time high-fidelity simulation and a one-time case-based discussion might not be enough to generate a difference in knowledge, whereas a series of simulations and case-based discussions potentially could do so. Finally, this study did not address clinical performance (Kirkpatrick level 4) or team-based skills (level 2b) and behaviors (level 3) 31. However, as medical students are not the sole healthcare providers responsible for patients and management decisions are routinely vetted by more senior physicians, it was unreasonable to assess clinical outcomes in this study. In any case, future studies would benefit from measuring team-based skills and behaviors, which are areas in which we would expect higher gains with simulation 32.

Reflecting on our results and study limitations, an appropriate next step would be to integrate multiple simulations and case-based discussions into our pediatric emergency clerkship curriculum and to prospectively evaluate groups randomized to each method regarding satisfaction, knowledge gains and both technical and non-technical skills and behaviors.

In conclusion, as a single teaching intervention, simulation is not significantly different from case-based discussion in terms of acquisition and retention of knowledge. However, greater satisfaction was found for simulation, showing a measurable benefit of simulation compared with case-based discussion.