Effectiveness of simulation-based training for obstetric internal medicine: Impact of cognitive load and emotions on knowledge acquisition and retention.

BACKGROUND: Simulation-based training's impact on learning outcomes may be related to cognitive load or emotions during training. We evaluated the association of validated measures of cognitive load and emotion with learning outcomes in simulation-based obstetric internal medicine cases.
METHODS: All internal medicine learners (n = 15) who completed the knowledge test pre-training, post-training (knowledge acquisition), and at 3-6 months (knowledge retention) for all three simulation cases were included.
RESULTS: Mean knowledge scores differed over time in all three cases (p < 0.0001 for all). Knowledge retention scores were significantly higher only for cases 1 and 3. Cognitive load associated with frustration was positively associated with knowledge acquisition for case 2 (beta = 5.18, P = 0.007), while excitement was negatively associated with knowledge retention in case 1 (beta = -33.07, p = 0.04).
CONCLUSION: Simulation-based education for obstetric internal medicine can be effective in select cases. Attention to cognitive load and emotion may optimize learning outcomes.

Background

According to the Royal College of Physicians and Surgeons of Canada, the management of medically complex pregnant women is considered a core competency for both internal medicine and general internal medicine trainees.[1,2] However, a survey of graduates from General Internal Medicine training programs across Canada demonstrated that trainees felt unprepared in the management of medical disorders in pregnancy. This is not surprising given the complexity of obstetric medicine cases, the risk of severe maternal and fetal morbidity and mortality, and limited exposure to obstetric medicine cases throughout training. Simulation may be an effective educational tool to provide trainees with exposure to medically complex obstetric cases not routinely encountered during training to assist in developing competency in obstetric medicine.

Simulation education is widely used for training healthcare professionals and has shown a moderate positive educational effect size when compared to other forms of instruction in healthcare.[4-6] It is an effective method to teach skills in stressful scenarios such as advanced cardiac life support (ACLS) and resuscitation.[6,7] However, the evidence for using simulation in obstetric medicine is limited. We previously conducted a randomized control trial evaluating the use of scenario-based simulation in an obstetric medicine case. Our study did not demonstrate improved knowledge acquisition and retention compared to conventional case-based instruction and even demonstrated a non-statistically significant decrement in knowledge acquisition and retention. We hypothesized that the absence of a positive learning outcome may have been due to high anxiety and cognitive overload experienced during the obstetric simulation. Indeed, simulation-based training has been shown to increase stress levels compared to conventional tutorial-based training, and the impact of stress on attention, decision making, and memory may affect learning and performance in unpredictable ways. The simulated environment requires the learner to use their knowledge and skills in real time, respond to abrupt changes in the status of the standardized patient/mannequin, communicate with other health care professionals, and utilize crisis resource management skills, all of which may serve to impair learning. However, existing data examining emotion and cognitive load on knowledge and performance in simulation is conflicting. High cognitive load in simulation has been shown to be a potential barrier to learning and performance in some studies.[11-14] In contrast, an observational study examining trainees participating in an ACLS simulation found that added anxiety correlated with enhanced performance, whereas other studies have shown poor correlation between self-reported stress and performance.[16,17] In another study on medical students, stressors improved clinical performance.

To examine the relationship between cognitive load, emotion, and learning, our study seeks to evaluate the effectiveness of simulation-based training on learning outcomes across three different simulated obstetric medicine case presentations and to analyze the effect of cognitive load and emotion on knowledge acquisition and retention. Our study utilized the NASA Task Load Index (NASA-TLX) to measure cognitive load. This estimate of mental workload has been used in a variety of medical and medical education settings, including surgical procedures and learning,[20,21] resuscitation, endoscopic procedural training, and point-of-care ultrasound performance. Higher workload has previously been demonstrated to be associated with poorer performance. For learner emotions, we measured the dimensions of affect proposed by Feldman Barrett and Russel, whereby more negative emotions had previously been found to be associated with worse learning outcomes.

Methods

Participants

All postgraduate year (PGY)-2 and -3 internal medicine residents from the University of Calgary were invited to participate in this study between 2018 and 2020. This study was approved by the University of Calgary Conjoint Health Research Ethics Board.

Simulation scenarios

Between September 2018 and September 2019, each participant completed three simulation-based training sessions, in groups of 2–3 learners per group. Only participants who provided written informed consent were included in this study. Prior to the session, participants were provided with a five-minute orientation to the simulated environment and features of the computer-based mannequin (Laerdal SimMan® 3 G). Each simulation session was led by three simulation-trained instructors: one to operate the mannequin, one to act as the patient’s voice and consulting specialists, and one to act as the nurse confederate. Participants were instructed to evaluate the simulated patient as a team and each participant was encouraged to perform as a team leader for at least one simulation case. Each interactive case took 15–20 min. This is followed by a 15-min debrief,[27,28] whereby the first 5 min, facilitated by a simulation-trained instructor, were devoted to addressing learner reactions to the simulation case and/or environment. In the remaining 10 min, a didactic lecture was provided by an obstetric general internist, with discussion of learner performances and a summary of key teaching points (Supplementary Table 1). Content of the lecture covered key obstetric medicine learning objectives pertaining to the cases, based on the Royal College of Physicians and Surgeons of Canada Objectives of Training for the specialty of internal medicine.

The obstetric internal medicine simulation cases were created based on chart reviews from patients. Case 1 was on severe hypertension in pregnancy, case 2 was on thromboembolic disease in pregnancy, and case 3 involved cardiac arrest in pregnancy (Supplementary Table 2a–c). These cases were piloted in 2017 on PGY-4 and PGY-5 (n = 6) general internal medicine trainees to ensure optimal content and flow prior to starting the study. All learners were presented with these three cases in the same order.

Outcome assessment

Knowledge was assessed using a multiple-choice test administered before and after the simulation case and didactic session. Number of items on the multiple-choice test ranged from 8 to 12, depending on the topic assessed. The same multiple-choice test was then administered 3–6 months after completing the simulation case. Knowledge assessment scores are presented as the number of correct responses divided by the total number of questions, presented as a percent. Knowledge acquisition was defined as the difference in test scores between post- and pre-simulation. Knowledge retention was defined as the difference in test scores between 3 and 6 months post-simulation and pre-simulation. The multiple-choice tests were created by an expert panel consisting of two obstetric medicine specialists and one specialist with expertise in general internal medicine and medical education. The tests were constructed based on a blueprint to assess multiple cognitive domains including knowledge, comprehension, application, analysis, and synthesis of information pertaining to the obstetric medicine simulation case.

Self-reported cognitive load post-simulation was assessed using the NASA-TLX, a validated subjective assessment tool that rates perceived workload of a task. This tool measures cognitive load on six domains (mental demand, physical demand, temporal demand, performance, effort, and frustration) using a 100-point scale where 0 = very low and 100 = very high. A score of 60 or higher would be considered high. Self-reported emotion post-simulation was measured using a validated tool that requires participants to rate eight different emotions on a five-point Likert scale to describe their emotional state.[25,26] Change in self-reported comfort and confidence in the evaluation and management of medically unwell pregnant women was assessed utilizing a three-item questionnaire conducted before and after each simulation case. Comfort and confidence were rated using a five-point Likert scale, where 1= strongly disagree and 5 = strongly agree with the statements regarding being comfortable or confident.

Statistical analysis

Differences in scores over the three time points were evaluated using repeated-measures ANOVA, with post hoc Tukey analyses corrected for multiple comparisons. Comparisons of measures pre- and post-session were performed using paired t-tests, while comparisons of measures across cases were performed using ANOVA testing. The association of baseline variables, cognitive load, and emotions with knowledge acquisition and knowledge retention was assessed using linear regression analyses. Variables that were significant at a univariate level (P < 0.05) were entered into a multivariate model whereby all domains of that variable (cognitive load or emotions) were included, in order to evaluate for the independent association of that variable. All analyses were performed using SAS version 9.2 (SAS Institute Inc., Cary, NC).

Results

Twenty-one trainees consented to participate and 15 (71%) completed the study protocol. Of the 15 trainees, 60% were PGY-2 (n = 9), 40% were PGY-3 (n = 6), 53% were male (n = 8), and 47% were female (n = 7). None of the participants had previously completed a rotation in obstetric internal medicine.

Mean knowledge assessment scores over the three time points were significantly different in all three cases (pre-test 44.0 ± 19.2, post-test 78.0 ± 10.1, 3–6 months 59.3 ± 14.9, P < 0.001 for case 1; pre-test 54.0 ± 19.6, post-test 84.2 ± 13.9, 3–6 months 64.9 ± 15.1, P < 0.0001 for case 2; pre-test 51.9 ± 19.9, post-test 87.7 ± 12.4, 3–6 months 76.9 ± 9.3, P < 0.0001 for case 3, Figure 1). Post hoc analysis suggests that retention scores were significantly higher only for case 1 and case 3, but not for case 2. In case 2, only the post-test score was significantly higher than the pre-test score (P < 0.0001); retention score was not different from pre-score (P = 0.06).

Figure 1.

Mean knowledge assessment scores for three simulation cases, each conducted at three time points (pre-simulation, post-simulation, and at 3–6 months). Error bars indicate standard deviation. (a) Knowledge assessment scores for case one. (b) Knowledge assessment scores for case two. (c) Knowledge assessment scores for case three.

Cognitive load scores and self-reported emotion across the three cases are presented in Table 1. Cognitive load was significantly higher in case 3 than cases 1 and 2 for five of the six domains and emotions were more negative for case 3 in four out of eight emotions. Overall mean cognitive load was below 60 in all instances except mental demand, temporal demand, and effort in case 3.

Table 1.

NASA-TLX cognitive load scores and self-reported emotion post-simulation cases, presented as mean (standard deviation).

	Case 1	Case 2	Case 3	P-value
Cognitive load
Mental demand	55.0 (18.3)	56.0 (11.1)	79.5 (12.6)	<0.0001
Physical demand	14.6 (21.6)	21.2 (19.4)	58.5 (31.7)	<0.0001
Temporal demand	27.7 (17.8)	30.0 (21.3)	65.5 (18.7)	<0.0001
Performance	50.2 (20.3)	45.3 (20.9)	52.7 (13.4)	0.66
Effort	50.8 (16.5)	50.3 (18.5)	72.3 (17.3)	0.0004
Frustration	39.6 (19.8)	22.8 (14.7)	56.5 (23.0)	0.0002
Emotion
Tense/calm (1–5)	3.5 (0.9)	3.7 (0.9)	2.4 (1.1)	0.002
Nervous/relaxed	3.2 (1.0)	3.7 (0.9)	2.6 (1.2)	0.05
Stressed/serene	3.3 (0.9)	3.5 (0.7)	2.6 (1.1)	0.04
Upset/content	4.2 (0.7)	4.2 (0.7)	3.5 (0.7)	0.02
Sad/happy	3.9 (0.8)	4.1 (0.7)	3.6 (1.1)	0.32
Depressed/elated	3.8 (0.6)	3.8 (0.6)	3.4 (0.6)	0.049
Lethargic/excited	3.7 (0.6)	3.6 (0.6)	3.3 (1.1)	0.36
Bored/alert	4.2 (0.6)	4.1 (0.7)	4.3 (0.7)	0.54

On univariate analyses, knowledge acquisition was not associated with baseline variables (gender, training level), individual cognitive load domains or emotions (all P > 0.05) except for frustration (beta = 3.68, SE 1.43, P = 0.02) in case 2 (Supplementary Table 3). Adjusted for all cognitive load domains, frustration remained positively associated with knowledge acquisition (beta = 5.18, SE 1.44, P = 0.007, Supplementary Table 4).

Knowledge retention was not associated with gender, training level, or individual cognitive load domains, except for mental demand for case 3 (beta = 5.28, SE 1.73, P = 0.009, Supplementary Table 5), which after adjusting for the effects of other cognitive load domains, was no longer significant (beta = 6.51, SE 2.95, P = 0.06; Supplementary Table 6). Knowledge retention was not associated with individual emotion except for lethargy-excited for case 1 (beta = –17.97, SE 8.26, P = 0.048, Supplementary Table 5). After adjusting for the effects of the other emotions, lethargy-excited remained independently associated with knowledge retention in case 1 (beta = –33.07, SE 12.51, P = 0.04, Supplementary Table 6).

Mean self-reported comfort in participating and confidence in the evaluation and management of all three cases were statistically higher post-simulation compared with pre-simulation with the exception of comfort in participating in the care of medically complicated pregnant women after the cardiac arrest in pregnancy case (Table 2).

Table 2.

Mean self-reported comfort measures (± standard deviation) pre- and post-simulation based training, where 1 = strongly disagree and 5 = strongly agree.

	Pre	Post	P-value
I am comfortable participating in the care of medically complicated pregnant women
Case 1	1.9 (±0.8)	2.9 (±1.1)	0.0004
Case 2	2.3 (±0.5)	3.5 (±0.6)	<0.0001
Case 3	3.2 (±0.9)	3.7 (±0.5)	0.07
I am confident in my ability to appropriately evaluate a pregnant woman with medical complaints
Case 1	2.9 (±0.9)	4.1 (±0.3)	0.0003
Case 2	3.1 (±1.0)	4.0 (±0)	0.002
Case 3	2.9 (±1.1)	3.9 (±0.5)	0.004
I am confident in my ability to manage a pregnant woman with medical complaints
Case 1	2.5 (±0.9)	3.7 (±0.6)	0.0009
Case 2	2.5 (±0.6)	4.1 (±0.3)	<0.0001
Case 3	2.3 (±0.9)	3.5 (±0.8)	0.002

Conclusion

In this study, we designed three simulation-based training cases covering obstetric internal medicine topics. While knowledge acquisition was demonstrated immediately post-teaching in all three cases, knowledge retention was only significant in two out of three cases. Further, in all three cases, scores at 3–6 months were lower than scores immediately post-training, suggesting some degree of knowledge decrement over time post-training. We were unable to consistently demonstrate the impact of cognitive load and emotion on knowledge acquisition and knowledge retention, although a high cognitive load imposed by frustration appeared associated with a higher knowledge acquisition in case 2, while learners who reported higher excitement for case 1 had surprisingly lower knowledge retention. Lastly, learners reported a significant increase in comfort in participating in the care of medically complicated pregnant women after two out of three cases. They also reported an increase in their confidence in evaluating and managing these women after all three cases. Altogether, our results suggest the following: with careful attention to cognitive load and learner emotions during the design and execution of simulation-based training, significant knowledge retention can sometimes be demonstrated, but by no means a guaranteed learning outcome. While it is likely that cognitive load and emotion plays a role in knowledge acquisition and retention, the exact individual contribution by these domains remains unknown. No specific pattern of association can be identified, but our study was limited both in sample size (n = 15) and in scope (number of cases explored).

Our results demonstrating the association of higher frustration with increased learning is not novel. For example, in a study of math teaching, students experiencing greater frustration demonstrated higher learning gains. However, the relationship between frustration and learning is likely not straightforward. For example, others have found that while brief episodes of frustration were associated with improved learning, lengthy periods impaired learning. In addition, our results in case 1 showed that those who reported higher excitement had lower knowledge retention. While these results seem to be contradictory to the principles of active learning, they may nevertheless be a reflection of the complexity of the learning process.

In addition to the aforementioned sample size and scope limitations, our study has further limitations that must be considered in the interpretation of the results. First, because this study was conducted in a single center, generalizability may be limited. Our cases were solely obstetric medicine cases, to which the medical learners at our center have limited exposure. Moreover, our cases were designed by educational experts and executed by those with expertise in simulation, and cases were delivered in the same order each time to each learner (starting with severe hypertension and ending with the cardiac arrest case). Educational outcomes may differ if the educational context significantly differs from ours. Second, despite careful attention to issues pertaining to case design, the cognitive load in case 3 did exceed our intended maximum of 60 on half of the domains. Nonetheless, knowledge was significantly retained for this case, suggesting that learning may not have been significantly impaired as a result. Third, our cases were executed with small groups of learners rather than individual learners. The presence of other learners may alter the cognitive load and/or emotion perceived by individual learners in ways that are not always predictable. However, given the time and resource-intensiveness of simulation-based education, having two or three educators for every individual learner was simply not feasible in our educational setting. Fourth, our self-reported comfort and confidence measures were on the general evaluation and management of medically unwell pregnant women, rather than case-specific measures. Fifth, our learning outcomes involved only knowledge acquisition and retention. Logistical issues prevented the evaluation of higher-level outcomes such as learner performance. It is conceivable that cognitive load and emotion may affect learner performance differently than knowledge outcomes. Lastly, although our initial study intended to assess knowledge retention at three months, scheduling challenges resulted in delays such that knowledge in some cases was assessed as late as six months, thus modifying our learning outcome to the range of between three and six months.

Despite these weaknesses, our study has a number of strengths. First, our learning outcomes included the evaluation of knowledge retention rather than simply knowledge acquisition. Evaluation of knowledge acquisition only would have led to an over-estimation of the benefits of simulation-based training. Second, our cases represented a range of content difficulty as well as acuity in patient presentation. This allows us to have a broader look at the impact of cognitive load and emotion on learning outcomes in more than one setting. Our results are consistent with our suspicion that while cognitive load and emotions may play a role in learning, how they do so is likely context-specific. Thus, it remains prudent for educators to take these contributors into account when designing and executing training cases. Specifically, from a practical standpoint, educators should adhere to design principles and strategies that minimize cognitive overload for the learner. For example, the order of case introduction should begin with the simpler cases to those of increasing complexity. Second, cognitive load and emotions should be monitored and measured. In our cardiac arrest case, the mean cognitive load exceeded 60 in three of the six domains. In such instances, at a minimum, educators should ensure that learning outcomes are not negatively impacted. In the event that learning is impaired, case modification should occur or training environment be modified with cognitive load and learning outcomes re-measured. Alternatively, educators can prophylactically modify the training to ensure that the cognitive load is not high and compare learning outcomes before and after modification to clarify the impact of the high cognitive load.

In conclusion, our results suggest that simulation-based education for obstetric internal medicine can be effective in select cases. In the design and execution of simulation-based training, careful attention to cognitive load and emotion may help optimize learning outcomes.

Click here for additional data file.

Supplemental material, sj-pdf-1-obm-10.1177_1753495X211011915 for Effectiveness of simulation-based training for obstetric internal medicine: Impact of cognitive load and emotions on knowledge acquisition and retention by Kelle D Hurd, Meghan EO Vlasschaert, T Lee Ann Hawkins, Jolene Haws, Julia N Kupis and Irene WY Ma in Obstetric Medicine