Evaluation of cognitive load and emotional states during multidisciplinary critical care simulation sessions.

BACKGROUND: The simulation in critical care setting involves a heterogeneous group of participants with varied background and experience. Measuring the impacts of simulation on emotional state and cognitive load in this setting is not often performed. The feasibility of such measurement in the critical care setting needs further exploration.
METHODS: Medical and nursing staff with varying levels of experience from a tertiary intensive care unit participated in a standardised clinical simulation scenario. The emotional state of each participant was assessed before and after completion of the scenario using a validated eight-item scale containing bipolar oppositional descriptors of emotion. The cognitive load of each participant was assessed after the completion of the scenario using a validated subjective rating tool.
RESULTS: A total of 103 medical and nursing staff participated in the study. The participants felt more relaxed (-0.28±1.15 vs 0.14±1, P<0.005; d=0.39), excited (0.25±0.89 vs 0.55±0.92, P<0.005, d=0.35) and alert (0.85±0.87 vs 1.28±0.73, P<0.00001, d=0.54) following simulation. There was no difference in the mean scores for the remaining five items. The mean cognitive load for all participants was 6.67±1.41. There was no significant difference in the cognitive loads among medical staff versus nursing staff (6.61±2.3 vs 6.62±1.7; P>0.05).
CONCLUSION: A well-designed complex high fidelity critical care simulation scenario can be evaluated to identify the relative cognitive load of the participants' experience and their emotional state. The movement of learners emotionally from a more negative state to a positive state suggests that simulation can be an effective tool for improved knowledge transfer and offers more opportunity for dynamic thinking.

Introduction

Cognitive load theory provides a convenient framework for exploring the relationship between the design of simulation training sessions and learning outcomes.1 This theory assumes that working memory has limited capacity and that learning is impaired when an experience overloads its capacity to process and transfers knowledge to long-term memory.1–3 A scenario is a platform for education that, when complicated, may overload this capacity of working memory and risk knowledge transfer for the learner. Cognitive load is comprised of three components: intrinsic, extraneous and germane load. Intrinsic load is defined by the elements specific to a given task and reflects the inherent difficulty of that task. The work of processing the intrinsic load can be lessened by the learner’s prior experience with this task.4 Extraneous load is the working memory consumed during task completion that is imposed on learners by the structure of the activity that does not enhance learning.5 The educator can have the most direct influence on decreasing extraneous load through tight tailoring of the curriculum as well as considering the psychological safety of the learning environment.6 Germane load refers to the intentional cognitive effort from the working memory dedicated to learning the new task and transferring the knowledge from the short-term memory to the long-term memory.7 These components are additive, and learning is reduced when the total cognitive load exceeds the capacity of working memory. Thus, the optimal instructional design for a simulation is one that avoids cognitive overload, minimises extraneous load and maximises germane load.1–3

Psychology literature suggests that emotions during highly demanding activities impair cognitive processing efficiency.8 Similarly, the emotions experienced by participants, classified as an extraneous load, during simulation training may negatively affect knowledge transfer. Fraser et al applied a validated methodology for reporting emotions and identified that negative emotional experiences increased the cognitive load of simulation activities and resulted in reduced learning outcomes.9

Education in intensive care is evolving with more emphasis on multidisciplinary team-based learning involving complex tasks and human interactions. Simulation-based education in critical care promotes skills acquisition, aids development of clinical judgement and teaches learners about complex critical care situations with lifetime experiences. The functional task alignment,10 that is, alignment of functional properties of entire simulation context with the learning objectives provides an invaluable safety net for learning, as participants can practise in a setting where errors do not have consequence and in which they can have direct supervision with immediate feedback. Multidisciplinary simulation allows team training and helps to address ‘non-technical skills’ and human factors. Scenarios for these sessions can be tailored to reflect local case-mix and critical incidents and thus help to improve patient care.

The simulation experience significantly influences emotional state of the participants and potentially overwhelms their cognitive load. If participants have not been briefed properly prior to a scenario or the scenario is not well tailored to optimise knowledge transfer, there is risk of subjecting learners to undue emotional stress and excessive extraneous load on their working memory.11 Paas and Van Merriënboer have created a tool to evaluate the relative load on working memory of an educational experience. This tool ranges from 1 (very, very small effort) to 9 (very, very high effort).11 Applying this tool, Fraser et al found that performance declines at a load of 7 or more.9 As an educator, it is imperative to optimise the cognitive load of an experience to maximise the learning potential. Although previous studies have evaluated the effect of simulation on participants’ emotional state and cognitive load, studies addressing complex simulation scenarios in critical care setting are lacking. Tailoring scenarios with cognitive load and emotional state in mind will vary according to clinical setting, team dynamic as well as experience level.

The aim of this study was to assess the emotional states and relative overall cognitive load of participants by focusing on and optimising elements of cognitive load during high fidelity critical care simulation sessions.

Methods and materials

Study design

This study was a prospective observational study conducted in a single tertiary referral intensive care unit (ICU) between 1 January 2015 and 31 December 2015.

Written informed consent was obtained from all participants after obtaining ethics approval from the regional ethics committee.

Setting

Scenario

The simulation session was conducted in our simulation centre. For each training session, the same scenario was used. The scenario consisted of an 80-year-old man on the medical ward requiring attention by the ICU team for hypotension. The underlying diagnosis was upper gastrointestinal bleeding. During the scenario, the patient suffers a cardiac arrest and requires resuscitation.

The scenario was piloted, revised and retested by the investigators who were all trained in medical simulation. Revisions addressed the script for historical data, relevant physical examination findings and additional data from investigations (eg, arterial blood gas and other laboratory test results) that supported the underlying diagnosis aiming for consistency in learner experience. Emphasis was placed on the flow of information (blood results, blood gases and so on) to make sure it was consistent with how learners would receive that information in a real-life setting.

The scenario concluded with a return of spontaneous circulation at which time the patient is transferred to the ICU for further management. A particular element of extraneous load that was avoided was unexpected patient death.

Each session concluded with a 45 min debriefing by two experienced facilitators. The facilitators were involved in the simulation as an embedded participant, portraying the ward nurse and a simulation technician, operating the manikin and providing the patient voice.

Participants

Each group was comprised of four to five participants (ICU registrars and nurses). All participants were briefed about the simulated environment prior to the scenario to familiarise them with the equipment and establish a fiction contract to maximise learner engagement and theoretically optimise intrinsic load and germane load.6 During the scenario, they worked together to gather historical data, examine the simulated patient, order and interpret the results of investigations and initiate the treatment simultaneously.

Equipment

For each training session, the same human patient simulator (SimMan 3G; Laerdal Medical) was used. All other equipment required for scenario was made available in simulated arrest trolley. Participants were briefed about the location of equipment and environment prior to commencement of the scenario.

Assessment

We assessed emotion experienced by participants using the validated tool described by Feldman Barret and Russell.12 This tool has eight items and each one describes bipolar opposite affect or emotional state. The eight items were tense/calm, nervous/relaxed, stressed/serene, upset/contended, sad/happy, depressed/elated, lethargic/excited and broad/alert. We asked participants to rate their emotion for each item on a five-point Likert scale which ranged from −2 to +2. We assigned a positive value to the positive emotional state and a negative value to the negative emotional state as previously reported by Fraser et al.13 Participants were asked to rate their emotion before and after the simulation scenario (figure 1).

Figure 1

Subjective rating tool based on circumplex model of emotion.

We assessed the cognitive load of the participants on a nine-point symmetrical category scale which ranged from very, very low mental effort (1) to very, very high mental effort (9), as described by Paas and Van Merriënboer.11 The participants were asked to rate their cognitive load after the completion of the simulation scenario (figure 2).

Figure 2

Cognitive load rating tool by Paas and Van Merriënboer—using the rating scale, please rate the amount of mental effort that was required to complete this simulation scenario.

Results

A total of 103 (40 medical and 63 nursing) staff participated in the study. The median age of participants was 29 (IQR 22–52) years. Most participants had prior experience of <5 simulation sessions (table 1).

Table 1

Participant characteristics

	n=103
Age (years, median (IQR))	29 (22–52)
Participating staff
Nursing staff	63
Medical staff	40
Prior simulation experience (n=99)
0–5 simulation sessions	72
6–20 simulation sessions	27

After completion of simulation session, the mean subjective ratings for the descriptor assessing the emotion tended to be higher (positive) compared with those before the beginning of scenario. The difference was significant for three of eight items: nervous/relaxed (P=0.002, d=0.39), lethargic/excited (P=0.003, d=0.35) and bored/alert (P<0.00001, d=0.54) (table 2).

Table 2

Emotional states before and after simulation

	Before simulation	After simulation	P	Cohen’s d
Tense/calm	−0.08±1.17	−0.10±1.0	0.88	−0.02
Nervous/relaxed	−0.28±1.15	0.14±1.0	0.002	0.39
Stressed/serene	−0.09±0.99	−0.01±1.01	0.54	0.08
Upset/contended	0.59±0.98	0.57±0.84	0.82	−0.03
Sad/happy	0.78±0.84	0.70±0.81	0.39	−0.10
Depressed/elated	0.53±0.75	0.51±0.72	0.87	−0.02
Lethargic/excited	0.25±0.89	0.55±0.92	0.003	0.35
Bored/alert	0.85±0.87	1.28±0.73	<0.00001	0.54

This suggests transition from negative to positive mood relative to the educational intervention.

The mean cognitive load across all participants was 6.61±1.47. There was no significant difference between cognitive loads of medical and nursing staff (6.61±2.3 vs 6.62±21.7, P=0.73). Similarly, cognitive load of participants with prior experience of <5 simulation sessions did not differ significantly from those with experience of >5 simulation sessions (table 3).

Table 3

Cognitive load scores

	N	Cognitive load score Mean±SD	P
Participants (n=103)
Nurses	63	6.62±1.7	0.73
Doctors	37	6.61±2.3
Prior simulation experience (n=99)
0–5 sessions	72	6.57±1.38	0.76
6–20 sessions	27	6.67±1.47

There was no correlation between cognitive load and emotional descriptors (table 4).

Table 4

Correlation of emotional states with cognitive load

	Before simulation		After simulation
	Correlation coefficient	P	Correlation coefficient	P
Tense/calm	−0.12	0.22	−0.19	0.049
Nervous/relaxed	−0.09	0.35	−0.06	0.57
Stressed/serene	−0.06	0.54	−0.03	0.76
Upset/contended	0.08	0.43	0.13	0.20
Sad/happy	0.08	0.44	0.10	0.33
Depressed/elated	0.06	0.52	−0.03	0.78
Lethargic/excited	0.11	0.27	0.12	0.24
Bored/alert	0.18	0.07	0.13	0.18

Discussion

Critical care scenarios are complex and involve complex procedural skills along with non-technical skills such as teamwork and communication. This study demonstrated that the measurement of cognitive load and emotional experience of participants in well-designed complex simulation scenario in the ICU environment is feasible. The participants in this study demonstrated relatively higher cognitive load with positive emotional state after completion of scenario. That is, it is a positive experience despite its complexity.

Despite its pedagogic appeal, in practice, a learning gap is associated with simulation training whereby up to 25% of students fail to improve their performance after training.7 8 The possible explanations are inappropriate content, ineffective delivery and participant cognitive overload due to the highly interactive nature of the simulation learning environment.8 9 While it is difficult to test the learning gap, by testing, reviewing and retesting our scenario, we tried to minimise the confounders and attempted to focus on cognitive load of the experience and evaluated the emotional effect. By using measures to evaluate the cognitive load and emotional impact of a scenario, educators can better tailor curricula to hopefully bridge this learning gap.

According to Fraser et al, cognitive load between 3 and 6 out of 9 is associated with maximal learning experience and a score above 7 results in declined performance.9 In our study, cognitive load was near optimal. Educators who take the time to craft a scenario with cognitive load in mind will be more successful in optimising the amount of strain imposed on learner working memory.14

Interestingly, our study found no difference between the cognitive loads experienced by the medical and nursing staff despite the complexity of the clinical situation. Also, there was no significant difference in cognitive loads of participants with more exposure to simulation in the past versus those who are relatively simulation naïve. This could be attributed to participants' overall experience and familiarity with similar scenarios in real life, working in multidisciplinary teams, rather than a high level of simulation experience as most participants had experienced less than five previous simulations. Within cognitive load theory, this experience is termed as ‘schemata’ and allows the participants to more readily manage complexity without overloading the working memory.5 13 15 16

In our study, we focused on reducing the extraneous load due to patient’s death, which may cause an overwhelming emotional effect and therefore did not end our scenario with the patient’s death. The study by Fraser et al compared the effect of unexpected simulated patient death on medical students' emotional state and cognitive load.13 Participants experienced more negative emotional state in the group with unexpected simulated patient death compared with the group in which patient survived. It is difficult to make a direct correlation that this individual adjustment led to the reported outcomes in our study as there is no control group. However, it does imply that the cognitive load can be measured and adjustments to the scenario by applying different strategies may affect that measurement.

When considering the cognitive load of an experience, educators have many opportunities to reduce extraneous load and optimise focus on intrinsic load. An opportunity to reduce extraneous cognitive load in the critical care setting includes avoiding split attention and expertise reversal effect by focusing information sourcing during the scenario to be more aligned with how teams gather information. Another way to reduce extraneous load is to offer learners worked examples to better inform learners with modelled behaviours. Focusing on intrinsic load elements, an educator can be mindful to familiarise the learners to the environment and manikin. If novice learners, scaffolding the complexity of the scenarios will allow learners to gradually build their working knowledge around the intrinsic load over time.3 5 6 These and other approaches should be factored into scenario design to optimise knowledge transfer.

The focus on emotion as the manipulated extraneous load stems from research exploring how emotion can hinder or promote knowledge transfer. Positive emotional states have important influences on participants’ learning and performance.17 18 Because learners contribute to the creation of their learning environment through their own experiences and emotions, emotional states likely influence what participants learn and how readily they can transfer that learning to a new situation.19 Previous research has shown that negative emotions encourage individuals to focus on the individual details associated with a learning scenario, which may be beneficial in tasks that require a strong attention to details. On the other hand, positive emotions encourage individuals to focus on big picture of a learning event. This global processing style may enable learners to create associative and relational connections between learning events, thereby increasing likelihood of transferring knowledge and skills to new situation.20–26

Cognitive and emotion research suggests that negative emotions increase an individual’s reliance on familiar problem-solving strategies, while positive emotions facilitate cognitive flexibility and openness to information. Thus, negative emotions may make it more difficult for participant to adapt when they do not have adequate problem-solving strategies. On the contrary, positive emotions enable participants to detect similarities between tasks, thereby enhancing the transfer of clinical skills and knowledge.27–29 As supported by our study, simulation education offers a positive emotional response by participants and therefore primes learners for enhanced learning, dynamic thinking and hopefully eager for future simulation educational experiences. Anecdotally, we are finding that the more routinely we perform simulation training in our unit, the more that it is requested by clinical staff. One could also consider that evaluating the emotional impact of a simulation learning experience may be justified. It is unclear from our study if evaluating cognitive load and emotional state is a redundant or additive task for refining a simulation scenario. Our study does conclude, however, that scenarios can be measured in terms of emotional and cognitive load impact.

Strengths and limitations

This is the first study in the critical care setting exploring measurement of emotional state and cognitive load. The team structure and participant characteristics (age, level of experience, medical and nursing mix) were consistent with a real-world situation. The study effectively details that emotional and cognitive load can be measured using the scale developed by Paas and Van Merriënboer.11

We specifically altered the scenario to have the patient survive as a means of reducing extraneous load. The study is limited as there is no control group to compare to when considering this strategy to reduce load. Correlating the score of cognitive load to the transfer of knowledge would have further strengthened the study. This would be a challenge as it would involve linking the simulation experience to measured improvements in clinical performance for each participant. Future studies may focus on optimised and tailored scenarios for team training with emphasis on particular outcomes. These studies could look for relative clinical outcome improvements as a demonstration of knowledge transfer.

Another limitation of the study was not measuring the effect of debriefing on the emotional state. The emotional state was assessed before debriefing; therefore, we were unable to evaluate its impact on emotional state. Other future considerations include team dynamics and their impact on cognitive load.

Conclusion

Measurement of cognitive load and emotional impact of simulation education in an ICU environment is feasible. Moreover, a well-designed, high fidelity simulation scenario can result in a positive effect on participants’ emotional state. The movement of learners emotionally from a more negative state to a positive state suggests that simulation is an effective tool for improved knowledge transfer and offers more opportunity for dynamic thinking. Further studies are needed to assess different components of cognitive load and to determine their relationship with learning ability.