[Acquiring skills in malignant hyperthermia crisis management: comparison of high-fidelity simulation versus computer-based case study].

INTRODUCTION: The primary purpose of this study was to compare the effect of high fidelity simulation versus a computer-based case solving self-study, in skills acquisition about malignant hyperthermia on first year anesthesiology residents.
METHODS: After institutional ethical committee approval, 31 first year anesthesiology residents were enrolled in this prospective randomized single-blinded study. Participants were randomized to either a High Fidelity Simulation Scenario or a computer-based Case Study about malignant hyperthermia. After the intervention, all subjects' performance in was assessed through a high fidelity simulation scenario using a previously validated assessment rubric. Additionally, knowledge tests and a satisfaction survey were applied. Finally, a semi-structured interview was done to assess self-perception of reasoning process and decision-making.
RESULTS: 28 first year residents finished successfully the study. Resident's management skill scores were globally higher in High Fidelity Simulation versus Case Study, however they were significant in 4 of the 8 performance rubric elements: recognize signs and symptoms (p = 0.025), prioritization of initial actions of management (p = 0.003), recognize complications (p = 0.025) and communication (p = 0.025). Average scores from pre- and post-test knowledge questionnaires improved from 74% to 85% in the High Fidelity Simulation group, and decreased from 78% to 75% in the Case Study group (p = 0.032). Regarding the qualitative analysis, there was no difference in factors influencing the student's process of reasoning and decision-making with both teaching strategies.
CONCLUSION: Simulation-based training with a malignant hyperthermia high-fidelity scenario was superior to computer-based case study, improving knowledge and skills in malignant hyperthermia crisis management, with a very good satisfaction level in anesthesia residents.

Introduction

Anesthesiology practice has become increasingly challenging, especially in teaching environments where residents have to learn new skills in demanding and complex professional environments with more difficult patients from a technical perspective. Furthermore, it is now widely acknowledged that in order to achieve a successful clinical outcome it is also required that anesthesiologists have a wide range of non-technical skills such as effective communication, teamwork and proper resources management.

Anesthesia Crisis Resource Management (ACRM) has been defined by Gaba as the articulation of principles of individual and crew behavior that focuses on skills of dynamic decision-making, interpersonal behavior, and team management. High fidelity simulation has been well established in anesthesia where it is currently considered a critical teaching tool.4, 5 Full-scale high fidelity mannequin simulators through simulated scenarios can be used for several training purposes, including teaching technical skills and ACRM, advanced life support algorithms, and particularly simulating rare events, among others.

Despite the widespread acceptance of simulation-based training in anesthesiology, there is still some degree of skepticism about its cost-effectiveness. In a recent meta-analysis of the current state of the evidence on simulation-based training in anesthesiology, simulation failed to demonstrate superiority over other simulation educational tools. They conclude that simulation seems to be more effective than no intervention and non-inferior to non-simulator training (computer-based instruction, small group discussion, one-on-one instructional sessions, assigned readings, and instructional videos).

Accordingly, we decided to study whether a simulation based approach such as high-fidelity simulation scenario performed better than a computer-based case study, a well known non-simulator based strategy. In order to assess our hypothesis, we evaluated the acquisition of skills to treat malignant hyperthermia by first year anesthesiology residents. As a secondary outcome, we evaluated clinical reasoning and decision making during the crisis management using qualitative analysis.

Methods

Participants

First year anesthesiology residents at Pontificia Universidad Catolica de Chile, School of Medicine were invited to participate as study subjects. There were no exclusion criteria and students were allowed to decline to participate. All individuals signed an informed consent form before enrolling the study. The approval for this study was obtained from the Research Ethics Committee of Pontificia Universidad Catholic de Chile, School of Medicine.

The anesthesia diploma is a one year duration program, created for first year residents, in order to teach basic theoretical contents about anesthesia. It is an open program, where anesthesia residents from other universities are eligible, as well. All the participants had previous exposure to simulated scenarios form crisis management, as a part of their academic activities in the diploma.

Study design

This was a prospective, randomized study with two groups: High Fidelity Simulation-based training (HFS) versus a computer-based Case Study (CS). Participants were assigned to develop the same case stem, about a Malignant Hyperthermia (MH) crisis, in one of the two educational strategies. In order to assure that both groups receive the same information and contents, the investigators performed a review of literature and prepared both cases by consensus. Randomization was performed using a computer random number generator.

Intervention

Before intervention, all participants attended to a lecture about MH and read 2 complementary literature reviews about MH.9, 10 One week later, all students were randomized to their respective intervention (Fig. 1). In the simulation-based training group, students were divided into 4 groups of four residents. Each student participated individually in a high-fidelity simulated scenario, as the primary anesthesiologist. In addition, three confederates participated in the case, one anesthesia resident and two operating room nurses. The scenario was a MH crisis during a general anesthesia (Case 1). After the scenario, a trained instructor guided a debriefing session focused on both, technical and nontechnical skills performance. All debriefings were conducted by the same anesthesiologist, using the approach described by Rudolph.

Figure 1

Protocol timeline.

The computer-based case study, consisted in a computer-based written case. Instructions, development guidelines and information of the case, were prepared in a virtual platform. Each student received via email a username and a password to develop the case through their own computer.

Assessment

All participants performed a test questionnaire before entering the intervention, after the lecture and after reading the papers (Fig. 1). The questionnaire was designed according to the learning objectives of the ACGME frame, and 2 complementary literature reviews.9, 10 The questionnaire was designed by one anesthesiologist with convergent questions, intending to elicit a specific response or a narrow list of possible responses. The questionnaire was evaluated by 3 anesthesia faculties, in order to determine the content validity. Finally, the questionnaire was tested with a second year anesthesia resident (not involved in the study), in order to assess practical topics such as questions comprehension, phrasing, clarity, and time necessary to answer.

A satisfaction survey was distributed immediately post interventions to rate both educational experiences. The answers were processed anonymously.

All participants attended to the simulation center 2 weeks after the intervention for an assessment session. The session consisted in a high-fidelity simulation malignant hyperthermia case, which was completely different to the first case (Case 2). Each student participated individually in the scenario, as a primary anesthesiologist with the same three confederates (one anesthesia resident and two operating room nurses). The simulation session was videotaped. The performance assessment was done using a performance rubric. Two blinded and independent raters reviewed all the video performances and the behaviors observed were rated using the rubric described below. Additionally, after the simulated scenario, all participants performed a post-test knowledge questionnaire (Fig. 1).

Finally, a semi-structured interview was done in order to assess self-perception of factors influencing participant's learning, reasoning process and decision-making during the crisis management. As described by Higgs, every clinical practice situation decision is characterized by a unique combination of decision attributes (uniqueness, certainty, importance, stability, urgency, familiarity, congruence, risk, relevance and number of variables). The interview was guided by a short questionnaire specially designed for this study, taking in consideration these decision attributes and the dual-process theory. This theory highlights the importance of physicians’ intuition and the high level of interaction between analytical and non-analytical processes. The same investigator interviewed all participants individually, in a 30 minutes session.

Instrument development

A performance rubric was developed specifically for the purpose of this study. It was designed with the aim to evaluate both, technical and nontechnical skills. It was developed according to the structure established by Taggart, using the elements of the ACGME frame. The specific content was developed using the MHAUS treatment protocol, and the checklist for MH published by Arriaga. For each element of the rubric (Table 4), a detailed observable behavior was described, with levels of performance ranging from 1 to 3.

Table 4

Performance rubric elements.

1) Demonstrate ability to recognize differential diagnoses of MH.

2) Demonstrate ability to recognize signs and symptoms of MH crisis.

3) Demonstrate ability to establish and prioritize the initial actions of a MH crisis management.

4) Demonstrate the ability to dissolve and administer Dantrolene.

5) Demonstrate ability to recognize MH complications.

6) Demonstrate the ability to treat the patient under ethical principles.

7) Demonstrate communication skills, verbal and attitudinal interaction with the health team.

8) Demonstrate the ability to adopt an attitude of communication and interaction with the surgeon.

MH: Malignant Hyperthermia.

The instrument was validated before implementation as follows: A pilot study including two anesthesiologists trained by the primary investigator it was done. These individuals applied the instrument independently to rate performances during MH simulated scenarios using videos stored in our laboratory. To analyze the reliability of the instrument, a Cohen kappa coefficient was calculated. Iterative revisions of the rubric were made until to achieve a correlation level of 0.7.

Statistical analysis

Statistical analysis was performed using SPSS16.0 (Chicago, IL). Demographic data were analyzed using the Chi-square test. The inter-rater reliability was assessed using the Cohen kappa coefficient. ANOVA test was used for the analysis of pre and post knowledge tests. Descriptive statistics mean scores and a Mann–Whitney test were used to analyze the performance rubric scores. The information arising from the application of the satisfaction survey was analyzed with and a Mann–Whitney test. A p-value of 0.05 was considered significant. Content analysis of the semi-structured interview was handled with analytical Phenomenology, through a process of encoding information to reach a development scheme of representative codes of participant's perception.

Results

Instrument agreement analysis

As described in methods, two raters assessed individually 2 videos of MH cases stored in our simulation laboratory. The inter-rater reliability for the first instrument was 0.385. After this first agreement calculation, a meeting was done with the purpose of reviewing discordant elements. The rubric was improved with the suggestions that emerged from the discussion. Raters reviewed another 2 videos of MH simulation scenarios, stored in our laboratory. The second Cohen kappa coefficient was 0.619. This iterative process was done 1 more time with a final inter-rater reliability value of 0.75.

Of the 31 residents recruited, 28 finished successfully the study. Demographic data are listed in Table 1. Regarding previous experience in high fidelity simulation, all residents have been participated in scenarios during a period of 6 months previous to this study.

Table 1

Participant demographics.

	CS, mean (SD)	HFS, mean (SD)	X2 test
Age	31.00 (8.18)	30.57 (6.98)	0.085
Years of postgraduate training	5.23 (2.04)	6.50 (8.64)	0.686
Number of HFS sessions previous to this study	4.69 (2.84)	5.57 (2.56)	0.406
Number of HFS MH cases previous to this study	0.08 (0.077)	0.00 (0.00)	0.337
Number of real MH cases previous to this study	0.08 (0.77)	0.07 (0.27)	0.959
Prior academic performance	6.05 (0.24)	6.25 (0.32)	0.084

CS: Computer-based Case study; SD: Standard Deviation; HFS: High Fidelity Simulation Scenario; MH: Malignant Hyperthermia.

The time between the intervention (HFS case or CS) and the assessment simulation session was 2 weeks for all the participants. Average scores from questionnaires assessing knowledge improved from 74% to 85% in the HFS group, and decrease from 78% to 75% in the CS group (Fig. 2). A comparison between the post-tests was significant in favor of HFS (p = 0.032).

Figure 2

Participants’ knowledge scores for both interventions. Median values of post-test are presented in box plot with mean scores and 95% CI.

Table 2 shows mean performance rubric scores of both interventions. HFS scores are significantly higher than CS scores, in 4 rubric elements, recognize signs and symptoms of MH crisis (p = 0.025), prioritization of initial actions of management (p = 0.003), recognize malignant hyperthermia complications (p = 0.025) and communication with health team (p = 0.025).

Table 2

Participants’ performance for both interventions.

Performance rubric elements	CS, mean (SD)	HFS, mean (SD)	Mann–Whitney
Recognize differential diagnoses of MH	1.38 (0.58)	1.64 (0.82)	0.685
Recognize signs and symptoms of MH crisis	1.73 (0.44)	2.25 (0.55)	0.025a
Establish and prioritize the initial actions of management	2.54 (0.59)	2.96 (0.13)	0.003a
Ability to dissolve and administer Dantrolene	1.96 (0.75)	2.14 (0.93)	0.685
Recognize MH complications	1.88 (0.71)	2.53 (0.63)	0.025a
Treat the patient under ethical principles	2.35 (0.66)	2.67 (0.42)	0.141
Communication skills with health team	2.2 (0.66)	2.78 (0.47)	0.025a
Communication and interaction with the surgeon	2.92 (0.19)	2.89 (0.28)	1

CS: Computer-based Case study; SD: Standard Deviation; HFS: High Fidelity Simulation Scenario; MH: Malignant Hyperthermia.

p < 0.05.

Several aspects of both interventions were highly rated in the satisfaction survey. There was a significant difference in favor of HFS, in four of eight elements of the satisfaction scores: “The session was funny”, “I feel confident that I achieved the objectives of the session”, “The debriefing/reflection conduction was clear” and “Participating in this session was a positive experience” (Table 3).

Table 3

Participants’ satisfaction survey scores.

Satisfaction survey	CS, mean (SD)	HFS, mean (SD)	t-test
The session was funny	4 (0.894)	4.79 (0.426)	0.008a
The session was realistic	4.36 (0.674)	4.57 (0.646)	0.385
The session developed important skills for my future	4.73 (0.467)	4.93 (0.267)	0.182
The time devoted to the session was appropriate	3.91 (0.944)	4.43 (0.756)	0.14
The debriefing/reflection conduction was clear	4.18 (0.751)	4.71 (0.611)	0.045a
The debriefing/reflection session fulfilled the objectives	4.45 (0.522)	4.71 (0.469)	0.143
I feel confident in achieving the objectives of the session	4 (0.632)	4.57 (0.646)	0.03a
Participating in this session was a positive experience	3.2 (0.632)	3.85 (0.376)	0.008a

CS: Computer-based Case study; SD: Standard Deviation; HFS: High Fidelity Simulation Scenario.

p < 0.05.

Regarding the qualitative analysis, there was no difference in the most important factors that influence the student's process of reasoning and decision-making, between both teaching strategies (Fig. 3).

Figure 3

Factors influencing the decision-making process in a 5 Likert Point Scale.

Discussion

Our results demonstrate that simulation-based training using a MH high-fidelity scenario is better than computer-based case study, in terms of improving technical training and skills of MH crisis management.

Regarding crisis management skill acquisition, this study showed a statistically significant difference in 4 of the 8 items assessed. Several reasons may explain why not all the evaluated items were improved with MH high fidelity; one could be the relatively short simulation intervention. This situation have been previously described by Fletcher et al. not demonstrating any appreciable improvement in Non-Technical Skills (NTS) of practicing anesthesiologists using the ANTS tool. Likewise, Morgan et al. concluded that a relatively short intervention accounted for poor results in terms of improving NTS in their study. These data are consistent with a recent meta-analysis suggesting that measures of NTS vary widely between studies and that NTS training might have been insufficiently intensive to effectively improve it. Other explanation for our results might be the period of time between the intervention and the assessment scenario. Previous studies including anesthesia residents demonstrated an improvement in ANTS scores after simulation-based education.24, 25, 26 Nevertheless, in those studies the intervention consisted in only one scenario and the posttest assessment to evaluate new skills, was performed immediately after the intervention.

In our study, the assessment scenario was done 2 weeks after the intervention, in order to have enough time for reflection. Reflection process has been described as a crucial point in the context of adult learning, according to Kolb. However, it is plausible that 2 weeks could have been an exceedingly long period of time, resulting in skills decay, a situation previously described for non-technical and motor skills.23, 28 Indeed, all these possibilities could be definitely worth to explore in the future. Furthermore, in order to plan a sustained intervention over time, it is extremely relevant to define the admissible timing between the simulation sessions and the assessments, considering the decay of the acquired clinical or non-technical skills. Finally, we cannot rule out that the assessment tool used in this study was not the optimal instrument to evaluate the proposed outcome. It has been described that creating rubrics is certainly one of the main challenges for a proper assessment. Efforts to develop scoring metrics for high fidelity simulators are currently expanding very fast.22, 30, 31, 32 Ross described that changes in the use of structured assessment tools to assess performance, is mainly observed in the non-technical domain. In general, the choice of metrics will depend on what we want to measure. If we want to assess technical skills, it is usually possible to identify key actions and to develop checklists. On the other hand, for NTS such as communication and teamwork, rating scales are usually more appropriate. In this specific case, we wanted to measure both, technical and NTS. Nevertheless, a specific validated tool to evaluate the performance in a high fidelity simulation MH crisis scenario, have not been previously reported.

In this context, the tool we have developed has different components, aiming to evaluate technical and non technical skills: the first 5 items of the performance rubric evaluated technical skills (medical management) and the 3 remaining assessed Non-Technical Skills (NTS), as described in Table 4.

Noteworthy, regarding medical management skills, our results show despite that the prioritization of initial actions and the recognition of MH complications improved significantly, the ability to dissolve and administer Dantrolene did not improve as we expected. Because reconstituting Dantrolene fast enough to treat an MH crisis has proven to be a difficult task, this point was extensively discussed during our debriefings. Adding this specific task to the scenario forced our residents to rationally manage the human resources available during the crisis. The same situation took place regarding NTS: although the ability to adopt an attitude of communication and interaction with the surgeon was extensively discussed during debriefings, there were no differences in this element. Our simulation curriculum intends to train both, technical and NTS, in order to provide a balanced education. Therefore, simulation scenarios have been designed with the objective to talk about technical and NTS. Since the time dedicated to each debriefing is limited, it is possible that not all items were covered extensively enough, which could account for the absence of significant improvement observed in our study.

Finally, since that technical skills and NTS are obviously intertwined, it becomes very difficult to find differences when learners start with a high level of technical skills.

Our study has several limitations. First, we did not determine a sample size and we included in our cohort including all the first year anesthesiology residents. Therefore, it is possible that the number of participants could be insufficient to demonstrate a significant improvement in MH management skills. We have only 28 complete data sets, because 3 residents did not attend to the assessment session. The difference in favor of HFS performance scores of residents may be secondary to an increased familiarity with the simulation environment during the assessment process. However, demographic data supports that all residents have participated in HFS scenarios previous to this study, minimizing any possible impact of this issue.

In conclusion, our results are consistent with previous literature, in a setting where it is difficult to find improvements in residents’ performance skills. From our point of view, the lack of significant differences in the remaining performance elements could be explained by the relatively short simulation intervention in individuals with previous well-developed technical and NTS. These results highlight the need to carefully consider the design of a study to evaluate improvement of skills and the assessment tools that should be used.

Finally, the anesthesiology education community needs to continue working to clear how and how much to use high fidelity simulation in anesthesiology teaching, if adding non-technical skills training to enhance simulation-based learning, altogether with a rigorous cost-effectiveness analysis.

Conflicts of interest

The authors declare no conflicts of interest.