A cost-effectiveness analysis of self-debriefing versus instructor debriefing for simulated crises in perioperative medicine in Canada.

PURPOSE: High-fidelity simulation training is effective for learning crisis resource management (CRM) skills, but cost is a major barrier to implementing high-fidelity simulation training into the curriculum. The aim of this study was to examine the cost-effectiveness of self-debriefing and traditional instructor debriefing in CRM training programs and to calculate the minimum willingness-to-pay (WTP) value when one debriefing type becomes more cost-effective than the other.
METHODS: This study used previous data from a randomized controlled trial involving 50 anesthesiology residents in Canada. Each participant managed a pretest crisis scenario. Participants who were randomized to self-debrief used the video of their pretest scenario with no instructor present during their debriefing. Participants from the control group were debriefed by a trained instructor using the video of their pretest scenario. Participants individually managed a post-test simulated crisis scenario. We compared the cost and effectiveness of self-debriefing versus instructor debriefing using net benefit regression. The cost-effectiveness estimate was reported as the incremental net benefit and the uncertainty was presented using a cost-effectiveness acceptability curve.
RESULTS: Self-debriefing costs less than instructor debriefing. As the WTP increased, the probability that self-debriefing would be cost-effective decreased. With a WTP ≤Can$200, the self-debriefing program was cost-effective. However, when effectiveness was priced higher than cost-savings and with a WTP >Can$300, instructor debriefing was the preferred alternative.
CONCLUSION: With a lower WTP (≤Can$200), self-debriefing was cost-effective in CRM simulation training when compared to instructor debriefing. This study provides evidence regarding cost-effectiveness that will inform decision-makers and clinical educators in their decision-making process, and may help to optimize resource allocation in education.

RCT Entities:   Population Interventions Outcomes

Introduction

Crisis resource management (CRM) relies on skill sets such as leadership/followership, communication, and resource allocation, which contribute to teamwork for effective patient care during life-threatening emergencies. CRM is essential to several acute care specialties, such as emergency medicine, anesthesiology, and obstetrics.

High-fidelity simulation training is effective for residents and staff to learn and retain CRM skills [1-5]. A simulation session usually includes practice using a full-body mannequin-based simulator immediately followed by a debriefing phase. Debriefing is crucial for learning CRM and is typically administered by a trained instructor [1,6,7].

The movement of residency training programs design towards competence by design will increase demand for medical simulations within a resource-constrained environment. Due to its human resource intensity, cost is a major barrier to the implementation of high-fidelity simulation training into the curriculum [8,9]. However, little research has been conducted into the cost of simulation programs [8], except for a few studies examining cost savings after simulated interventions [10,11]. The simulation community usually assumes that a large portion of costs are related to human resources, including faculty time, along with facility and hardware costs [8]. Instructor fees aim to compensate instructors for leave from clinical service and recognize the need for instructors to be trained to effectively and safely debrief learners.

In an effort to reduce the direct costs related with medical simulations, several studies have investigated various types of debriefing as alternatives to the traditional resource-intense instructor debriefing [1,2,12]. Self-debriefing has been shown to be effective in learning CRM, and it was found to be similarly effective to traditional instructor debriefing [2]. Despite several calls for cost-effectiveness analyses in the peer-reviewed literature [8,13] and although the cost of simulation-based education has been cited as a key criticism of simulation training [8], cost-effectiveness analyses of educational interventions are rare [13]. Assessing the cost-effectiveness of educational interventions, such as simulation-based education, is crucial to assist decision-makers and health providers in their resource allocation process. Given current economic pressures and fiscal constraints, it is important to analyze the cost-effectiveness of self-led versus instructor-led debriefing in learning CRM.

The aim of this study was to compare the cost and effectiveness of self-debriefing in comparison to the traditional standard of instructor-led debriefing for residents who learn CRM through simulation, using a net benefit (NB) regression analysis. Additionally, a goal of this study was to calculate the minimum willingness-to-pay (WTP) value (i.e., the turning point) when one debriefing type becomes more cost-effective than the other. This study builds upon a previously published article examining the effectiveness of self-debriefing [2]. Here, we hypothesized that self-debriefing would be less costly than instructor-led debriefing for residents learning CRM through simulation, but with similar efficacy.

Methods

Study design

This cost-effectiveness analysis used data from a previously conducted randomized controlled trial [2]. Institutional Research Ethics Board approval was granted for the original study with appropriate written informed consent, and the ethics board determined that no other approval was necessary to perform this cost-effectiveness analysis study (St. Michael’s Hospital, Toronto, Ontario, Canada).

Subjects

Fifty anesthesia residents were randomized to the control (instructor debriefing) or intervention (self-debriefing) groups [2]. All participants were volunteers and were in their second to fifth postgraduate years (University of Toronto, Canada). A pretest briefing reminded them of the usual non-technical skills for CRM as well as the technical aspects of conducting a simulation session. All participants then individually managed a pretest crisis scenario. Participants randomized to the intervention group self-debriefed, with no instructor present during the debriefing. They were allowed to review their pretest video. As a reminder of the skills targeted in the session, a brief description of non-technical skill categories (decision-making, situational awareness, task management, and teamwork) was left with them during the debriefing. Participants from the control group were debriefed by a trained instructor, and also used their pretest video. During the instructor debriefing, the instructor guided the trainee, fostering their reflection on the pretest performance. Immediately after their debriefings, all participants managed a second simulated crisis scenario (post-test). Both types of debriefing lasted 20 minutes. The 2 high-fidelity simulated scenarios were cardiac arrests occurring during surgery, and lasted for precisely 5 minutes. Each scenario was randomly assigned either for pretest or post-test for each subject. The high-fidelity scenarios took place in a simulation center that reproduced an operating environment with the necessary equipment, such as a respiratory machine and a crash cart. One was ventricular tachycardia due to myocardial infarction, and the other was ventricular fibrillation secondary to hyperkalemia. Two confederates played the roles of a circulating nurse and surgeon, following a predetermined script. Data collection occurred between November 2008 and June 2009.

Technical information

Participants’ CRM performance was assessed using the reliable and valid Anesthetists’ Non-Technical Skills (ANTS) scoring tool [14] (Supplement 1). ANTS assesses 4 categories of non-technical skills, each scored from 1 (low performance) to 4 (high performance), using half-point increments. Each of the 4 categories included several elements describing each non-technical skills category. Situational awareness encompassed gathering information, recognizing and understanding, and anticipating; teamwork encompassed coordinating activities with the team, exchanging information, using authority and assertiveness, assessing capabilities and supporting others; task management encompassed planning and preparing, prioritizing, providing and maintaining standards, and identifying and utilizing resources; and decision-making encompassed identifying options, balancing risks and selecting options, and re-evaluating (https://www.abdn.ac.uk/iprc/documents/ANTS%20Handbook%202012.pdf). In our study, subjects’ performance was rated at the level of the category. The elements within each category were not rated, but were simply used to guide the scoring of the categories. The total ANTS score was the sum of the 4 category scores, and ranged from 4 (low) to 16 (high). All scenarios were video-recorded and later rated by 2 simulation experts. The evaluators rated all performances, blinded to group allocation and scenario order (pretest or post-test). For the cost-effectiveness analysis, the ANTS score represented the clinically relevant outcome (i.e., the effect variable).

Costs

From the hospital’s perspective, the total cost included the cost associated with the instructor’s time, the resident’s time, and the space used during the training. The length of time the instructor and residents spent during the training, combined with the residents’ postgraduate training year, was used to calculate their respective hourly rates. In addition to an average space rental cost for each individual [15], the cost for each individual was estimated as the product of time and unit cost. The unit cost was obtained from standard costing sources [15, 16]. All costs were reported in 2012 Canadian dollars.

Statistics

The sampling frame was a convenience sample from all 50 subjects reported in the original analysis. All analyses were conducted using STATA ver. 12.0 (SAS Institute Inc., Cary, NC, USA). A 2-sided P-value of <0.05 was considered significant for all analyses. The cost-effectiveness analysis was conducted from the perspective of the hospital using NB regression [17].

A NB regression model was used to compare the cost and effectiveness of the self-debriefing program with that of the traditional instructor debriefing. The outcome of the analysis was the incremental net benefit (INB) of the self-debriefing compared to the instructor debriefing at a specified WTP. The NB approach is summarized in Table 1.

Table 1.

Key terms and their definitions

Term	Acronym	Description
Cost	C	The cost associated with the program of interest
Effect	E	The clinical outcome of interest
Incremental net benefit	INB	The difference in net benefit between the 2 programs
Willingness to pay	WTP	How much a person is willing to pay for 1 more unit of effect

The first step of the NB regression approach was to generate a NB value for each study participant using the following equation:

NBi=WTP(Ei)–Ci

Where the subscript i refers to participant i. Ei and Ci represent the observed effect and cost, respectively. The effect was measured by the ANTS score. In economic evaluations, WTP is a value that represents the maximum amount that the decision-makers would be willing to pay in order to receive 1 more unit of outcome [18]. In this study, WTP was defined as the monetary value that a decision-maker would be willing to pay for a unit increase in the ANTS score. We calculated the NB value for each participant in the study using different WTP values arbitrarily chosen between Can$0 and Can$10,000. Each participant had different NB values at different WTP values (i.e., if we had 10 WTP values, we would have 10 different NB values for each participant). The WTP values ranged from very low (Can$0) to very high (Can$10,000).

With the NB values for each participant, we estimated the INB using a single NB regression model for each WTP. In a simple linear regression model, the NB value for a specific WTP was the dependent variable, and the NB regression could be estimated as:

NBi=β0+β1(TX)i+εi

Where β0 is an intercept or constant term, TX is an intervention variable (1=self-debriefing and 0=instructor debriefing), and εi is a stochastic error term. In the regression model, the coefficient estimate of the program variable (i.e., the regression estimate of β1) represents the INB. A negative INB indicates that the program of interest is not cost-effective, whereas a positive INB indicates that it is cost-effective when compared to the standard debriefing at the specified WTP [17].

We examined the INB value for each regression model at a specified WTP to determine when the self-debriefing program was cost-effective (i.e., at which WTPs was the INB value positive). We also adjusted our NB regression models for 2 covariates: residents’ postgraduate training year and the ANTS pretest score, both of which are believed to be significant covariates of the ANTS score, (i.e., the residents’ non-technical skills) [1,14]. The final NB regression model included the program variable and 2 covariates as follows: NBi=β0+β1(TX)i+β2(PGY)i+β3(pretest)i+εi

Where PGY is the postgraduate training year and the coefficient estimate of the TX variable (β1) is the INB, representing the cost-effectiveness of implementing the self-debriefing program, adjusting for covariates.

In addition, we used the results from the NB regression model (i.e., the coefficient estimates of the TX variable and P-values) to create a cost-effectiveness acceptability curve (CEAC) [19]. In general, a CEAC provides information regarding the probability that an intervention would be cost-effective when compared to the comparator at given WTP values. In a CEAC, the y-axis represents the probability that the intervention is cost-effective, and the x-axis shows the range of WTP values [19]. To characterize the uncertainty of the findings using a non-parametric approach, we created another CEAC using 1,000 bootstrap estimates and compared it to the CEACs obtained using both parametric and non-parametric approaches.

Results

All 50 patients completed the original study protocol, and there were no missing data. Results of the relative effectiveness of self-debriefing versus instructor debriefing have been previously reported elsewhere [2]. In short, the characteristics of the participants were similar in both groups. Each type of debriefing was significantly effective in improving CRM performance (F1,48=13.28, P<0.01), and the degree of effectiveness was similar when comparing self-debriefing and instructor debriefing (F1,48=0.31, P<0.58). The CEA data set is shown in Supplement 2. The average cost for the self-debriefing group (Can$233.50±2.30) was lower than that of the instructor-debriefing group (Can$437.60±2.26) (P<0.0001).

Table 2 provides the results from the NB regression models with WTP values of Can$0, Can$100, Can$200, Can$300, Can$500, Can$1000, and Can$5000. The INB values (i.e., the coefficient estimates of the TX variable) were positive when decision-makers’ WTP for 1 more unit of effect (i.e., an increase of 1 point in the ANTS score) was less than Can$200. With WTP values of Can$300 or higher, the INB was negative.

Table 2.

Results from net benefit regression models for various willingness-to-pay values

Variable	WTP = CanSO	WTP = CanS 100	WTP = Can$200	WTP = Can$300	WTP = Can$500	WTP = CanS 1,000	WTP = CanS5,000 1
Constant term	-430.68	933.91	2,298.49	3,663.08	6,392.25	13,215.18	67,798.64
Covariates
Postgraduate training year (P-value)	0.01 (0.00)	-49.07 (0.03)	-96.02 (0.03)	-142.98(0.03)	-236.89(0.03)	-471.68(0.04)	-2,349.93 (0.04)
Anesthetists'Non-Technical Skills pretest (P-value)	-2.11 (0.42)	10.03(0.35)	20.06 (0.35)	30.08 (0.35)	50.14(0.35)	100.27 (0.35)	501.32(0.35)
Program variable
Self-debriefing (P-value)	204.38 (0.00)	131.42(0.01)	58.47 (0.52)	-14.48(0.92)	-160.39(0.48)	-525.16(0.25)	-3,443.31 (0.13)
Adjusted R2	1.00	0.19	0.08	0.08	0.09	0.10	0.12
F (df)		4.83 (3,46)	2.37 (3,46)	2.32 (3,46)	2.56(3,46)	2.90(3,46)	3.27(3,46)
Prob > F	0.0000	0.0052	0.0823	0.0873	0.0664	0.0449	0.0294
Meaning	CE	CE	CE	Not CE	Not CE	Not CE	Not CE

WTP, willingness to pay; CE, cost-effective (because INB>0); not CE, self-debriefing is not cost-effective compared to instructor debriefing (because INB<0); INB, incremental net benefit.

Results from the NB regression models for various WTP values are presented in Supplement 3. The cost-effectiveness plane with 1,000 bootstrap estimates is shown in Fig. 1. The plot showed estimates located in either the bottom left or bottom right quadrants, suggesting that self-debriefing was less costly than instructor debriefing. Most point estimates were in the bottom left quadrant, suggesting that self-debriefing was less costly and less effective than instructor-debriefing, whereas a portion of the estimates were in the bottom right quadrant, indicating that self-debriefing was less costly and more effective than instructor debriefing (Fig. 1).

Fig. 1.

One thousand bootstrap ICER estimates. ICER, incremental cost-effectiveness ratio.

Fig. 2 shows the probability that the self-debriefing program was cost-effective in comparison to the instructor debriefing. At a WTP of Can$100, there was 99% chance that self-debriefing would be cost-effective in comparison to instructor debriefing. As the WTP increased, the probability that self-debriefing was cost-effective decreased. With a WTP of Can$300, there was approximately a 46% chance that self-debriefing was cost-effective compared to instructor-debriefing. The CEACs created from the parametric and non-parametric approaches were similar, and for this reason only one was presented.

Fig. 2.

Cost-effectiveness acceptability curve.

Discussion

This study aimed to compare the cost and effectiveness of self-debriefing versus instructor debriefing for anesthesia residents learning CRM through a full-scale simulation from the perspective of a decision-maker (i.e., a hospital or university). Self-debriefing costs less; and consequently, as a decision maker’s WTP increased, the probability that self-debriefing would be cost-effective decreased. The results showed that with a WTP ≤Can$200, self-debriefing was cost-effective (INB >0). As the decision makers’ WTP increased to ≥Can$300, the INB became negative, which means that self-debriefing was not cost-effective compared to the traditional instructor debriefing. If the decision-maker’s WTP is ≤Can$200, then self-debriefing could be considered as an economically attractive option. However, when effectiveness is priced higher than cost savings and WTP >Can$300, instructor debriefing is the preferred alternative.

This study contributed to fill a knowledge gap in the literature, as outlined by several calls for more cost-effectiveness studies in medical education/simulation [8]. A recent systematic review further identified the components related to faculty costs to be staff fees, time, quantity, and training [8]; only a limited number of studies have attempted to conduct a cost analysis of personnel/staff costs associated with simulation, and have used inconsistent methodologies [8]. Our study has taken the first step in attempting to provide economic evidence by comparing self-debriefing to instructor debriefing. In addition, we have contributed to the limited literature on cost-effectiveness in healthcare education, producing a robust analysis and indicating potential directions for future work.

Implications of the findings for decision-makers

These results have implications for administrative decision-makers and clinical educators. As instructor debriefing is no more effective than self-debriefing for anesthesia residents to learn CRM [2], cost may also serve as a deciding factor between these methods. This study provided a foundation for self-debriefing to be seen as a cost-effective alternative to instructor debriefing, without sacrificing the effectiveness of the learner’s CRM educational simulation experience. Access to simulations may become less dependent on factors related to instructors such as availability, training, and cost. Additionally, cost savings from debriefing may allow for more resource allocation to other areas of the curriculum in order to further improve the learning opportunities of trainees, or may allow trainees to be exposed to additional simulation sessions.

Applicability of industry cost-effectiveness methods to simulation-based education

One strength of this study was that a cost-effectiveness analysis approach was used to examine the cost-effectiveness of different types of debriefings. Other forms of debriefings have been investigated as alternatives to instructor debriefing, including the use of video review, multimedia debriefing, and within-team debriefing. Future studies can apply a similar approach to examine the cost and effectiveness of other debriefing interventions. Furthermore, the NB regression approach provides information for a range of theoretical WTPs in a regression format and in the CEAC to assist decision-makers with the choice of whether to adopt a new program.

Limitations

This study is limited in that data were collected from a single simulation center. The monetary values utilized for the analysis were based on specific institutions and could be different elsewhere. Simulation and faculty costs can vary across regions and centers, which could potentially influence the results of the analysis if conducted at other sites. Our analysis focused solely on non-technical skills, based on the study objectives and data available; if other skills were to be included, the findings may change. However, we previously showed that technical and non-technical skills in medical crisis management were related to each other [20]. We elected to use the ANTS score as a surrogate for performance, and therefore reported the cost corresponding to an increase in performance by 1 ANTS score increment (ANTS scores range from 4 to 16). The translation of 1 extra ANTS point into patient outcomes is unknown. In addition, due to limited data on cost, the variation in cost among participants was minimal, supporting the similarity of CEACs between the parametric and non-parametric approaches. However, this study used an established model for exploring the cost-effectiveness of simulation training and has taken the first step in this field of study by examining the cost-effectiveness of debriefing within simulation and CRM training.

In conclusion, this study showed that with a lower WTP (≤Can$200), self-debriefing was cost-effective in comparison to instructor debriefing in CRM simulation training for anesthesia residents. This study provides evidence to inform decision-makers and clinical educators in their decision-making processes, especially in term of cost-effectiveness, and may help to optimize resource allocation in education. The health economics approach used in our study may be applied to examine the cost and effect of alternate forms of debriefing in simulation-based education.