Impact of a novel, resource appropriate resuscitation curriculum on Nicaraguan resident physician's management of cardiac arrest.

PURPOSE: Project Strengthening Emergency Medicine, Investing in Learners in Latin America (SEMILLA) created a novel, language and resource appropriate course for the resuscitation of cardiac arrest for Nicaraguan resident physicians. We hypothesized that participation in the Project SEMILLA resuscitation program would significantly improve the physician's management of simulated code scenarios.
METHODS: Thirteen Nicaraguan resident physicians were evaluated while managing simulated cardiac arrest scenarios before, immediately, and at 6 months after participating in the Project SEMILLA resuscitation program. This project was completed in 2014 in Leon, Nicaragua. The Cardiac Arrest Simulation Test (CASTest), a validated scoring system, was used to evaluate performance on a standardized simulated cardiac arrest scenario. Mixed effect logistic regression models were constructed to assess outcomes.
RESULTS: On the pre-course simulation exam, only 7.7% of subjects passed the test. Immediately post-course, the subjects achieved a 30.8% pass rate and at 6 months after the course, the pass rate was 46.2%. Compared with pre-test scores, the odds of passing the CASTest at 6 months after the course were 21.7 times higher (95% CI 4.2 to 112.8, P<0.001). Statistically significant improvement was also seen on the number of critical items completed (OR=3.75, 95% CI 2.71-5.19), total items completed (OR=4.55, 95% CI 3.4-6.11), and number of "excellent" scores on a Likert scale (OR=2.66, 95% CI 1.85-3.81).
CONCLUSIONS: Nicaraguan resident physicians demonstrate improved ability to manage simulated cardiac arrest scenarios after participation in the Project SEMILLA resuscitation course and retain these skills.

Introduction

Meaney et al. undertook a review of resuscitation training in developing countries as part of the educational review of the International Liason Committee of Resuscitation (ILCOR) [1]. In addition to reporting a lack of evidence for adult resuscitation, they also reported inconsistently defined and reported outcomes for resuscitation education and a lack of local input into course appropriateness and modifications. The group encouraged all future resuscitation education in low and middle-income countries (LMIC) to account for the local environment and experience of the participants. In response to the need for low cost and resource appropriate resuscitation trainings in Nicaragua, a volunteer group of emergency care practitioners from the United States and Nicaragua developed the Project Strengthening Emergency Medicine, Investing in Learners in Latin America (SEMILLA) Resuscitation Program. It is a novel, resource-appropriate curriculum for Nicaraguan physicians in the management of adult cardiac arrest victims. The Project SEMILLA curriculum emphasizes treatment options that are available in Nicaraguan hospitals, hands-on skill practice, and leadership skills. The differences in resources were determined in conjunction with Nicaraguan medical school faculty. Notable differences include lack of pre-hospital care, no access to cardiac catheterization, no access to thrombolytics, and limited availability of pharmacotherapeutics. The Project SEMILLA resuscitation program uses no copyrighted algorithms. The focal point of the course is a management table that summarizes interventions by the initially identified cardiac rhythm. The management table and the curriculum overview are publically available on the resources page of the Project SEMILLA website (available from: http://www.projectsemilla.org). When piloted in 2013, knowledge of the core concepts of resuscitation improved. On a multiple-choice exam of the concepts of resuscitation, the median pre course score was 24% (interquartile range 20-32%) and improved post course to 68% (interquartile range 64-76%) (P<0.01). To evaluate the true impact of a training program, however, higher level endpoints must be assessed to assure that this knowledge translates to patient care.

According to Kirkpatrick’s framework of evaluation, the evaluation of training programs should assess whether the training successfully navigates the transfer of learned knowledge into behavioral change in the setting where the knowledge is applied [2]. This study aimed to evaluate the impact of the novel curriculum in adult cardiac arrest resuscitation on Nicaraguan resident physician behavior in a simulated cardiac arrest both immediately and at 6 months after the course.

Methods

Study design

The study is a comparison of resident management of simulated adult cardiac arrest scenarios at three time points; immediately pre-course, immediately post-course and at 6 month time point. The study uses standardized Cardiac Arrest Simulation Test (CASTest) scenarios and evaluation matrix to grade performance of a set of predetermined actions.

Setting

The setting of the research was at Hospital Escuela Oscar Danilo Rosales Argüello (HEODRA), the teaching hospital of the medical school of Universidad Nacional Autónoma de Nicaragua (UNAN) in León, Nicaragua. It is a 400-bed tertiary referral center that services the population of Northwest Nicaragua with multiple residency programs. The hospital has an Intensive Care Unit (ICU) and sees an average of 300 emergency department visits per day. The pre-course assessment, resuscitation course participation and immediate post course assessment were all completed during February, 2014. The 6 month post course assessment was completed in August, 2014.

Subjects

In total, we were able to retain 13 of 15 initial subjects and 13 completed the simulation at all three study time points. Attrition of the 2 subjects was secondary to call schedules that did not allow them to participate in the final simulation. These residents have completed a medical degree and are now enrolled in post-graduate training at UNAN in internal medicine, surgery, gynecology, anesthesia, and pediatrics. Their average age was 28.5 years-old and standard deviation was 3.3 years-old. Their other characteristics are presented in Table 1.

Table 1.

Characteristics of 13 Nicaraguan resident physicians who participated in 2014 Project Strengthening Emergency Medicine, Investing in Learners in Latin America (SEMILLA) resuscitation course, 2014

Characteristics	Number (%)
Female, n (%)	9 (69)
Year of residency, n (%)
1st	7 (54)
2nd	5 (38)
3rd	1 (8)
Specialty, n (%)
Anesthesia	4 (31)
Gynecology	4 (31)
Internal medicine	1 (8)
Pediatrics	2 (15)
Surgery	2 (15)
Prior resuscitation course, n (%)	2 (15)
Prior airway course, n (%)	2 (15)
Prior Trauma course, n (%)	0
Managed a cardiac arrest, n (%)	10 (77)
Defibrillated a patient, n (%)	4 (31)
Course on EKG, n (%)	0

Measures

Demographic characteristics of the participants were collected including age in years, gender, year of residency (1st through 3rd), and specialty. Participants were also asked about their prior exposure to resuscitation courses. They were asked if they had ever participated in a resuscitation course in the past and, if so, what type of course. Participants were also asked about prior exposure to difficult airway, electrocardiography, and trauma resuscitation courses. Finally, they were polled about their experience with managing cardiac arrest within the hospital and whether they had ever used a defibrillator on a real patient.

Outcome measurement methods

Each subject was observed as a team leader in a clinical simulated cardiac arrest scenario. There was a team available for the subject to direct which was composed of 4 nurses from the Nicaraguan hospital who were blinded to the purpose of the study. The nurses were instructed to participate in the study as if on the hospital ward when an arrest occurs, but not to interpret cardiac rhythms or suggest forgotten actions for the subject.

The primary outcome is the percent of participants achieving a passing score on the CASTest scoring sheet as assessed by a local expert at 6 months after the course as compared to their pre-course score. The CASTest is a validated measure of simulation performance used by the Resuscitation Council (UK) to assess skill performance after ACLS courses [3]. The Resuscitation Council of the United Kingdom (available from: https://www.resus.org.uk/) granted permission for use of the CASTest score card and cases for research purposes to evaluate the novel curriculum. Their cases have also been assessed for inter-rater reliability in scoring with good result [4]. Each case consists of a patient history in which their presenting condition then leads to arrest. There are a total of 24 actions rated by the instructor, 19 of which are considered “critical.” Each action is scored on a 4 point Likert scale as “excellent, acceptable, borderline, or not achieved.” To pass the exam, the student must receive a mark of excellent, acceptable, or borderline on each of the 19 critical actions. To reduce the possibility of bias, each case simulation was observed and scored by three Nicaraguan faculty members with expertise in resuscitation that were not involved in course development and were unaware of the overall purpose of the study.

Statistical analysis

Redcap was used for data entry and storage [5] and SAS v 9.4 (SAS Institute, Cary, NC) was used for the statistical analysis. Descriptive statistics were used to characterize demographic information about the subjects. Mean percent of correct actions (excellent, acceptable, or borderline marks) out of the 19 critical actions in the CASTest cases were calculated. Mixed effect logistic regression models were fitted for each of four outcomes from the CASTest to asses time effects, accounting for evaluator, and subject effects on the primary outcome (passing the CASTest) and each of the secondary outcomes (number of critical actions achieved, total actions achieved, and number of actions scored as excellent). A binary logistic regression model was used for the primary outcome, and binomial logistic regression models were used for the secondary outcomes. Finally, interrater reliability was assessed using Fleiss’s Kappa for the binary outcome of pass/fail. Intraclass correlation coefficient was used to assess interrater reliability for each of the secondary outcomes. A P-value<0.05 was considered statistically significant.

Sample size

Very few prior studies of adult resuscitation training programs report a pre-course assessment of skills which made calculating a sample size difficult [1]. The one prior study was written by Boonmak et al. and presented data on pre- and post-course resuscitation skills assessment for providers in Thailand. They demonstrated an increase in skill performance from 65% to 80% (SD=11%) [6]. Using repeated measures two sample comparison of means, a sample size of 12 subjects would give 90% power to detect a difference in the post-course evaluation with a two sided α of 0.05. To lessen the chance of attrition at 6 months impacting the result of the study, we planned to enroll a total of 15 subjects.

Ethical approval

This study was approved by Institutional Review boards in both the Olive View UCLA Medical Center Institutional Review Board (#13E-775950) and the Universidad Nacional Autónoma de Nicaragua (UNAN) León Bioethics Committee (Acta #92).

Results

On the CASTest, there are 19 specific critical actions that the resident must achieve to obtain a passing score. “Achievement” of the action is defined as obtaining a rating of borderline, acceptable, or excellent on that action, i.e. they cannot be given the rating “not achieved”. On the pre-course simulation exam, only 1 out of 13 subjects (7.7%) passed the test. Immediately post-course, the subjects achieved 4 out of 13 subjects (30.8%) passed, and 6 months after the course, the pass rate was 46.2% (6/13) (Table 2).

Table 2.

Comparison of outcomes at pre-test, post-test, and 6 months after post-test of 13 Nicaraguan resident physicians who participated in 2014 Project Strengthening Emergency Medicine, Investing in Learners in Latin America (SEMILLA) resuscitation course

	Outcome	Pre-test	Post-test	6 months after post-test
1	Pass Rate	1/13 (7.7%)	4/13 (30.8%)	6/13 (46.2%)
2	Percent of critical actions completeda)	78 (67-84)	94 (84-100)	95 (84-100)
3	Percent of total actions completeda)	74(65-83)	92 (88-100)	96 (67-100)
4	Number of actions scored as "excellent”a)	6 (4-8)	15 (10-20)	11 (8-13)

Presented as medians with interquartile ranges.

For the primary outcome of passing the CASTest exam, after the course, the participants were significantly more likely to pass the CASTest when accounting for evaluator and subject effects in a logistic regression model (OR=13.1, 95% CI 2.5-68.5). Furthermore, subjects maintained their improved ability to pass the CASTest at 6 months after the course. (OR= 21.7, 95% CI 4.2-112.8) (Table 3). There was no significant difference in the passing scores at 6 months as compared to immediately post course, signifying very little decay in acquired skills (OR 1.65, 95% CI 0.61-4.48). In addition, for each of the secondary outcomes, number of critical items completed, number of items completed and number of items scored as excellent, each also saw an improvement post-course and a maintenance of skills at 6 months after the course (Table 3).

Table 3.

Mixed effects logistic regression models to compare results at pre-test, post-test, and six months after post-test of 13 Nicaraguan resident physicians who participated in 2014 Project Strengthening Emergency Medicine, Investing in Learners in Latin America (SEMILLA) resuscitation course

Evaluation period	Passa)		No. of critical items completedb)		No. of items completedb)		No. of excellentemsc)
	OR (95% CI)	P-value	OR (95% CI)	P-value	OR (95% CI)	P-value	OR (95% CI)	P-value
Post-Test vs. Pre-Test	13.1 (2.5, 68.5)	0.003	4.11 (2.94, 5.74)	< 0.001	4.48 (3.35, 5.99)	< 0.001	4.85 (3.35, 7.00)	< 0.001
Six Months vs. Pre-Test	21.7 (4.2, 112.8)	< 0.001	3.75 (2.71, 5.19)	< 0.001	4.55 (3.40, 6.11)	< 0.001	2.66 (1.85, 3.81)	< 0.001
Six Months vs. Post-Test	1.65 (0.61, 4.48)	0.322	0.913 (0.617, 1.351)	0.647	1.02 (0.72, 1.45)	0.921	0.548 (0.389, 0.773)	< 0.001

Binary model;

Binomial model;

Overdispersed binomial model.

Interrater reliability of the CASTest scores assigned by the three expert observers was assessed. Although the binary outcome of pass/fail on the CASTest showed moderate interrater reliability (κ=0.461), the secondary outcomes that were all numeric, had high interrater reliability (critical items, κ=0.991, all items, κ=0.999, excellent items κ=0.974).

Discussion

Basic resuscitation skills are essential to providers and resuscitation education should be available to all practitioners in LMICs. Research is limited on how and what to teach in LMICs, especially for adult cardiac arrest. When developing a resuscitation programs in LMICs, organizers must define and understand baseline of the providers and the system prior to initiation [1]. Urbano et al. described a model of the successful implementation of pediatric resuscitation courses in Honduras [7]. Data on adult resuscitation training is sparse. This study is the first evidence that participants in a resource and language appropriate adult resuscitation course demonstrated improvement on a previously published and validated instrument from the Resuscitation Council of the United Kingdom. We believe the success of the course is the result of a long-term investment in and collaboration with local providers prior to course initiation.

An important aspect of the success of this program is the exemption of registration and textbook fees for participants. As noted by Urbano et al., access to high quality resuscitation training in LMICs is limited to an elite group of physicians who are able to afford the high costs of courses [7]. We hope that groups in other LMICs will be encouraged to create locally appropriate and open access courses.

Few prior studies of resuscitation education in LMICs use validated scoring systems to evaluated performance on psychomotor skills [1]. The CASTest was chosen because of its thoughtful development, validation and availability in the academic literature [3, 4]. Although the improvement was significant, Nicaraguan residents had a lower pass rates when compared to historic data from the United Kingdom [3]. These Nicaraguan residents have a different baseline as they have no prior training; therefore, it is expected that the overall result would not be equivalent. Also, the pass-fail outcome of the CASTest had a lower inter-rater reliability than that from the United Kingdom, although it was still within an acceptable range.

The improvement in scores post-course suggests that the Project SEMILLA Resuscitation Program is able to impact behavior. The ultimate outcome is improvement in patient survival. Although these studies are extremely resource intensive, they are possible [8] and will be a future goal for this group.

There were limitations to this small study. There were only 13 participants. In addition, the study used 6 months as the time point to measure retained skills, which does not preclude decay of skills at future time points. Finally, simulated cases are not equivalent to performance when faced with a patient in arrest, although recent evidence suggests that simulation training does correlate with improved skills when managing real patients [9]. We did not evaluate the cost effectiveness of this training.

Further research and documentation is needed to bring resuscitation education to the attention of ministries of health in LMICs. Though this study is small, it offers a model of a theoretically grounded evolution of a locally relevant resuscitation training program. This is an initial step toward the goal of high quality, locally relevant resuscitation training for all health care providers.

In conclusion, The Project SEMILLA Resuscitation Program improves the ability of Nicaraguan resident physicians to manage simulated cardiac arrest scenarios using a previously validated tool. The use of simulation in training and research is feasible within LMICs. Locally adapted resource appropriate resuscitation training courses are feasible and necessary to meet the needs of providers in LMICs. This low cost training program is able to improve critical resuscitation skills.