Prompting with electronic checklist improves clinician performance in medical emergencies: a high-fidelity simulation study.

BACKGROUND: Inefficient processes of care delivery during acute resuscitation can compromise the "Golden Hour," the time when quick interventions can rapidly determine the course of the patient's outcome. Checklists have been shown to be an effective tool for standardizing care models. We developed a novel electronic tool, the Checklist for Early Recognition and Treatment of Acute Illness (CERTAIN) to facilitate standardized evaluation and treatment approach for acutely decompensating patients. The checklist was enforced by the use of a "prompter," a team member separate from the leader who records and reviews pertinent CERTAIN algorithms and verbalizes these to the team. Our hypothesis was that the CERTAIN model, with the use of the tool and a prompter, can improve clinician performance and satisfaction in the evaluation of acute decompensating patients in a simulated environment.
METHODS: Volunteer clinicians with valid adult cardiac life support (ACLS) certification were invited to test the CERTAIN model in a high-fidelity simulation center. The first session was used to establish a baseline evaluation in a standard clinical resuscitation scenario. Each subject then underwent online training before returning to a simulation center for a live didactic lecture, software knowledge assessment, and practice scenarios. Each subject was then evaluated on a scenario with a similar content to the baseline. All subjects took a post-experience satisfaction survey. Video recordings of the pre-and post-test sessions were evaluated using a validated method by two blinded reviewers.
RESULTS: Eighteen clinicians completed baseline and post-education sessions. CERTAIN prompting was associated with reduced omissions of critical tasks (46 to 32%, p < 0.01) and 12 out of 14 general assessment tasks were completed in a more timely manner. The post-test survey indicated that 72% subjects felt better prepared during an emergency scenario using the CERTAIN model and 85% would want to be treated with the CERTAIN if they were critically ill.
CONCLUSION: Prompting with electronic checklist improves clinicians' performance and satisfaction when dealing with medical emergencies in high-fidelity simulation environment.

Background

Acute critical illness is routinely treated by highly trained staff in specialized care units. However, the initial resuscitation, the “Golden Hour” (after the initial hour following physiological insult or trauma is most crucial for successful resuscitation) can be impeded by inefficient processes of care delivery [1]. A key factor contributing to this is that critically ill patients continually generate vast quantities of clinical data [2]. This information can overwhelm providers, especially those not specifically trained to work in fast-paced, high-stress environments [3]. It is crucial to develop strategies that streamline the processes of care to minimize clinical misjudgment [4].

Checklists are a validated tool for dealing with such challenges; long adopted by aviation and nuclear industries, they have recently been demonstrated invaluable in standardizing and improving clinical care [5]. Haynes et al. showed a decreased in complications and 30-day mortality after non-cardiac surgery by implementing a surgical safety checklist [6].

However, checklist implementation has been slow in many settings, often due to a perception that the tool disturbs existing workflows. One approach to reduce the disruption is the presence of an additional team member, a checklist “prompter” specifically tasked with ensuring checklist completion. This has been tested during ICU [7] rounds, and it was associated with decreased mortality and shorter length of stay [8].

Proper care and timely interventions are critical in initial resuscitation [1, 9–12] especially in situations with diverse provider background and training [13]. Algorithms like advance cardiac life support (ACLS) and advance trauma life supports (ATLS) have been created to structure the care provided in emergent cases. However, none of these algorithms address the more frequently encountered clinical problems like altered mental state, respiratory distress, syncope, and sepsis with a prescribed structured approach. Furthermore, these packages are based on memorizing and recall rather than providing easy access to informational cues to guide the resuscitation workflow.

It seems evident that providing an interface capable of summarizing this information and providing decision support and validated management algorithms would address a critical need [14]. We have recently designed and developed a novel electronic tool, the Checklist for Early Recognition and Treatment of Acute Illness (CERTAIN), to apply a standard approach to evaluation and management of the acutely decompensating patient which included process of care workflows and a designated prompter [8]. We hypothesized that a standardized approach to the evaluation and management of acutely decompensating patients using the CERTAIN model would improve clinicians’ performance and satisfaction in a simulated acute care environment.

Methods

Study participants

Participants were recruited from the trainees and staff of a tertiary care teaching medical center in Rochester Minnesota. Subjects included medical students, medical and surgical residents and fellows, nurse practitioners, and physicians. To ensure a common minimum level of competence and training, all volunteers were required to have ACLS/BLS certification to be able to participate. All subjects consented to being recorded and having their performance analyzed as part of this study and the Institutional Review Board approved the study protocol. Each participant attended two sessions, a baseline evaluation without the use of the CERTAIN and a second session with the CERTAIN after receiving training.

Study setting

The study was performed at a high-fidelity multidisciplinary simulation center equipped with technologically advanced mannequins programmed to show complex findings and react just as a patient would to treatment decisions and with video/audio recording capabilities [15].

Study design

Subjects came to the simulation center in groups of two or three for baseline testing. During the first session, the subjects would be fully oriented to the simulation center, the mannequin’s capabilities, and the study goals and expectations. After that, each was asked to perform evaluation and treatment of a simulated patient based on a standard 10-min clinical scenario representing close to 30 min of “real-time” action. During the scenario, two members of the research team would act as confederates, playing the role of nurse, respiratory therapist, or other resuscitation staff as needed. This study staff would perform actions such as starting IVs, obtaining labs or studies, and providing background as needed. In this role, they were instructed to act only on commands given by the team lead, not taking independent decisions on their own. To enhance the fidelity of the simulation, standard delays were part of each diagnostic test, e.g., laboratory results would be made available 2 min after the provider requested to reflect point-of-care laboratory testing. All encounters were videotaped, and later, they were scored based on the proportion of critical tasks completed in that particular scenario. Participants were provided with didactic material to be reviewed on their individual time consisting of a presentation describing the CERTAIN method and work flow, “knobology” video that described how to use the CERTAIN software, and an example video of a simulation case where research personal modeled the used of the CERTAIN model on a standardized clinical scenario. They were also provided with access to the CERTAIN software for their own independent practice in order to get familiarized with the use of the tool (Fig. 1).

Fig. 1

Graphic depiction of study design

Participants were asked to come back to the simulation center for a final session in groups of two to three clinicians in a minimum of 2 weeks for “washout” time. These 90-min sessions consisted of a brief 20-min didactic session, where we reviewed the most important components of the CERTAIN methodology and use, followed by a practical examination. The practical examination evaluated the participant’s ability to navigate the tool before allowing hands on the use of the CERTAIN in the simulator. A minimum passing grade of 80% was required before moving on. The participants were then allowed to practice with two to three nongraded scenarios to experience the team leader and prompter roles. A debrief period of 5 to 10 min was done after each practice session in order to give positive or corrective feedback as indicated. A final videotaped testing scenario was then evaluated, where the scenario was designed to reflect the same critical tasks as the baseline evaluation for that subject. Each scenario was designed to have two variations with similar clinical progression and scorable points but disguised with a different clinical vignette to minimize recall bias (see Additional files 1 and 2). Even though the clinicians were given feedback for usability, the actual testing components were not revealed to them at any point, prior to the final test. Ultimately, the participants were asked to complete an online survey regarding their opinion of the CERTAIN method on clinical practice [16] (see Additional files 3 and 4).

Statistical analysis

All video recordings were evaluated by two independent reviewers. In cases of disagreement, the reviewers had the opportunity to replay the video and clarify based on discussion. However, if disagreement persisted, a senior critical care physician, using the definition on the SOP, would review the video and adjudicate disagreements. Each scorable item was graded as either “done” or “not done,” and the time from simulation start to item completion was recorded. The data were analyzed with the use of JMP statistical software (JMP Version 7. SAS institute Inc. Cary, NC, 1989–2012). All reported P values are two-sided, and P values of less than 0.05 were considered to indicate statistical significance. Each subject served as its own control, so paired comparisons using McNemar and Wilcoxon signed rank test were used as appropriate. Pooling data analysis of total task, specific, and generic data was also presented as indication of group performance.

Results

From 24 participants enrolled in the simulation study, 18 completed both baseline and follow-up assessments; 11 critical care fellows, two residents, two visiting clinicians, and three medical students. Two thirds of the study participants were clinicians with at least 6 months of formal critical care training and were considered with high level of expertise.

Prompting with the CERTAIN effectively decreased the number omissions in both general (15 vs 29%, P < 0.01) and scenario-specific tasks (42 vs 59%, P< 0.01). The proportion of individual task completion is presented in Table 1. The overall pooled analysis of task completion was higher using the CERTAIN process, with omissions dropping by approximately one third (46 vs 32%, P< 0.01).

Table 1

Proportion of individual task completion with and without CERTAIN prompting

Item	Without CERTAIN (N = 13)	With CERTAIN (N = 18)	P value
Code status discussion	7 (39%)	12 (67%)	0.09
Airway assessment	10 (56%)	18 (100%)	*< 0.01
Breathing assessment	18 (100%)	18 (100%)	0.99
Cardiac assessment	18 (100%)	18 (100%)	0.99
Disability assessment	14 (78%)	13 (72%)	0.50
Exposure assessment	9 (50%)	15 (83%)	0.05
Evaluation of vital sign	18 (100%)	18 (100%)	0.09
Evaluation of temperature	10 (56%)	13 (72%)	0.23
Review of past medical history	12 (67%)	12 (67%)	0.62
Review of home medication	7 (39%)	11 (61%)	0.14
Review of allergies	8 (44%)	16 (89%)	*< 0.01
Order initial basic lab test	17 (94%)	18 (100%)	0.50
Start oxygen supplementation	18 (100%)	18 (100%)	0.99
Review of differential diagnosis	14 (78%)	15 (83%)	0.50
Total omissions (mean ± SD)	13 ± 4.3 (71%)	15 ± 2.7 (85%)	*0.01

*Statstically signficant p-value

Prompting with the CERTAIN leads to faster completion rate in most key assessment tasks (12 out of 14), suggesting a more efficient care on the group using the tool (Table 2). In the subgroup analysis by level of previous training, both groups (novice and expert) had similar improvement on the general tasks (expert n = 11, 3.5 vs 1.8 task omissions, P < 0.01; novice n = 7, 4.9 vs 2.4 task omissions, P = 0.03).

Table 2

Time to task completion on those cases that have task completion on the pre- and post-intervention test

Item	Number of cases available for assessment	Mean time change in seconds	P value
Code status discussion	5	− 181	0.04
Airway assessment	10	− 114	0.07
Breathing assessment	18	− 41	0.16
Cardiac assessment	18	− 92	0.05
Disability assessment	11	− 166	0.16
Exposure assessment	7	51	0.39
Evaluation of vital sign	18	− 31	0.56
Evaluation of temperature	8	− 73	0.35
Review of past medical history	7	51	0.34
Review of home medication	5	− 91	0.31
Review of allergies	8	− 5	0.92
Order initial basic lab test	17	− 79	*< 0.01
Start oxygen supplementation	18	− 53	0.27
Review of differential diagnosis	13	− 49	0.47
Total time to task completion	11.7 ± 5.2	− 62.4 ± 68	*< 0.01

*Statstically signficant p-value

Upon completion of their participation on the clinical scenarios, the volunteers completed a survey regarding their impression using the CERTAIN method to evaluate and treat critical ill patients. Seventy-two percent of them felt well prepared (four or more on a 5-point scale) when using the CERTAIN model. Eighty-three percent indicated that they would want to be treated by the CERTAIN model if they were critically ill. However, only one third (six of 18) of the subjects thought the software was easy to use (see Table 3).

Table 3

Survey result

Survey statement	Response score (N = 18)	Percentage of score with four points or higher (%)
The CERTAIN approach helps you feel better prepared during the emergency scenario	4.2 ± 1.0	72
The CERTAIN software was easy to use	2.7 ± 1.3	33
I would want to be treated by CERTAIN approach if I were critically ill or injured	4.2 ± 0.9	83
I think that checklist are useful in a medical emergency	4 ± 0.9	67

Discussion

In a high-fidelity simulation environment prompting with the CERTAIN improved clinical task completion and decreased omission rates of critical tasks. Clinician satisfaction was high, with majority of participants wanting to incorporate this method to their own clinical practice.

Traditionally, resuscitation teams have been formed and trained to perform under stressful situations where they have to rapidly coordinate evaluation and treatment efforts [17, 18]. However, they have been structured around specific types of illness like trauma or cardiac arrest and largely rely on memory recall [18, 19]. Our experience demonstrated the performance of clinicians under stress is suboptimal at best, with an overall task completion rate of 50% at baseline. This is consistent with the findings of Smith et al. showing a decline in skill retention and loss of ability to perform ACLS and BLS skills to standard level when re-tested at 12 months [20].

As a potential solution, other studies have evaluated memory aids to improve health team performance in other situations. Haynes et al. tested the use of a checklist applied to regular operating room workflow on elective surgeries. His work showed a decrease on preventable surgical-related complications in the operating room [6]. Two studies designed for emergency teams dealing with late-phase resuscitation, one with a smart phone application [21] and the other a traditional checklist [22], had promising results in simulated environments. However, their designs did not include a prompter, which likely reduced their team’s compliance. Prompting with the CERTAIN is aimed to approach critical illness resuscitation earlier in the natural course of the disease, targeting to standardize the care on the so-called Golden hour. Early structured treatment has been shown to give better outcomes in simulated operating room crises [16], sepsis [23], myocardial infarction [24], and other critical illness states and providing a unified approach to decompensation may prevent the need of cardiopulmonary resuscitation efforts.

In a different setting, Weiss et al. tested the usefulness of prompting in critical care practice by implementing the use of checklist with a prompter versus checklist alone during daily ICU rounds. This study showed improvement in compliance with process of care, decreased length of stay, and a decrease in mortality [7]. These findings clearly demonstrate that important role prompting can play in facilitating complex process of care. The present study differs, however, in using a prompter in a higher stress environment with simulated acute medical emergencies. This stress imposes an extra burden on providers which could increase the risk of task omission. In this sense, the choreography of the CERTAIN with a prompter combines the lessons of prior studies on checklists and prompting with leadership “best practices” [25], such as egalitarian leadership [26] and closed-loop communication [27].

Ideally, one of the existing team members should be able to play a role of prompter, without any added cost. However, in resource constraint situations, the team lead should focus on resuscitation and as soon as the time allows review the checklist to see if anything has been missed.

The CERTAIN approach was well received by the participants. However, it is worth noting that software usability limitations were evident in the post-intervention survey. With only 33% of participants feeling the software is easy to use, a combination of improved training and interface may be necessary to make this practical in real high-stress environments. In this simulation study, due to volunteers’ time and schedule constraints, the training was limited to 90 min. Most initial training in clinical resuscitation models (ACLS, BLS, or ATLS) are 12 h courses (usually two full training days) [28].

Another potential limitation could be that, even though the didactic sessions and practice scenarios were focused on tool usability rather than the scenario performance, having these done just prior to final testing could have influenced the performance.

The inferences from our results are further limited due to the simulation nature of this study, as well as the small sample size. Simulation training has been increasingly suggested as a valid research and training tool paired with good outcomes which makes it the ideal scenario to test a new method and clinical software [29]. The spectrum of scenarios encountered by our test subjects was limited to three common types: respiratory distress and hypoxia, hypotension due to severe sepsis/septic shock, and chest pain secondary to acute coronary syndrome. These cases were chosen as they are the most common clinical presentations in hospitalized medical patients [30-33]. Each clinician serving as his/her own control minimized the effect of variability in general medical knowledge. However, the absence of the control group of clinicians who were simply re-tested may limit the ability to discern the effectiveness of CERTAIN prompting vs training.

Conclusion

Prompting with an electronic checklist (CERTAIN) improves clinical performance, subjective perceptions, and confidence of bedside clinicians confronted with typical emergency medical scenarios in high-fidelity simulation environment. Enhanced design/usability and better training are needed to leverage potential benefits of electronic checklist at the bedside of critically ill patients.

Structure of the study design. (DOCX 46 kb)

Example of standardized clinical scenario. (DOCX 99 kb)

Satisfaction survey: a post-CERTAIN survey. (DOCX 33 kb)

Snapshot of CERTAIN tool. (DOCX 122 kb)