Comparison of methods of alert acknowledgement by critical care clinicians in the ICU setting.

BACKGROUND: Electronic Health Record (EHR)-based sepsis alert systems have failed to demonstrate improvements in clinically meaningful endpoints. However, the effect of implementation barriers on the success of new sepsis alert systems is rarely explored.
OBJECTIVE: To test the hypothesis time to severe sepsis alert acknowledgement by critical care clinicians in the ICU setting would be reduced using an EHR-based alert acknowledgement system compared to a text paging-based system. STUDY
DESIGN: In one arm of this simulation study, real alerts for patients in the medical ICU were delivered to critical care clinicians through the EHR. In the other arm, simulated alerts were delivered through text paging. The primary outcome was time to alert acknowledgement. The secondary outcomes were a structured, mixed quantitative/qualitative survey and informal group interview.
RESULTS: The alert acknowledgement rate from the severe sepsis alert system was 3% (N = 148) and 51% (N = 156) from simulated severe sepsis alerts through traditional text paging. Time to alert acknowledgement from the severe sepsis alert system was median 274 min (N = 5) and median 2 min (N = 80) from text paging. The response rate from the EHR-based alert system was insufficient to compare primary measures. However, secondary measures revealed important barriers.
CONCLUSION: Alert fatigue, interruption, human error, and information overload are barriers to alert and simulation studies in the ICU setting.

Introduction

Electronic health record (EHR)-based, automated sepsis alert systems have failed to demonstrate improvements in clinically meaningful endpoints, such as Intensive Care Unit (ICU)/hospital length of stay (LOS) and mortality (Hooper et al., 2012; LaRosa et al., 2012; Nelson et al., 2011; Sawyer et al., 2011). This includes studies of ICU-specific and non-ICU specific alert systems. This also includes studies ranging in variation regarding degree of distinction between the detection of sepsis, severe sepsis, and/or septic shock (Dellinger et al., 2013). Clinically meaningful endpoints range from compliance with the international Surviving Sepsis Campaign (SSC) guidelines to hospital LOS, ICU LOS, and mortality. There are ICU-based and hospital wide means to trigger an alert for the early recognition of sepsis. Most EHRs now have a built in system to support this alert

Time to alert acknowledgement has been validated as one proxy for time to recognition of sepsis by critical care clinicians (Dziadzko et al., 2016). The failure of EHR-based, automated sepsis alert systems to be directly correlated with improvements in clinically meaningful endpoints is frequently attributed to limitations of detection algorithms and/or the need for clinical decision support (CDS) systems (Semler et al., 2015). Human factors, such as the impact of workflow changes or the influence of method of alert delivery, are known to be barriers to the implementation of new alert systems in the clinical setting (Harrison, Herasevich & Gajic, 2015).

As “alarm hazards” have been ranked as the top health technology hazard in the United States (ECRI-Institute, 2013), it is important to explore the effect of implementation of new alert systems on workflow changes and other human factors in the clinical setting. Despite outcome improvements in recent decades (Kaukonen et al., 2014), sepsis remains one of the most expensive in-hospital conditions (Torio, 2013). As one of the most technologically sophisticated hospital environments, the critical care setting serves as a model to explore the impact of implementation of new alert systems. Despite these technological advances, including widespread utilization of text messaging and smartphones by clinicians, text paging remains standard practice in the hospital setting (HIMSS Analytics, 2016). We hypothesized that time to severe sepsis alert acknowledgement by critical care clinicians in the ICU setting would be reduced using an EHR-based alert acknowledgement system compared to a text paging-based system.

Methods

Study design and setting

This study was performed in February 2015 in the medical ICU at Mayo Clinic in Rochester, MN, USA (Fig. 1). This study was not performed in a laboratory setting or simulation environment. This medical ICU has been described previously as a benchmark for evaluation of institutional performance (Afessa et al., 2005). Severe sepsis alerts were delivered to critical care clinicians, including attending physicians, fellows, residents, and nurse practitioners/physician assistants (NPs/PAs) using traditional HIPAA-compliant text paging. This study was approved by the Mayo Clinic Institutional Review Board (IRB) for clinician-participant enrollment by oral consent.

Figure 1

Schematic illustration of study design.

Study participants and medical ICU workflow

The medical ICU at Mayo Clinic consists of two physically adjacent 12-bed units, in close proximity to a nearby 9-bed respiratory care unit (RCU). For any given month, there are approximately 15 critical care attending physicians, six critical care fellows, four postgraduate year three internal medicine residents (PGY-3), six PGY-1 interns, and nine dedicated medical ICU NPs/PAs. There are 2 shifts: 6 am–6 pm (AM) and 6 pm–6 am (PM). On any given day, the morning shift is further divided into two teams. Team 1 is assigned to the majority of the medical ICU patients. Team 2 is assigned the remaining patients, as well as the RCU, which is further staffed by an additional fellow and dedicated NP/PA from the same group of approximately 40 clinicians in the medical ICU that month.

AWARE (Ambient Warning and Response Evaluation)

AWARE is the ICU-specific EHR system used in this study for patient viewer/monitoring. It was developed at Mayo Clinic and has been in routine clinical use in the medical ICU at Mayo Clinic since July 2012 (Pickering et al., 2015; Pickering et al., 2010). AWARE has been demonstrated to improve clinician task load, errors of cognition, and performance (Ahmed et al., 2011). AWARE is accessible from every computer workstation in this medical ICU, including bedside desktops, nursing stations, and clinician workrooms.

Severe sepsis alert system

The severe sepsis detection algorithm was developed at Mayo Clinic and implemented into AWARE in December 2014 (Harrison et al., 2015). Within AWARE, the severe sepsis alert system displays a passive, yellow alert icon when severe sepsis is detected. This yellow alert icon can also be activated manually by clinicians for specific patients, when severe sepsis is suspected, but not detected by the automated alert system. This yellow alert icon is automatically updated to display a passive, green alert icon within AWARE after completion of the four elements of the 3-hour Surviving Sepsis Campaign (SSC) bundle (Dellinger et al., 2013). Once activated, the yellow alert icon will persist for at least 6 h, unless completion of the 6 h SSC bundle is detected (green alert icon) or manual deactivation by clinicians occurs. In the context of prolonged severe sepsis and/or septic shock, the yellow alert icon can persist indefinitely. The green alert icon automatically reverts back to “no sepsis detected” after 3 h, unless additional automatic (or manual) activation occurs.

Study procedures

Clinicians agreed to participate in this study from February 02 through February 28 in 2015. The evening before each AM shift and the next PM shift, clinician participants for these upcoming shifts received a detailed email reminder with instructions (Fig. 2). The number of severe sepsis system alerts per shift through AWARE (yellow or green icon alerts) was entirely dependent on the number of septic patients in the medical ICU during any specific shift. Clinician participants randomly received no more than 3 simulated severe sepsis alerts per shift via text paging. In both cases, clinician participants were instructed to acknowledge all AWARE and traditional text paging severe sepsis alerts by email response. The difference between the time to severe sepsis alert activation in AWARE (or alert delivery via text page) and email response was defined as the time to alert acknowledgement.

Figure 2

Detailed daily email reminder to clinician participants with complete instructions.

Survey design

To compare clinician satisfaction between the EHR-based alert acknowledgement system and text paging-based system, clinician participants completed a structured, mixed quantitative/qualitative survey, upon completion of the severe sepsis alert acknowledgement portion of this study (full survey facsimile in Results, Fig. 3). These questions were designed partially on existing clinician satisfaction surveys of alert methods for use in the hospital and critical care settings (Embi, Jain & Harris, 2008; Wagner et al., 1998).

Figure 3

Facsimile of the structured, mixed quantitative/qualitative survey provided to the clinician participants with all quantitative results overlaid: median (IQR).

Statistical analysis

Severe sepsis alert system data was extracted directly from AWARE using METRIC Data Mart, a near-real time relational database of the complete EHR, which was developed at Mayo Clinic and has been described previously (Herasevich et al., 2010). Data was queried using JMP Pro (SAS Institute, Inc). Data collection and statistical analyses, such as the two-sided Student’s t-test and the Chi-squared test, were also performed in JMP Pro. For all statistical analyses, a p-value of less than 0.05 was considered to be statistically significant. For all median values from the survey results, interquartile range (IQR) was reported.

Results

Prior to initiation of this study, a 1-day feasibility pilot was performed in January 2015 using seven medical ICU clinicians. Based on the result of this feasibility study (data not shown), it was determined that a sufficiently high clinician participant alert acknowledgement rate could be obtained from both severe sepsis system alerts through AWARE (yellow or green icon alerts) and simulated severe sepsis alerts through traditional text paging in the ICU setting. Based on the results of this feasibility pilot, participant instructions were optimized (Fig. 2).

Of the 40 clinicians staffing the medical ICU in February 2015, 13 (32%) were recruited to participate in this study. However, it was decided after two weeks (February 02 AM through February 15 PM) to prematurely terminate this study, due to sufficient statistical power for time to alert acknowledgement analysis, as well as feedback from clinician participants. As a result, it was necessary to exclude one NP/PA due to unavailability in the medical ICU during this shortened study period (RCU only). Ultimately, 12 clinicians participated: five NPs/PAs (out of nine), three attending physicians (out of 15), two fellows (out of six), two PGY-3s (out of four), and zero PGY-1s (out of six). The median number of potential AWARE alert acknowledgements per shift was two (IQR 1–4). The minimum and maximum numbers were zero and five. The number of patients who trigged at least 1 severe sepsis system alert through AWARE (yellow or green icon alert) was 28. Of the 28 shifts that occurred during this shortened study period, 23 shifts (82%) were covered by at least one participant (Table 1).

Table 1

Number of shifts per clinician participant and number of participants per shift.

	Total AM Shifts	Total PM Shifts	Total Shifts	Shift, part 1	Number of Providers	Shift, part 2	Number of Providers
NP/PA #01	5	–	5	02/05 Thu PM	1	02/12 Thu PM	1
NP/PA #02	4	–	4	02/06 Fri AM	2	02/13 Fri AM	4
NP/PA #03	3	1	4	02/06 Fri PM	2	02/13 Fri PM	–
NP/PA #04	3	–	3	02/07 Sat AM	1	02/14 Sat AM	6
NP/PA #05	1	–	1	02/07 Sat PM	2	02/14 Sat PM	–
Attending #01	7	–	7	02/02 Mon AM	4	02/09 Mon AM	2
Attending #02	1	–	1	02/02 Mon PM	–	02/09 Mon PM	1
Attending #03	–	1	1	02/03 Tue AM	4	02/10 Tue AM	5
Fellow #01	7	2	9	02/03 Tue PM	1	02/10 Tue PM	–
Fellow #02	6	–	6	02/04 Wed AM	3	02/11 Wed AM	5
Resident #01	5	5	10	02/04 Wed PM	1	02/11 Wed PM	–
Resident #02	7	2	9	02/05 Thu AM	2	02/12 Thu AM	4
				02/08 Sun AM	3	02/15 Sun AM	4
				02/08 Sun PM	1	02/25 Sun PM	1
	49	11	60		27		33

The alert acknowledgement rate from the severe sepsis alert system through AWARE was 3% (N = 148) and 51% (N = 156) from simulated severe sepsis alerts through text paging (Table 2). Time to alert acknowledgement from the severe sepsis alert system through AWARE was median 274 min (N = 5) and median 2 min (N = 80) from simulated severe sepsis alerts through text paging. The 5 alert acknowledgements from the severe sepsis alert system through AWARE came from only three clinician participants (NP/PA #01, NP/PA #04, and NP/PA #05), while all 12 participants acknowledged at least one simulated severe sepsis alert through text paging.

Table 2

Comparison of alert response rate and median time to alert acknowledgement between the severe sepsis alert system through AWARE and simulated severe sepsis alerts through traditional text paging.

	Text paging (N = 156)	AWARE (N = 148)	p-value
Alert response rate (N)	51% (80)	3% (5)	0.001
Median time to alert acknowledgement (IQR)	2 mins (1–32)	274 mins (130–517)	0.053

All participants completed a structured, mixed quantitative/qualitative survey. For the quantitative portion of the survey (Fig. 3), clinicians found alert by AWARE to be slightly less disruptive than alert by text paging. Clinicians found acknowledgement of AWARE and text paging alerts to be equally disruptive. When AWARE and text paging alerts were directly compared, a clear preference for text paging for both “urgent” and “non-urgent” alerts was present. When asked to “select one or more” (text paging, AWARE, email, phone call, text message, or other), the results for non-urgent alerts were mixed. However, when asked the same question for urgent alerts, the preference was once again clearly for text paging.

For the qualitative portion of the survey (Fig. 4), 11 out of 12 clinician participants provided “at least one suggestion for improving alert/notification delivery”. Clinicians commented on inhomogeneous overall use of AWARE in the medical ICU, despite implementation several years prior (July 2012). Of the same 11 clinicians, four provided “any additional comments”: the same three NPs/PAs who responded to at least one alert acknowledgement from the severe sepsis alert system through AWARE, as well as Attending #03. A clear theme concerning alert fatigue, interruption, human error, and information overload was present.

Figure 4

All qualitative responses to the structured, mixed quantitative/qualitative survey reproduced in their entirety, including typographical errors.

An informal group interview in the form of a noon pizza party was held to thank all clinician participants and gather additional feedback on the barriers to clinician participation and engagement in this implementation study. The four clinicians who attended were once again the same four clinicians who provided “any additional comments” on the survey. The statements regarding alert fatigue, interruption, human error, and information overload were reinforced, despite a strong interest from these clinicians to participate. Regarding inhomogeneous overall use of AWARE in the medical ICU, specific attention was drawn to a particular lack of interest from residents to use AWARE, as well as a lack of interest from both residents and fellows to participate in any research study during their required rotations through the medical ICU, including implementation of the severe sepsis alert system.

Discussion

We hypothesized that time to severe sepsis alert acknowledgement by critical care clinicians in the ICU setting would be reduced using an EHR-based alert acknowledgement system compared to a text paging-based system. Based on the limited alert acknowledgement response rate using the severe sepsis alert system compared to traditional text paging, it was not possible to answer this hypothesis. However, feedback from the structured, mixed quantitative/qualitative survey, as well as the informal group interview, provided invaluable insight into the sources of this limited acknowledgement response rate. Implementation barriers included human factors, such as alert fatigue, interruption, human error, and information overload.

With the implementation of increasingly sophisticated EHR systems, interest in the development of novel automated detection and alert systems has increased (Bourgault et al., 2014). However, investigation into best methods of alert delivery (text paging, EHR systems, email, phone calls, and/or text messaging) for urgent and non-urgent alerts in the hospital setting is limited (Gill, Kamath & Gill, 2012). Investigation into the most appropriate clinician for alert delivery is also limited (Zhang et al., 2003). Monitoring and alert systems have been developed for patient use in the home setting (Steinman et al., 2011; Tchalla et al., 2012). However, there has been comparatively limited investigation into methods of alert delivery to clinicians in the hospital setting (Loo et al., 2011). Interestingly, many of these studies have been performed in the geriatric patient population, but not in the ICU setting, where the average patient age is often 65 or older (Seferian & Afessa, 2006). Thus, there is a clear need for further systematic exploration of human factors barriers to the implementation of new alert systems in the ICU setting, such as the impact of workflow changes and the influence of method of alert delivery.

Implementation of automated detection and alert systems without consideration of these factors is known to have the potential to result in alert fatigue (Singh et al., 2013), interruption (Hodgetts & Jones, 2007), human error (Bates et al., 1998), and information overload (Stokstad, 2001). Recognition of the importance of alert fatigue in the hospital setting has increased significantly in recent years (Herasevich et al., 2013). However, implementation of automated alert systems generally must be performed in the context of information overload and complex task interruption (Eppler & Mengis, 2004). It is also known that information overload can alter alert perception in the medical setting (Glassman et al., 2006). This can cause clinicians to perceive alert systems negatively and deter future use (Harrison et al., 2016). Thus, the task of generating clinically meaningful alerts while concurrently minimizing information overload and task interruption is challenging.

Clinician-participant comments provided valuable insight regarding preferences for method of alert delivery. Although there was a clear preference for receiving urgent alerts through text paging, additional investigation is required to specifically explore the rationale for this preference. Understanding the rationale for this preference may reduce the barriers to answering the primary objective of this study, which was comparison of time to severe sepsis alert acknowledgement methods by critical care clinicians in the ICU setting. These secondary outcomes revealed important barriers to the inability to answer the primary outcome, which are applicable and generalizable to future studies.

Even the reluctance of clinicians to participate in this research study, as revealed during the post-study discussion, has the potential to confound interpretation of the results of this study. Whether this reluctance is the result of information overload, the Hawthorne effect, and/or some other factor(s), clinicians are currently struggling to find a balance between understanding of the importance of their participation in clinical research studies and “study fatigue.”

This study has several limitations: (1) This was a single-center study at an academic medical center. Well-established biases and potential confounders are known to be present with this particular study design (Straus et al., 2005); (2) Unlike the severe sepsis system alerts through AWARE (yellow or green icon alerts), the severe sepsis alerts through text paging were simulated. Comparing non-simulated alerts to simulated alerts may introduce additional confounders into the interpretation of the results of this study; (3) Although not investigated in this study, the feasibility of severe sepsis alert delivery using an EHR-based, automated mobile app for smartphones has been validated (Dziadzko et al., 2016); (4) The significant range of clinical experience of clinician-participants introduces study bias. The potential application of this technology for the future of clinical practice and clinical research should not be ignored. Ultimately, a multi-center, non-simulated study in the ICU setting is required to address various aspects of these limitations.

Conclusion

It could not be determined whether an automated alert for severe sepsis reduced time to alert acknowledgement by critical care clinicians in the ICU setting compared to text paging. This was due to an extremely limited alert acknowledgement response rate using the severe sepsis alert system compared to traditional text paging. Implementation barriers, including human factors—such as alert fatigue, interruption, human error, and information overload—were determined to be an important source of this finding.