Validation of the VitalPAC Early Warning Score at the Intermediate Care Unit.

AIM: To assess the performance and clinical relevance of the Early Warning Scoring (EWS) system at the Intermediate Care Unit (IMCU).
METHODS: This cohort study used all the VitalPAC EWS (ViEWS) scores collected during each nursing shift from 2014 through 2016 at the mixed surgical IMCU of an academic teaching hospital. Clinical deterioration defined as transfer to the Intensive Care Unit (ICU) or mortality within 24 h was the primary outcome of interest.
RESULTS: A total of 9113 aggregated ViEWS scores were obtained from 2113 admissions. The incidence of the combined outcome was 272 (3.0%). The area under the curve of the ViEWS was 0.72 (CI: 0.69-0.75). Using a threshold value of six, the sensitivity was 68% with a positive predictive value of 5% and a number needed to trigger (e.g., false alarms) of 19%.
CONCLUSION: The ViEWS at the IMCU has a discriminative performance that is considerably lower than at the hospital ward. The number of false alarms is high, which may result in alarm fatigue. Therefore, use of the ViEWS in its current form at the IMCU should be reconsidered.

Core tip: This study used all the routinely collected Early Warning Scores (EWS) in every nursing shift from 2014 to 2016 (n = 9113) at the standalone Intermediate Care Unit to assess the performance and clinical relevance of the EWS to detect clinical deterioration amongst patients admitted in this critical care facility. It follows that although the discriminative performance was acceptable (AUC 0.72), the clinical relevance is limited as 19 false alarms were needed to detect one event. As this may result in alarm fatigue, its use in this setting should be reconsidered.

INTRODUCTION

In the past decade, increasing attention has been raised for the concept of Early Warning Scoring (EWS) systems, in order to timely detect the clinically deteriorating patient in need of intervention[1]. Frequently the EWS is used, combined with structured communication systems to allow adequate information transfer[2]. EWS systems were initially developed and validated at general hospital wards, but are currently also integrated in other settings, such as the emergency department and the prehospital setting[1,3-6]. It is however unclear if the performance of the EWS obtained in the general ward is representative for such settings, as differences in case-mix between settings are likely present[7].

One setting where EWS is used, but to our knowledge not validated, is the Intermediate Care Unit (IMCU). Although the exact extent to which EWS systems are applied at IMCUs has not been investigated, EWS systems are widely applied in the hospital setting, for example within a broad national safety program in The Netherlands[8], and the National Health Service in the United Kingdom[9]. As within the continuum of care the IMCU is positioned between the hospital ward and the intensive care unit[10,11], it has a remarkably different case-mix, which is likely to influence the performance of the EWS. Also, the haemodynamic and respiratory support provided at IMCUs directly influences the vital parameters used in (most) EWSs, and may affect its performance[10].

This validation study aims to assess the performance and clinical relevance of the EWS at the IMCU. Additionally, it investigates the potential of several IMCU-specific clinical parameters to improve the performance of EWS within this setting.

MATERIALS AND METHODS

Data sources and participants

This cohort study used prospectively collected data of all admissions at the surgical IMCU of the University Medical Centre in Utrecht, a tertiary referral hospital in the Netherlands from the period of January 1st, 2014 through December 31st, 2016. This stand-alone, mixed surgical IMCU admits patients from all surgical disciplines (except cardiothoracic surgery and neurosurgery), providing haemodynamic monitoring and cardiovascular- and respiratory-support, including inotropic use and supplementary (high flow) oxygen administration. Multiple vasoactive medications, haemodialysis, and mechanical ventilation are not performed at this IMCU. According to the Institutional Review Board the study was not subject to the Medical Research Involving Human Subjects Act and therefore the necessity of informed consent was waived (protocol No. 17-326/C).

Clinical parameters

Because the VitalPAC-Early Warning Score (ViEWS) has the highest discriminative ability compared to 34 other EWS[1], we chose to validate this EWS. The ViEWS is based on the heart frequency, respiratory rate, temperature, systolic blood pressure, oxygen saturation, inspired oxygen (yes/no) and consciousness (supplementary material). As a threshold, it often uses an aggregated score of 5[4]. In our hospital, a slightly modified score was used in routine care, to which we will refer to as the University Medical Center Utrecht-EWS (UMC-EWS). This UMC-EWS used almost similar parameters as the ViEWS. Therefore, the performance of this score (with the currently used threshold of 3) was also assessed. Additionally, possible clinically important IMCU-specific predictors were added to the ViEWS.

Data collection

The parameters required for the UMC-EWS were collected in the Electronic Health Record in every nursing shift (every 8 h), as required by the Dutch authorities. The blood pressure and saturation were additionally required to calculate the ViEWS. These were collected from nursing measurements within 2 h before the UMC-EWS measurements.

Additional clinical predictors collected were noradrenalin (norepinephrine) use (μg/kg per minute), oxygen use (L/min), nurse worry (yes/no), urine output (≤ 75 mL over the last 4 h) and the change in ViEWS, which was the ViEWS minus the previous ViEWS at the IMCU (within two nursing shifts, defined as within 18 h). Noradrenalin (μg/kg per minute) and oxygen (L/min) were collected from nursing measurements within 2 h before the UMC-EWS measurements. Nurse worry (yes/no) and urine output (≤ 75 mL over the last 4 h) were part of the UMC-EWS and hence, did not require additional data collection.

Outcome

The primary outcome was clinically relevant deterioration, defined as transfer to the Intensive Care Unit (ICU) or death within 24 h of the EWS measurement. Secondary outcome was transfer to the ICU or death within 12 h.

Statistical analysis

Descriptive statistics presented are the mean and 95% CI for normally distributed continuous variables, and the median and interquartile range (IQR) for non-normally distributed continuous variables[12]. Categorical variables are described as proportions. The aggregated ViEWS was calculated from the separately collected parameters from the UMC-EWS and the nursing measurements. To assess the discriminative performance of the ViEWS score, i.e., the extent to which deteriorating patients can be distinguished from patients who are not deteriorating, we calculated the area under the receiver operating characteristics curve (AUROC) with its 95% CI. Subsequently, the optimal threshold value was deducted from the ROC curve by the Q-point method (i.e., the threshold value with the minimum distance to the upper left corner). For this threshold value, the parameters sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and number needed to trigger (NNT) were calculated. The latter was of interest to assess the possibility of alarm fatigue[13]. Furthermore, to analyse if the discriminative performance could be improved, other possible clinical predictors were, first separately and then jointly, added to the ViEWS in logistic regression analyses. To this end, the ViEWS was included as an offset term. For these models, again discriminative performance measures using the optimal threshold were calculated. As the amount of missing data was small, complete case analyses were performed. To determine whether the occurrence of multiple measurements per admission affected the discriminative performance, an additional analysis was performed using only the first observation per IMCU admission. As the UMC-EWS was not always performed at 8 h intervals, another additional analysis was performed to determine the effect of the measurements at irregular time intervals. To this end, only regular time interval measurements (within 6-10 h after the previous measurements, thus in the consecutive nursing shift) were taken into account for this analysis. All statistical analyses were performed using R software for statistical computing version 3.3.2[14]. Where applicable, the reporting of this article follows the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) statement[15].

RESULTS

Participants

During the study period, there were 2113 admissions (1782 different patients) at the IMCU with a total of 9113 aggregated EWS. The mean age of the patients was 61.2 (SD 17.0) years. In total, 1150 (64.5%) patients were male.

The median ViEWS was 5 (IQR 3-6). Most common were ViEWS of 3 (n = 1754, 19.3%), 4 (n = 1500, 16.5%), 5 (n = 1398, 15.4%) and 6 (n = 1324, 14.5%). The median interval between subsequent EWS was 8.6 h (IQR 6.8-13.4). This was 8.1 h (IQR 6.4-11.5) for observations followed by ICU transfer within 24 h, as compared to 8.6 h (IQR 6.8-13.5) for observations that were not followed by ICU transfer within 24 h.

Figure 1 shows the percentages of observations for the separate ViEWS parameters. It should be noted that during almost all of the observations (n = 7530, 82.63%) patients received oxygen, hence nearly always three points were received for this item. There were 6602 (72.4%) observations during which patients received oxygen through a nasal cannula, 748 (8.2%) observations during which patients received oxygen through a non-rebreathing mask and 180 (2.0%) observations during which patients received oxygen through a high-flow nasal cannula.

Figure 1

Frequency of Scores for Separate VitalPAC Early Warning Score Parameters. This figure shows the relative contribution of separate VitalPAC Early Warning Score (ViEWS) Parameters towards the aggregated scores at the Intermediate Care Unit. bpm: Breaths per minute; BP: Blood pressure; CNS: Central nervous system; AVPU: Alert, verbal, pain, unresponsiveness scale.

There were 687 (7.5%) observations during which patients received noradrenalin (251 admissions). Nurses indicated to be worried in 895 (9.8%) of observations, and reported a decreased urine output (≤ 75 mL over the last 4 h) in 304 (3.3%) observations. The mean change in ViEWS was -0.15 (SD 2.29) per subsequent measurement, using 6115 (67.10%) measurements.

The number of observations that were followed by clinical deterioration (ICU transfer or mortality within 24 h) was 272 (3.0%). Of these, 248 (90.1%) were ICU transfers within 24 h and 27 (8.8%) were deaths within 24 h. In three (1.1%) observations, both outcomes occurred within 24 h after the observation.

Figure 2 shows the percentage and number of observations per ViEWS and the event rate within 24 h. Two observations with a ViEWS of 16 were not followed by an event. During these observations, one admission was a patient who had sepsis due to pneumonia who stabilized, until secondary deterioration (respiratory insufficiency) and ICU transfer 3 d later. The other admission was a patient who suffered from respiratory insufficiency due to cardiac failure and pneumonia with a restricted policy of no ICU admission.

Figure 2

Distribution of VitalPAC Early Warning Score and transfer to the Intensive Care Unit rates. This figure shows distribution of the VitalPAC Early Warning Score aggregated values (bar chart) in relation with the occurrence of the event (ICU transfer or mortality within 24 h), with 95%CIs. ICU: Intensive Care Unit.

Discriminative performance

The AUROC of the ViEWS for the prediction of clinical deterioration within 24 h was 0.72 (0.69-0.75). For a threshold of five (as is recommended for hospital wards[9]), the sensitivity was 79%, PPV 4%, NPV 99%, and NNT 23. For the optimal threshold value at the IMCU of 6, the sensitivity was 68%, with a PPV of 5%, NPV 98% and the NNT 19. For the outcome ‘clinical deterioration within 12 h’ the AUROC was 0.74 (0.70-0.79).

The AUROC of the hospital-adjusted EWS (UMC-EWS) was 0.71 (0.67-0.74). With the currently used threshold value of 3, the sensitivity of UMC-EWS was 58%, specificity was 73%, PPV 6%, NPV 98% and NNT 17.

Addition of all the variables noradrenalin use, oxygen (L/min), nurse’s worry, urine output (≤ 75 mL over the last 4 h) and the change in ViEWS, increased the AUROC to 0.78 (0.74-0.81), with a NNT of 11 at the optimal cut-off value (Table 1).

Table 1

Predictive Performance of the regular VitalPAC Early Warning Score and updated models at the Intermediate Care Unit

Predictors	AUROC	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	NNT
ViEWS, with regular threshold (5)	0.72 (0.69-0.75)	79	49	4	99	23
ViEWS, with optimal threshold (6)	0.72 (0.69-0.75)	68	64	5	98	19
ViEWS and noradrenalin	0.72 (0.69–0.75)	68	64	5	98	19
ViEWS and oxygen (continuous)	0.74 (0.71-0.77)	61	76	7	98	14
ViEWS and nurse worry score	0.73 (0.70-0.76)	70	63	5	99	21
ViEWS and change in ViEWS	0.74 (0.70-0.78)	65	70	7	99	16
ViEWS and noradrenalin, nurse worry score, oxygen (continuous) and change in ViEWS	0.78 (0.74-0.81)	62	78	10	99	11

The numbers presented indicate the area under the curve (AUROC), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and number needed to trigger (NNT). This is shown for the original VitalPAC early warning score model (ViEWS) with both the regularly used and optimal threshold, and for the updated models containing additional parameters, all with their optimal thresholds as determined by the Q-point method. ViEWS: VitalPac Early Warning Score; NNT: Number needed to trigger.

When analysing only the first observation of the ViEWS per IMCU admission (n = 2113), the AUROC was 0.72 (0.65-0.79) using the optimal threshold. When including only the observations that were part of a regular measurement interval (n = 5255), the AUROC was 0.70 (0.66-0.74) for the optimal threshold.

Calibration

The calibration of the ViEWS and the ViEWS with additional parameters (noradrenalin, nurse worry score, oxygen (continuous) and change in ViEWS) is shown in Figure 3. From this, it follows that the ViEWS underpredicts clinical deterioration for Probability < 0.05, while it overpredicts clinical deterioration for Probability > 0.05.

Figure 3

Calibration plots of the VitalPac Early Warning Score and the VitalPac Early Warning Score with additional parameters at the Intermediate Care Unit. This figure shows the calibration plot of the ViEWS (lower plot) and the ViEWS with the additional parameters noradrenalin, nurse worry score, oxygen (continuous) and change in ViEWS (upper plot). ViEWS: VitalPac Early Warning Score.

DISCUSSION

This study shows that the discriminative performance of the (Vi)EWS at the IMCU is moderate. However, its clinical relevance is debatable as its use carries the risk of alarm fatigue (NNT = 19).

In comparison, considerably higher AUROCS have been described for the ViEWS at the hospital ward. One study using the same composite outcome found an AUROC of 0.87 (0.87-0.87)[4]. At external validation studies AUROCs of 0.87 (no confidence interval) and 0.89 (0.83-0.95) for 24 h mortality were observed in, respectively, first world and lower-income countries[3,16]. In the pre-hospital setting an AUROC of 0.815 (0.730-0.990) was reported in external validation for the composite outcome of ICU transfer or mortality within 48 h[5]. A possible explanation for the lower discriminative performance of the (Vi)EWS at the IMCU is the considerably different case-mix when compared to the hospital ward: IMCU patients are already closely monitored and often receive haemodynamic and respiratory support. Hence, many patients achieve a relatively high ViEWS score, which diminishes the discriminative potential of the score. Furthermore, the supervising clinicians at the IMCU are trained and experienced in supporting the deteriorating patient, and thus deterioration (as measured by the EWS) may not directly lead to ICU transfer and mortality, as a large part of the resuscitation can be provided at the IMCU.

Subsequently, this study shows that the addition of IMCU-specific clinical parameters increases the discriminative performance (AUROC 0.78, CI: 0.74-0.81) and decreases the false alarm rate (NNT = 11). We did however not intend to develop and present a revised IMCU-specific EWS. First further external validation would be necessary, and second, it would require collection of extra parameters, which would make the EWS less easy to use. It should be noted that the moderate performance of the EWS was not specific for the ViEWS, as the observed performance of the UMC-EWS was lower than the ViEWS (AUROC of 0.71, CI: 0.67-0.74) in the IMCU.

Clinically, we do not consider a prediction model with a NNT of 19, i.e. only 1 out of 19 triggers, does indeed indicate clinical deterioration, to be feasible. This carries a high risk of alarm fatigue and therefore has the potential to compromise patient safety while increasing the (administrative) burden at the IMCU. In its current design, we therefore believe that it should not be used. We would regard a NNT of 4 or less as ideal (with a sensitivity of ≥ 50%), as this would indicate that at least 25% of the alarms are justified. It should be noted that the addition of the other IMCU-specific clinical parameters increased the amount of data to be collected, still resulting in a higher NNT; therefore, more detailed modelling, likely connected with automated data collection to reduce administrative burden, would be necessary to achieve this goal. Furthermore, as these additional IMCU-specific clinical parameters are used to fit a model in our own cohort, there is a risk of overfitting and thus the real performance of this updated model is likely even less.

We recognize that this clinical relevance highly depends on the organisation of the IMCU and, in particular, on the extent to which the IMCU is independent. IMCUs that are an extension of the hospital ward perhaps do benefit from the ViEWS, whereas independent (stand-alone) IMCUs with extensive monitoring and supportive options probably do not. The clinical relevance of the ViEWS at the IMCU also depends on the hospital organization, as “transfer to the ICU” is a process measure which is potentially susceptible to logistical factors (the occupancy rates at both units). Although the in-hospital mortality rate is preferably an outcome measure[1], its use as single outcome parameter was not considered feasible due to its rare occurrence (0.30%) and susceptibility to treatment restrictions. Additionally, as necessity for ICU transfer often is due to mechanical ventilation[10], its use as an outcome measure for deterioration at IMCUs can likely be generalized.

This study has multiple strengths. First, it is the first to investigate the validity and clinical relevance of the ViEWS at the IMCU. Second, it analyses a large dataset, using a standardized data collection protocol in which the EWS is consistently collected on a regular basis.

Its limitations are that physicians have likely acted upon EWS values to prevent further deterioration. Although we have partly taken this into account by using ICU transfer as an outcome, we did not include other treatment changes, such as increasing the amount of noradrenalin. Therefore, we were unable to assess the relevance of the ViEWS for the prediction of these treatment changes. Another limitation is, as indicated before, that we cannot generalize our findings to IMCUs with a case-mix which is more comparable to the hospital ward, in particular as our IMCU may be unique in its possibility to administer vasopressors.

Further research to detect the deteriorating patient at the IMCU should probably not solely focus on cross-sectional measurements of vital signs (as is done in the EWS). Rather, focus should shift towards using (automated) repeated measurements of these vital signs to adequately and timely detect the deteriorating patient at the IMCU. By optimal support through ICT and electronic medical record systems, this preferably results in a real-time trend analysis and score visualization with an easy read-out of clinical deterioration and adjustable thresholds per patient. Especially in an environment with constant monitoring, this could possibly lead to earlier interventions and improved patient outcomes.

The ViEWS at the IMCU has a discriminative performance that is considerably lower than at the hospital ward. As the number of false alarms is high, it carries the risk of alarm fatigue. Therefore, its application in this setting should be reconsidered. As the addition of IMCU-specific clinical predictors does not sufficiently improve the performance, further research should focus on more complex models incorporating repeated measurements of vital functions.

ARTICLE HIGHLIGHTS

Research background

Early Warning Scoring (EWS) systems to recognize the clinically deteriorating patient are widely used in the clinical setting, including in Intermediate Care Units (IMCUs). However, they have been developed and validated for the general hospital ward population and hence their applicability within the IMCU population is unclear.

Research motivation

The application of prediction models (EWS) at a different setting than the setting at which they were developed (IMCU instead of hospital ward), could lead to an inefficient use of scarce resources and may compromise patient safety. To justly consider the (ongoing) use of the EWS at the IMCU, its discriminative performance and applicability need to be investigated.

Research objectives

This validation study aims to assess the performance and clinical relevance of the VitalPAC-EWS (ViEWS) at the IMCU. Further, it aims to improve the EWS for its use at the IMCU.

Research methods

Electronically collected data from 2014 to 2016 at the IMCU were used to obtain the area under the receiver operating curve (AUC) and the number needed to trigger (false alarm rate) at the current and the optimal threshold.

Research results

The AUC of the ViEWS was 0.72 (CI: 0.69-0.75). The number needed to trigger was 19 per one event. Although the discriminative performance is acceptable, the clinical relevance is limited as 19 false alarms are needed per one event. This carries the risk of alarm fatigue. Therefore, this study contributes to this research field that the use of the EWS at the stand-alone IMCU should be reconsidered. The main problem that remains to be solved are that an alternative system needs to be developed to timely detect clinical deterioration at the IMCU.

Research conclusions

The new findings of this study are that the use of the ViEWS at the IMCU should be reconsidered. It proposes that this is due to remarkable case-mix differences between the hospital ward and the IMCU. This study proposes to use new methods to detect clinical deterioration at the IMCU, using automated data collection and perhaps more sophisticated statistical methods. The implication for clinical practice is that the EWS in its current form at the IMCU should perhaps not be used.

Research perspectives

General experiences and lessons that can be learned from this study are that prediction models should not be used in different settings without prior validation. Further research should focus on alternative methods to detect the clinically deteriorating patient at the IMCU, through the modelling of repeated measurements in prediction models. Also, further research should focus on the use of the EWS in differently formatted IMCUs, such as the IMCU that is integrated into the ICU.

ACKNOWLEDGEMENTS

We would like to thank Melvin van Maurik for his efforts in extracting the required data.