Comparison of the National Early Warning Scores and Rapid Emergency Medicine Scores with the APACHE II Scores as a Prediction of Mortality in Patients with Medical Emergency Team Activation: a Single-centre Retrospective Cohort Study.

INTRODUCTION: The medical emergency team enables the limitation of patients' progression to critical illness in the general ward. The early warning scoring system (EWS) is one of the criteria for medical emergency team activation; however, it is not a valid criterion to predict the prognosis of patients with MET activation. AIM: In this study, the National Early Warning Score (NEWS) and Rapid Emergency Medicine Score (REMS) was compared with that of the Acute Physiology and Chronic Health Evaluation II (APACHE II) score in predicting the prognosis of patients who had been treated a medical emergency team.
MATERIAL AND METHODS: In this single-centre retrospective cohort study, patients treated by a medical emergency team between April 2013 and March 2019 and the 28-day prognosis of MET-activated patients were assessed using APACHE II, NEWS, and REMS.
RESULTS: Of the 196 patients enrolled, 152 (77.5%) were men, and 44 (22.5%) were women. Their median age was 68 years (interquartile range: 57-76 years). The most common cause of medical emergency team activation was respiratory failure (43.4%). Univariate analysis showed that APACHE II score, NEWS, and REMS were associated with 28-day prognostic mortality. There was no significant difference in the area under the receiver operating characteristic curve of APACHE II (0.76), NEWS (0.67), and REMS (0.70); however, the sensitivity of NEWS (0.70) was superior to that of REMS (0.47).
CONCLUSION: NEWS is a more sensitive screening tool like APACHE II than REMS for predicting the prognosis of patients with medical emergency team activation. However, because the accuracy of NEWS was not sufficient compared with that of APACHE II score, it is necessary to develop a screening tool with higher sensitivity and accuracy that can be easily calculated at the bedside in the general ward.

Introduction

Despite advances in modern medicine, many patients are at risk of experiencing deterioration in their condition during hospitalization. Failure or delay of recognition and intervention in clinical deterioration are significant problems in healthcare management [1]. The rapid response system is a patient safety system that aims to monitor vital signs, recognize and respond to abnormalities earlier, and educate medical staff. The introduction of a rapid response system is reported to be associated with a decrease in the incidence of unexpected cardiopulmonary arrest, unplanned intensive care unit admission, and death [2, 3, 4].

A rapid response system involves recognizing and triggering early warning scores (EWS) for acutely ill patients. In addition to monitoring and tracking patients’ vital signs and physical conditions, Early warning scores are used to identify patients at risk of clinical deterioration [5]. Thus, these track and trigger systems play a crucial role in preventing deterioration in hospitalized patients [6].

Among many track and trigger systems, the efficacy of the National Early Warning Score (NEWS) [7] in the prevention of unplanned ICU admission and death was evaluated in patients attending in-hospital or emergency departments. In contrast, the Rapid Emergency Medicine Score (REMS) [8] is a scoring system derived from the Acute Physiology And Chronic Health Evaluation II score [9]. It was developed to distinguish patients at risk of ICU admission and death by including age as a variable. It can also be used in emergency departments [10]. However, it remains unclarified whether REMS effectively distinguishes patients at risk of deteriorating conditions during hospitalization.

Therefore, the present study aimed to determine the effectiveness of the National Early Warning Score and the Rapid Emergency Medicine Score in improving 28-day mortality in patients who were treated by a medical emergency team during hospitalization. We also assessed the severity of illness in patients who received medical emergency team activation calls using the APACHE II score and compared the predictive accuracy of National Early Warning Scores and the Rapid Emergency Medicine Scores on 28-day mortality with the APACHE II score.

Methods

A single-centre, retrospective, observational study in the General Hospital of Sapporo, Japan, was undertaken. The Institutional Review Board approved the study design and protocol of Sapporo Medical University (authorized number: 312-85, UMI000039993); as the study was performed retrospectively, the requirement for informed consent was waived.

Patients were enrolled who received treatment following the Medical Emergency Team activation calls after tracking and triggering the deterioration of patients’ conditions in the general ward between April 1st 2013 and March 31st 2019.

Data collection

Patient data were collected from the hospital’s electronic healthcare records.

Data of patient characteristics such as age, sex, and underlying disease were collected, and the Charlson comorbidity index [11], APACHE II Sequential Organ Failure Assessment (SOFA) scores [12] were calculated using the collected data.

The following variables related to vital signs and physical status, which were calculated for determining the early warning score, were evaluated: systolic and diastolic blood pressure, heart rate, respiratory rate, oxygen saturation determined by pulse oximetry (SpO2), body temperature, mental status evaluated by alert-verbal-pain-unresponsive (AVPU) code, and supplemental oxygen therapy.

Inclusion criteria

All patients with medical emergency team activation.

Patients older than 18 years, patients with cardiopulmonary arrest at the time of activation of the medical emergency.

Patients listed as Do Not Attempt Resuscitation.

Patients without detectable vital signs following the activation of the Medical Emergency Team.

The criteria for the activation of the medical emergency team

The medical emergency team was called in when the vital signs and physical conditions met at least one of the following criteria:

Respiratory rate: ≥30 breaths per minute, or <8 breaths per minute

Heart rate: ≥130 beats per minute

Systolic blood pressure: ≤90 mmHg, or >40 mmHg of decrease from the normal systolic blood pressure

Oxygen saturation: < 90% (under supplement of oxygen therapy)

Body temperature: cold sense or clammy sweat at peripherals of extremities

Consciousness: rapid alteration of the consciousness level

Any concerns

Statistical Analysis

Statistical analysis was performed using SPSS Statistics, version 27 (IBM Corp., Armonk, NY). The median and interquartile range of numerical variables and frequencies and the corresponding percentages (%) of categorical variables were recorded.

Patient characteristics were analyzed using the Mann-Whitney U test for continuous variables and the chi-square test or Fisher’s exact test for categorical variables. Scores (NEWS, REMS, and APACHE II scores) were calculated using MS EXCEL.

The significance level was set at α = 0.05.

Two-tailed p values of less than 0.05 were considered significant.

Discrimination of scoring systems was assessed using the receiver operating characteristic (ROC) curve analysis for calculating the sensitivity and specificity of each score and the corresponding area under the ROC curve (AUROC).

Results

Characteristics of patients

Table 1 shows the patients’ demographic characteristics. The number of patients in whom the MET call was activated during the study period was 211; 15 patients were excluded from this study.

Table 1

Patient characteristics

	Total (n=196)	Survivors (n=152)	Non-survivors (n=44)	p-value
Age (years), [Median, IQR]	68(57-76)	67.5(56-76)	67.7(63-77)	0.18
Male, n. (%)	152(77.5)	121(79.6)	31(70.4)	0.21
Transferred to ICU post-MET, n (%)	134(68.4)	96(49.0)	38(86.4)	<0.01
CCI, [Mean (SD)]	1.05(1.44)	1.06 (1.48)	0.97(1.29)	0.94
SOFA, [Median, IQR]	6(4-8)	5(4-7)	9(7-12)	<0.01
Illness category, surgical, n (%)	122(62.2)	98(64.5)	24(54.5)	0.29
Reason of MET triggers, n (%)
Respiratory	85(43)	62(41)	23(52)	0.22
Cardiovascular	45(23)	37(24)	8(18)	0.39
Neurology	18(9)	15(10)	3(7)	0.38
Haemorrhage	15(8)	12(8)	3(7)	0.06
Others	33(17)	26(17)	7(16)	0.85

Abbreviations: CCI, Charlson Comorbidity Index; SOFA, Sequential Organ Failure Assessment; MET, Medical Emergency Team; ICU, intensive care unit; IQR, interquartile range; SD, standard deviation

Of 196 patients, 44 (the non-surviving group) died, and 152 survived (the surviving group) within 28 days after MET call activation, and the 28-day mortality rate was 22.4%. There was no significant difference in age (p = 0.18) or sex (p = 0.21) between the surviving and non-surviving patients.

The median age of the eligible patients was 68 years (IQR, 57-76 years).

The ward in which the MET call was activated most frequently was the Surgical Ward (62%), followed by the Internal Medicine Ward (38%).

The most and the second most frequent reason for MET call activation was respiration-related issues such as a decrease in SpO2, increase in respiratory rate and complaints related to dyspnea, and circulation-related issues such as hypotension and tachycardia, respectively.

There were significant differences in SpO2 (p = 0.02), Glasgow Coma Scale (p = 0.01), and mental status based on the AVPU score (p < 0.01) between the surviving and non-surviving groups.

There was a higher percentage of patients with a poor Glasgow Coma Scale score of ≤8 in the non-surviving group. In addition, the unresponsiveness of the AVPU score was significantly higher (23%) in the non-surviving group than in the surviving group (p = 0.02).

Based on the prediction model using the scoring system with the Mann-Whitney U test, the probability of death was calculated and compared between the non-surviving and surviving groups. The results showed a significant difference in the scores of the target scoring system between the surviving and non-surviving groups (NEWS: p < 0.01, REMS: p < 0.01, APACHE II: p < 0.01).

APACHE II (AUROC, 0.76; 95% CI, 0.67–0.83) had the highest predictive power for 28-day mortality, followed by REMS (AUROC 0.7; 95% CI 0.61–0.78) and NEWS (AUROC 0.67; 95% CI 0.59–0.75) (Table 4, Fig. 1).

Table 4

Comparison of the predictive power and clinical utility of the APACHE II score, NEWS, and REMS in 28-day mortality

	Discrimination AUC (95% CI)	APACHE II vs p-value EWS	Clinical utility
			Optimal cut-off	Sensitivity	Specificity	PPV	NPV	PLR	NLR
APACHE II	0.76 (0.67	-0.83)	18	0.89	0.50	0.34	0.94	2.52	0.45
NEWS	0.67 (0.59-0.75)	0.10	7	0.70	0.62	0.35	0.88	1.84	0.76
REMS	0.70 (0.61-0.78)	0.26	11	0.47	0.81	0.42	0.84	0.95	0.65

Abbreviations: APACHE-II, Acute Physiology And Chronic Health Evaluation II; EWS, Early Warning Score; NEWS, National early warning score; REMS, Rapid emergency medicine score; PPV, Positive predictive value; NPV, Negative predictive value; PLR, Positive likelihood ratio; NLR, Negative likelihood ratio; AUC, area under the curve; CI, confidence interval

Fig. 1

Study inclusion flow chart.

There was no significant difference among the AUROC of the three scoring systems (p = 0.27).

The clinical usefulness of APACHE II, NEWS, and REMS was evaluated in terms of sensitivity, specificity, PPV, NPV, PLR, and NLR (Table 4). At the Youden Index cut-off point, APACHE II >18 had the highest sensitivity and the lowest specificity for 28-day mortality; NEWS > 7 had the best balance of sensitivity and specificity.

The PPV and NPV of all three scoring systems were similar. PLR and NLR were highest in APACHE II, followed by NEWS and REMS.

Discussion

In this study, the prognostic abilities of NEWS, REMS, and APACHE II were compared in patients who had received MET activation. The APACHE II score was used as a standard predictive tool for prognosis in critically ill patients. Although the statistical results showed that the AUROC was not significantly different among the three scoring systems, the APACHE II score had a higher AUROC than EWS in predicting the prognosis of patients in whom MET was activated. In general, the APACHE II score was calculated using multiple blood data, in addition to several vital signs. Therefore, it is difficult to calculate the APACHE II score quickly when the MET call is activated in patients. While EWS requires data on vital signs and the patient’s physical condition, it is easy to calculate by the bedside, and it does not take a long time to obtain results. However, the AUCs of NEWS and REMS are very close, in that the values were 0.67 (95% CI 0.59-0.75) and 0.7 (0.610.78), respectively; sensitivity for prognostic prediction was superior in NEWS rather than REMS. Therefore, NEWS could be a valuable tool for the decision of ICU admission when the MET call is activated.

Area under the receiver operator (ROC) for National Early Warning Score (NEWS), Rapid Emergency Medical Score and the Acute Physiology and Chronic Health Evaluation (APACHE) II score, calculated at the MET activation

NEWS has been widely used worldwide as a tool to track and trigger the clinical conditions of in-hospital patients [13]. Smith et al. [14] reported that NEWS had an AUROC of 0.89 (95% CI 0.89-0.90) for predicting in-hospital mortality in hospitalized patients with sudden deterioration of a condition, presenting a higher AUROC value than that of our results. Contrarily, in a study that assessed the predictive power of 28-day mortality in patients with MET activation, the AUROC of NEWS was 0.6 (95% CI 0.59-0.74) [15]. Therefore, the predictive power of NEWS in the prognosis of MET-activated patients varies depending on the clinical conditions of the studies. The sensitivity of NEWS for predicting the prognosis of patients with MET activation has been demonstrated to be approximately 0.7-0.8, and the differences between the studies are minimal [15, 16]. This indicates that our results may be compatible with those of other studies.

In general, an aggregate score of more than 7 for NEWS has been associated with an increased risk of developing critical conditions such as cardiopulmonary arrest and recommended intervention by the response team (NHS-NEWS definition). In this study, the median NEWS between the surviving and non-surviving groups was 7 (6-9) and 9 (7-10), respectively, which was higher than that reported in previous studies [16, 17]; therefore, the patients were considered to be in a critical condition. The higher degree of a critically ill condition of MET patients in our study may be explained by a delay in MET activation. The criteria for MET activation were set to a single-parameter system. This system is readily available, and the sensitivity of recognition of deterioration is thought to be high. However, highly sensitive vital signs, such as respiratory rate, are often excluded from ordinary observation [18]. Therefore, RRS in our hospital is possibly immature, resulting in a delay in MET activation.

The REMS was developed and established by modifying the APACHE II score as a tool for predicting patient prognosis in the emergency department in 2003 [8]. REMS has been reported to help predict the prognosis of septic patients in emergency departments [19, 20, 21]; however, its validity for predicting the prognosis of in-hospital patients has not been elucidated. The AUC of REMS (0.7) was not significantly different from that of NEWS; however, the sensitivity of REMS was the lowest at 0.47. Therefore, it did not help screen critically ill in-hospital patients with a deteriorating condition using REMS. The difference between NEWS and REMS is that the age score is included in the variables for REMS calculation. The inclusion of age in the EWS is controversial. Ageing in advanced countries is increasing each year, and high rates of a sudden deterioration in the patient’s condition and related mortality among elderly patients have been reported [22, 23]. Therefore, it is helpful to use a prediction model that includes age as a predictor of in-hospital cardiac arrest and in-hospital mortality [24, 25]. In addition, it has been reported that the prognosis of patients with MET activation is associated with the age of 75 years or older [17]. However, the mean age of the patients with MET activation in this study was approximately 67 or 68 years old, and there was no significant difference between the two groups, which may be because REMS did not discriminate the mortality risk.

Furthermore, Shamout et al. (2019) reported that it is important to consider changes in vital signs associated with ageing rather than simply including age as a variable [26]. The inclusion of age in the EWS may reduce the opportunity to recognize patients whose age score is small despite abnormal vital signs will be identified [27]. Therefore, much discussion is needed to consider age when using EWS to predict the prognosis of MET-activated patients.

This study has several limitations. First, this was a single-centre, retrospective, observational study conducted in a university hospital.

Second, the study did not adjust for the primary disease. The comorbidity in the surviving and non-surviving groups was not significantly different, but the primary disease may have influenced the prognosis. Large-scale and multicenter studies are needed to identify a more accurate scoring system.

Conclusions

We compared the prognostic performance of NEWS and REMS with that of the APACHE II score, which is used as a standard prognostic tool in patients with MET activation. There was no significant difference in AUROC among the studied scores; however, the APACHE II score had the highest sensitivity, followed by NEWS, in patients with MET activation. The sensitivity of REMS was lower than that of NEWS. Therefore, NEWS was a more sensitive screening tool than REMS for predicting the prognosis of patients with MET activation. It is necessary to develop a more sensitive and accurate screening tool that can be easily calculated at the bedside in the general ward. Therefore, a large-scale, prospective study is needed to develop an EWS that can predict patient prognosis when MET is activated (or in patients with MET activation).

Table 2

Comparison of vital signs and physical status in the survivor group and the non-survivor group

	Total (n=196)	Survivor group (n=152)	Non-survivor group (n=44)	p-value
Body temperature (°C), [Median, IQR]	37.2(36.5-37.5)	36.8(36.5-37.4)	37.0(36.7-37.7)	0.95
Pulse rate (/min), [Median, IQR]	100(80-120)	92(80-120)	110(80-121)	0.15
Respiratory rate (/min), [Median, IQR]	27(20-30)	24(20-30)	27(20-31)	0.10
SpO2 (%), [Median, IQR]	95(88-98)	94(90-98)	90(85-95)	0.02
Systolic arterial pressure (mmHg), [Median, IQR]	109(74-130)	100(74-130)	101(73-133)	0.88
Mean arterial pressure (mmHg), [Median, IQR]	76(52-89)	71(51-89)	72(39-90)	0.98
Glasgow Coma Scale≤8, n. (%)	35 (18)	21(13)	14(31)	0.01
Oxygenation, n. (%)	88 (44)	64(42)	24(54)	0.60
Altered mental status, n. (%)	58(29)	44(28)	14(31)	0.86
AVPU score, n. (%)
Alert	100(51)	88(58)	12(27)	<0.01
Verbal	41(21)	29(19)	12(27)	0.29
Pain	32(16)	22(14)	10(23)	0.06
Unresponsive	23(12)	13(9)	10(23)	0.02

Abbreviations: IQR, interquartile range; AVPU, alert-verbal-pain-unresponsive

Table 3

Median and IQR of the probability of death using the APACHE II score, NEWS, and REMS

	Total (n=196)	Survivor group (n=152)	Non-survivor group (n=44)	p-value
APACHE II Median, IQR]	21(15-26)	19(14-24)	25(21-31)	<0.01
NEWS [Median, IQR]	7(6-9)	7(5-9)	9(7-10)	<0.01
REMS [Median, IQR]	9(6-12)	9(6-11)	11(8-13)	<0.01

Abbreviations: APACHE-II, Acute Physiology And Chronic Health Evaluation II; NEWS, National early warning score; REMS, Rapid emergency medicine score; IQR, interquartile range