Validation of National Early Warning Score for predicting 30-day mortality after rapid response system activation in Japan.

AIM: Although rapid response systems (RRS) are used to prevent adverse events, Japan reportedly has low activation rates and high mortality rates. The National Early Warning Score (NEWS) could provide a solution, but it has not been validated in Japan. We aimed to validate NEWS for Japanese patients.
METHODS: This retrospective observational study included data of 2,255 adult patients from 33 facilities registered in the In-Hospital Emergency Registry in Japan between January 2014 and March 2018. The primary evaluated outcome was mortality rate 30 days after RRS activation. Accuracy of NEWS was analyzed with the correlation coefficient and area under the receiver operating characteristic curve. Prediction weights of NEWS parameters were then analyzed using multiple logistic regression and a machine learning method, classification and regression trees.
RESULTS: The correlation coefficient of NEWS for 30-day mortality rate was 0.95 (95% confidence interval [CI], 0.88-0.98) and the area under the receiver operating characteristic curve was 0.668 (95% CI, 0.642-0.693). Sensitivity and specificity values with a cut-off score of 7 were 89.8% and 45.1%, respectively. Regarding prediction values of each parameter, oxygen saturation showed the highest odds ratio of 1.36 (95% CI, 1.25-1.48), followed by altered mental status 1.23 (95% CI, 1.14-1.32), heart rate 1.21 (95% CI, 1.09-1.34), systolic blood pressure 1.12 (95% CI, 1.04-1.22), and respiratory rate 1.03 (95% CI, 1.05-1.26). Body temperature and oxygen supplementation were not significantly associated. Classification and regression trees showed oxygen saturation as the most heavily weighted parameter, followed by altered mental status and respiratory rate.
CONCLUSIONS: National Early Warning Score could stratify 30-day mortality risk following RRS activation in Japanese patients.

Introduction

The rapid response system (RRS) has become the standard international system for preventing unexpected cardiac arrests in hospital wards. , , , , Although the concept of RRS has gradually become accepted in Japan, low activation rates and high mortality have been reported after rapid response team/medical emergency team (RRT/MET) calls, especially during night‐shifts. To solve this problem, a system that overcomes the barriers in identifying dangerous changes in afferent limb conditions is needed.

Automated calculation of the early warning score (EWS) and proactive RRT/MET activation based on EWS is expected to be one solution. , , The majority of Japanese institutions have adapted single‐parameter activation criteria. Single‐parameter activation criteria are easy for medical professionals to understand, and they work effectively. , Although it is not clear whether single‐parameter or EWS is superior, EWS has increased RRT/MET calls and is better at the multiple‐layer risk stratification of inpatients , , , , , , , , and the National Early Warning Score (NEWS) is the most widely used EWS. However, to the best of our knowledge, it has not been validated for use with Japanese patients. Therefore, this study aimed to validate NEWS for Japanese patients using data from the In‐Hospital Emergency Registry in Japan (IHER‐J), a multicenter RRS online registry. Furthermore, this study also focused on the predictive weight of each NEWS parameter, which is important for obtaining a better understanding of NEWS and necessary for the development of a more sophisticated NEWS system.

Methods

Data source and participants

For this retrospective observational study, as previously described, data were drawn from the IHER‐J, a multicenter RRS online registry. All patients registered between January 2014 and March 2018 were eligible for inclusion in this study. One institution, in which half of the beds were allocated for long‐term care, was excluded from the analysis because the patient characteristics might be different from those obtained from other acute hospitals, particularly in terms of limitation of medical treatment (LOMT). Patients younger than the age of 16 years or of unknown age were excluded. Additionally, cases with incomplete data for 30‐day outcomes or NEWS parameters were excluded. The exclusion criteria in this analysis are shown in Fig. 1.

Fig. 1

Flowchart of study of the National Early Warning Score for predicting 30‐day mortality following rapid response system activation in Japan

Variables collected

Patient demographic data, data related to RRS, and physiological parameters for the calculation of NEWS were collected from IHER‐J. The NEWS parameters at the time of RRT/MET activation were used for this analysis. The IHER‐J used the Glasgow Coma Scale (GCS) for measuring the consciousness level; therefore, GCS scores were substituted using the following rules adopted from a previous study: a score of 15 on GCS was considered to be “alert”, and a score less than 15 was considered as VPU (voice, pain, unresponsive) on the AVPU (alert, voice, pain, unresponsive) scale. Information from the registry for oxygen supplementation was only available for the timing of RRT/MET interventions; thus, it was substituted for oxygen supplementation in the NEWS calculation.

Outcomes

The primary outcome was the mortality rate after 30 days of RRT/MET calls. The secondary outcome was the composite outcome, including unplanned transfers to an intensive care unit (ICU) soon after RRT/MET intervention and mortality 30 days after RRT/MET calls.

Statistical analysis

Demographic data were compared between survivors and non‐survivors using the Wilcoxon rank sum test for continuous variables and Fisher’s exact test for categorical variables. For validating NEWS total and NEWS grade for the prediction of outcomes, Spearman’s rank correlation was used on NEWS, and multiple comparisons with Fisher’s exact test were used on NEWS grade. The area under the receiver operating characteristic curve (AUC), sensitivity, specificity, positive predictive value, negative predictive value, and accuracy value were also calculated. To compare the predictive weight of each parameter used in NEWS, a multiple logistic regression model and a machine learning method called CART (Classification and Regression Trees) were used. For the multiple logistic regression model and CART model, the score of each parameter based on the NEWS cut‐off (score of 0–3) was used. A bootstrap method was applied to confirm the order of the importance on CART model. Statistical significance was defined as a P‐value of less than 0.05. All data manipulation were carried out using R version 3.6.1.

Results

Demographic details

A total of 6,784 cases were registered at 35 facilities during the study period. Of those, 2,255 cases at 33 facilities were eligible for this analysis (Fig. 1).

Patient characteristics are shown in Table 1. Compared with the non‐survivors, survivors were younger (68.3 versus 72.2, P < 0.001), fewer patients were male (57.5% versus 68.2%, P < 0.001), fewer were cancer patients (19.3% versus 31.6%, P < 0.001), more were in a postoperative state (14.9% versus 9.0%, P < 0.001), fewer were patients with LOMT (10.5% versus 42.6%, P < 0.001), and fewer were medical patients (48.1% versus 60.4%, P < 0.001). Additional information regarding RRT/MET calls, including the reasons for calls and interventions, is shown in Tables S1 and S2.

Table 1

Characteristics of adult patients registered in the In‐Hospital Emergency Registry in Japan, January 2014–March 2018 (n = 2,255)

Variable	All (n = 2,255)		Survivors (n = 1,755)		Non‐survivors (n = 500)		P‐value
	n	(%)	n	(%)	n	(%)
Age, mean ± SD	69.2 ± 16.5		68.3 ± 17.1		72.2 ± 14.0		<0.001
Male sex	1,350	(59.9)	1009	(57.5)	341	(68.2)	<0.001
Existing comorbidity
Cancer	496	(22.0)	338	(19.3)	158	(31.6)	<0.001
Postop	307	(13.6)	262	(14.9)	45	(9.0)	<0.001
Sepsis	257	(11.4)	190	(10.8)	67	(13.4)	0.110
LOMT †	308	(17.4)	146	(10.5)	162	(42.6)	<0.001
Admitted department
Medical	1146	(50.8)	844	(48.1)	302	(60.4)	<0.001
Surgical	733	(32.5)	623	(35.5)	110	(22.0)
Minor ‡	173	(7.7)	138	(7.9)	35	(7.0)
Ob/gyn	39	(1.7)	30	(1.7)	9	(1.8)
Psychiatric	11	(0.5)	11	(0.6)	0	(0.0)
Other	153	(6.8)	109	(6.2)	44	(8.8)
Outcomes §
Death	500	(22.2)	0	(0.0)	500	(100.0)	–
ICU transfer	529	(23.5)	416	(23.7)	113	(22.6)	0.63
Composite outcome	916	(40.6)	416	(23.7)	500	(100.0)	–

–, not applicable; LOMT, limitation of medical treatment; Ob/gyn, obstetrics and gynecology; Postop, postoperative patients; SD, standard deviation.

Data from 1,774 cases.

Minor: urology, otolaryngology, dermatology, ophthalmology.

Death, death at 30 days; ICU transfer, intensive care unit (ICU) transfer at rapid response team (RRT)/medical emergency team (MET) intervention; Composite outcome, Death at 30 days or ICU transfer at RRT/MET intervention.

Validation of NEWS and NEWS grade

Associations between NEWS and 30‐day mortality rate and the composite outcome rate are shown in Figure 2. Both mortality rate and composite outcome rate were strongly associated with NEWS, with correlation coefficients of 0.95 (95% confidence interval [CI], 0.88–0.98) and 0.98 (95% CI, 0.96–0.99), respectively. With the cut‐off set to 7 or more, as recommended for the activation threshold for RRT/MET, sensitivity, specificity, positive predictive value, negative predictive value, and accuracy values were 80.8%, 45.1%, 29.6%, 89.2%, and 53.0%, respectively. These values for NEWS and NEWS grade are shown in Table 2. The AUC of NEWS was 0.67 (95% CI, 0.64–0.69) (Fig. S1). In terms of NEWS grade, the mortality rates of low, low‐medium, medium, and high grade patients were 5.0%, 11.4%, 15.2%, and 29.6%, respectively (Table S3). Mortality rates were statistically different between each group except between low and low‐medium, and low‐medium and medium. Composite outcomes are provided as the statistical difference between each group, except for those between low and low‐medium (Fig. 3).

Fig. 2

Correlation between National Early Warning Score (NEWS) score and outcomes. This figure shows the strong association between NEWS and outcomes. Spearman’s correlation coefficient (R 2) of mortality and composite outcome rate were 0.95 (95% confidence interval, 0.88–0.98) and 0.98 (95% confidence interval, 0.96–0.99), respectively. (), 30‐day mortality; (), composite outcome. x‐axis, NEWS; y‐axis, mortality and composite outcome rate.

Table 2

Sensitivity (Sn), specificity (Sp), positive predictive value (PPV), negative predictive value (NPV), and accuracy of the National Early Warning Score (NEWS) and NEWS grade for death at 30 days (n = 2,255)

NEWS	Sn	Sp	PPV	NPV	Accuracy
0	–	–	–	–	–
1	99.4	4.8	22.9	96.6	25.8
2	98.6	8.7	23.5	95.6	28.6
3	97.6	12.3	24.1	94.7	31.2
4	95.2	19.5	25.2	93.4	36.3
5	92.2	27.0	26.5	92.4	41.5
6	86.4	35.0	27.5	90.0	46.4
7	80.8	45.1	29.6	89.2	53.0
8	72.0	53.6	30.7	87.0	57.7
9	62.4	62.5	32.1	85.4	62.4
10	50.8	70.8	33.2	83.5	66.4
11	38.8	78.0	33.4	80.6	69.3
12	28.2	85.0	34.9	79.9	72.4
13	20.4	90.1	37.0	79.6	74.6
14	12.0	93.5	34.5	79.9	75.4
15	8.8	96.5	41.9	78.8	77.0
16	5.2	98.3	46.4	78.4	77.6
17	3.2	99.2	53.3	78.2	77.9
18	1.0	99.8	62.5	78.0	77.9
19	0.2	100	100	77.9	77.9
20	0.2	100	100	77.9	77.9
NEWS grade †
Low	–	–	–	–	–
Low‐medium	97.0	10.5	24.8	95.0	34.3
Medium	92.2	16.4	26.5	92.4	41.5
High	80.8	45.1	29.6	89.2	53.0

–, not applicable.

Low, NEWS 0–4; Low‐medium, NEWS ≤4 and score of 3 in any individual parameter; Medium, NEWS 5–6; High, NEWS ≥7.

Fig. 3

Correlation between National Early Warning Score (NEWS) grade and outcomes. Risk ratio of 30‐day mortality and composite outcomes between groups are shown. () 30‐day mortality; (), composite outcome. x‐axis, NEWS grade. y‐axis, 30‐day mortality and composite outcome rate. *P < 0.05 with Bonferroni correction.

Predictive weight of each NEWS parameter

When compared using the adjusted odds ratio (OR) of multiple logistic regression, oxygen saturation showed the highest association (adjusted OR 1.36; 95% CI, 1.25–1.48), followed by altered mental status (adjusted OR 1.23; 95% CI, 1.14–1.32), heart rate (adjusted OR 1.21; 95% CI, 1.09–1.34), systolic blood pressure (adjusted OR 1.12; 95% CI, 1.04–1.22), and respiratory rate (adjusted OR 1.03; 95% CI, 1.05–1.26). Body temperature and oxygen supplementation were not associated (Table 3).

Table 3

Logistic regression model with National Early Warning Score parameters (n = 2,255)

	Adjusted OR	95% CI	P‐value
Oxygen saturation	1.36	1.25–1.48	<0.001
Altered mental status	1.23	1.14–1.32	<0.001
Heart rate	1.21	1.09–1.34	<0.001
Systolic blood pressure	1.12	1.04–1.22	0.005
Respiratory rate	1.03	1.05–1.26	0.002
Body temperature	0.92	0.78–1.09	0.350
Oxygen supplement	0.91	0.80–1.02	0.100

CI, confidence interval; OR, odds ratio.

The predictive value of each parameter was also analyzed with the CART model. Based on the tree algorithm in the CART model, cases were divided into five categories. The CART model showed that oxygen saturation, AMS, respiratory rate, and heart rate were important parameters (Fig. 4). A bootstrap method confirmed the result. These four parameters were also shown to be significantly associated in the multiple logistic regression model. Figure 5 shows the proportion of CART categories by NEWS grade. Category 4 and 5 proportions, which met the oxygen saturation criterion, increased with the increase in the risk of NEWS grade.

Fig. 4

Classification and regression trees (CART) model. This tree diagram shows the prediction model of CART for 30‐day mortality following rapid response system activation in Japan. Cut‐off points for oxygen saturation (SpO2), consciousness, and respiratory rate were the same criteria as a score of 3 on the National Early Warning Score (NEWS). *Number of cases, 30‐day mortality rate.

Fig. 5

Weight of National Early Warning Score (NEWS) parameters for NEWS grade for predicting 30‐day mortality following rapid response system activation in Japan. This figure shows the proportions of classification and regression trees (CART) categories by NEWS grade. The proportion of category 4 increased with increase in the risk of NEWS grade. (), CART category 1; (), CART category 2; (), CART category 3; (), CART category 4 and 5. x‐axis, NEWS grade; y‐axis, proportion of categories.

Discussion

National Early Warning Score validation

In this study, NEWS showed a high correlation between 30‐day mortality and ICU transfer after RRT/MET activation (Fig. 2). Previous studies of the validity of NEWS in other patient cohorts (not in Japan) include that of Smith et al., who reported that the AUC of NEWS for a serious event within 24 h including cardiac arrest, unanticipated ICU admission, or death, was 0.87 (95% CI, 0.866–0.879). Pimentel et al. reported that the AUC of NEWS for death within 24 h was 0.910 (95% CI, 0.907–0.912). , Our data showed lower accuracy compared with these previous studies carried out using time series physiological datasets; however, our study used only a single dataset that included the timing of RRS calls. Vital signs are easily modifiable with treatment, such as treatment with catecholamine or fluid bolus; thus, single‐point measurements might not measure a parameter that is significant for a particular patient prior to their RRT/MET calls.

Japanese hospitals have struggled with a very low rate of RRT/MET calls, and most institutions have been using single‐parameter activation criteria. Using RRT/MET activation or proactive rounds based on an automated track‐and‐trigger system with NEWS, in addition to the existing RRS system, would promote patient safety. In terms of stratifying and tracking all patients’ risk, NEWS is better than single parameter criteria because single parameter criteria only stratified patients into high‐ or low‐risk groups. Furthermore, NEWS is an easy‐to‐change activation threshold based on the analysis of each facility’s circumstances. Recently, EWS based on machine learning methods have been reported. , , , , Further evaluations of external validity and implementability are needed.

Predictive value of each NEWS parameter

Multiple logistic regression analysis indicated oxygen saturation as having the highest association with 30‐day mortality, followed by consciousness, heart rate, systolic blood pressure, and respiratory rate. Body temperature and oxygen supplementation were not associated with the outcome. The CART method showed oxygen saturation to be the most heavily weighted parameter and consciousness, respiratory rate, and heart rate to be associated parameters. These two analyses produced similar results. Our results differed from past studies that reported respiratory rate as the most predictive vital sign. , , Respiratory rate was the most unmeasured parameter in this registry; thus, this might affect the result. This is a serious concern in Japanese institutions.

In terms of the associations between individual parameters, the CART model was more descriptive. Although the CART model could provide a more complex regression tree, the model was adjusted to an understandable depth. Heart rate was important among patients with low oxygen saturation; AMS and respiratory rate were important among patients without low oxygen saturation. These findings might reflect the more sophisticated NEWS scoring system that incorporated changes such as adding an extra point with a combination of parameters. Figure 5 shows the proportion of CART categories by NEWS grade. The proportion of categories 4 and 5 of the CART model increased with the rise of NEWS grade risk, but the proportion of category 3 remained almost the same. This indicated that many parts of NEWS transitions were affected by the value of oxygen saturation.

To implement the results of this study in practice, it is important to understand that this analysis set mortality as the primary outcome and not early intervention. An important RRT/MET goal is not only preventing death but also providing early detection and intervention. Body temperature and oxygen supplementation were not associated with outcome in this analysis. However, body temperature is an important parameter for suspected sepsis, and oxygen supplementation easily modifies oxygen saturation. If the outcome had been set as early detection of deterioration, different parameter priorities might have been indicated. Although it is not easy to define early intervention, further research of predictive weight for early intervention of these parameters is needed.

Study strengths and limitations

The main strength of this study is that it is the first multicenter study for validating NEWS in Japan. Moreover, the CART machine learning method was used for analyzing the predictive weight of parameters for the outcome. Despite all the potential implications of its findings, this study has several limitations. First, there was a risk of selection bias. We analyzed the cases that were RRT/MET activated, and some cases were excluded because of incomplete data. If we had analyzed all inpatient data, different information might have been provided. Second, data before RRT/MET activation were not available for this analysis. Vital signs and oxygen supplementation data were only available at the timing of RRT/MET activation. Use of time series data would increase the accuracy of the analysis. Third, we did not consider the variety of RRS in each facility. Outcomes might be influenced by intervention of RRT/MET. Therefore, potential heterogeneity of patients’ intervention of RRT/MET might have affected this analysis. Fourth, possible confounders, such as age, gender, LOMT, and existing comorbidities were not adjusted because this analysis focused on the validity of NEWS. These possible confounders should be included in developing a new EWS. Finally, we did not consider the effect of NEWS on patients with LOMT. These patients might not have benefitted from early intervention for preventing death. Although RRT/MET could contribute to these patients, such as by providing better comfort care, the validation of NEWS for these outcomes were not evaluated in this study.

Implications for clinicians and policymakers

Unified activation criteria based on NEWS could provide a benchmark for Japanese hospitals to assess the quality of their in‐hospital emergency systems, similar to the implementation in England and The Netherlands. , Additionally, NEWS might facilitate communication of patient situations not only between hospitals but also private clinics, long‐term care facilities, and prehospital emergency systems across the country.

Future research

Future research can validate the accuracy of NEWS among all inpatients. The positive predictive value is expected to be different from the current results if applied to all inpatients and could be affected by the burden of the medical staff. Additionally, NEWS adjusted by the clinical assessment was considered as a strategy for improving the current systems. A prospective, multicenter, cluster‐randomized study for adjusted NEWS carried out according to the clinical assessment of the patient is in progress. Furthermore, the development of more simple and accurate prediction models including different information, such as patients’ existing comorbidities, provided medicines, and the text of medical records, is expected. Machine learning and artificial intelligence for real‐time analyses should also help advance the research in this area. , , , ,

Conclusions

National Early Warning Score is highly associated with 30‐day mortality and ICU admission after RRT/MET calls. Oxygen saturation was the most heavily weighted parameter for predicting 30‐day mortality. The results of this study will pave the way for constructing better in‐hospital emergency systems in Japanese institutions.

Disclosure

Approval of the research protocol: The study protocol was reviewed and approved by the ethics committee of all 35 participating institutions in Japan (IRB No. 2498, St. Marianna University School of Medicine, Japan, the representative for IHER‐J).

Informed consent: The need for written informed consent was waived because this study used an anonymous case registry and was conducted by the Patient Safety Committee.

Registry and registration no. of the study: This study was registered 16 October, 2013 in UMIN‐CTR, registration no. UMIN000012045 (https://www.umin.ac.jp/ctr/index‐j.htm).

Animal studies: N/A.

Conflict of interest: None.

Fig. S1. Receiver operating characteristic curve of National Early Warning Score (NEWS) and NEWS grade for 30‐day mortality.

Click here for additional data file.

Table S1. Reason for rapid response team/medical emergency team (RRT/MET) calls (n = 2,255)

Table S2. Interventions during rapid response team/medical emergency team (RRT/MET) calls (n = 2,255)

Table S3. Mortality rate and composite outcome rate in National Early Warning Score (NEWS)

Click here for additional data file.