Developing and validating a model for predicting 7-day mortality of patients admitted from the emergency department: an initial alarm score by a prospective prediction model study.

OBJECTIVES: To set up a prediction model for the 7-day in-hospital mortality of patients admitted from the emergency department (ED) because it is high but no appropriate initial alarm score is available.
DESIGN: This is a prospective cohort study for prediction model development.
SETTING: In a tertiary referred hospital in northern Taiwan. PARTICIPANTS: ED-admitted medical patients in hospitalist care wards were enrolled during May 2010 to October 2016. Two-thirds of them were randomly assigned to a derivation cohort for development of the model and cross-validation was performed in the validation cohort. PRIMARY OUTCOME MEASURED: 7-day in-hospital mortality.
RESULTS: During the study period, 8649 patients were enrolled for analysis. The mean age was 71.05 years, and 51.91% were male. The most common admission diagnoses were pneumonia (36%) and urinary tract infection (20.05%). In the derivation cohort, multivariable Cox proportional hazard regression revealed that a low Barthel Index Score, triage level 1 at the ED, presence of cancer, metastasis and admission diagnoses of pneumonia and sepsis were independently associated with 7 days in-hospital mortality. Based on the probability developed from the multivariable model, the area under the receiver operating characteristic curve in the derivation group was 0.81 (0.79-0.85). The result in the validation cohort was comparable. The prediction score modified by the six independent factors had high sensitivity of 88.03% and a negative predictive value of 99.51% for a cut-off value of 4, whereas the specificity and positive predictive value were 89.61% and 10.55%, respectively, when the cut-off value was a score of 6.
CONCLUSION: The 7-day in-hospital mortality in the hospitalist care ward is 2.8%. The initial alarm score could help clinicians to prioritise or exclude patients who need urgent and intensive care. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The six-factor prediction score was based on a large-scale cohort using cross-validation methods.

The prediction score for 7-day in-hospital mortality could be used before vital signs changed and focus on the subacute phase (2–7 days) after hospitalisation instead of the 24 hours mortality predicted by the National Early Warning Score.

The data of vital signs have not been incorporated into the model.

The results have not been proved to be generalisable to other ethnic groups and specialty care models.

Introduction

In-hospital mortality is a universally accepted indicator for the medical quality of inpatient care.1 Although mortality is usually very low for elective admission, it is comparably higher for admissions from the emergency department (ED).2 3 The 30-day mortality of emergency admission has been reported to be 3.7%–8.2% in a report from Ireland,4 and 4.7%–5.8% and 6.93%–7.04% in two UK reports, respectively.5 6 Notably, the risk of death is relatively high in the initial days from emergency stay to hospitalisation.4 Therefore, identifying patients at risk of deterioration during their initial hospitalisation days is a great challenge but important to clinicians.

One commonly used prediction score is the National Early Warning Score (NEWS), which is mainly composed of vital signs and consciousness level. However, changes in vital signs can indicate acute changes in the illness leading to a possibly irreversible late stage, and the NEWS is suggested for only very short prognosis, such as within 24 hours.7 However, there are scarce studies reported mortality prediction model for subacute phase (2–7 days) after hospitalisation. It is particularly crucial to predict in-hospital mortality within 7 days at the initial admission because emergency admissions have been reported to have high in-hospital mortality of 6.2%–12% in previous reports, because deterioration is not uncommon.8 9 In addition, for in-hospital prognosis before vital signs change, an initial predictive score that prioritises the inpatient’s care is needed for admission evaluation. Therefore, in this study, we conducted a prospective observational study in a hospitalist model for emergency-admitted patient care in Taiwan and developed a prediction model for 7-day in-hospital mortality using available information at admission, including patient demographics, admission diagnosis, underlying comorbidities and performance.

Methods

Source of data

We prospectively conducted a 6-year hospital-based cohort study in a tertiary referral medical centre in Taipei, Taiwan. Participants were limited to adult (≥20 years old) non-trauma patients admitted from the ED to hospitalist care general wards. We did not include patients needing surgery or subspecialty care such as care for HIV infection or chemotherapy for haematological or oncology diagnosis. The assignment of patient admission is decideded by ED physician, not by hospitalists, accordingly to aforementioned rules. We consecutively recruited patients from the period of 1 May 2010 to 31 October 2016. Written informed consent was waived due to the observational design. Patients transferred to other wards before discharge were excluded from the analysis of in-hospital outcomes.

The enrolled patients were divided into two groups by a computerised random process. Two-thirds of the patients were classified as a derivation cohort, and the remaining one-third were used as a validation cohort.

Patient and public involvement

Patients or the public were not involved in the design, or conduct, or reporting, or dissemination plans of our research. The study results were not disseminated to study participants.

Study settings and participants

The tertiary medical centre where our study was performed contained around 2450 beds, and 9500 patients visit the ED per month. Of these 9500, nearly 20% are admitted. The inpatient wards had 36 beds for one ward from October 2009 to May 2013 and then 71 beds for two wards from June 2013 to October 2016, which were tended by hospitalists.

Outcome and clinical characteristics

The outcome of interest was 7-day in-hospital mortality. The in-hospital duration was defined as the time interval from ED admission to death or discharge of the patient. The demographic and clinical data, including age, sex, season of ED admission, triage level at ED arrival, underlying comorbidities, Barthel index at ward admission and admission diagnosis were coded in a standard case report form with default options by trained assistants. In our study, the Taiwan Triage and Acuity Scale (TTAS) was used in the ED.10 One medical patient would be assorted as triage level 1 if one of the following conditions happened at ED arrival: (1) cardiopulmonary arrest, (2) severe respiratory distress with oxygen saturation <90% on the pulse oximeter, (3) signs of shock, (4) Glasgow Coma Scale ≤8 points, (5) body temperature >41°C and (6) the patient transferred with endotracheal tube. The patient assorted as triage level 1 got admitted to the general ward if the patient was stabilised at the ED.

Statistical analysis methods

After dividing the cases into derivation and validation cohorts, we used locally weighted least squares regression (LOWESS) curves to first find the inflection points of continuous predictors in the derivation cohort. We then transformed the continuous predictors into categorical predictors by using the inflection point on the curve. If the inflection point could not be found, we used threshold values adopted in either one or more of the existing EWSs or from clinical experience. Next, we performed univariable Cox proportional hazard regression for each predictor with the outcome. If the p value of one predictor was less than 0.10 in the univariable analysis, it was put into the multivariable stepwise Cox proportional hazard regression. The area under the receiver operating characteristic curve (AUROC) was used to examine discrimination in the derivation and validation cohorts.

To make the prediction model easy to use for medical personnel, we modified the beta coefficient of each statistically significant predictor in multivariable stepwise Cox proportional hazard regression into an integer by multiplying the same but smallest number and adopting the nearest integer. The integer point value system was then built up, and the best cut-off value was chosen using Youden’s Index. We then used this value to calculate the sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) when the model was applied separately to the derivation and validation cohorts. Then we tried to find the cut-off value with >85% sensitivity and the highest possible specificity, and that with >85% specificity and the highest possible sensitivity. Data were analysed in SAS software (V.9.4, SAS Institute) and R software (V.3.3.2). Statistical significance was defined as p<0.05.

Results

Enrolled participants

During the study period, a total of 9093 patients were admitted to the general ward. After excluding 444 (4.9%) patients with incomplete data (167 missed Barthel index, 275 missed triage level and two patients missed survival status), 8924 patients were included in the final analysis. Within 7 days of hospitalisation, 275 patients were transferred to other wards and exclude from the analysis. A flow chart of the cases is shown in figure 1.

Figure 1

Flow chart of participant recruitment and outcomes.

The characteristics of the derivation cohort, the validation cohort and all patients are shown in table 1. The mean age of the patients was 71.05 years, and 51.91% were male. Around 10.71% of the patients had triage levels of 1 at ED, and 24.94% had active cancer. Barthel Index Scores at admission were totally dependent (scores of <35) (30.67%) and severely dependent (scores of 35–60 score) (24.88%). The most common admission diagnosis was pneumonia (36%), followed by urinary tract infection (20.05%) and upper gastrointestinal bleeding (9.26%).

Table 1

Characteristics of study participants

Characteristics	All8649 (100%)	Derivation cohort5763 (66.63%)	Validation cohort2886 (33.37%)	P value
Age	71.05(15.58)	71.08(15.41)	70.99(15.92)	0.81
Male	4490 (51.91)	2953 (51.24)	1537 (53.26)	0.077
Season				0.66
Winter	2043 (23.62)	1359 (23.58)	684 (23.7)
Other seasons	6606 (76.38)	4404 (76.42)	2202 (76.3)
Triage				0.14
1	926 (10.71)	611 (10.6)	315 (10.91)
2–5	7723 (89.29)	5152 (89.4)	2571 (89.09)
Underlying disease
DM	2659 (30.74)	1.765 (30.63)	894 (30.98)	0.74
PAOD	220 (2.54)	149 (2.59)	71 (2.46)	0.73
CKD stages 3–5	111 (1.28)	75 (1.3)	36 (1.25)	0.83
Dialysis	537 (6.21)	355 (6.16)	182 (6.31)	0.79
Myocardial infarction	42 (0.49)	29 (0.5)	13 (0.45)	0.74
CAD	1048 (12.12)	697 (12.09)	351 (12.16)	0.93
CVA	1237 (14.3)	815 (14.14)	422 (14.62)	0.55
Dementia	610 (7.05)	391 (6.78)	219 (7.59)	0.17
Congestive heart failure	613 (7.09)	390 (6.77)	223 (7.73)	0.10
Proven liver cirrhosis	349 (4.04)	231 (4.01)	118 (4.09)	0.86
COPD	609 (7.04)	404 (7.01)	205 (7.1)	0.87
Chronic respiratory failure	45 (0.52)	30 (0.52)	15 (0.52)	1.00
Peptic ulcer disease	903 (10.44)	617 (10.71)	286 (9.91)	0.25
Cancer, active	2157 (24.94)	1406 (24.4)	751 (26.02)	0.10
Metastatic, presence	683 (7.9)	465 (8.07)	218 (7.55)	0.40
Haematologic malignancy	188 (2.17)	122 (2.12)	66 (2.29)	0.61
MDS	25 (0.29)	15 (0.26)	10 (0.35)	0.48
Barthel Index Score				0.33
<35	2999 (34.67)	1969 (34.17)	1.030 (35.69)
35–60	2152 (24.88)	1454 (25.23)	698 (24.19)
>60	3498 (40.44)	2340 (40.60)	1158 (40.12)
Admission diagnosis
Pneumonia	3161 (36.55)	2109 (36.6)	1052 (36.45)	0.90
COPD exacerbation	301 (3.48)	195 (3.38)	106 (3.67)	0.49
Asthma exacerbation	36 (0.42)	29 (0.5)	7 (0.24)	0.08
Bronchiectasis exacerbation	6 (0.07)	4 (0.07)	2 (0.07)	1.00
Thoracic empyema	31 (0.36)	17 (0.29)	14 (0.49)	0.16
Lung abscess	22 (0.25)	16 (0.28)	6 (0.21)	0.54
CHF with exacerbation	497 (5.75)	341 (5.92)	156 (5.41)	0.33
Biliary tract infection	199 (2.3)	134 (2.33)	65 (2.25)	0.83
Liver abscess	67 (0.77)	40 (0.69)	27 (0.94)	0.23
Acute pancreatitis	97 (1.12)	69 (1.2)	28 (0.97)	0.34
Acute cholecystitis	117 (1.35)	64 (1.11)	53 (1.84)	0.01
Upper GI bleeding	801 (9.26)	530 (9.2)	271 (9.39)	0.77
Lower GI bleeding	49 (0.57)	32 (0.56)	17 (0.59)	0.84
Ileus	308 (3.56)	203 (3.52)	105 (3.64)	0.78
Appendicitis	12 (0.14)	7 (0.12)	5 (0.17)	0.54
Diverticulitis	20 (0.23)	14 (0.24)	6 (0.21)	0.75
Cellulitis	264 (3.05)	175 (3.04)	89 (3.08)	0.90
Fever, unknown origin	160 (1.85)	114 (1.98)	46 (1.59)	0.21
Wound infection	46 (0.53)	30 (0.52)	16 (0.55)	0.84
Bacteremia	531 (6.14)	347 (6.02)	184 (6.38)	0.52
Sepsis	178 (2.06)	123 (2.13)	55 (1.91)	0.48
Ischaemic stroke	206 (2.38)	142 (2.46)	64 (2.22)	0.48
Meningitis	58 (0.67)	44 (0.76)	14 (0.49)	0.13
Febrile neutropenia	123 (1.42)	79 (1.37)	44 (1.52)	0.57
Diabetic ketoacidosis	20 (0.23)	9 (0.16)	11 (0.38)	0.04
Urinary tract infection	1734 (20.05)	1143 (19.83)	591 (20.48)	0.48
In-hospital mortality on 7 days	242 (2.80)	161 (2.79)	81 (2.81)	0.97

The data are represented as number (per cent) or mean (SD) and compared with χ2 test or t-test, respectively.

CAD, coronary artery disease; CHF, congestive heart failure; CKD, chronic kidney disease; COPD, chronic obstructive pulmonary disease; CVA, cerebral vascular accident; DM, diabetes mellitus; GI, gastrointestinal; MDS, myelodysplatic syndrome; PAOD, peripheral arterial occlusive disease.

We found no significant differences in the characteristics of the derivation and validation cohorts except that admission diagnoses of acute cholecystitis and diabetic ketoacidosis were higher in the validation cohort. A total of 242 (2.80%) patients died within the first 7 days of their hospital stays.

Model development, specification and performance

Using LOWESS curves, we converted values for age from continuous variables to binary variables by using the inflection point on the curve. Univariable and multivariable Cox proportional analyses are shown in table 2. In univariable Cox analysis, age (≥80 vs <80 years), Barthel Index Score (<35 vs 35–60 vs. >60), season (winter vs other seasons), triage level (level 1 vs other levels), cancer, metastasis and admission diagnoses of pneumonia, bronchiectasis and sepsis were found to be associated with 7-day in-hospital mortality. The factors with p>0.10 in unvariable analysis were listed in online supplemental table S1. Multivariable stepwise Cox proportional hazard regression revealed that Barthel Index Score, triage level at ED, presence of cancer, metastasis and admission diagnoses of pneumonia and sepsis were independently associated with 7-day in-hospital mortality. Based on the probability developed from the multivariable model, the area under the ROC curve in the derivation group was 0.81 (95% CI 0.79 to 0.85) (figure 2).

Table 2

Univariable and multivariable analyses for predictive factors of 7-day in-hospital mortality in the derivation cohort using stepwise model selection

Factors	Univariable analysis			Multibvariable analysis (n=5762)
	Beta coefficient	SE	P value	Beta coefficient	SE	P value
Barthel Index Score
1:<35	2.57	0.27	<0.0001	2.38	0.28	<0.0001
2: 35–60	1.48	0.30	<0.0001	1.39	0.31	<0.0001
3:>60	0			0
Season
Winter	0.30	0.15	0.045
Other seasons	0
Triage level
1	1.47	0.15	<0.0001	1.01	0.15	<0.0001
2–5	0			0
Age
≥80 years	0.43	0.14	0.0016
<80	0
Cancer
Y	0.61	0.14	<0.0001	0.68	0.14	<0.0001
N	0			0
Metastasis
Y	0.88	0.18	<0.0001	1.07	0.18	<0.0001
N	0			0
Diagnosis: pneumonia
Y	0.87	0.14	<0.0001	0.47	0.14	0.00094
N	0
Diagnosis: bronchiectasis
Y	2.13	1.00	0.034
N	0
Diagnosis: sepsis
Y	1.26	0.27	<0.0001	0.92	0.27	0.00070
N	0			0

Figure 2

Receiver operating characteristic curves to predict 7-day in-hospital mortality by probability developed from the multivariable model. AUC, area under curve.

After multiplication by 2 and adoption of the nearest whole number as the point value, the final model was developed (table 3). The sensitivities and specificities of each score are shown in online supplemental figure S1. In brief, using Youden’s index, we determined the optimal cut-off values for the model. The sensitivity, specificity, PPV and NPV for the derivation cohort were 88.03%, 58.81%, 4.91% and 99.51%, respectively, when a score of 4 was determined as the optimal cut-off value (online supplemental table S2). Similarly, we found that a cut-off value of 4 had >85% sensitivity and the highest possible specificity. On the other hand, a score of 6 had >85% specificity and the highest possible sensitivity (online supplemental table S2). The sensitivity, specificity, PPV and NPV were 50.69%, 89.61%, 10.55% and 98.69% when the cut-off value was 6. Then we divided the derivation cohort into three groups according to score ranges of 0≤score < 4, 4≤score < 6 and ≥6. The Kaplan-Meier survival curve showed significant differences between any two of the three groups (all p<0.001)(figure 3).

Table 3

Derivation of the prediction score for 7-day in-hospital mortality (data source: derivation cohort)

Risk factors	Beta coefficient	SE	P value	Point values(beta x 2)
Barthel index <35 vs >60	2.38	0.28	<0.0001	5
Barthel index 35–60 vs >60	1.39	0.31	<0.0001	3
Triage: 1 vs 2–5	1.01	0.15	<0.0001	2
Cancer: Y vs N	0.68	0.14	<0.0001	1
Metastasis: Y vs N	1.07	0.18	<0.0001	2
Diagnosis of pneumonia: Y vs N	0.47	0.14	0.00094	1
Diagnosis of sepsis: Y vs N	0.92	0.27	0.00070	2

Figure 3

Kaplan-Meier curves for 7 days in-hospital mortality for different ranges of the prediction score.

In the validation cohort, the area under the ROC curve was 0.78 (95% CI 0.73 to 0.82) (figure 2). Using the optimal cut-off value of 4 according to Youden’s index, the sensitivity, specificity, PPV and NPV were 87.06%, 57.24%, 4.35% and 99.50%, respectively (online supplemental table S3), which were similar to those for the derivation cohort.

Discussion

This present study found that the 7-day in-hospital mortality was 2.80% among medical patients admitted from the ED. We established a mortality prediction score at the initial admission using the Barthel Index Score, presence of cancer and metastasis, admission diagnoses of pneumonia and sepsis, and triage level on arrival at the ED. Scores of ≥4 had NPVs of up to 99% and could be used for exclusion screening. By contrast, scores of ≥6 had specificity of 89% and PPV of 10.55%, which could cover most of the frail patients.

To manage the mortality outcomes of patients admitted to general medical wards is never easy for hospitalists, although some mortality cannot be avoided. In particular, for patients admitted for unplanned causes leading to ED visits, it is difficult to stratify those who have a high risk for potential death at initial admission. Although prediction scores of NEWS7 or clinical alert system11 have been developed, the indications of the two scores are mostly regarding the change of vital signs and ensuing critical status that is not strange to initiate intensive care. However, to predict the targeted risk group before they exhibit unstable vital signs is important, for it would allow us to prepare further discussions regarding intensive care and prognosis explanations. By using the prediction score developed and validated in the present study with a large-scale sample, we could stratify the patients easily into three subgroups: low risk for prediction scores of <4, intermediate risk for scores of 4–6, and high risk for scores of ≥6. Those with scores of <4 can be excluded from the alarm status, and those with scores of ≥6 probably need intensive treatment.

The prediction score showed a high AUROC of 0.819 and a hazard ratio of 1.659 (95% CI 1.55 to 1.76 per 1 point increment) in the derivation cohort to predict in-hospital mortality within 7 days; the results were similar in the validation group. This score is the first mortality prediction score developed from general medical hospitalised patients and could be applied broadly. However, because the score element is relatively non-specific to diseases, the sensitivity and specificity are not >90%. Therefore, the score can be used in clinical practice to detect fragile patients at initial admission, but final judgement must be reserved for inpatient physicians.

Among the prediction model, the Barthel index, which measures performance in activities of daily living, can be used to represent general condition and disease severity, and it is one of the important factors in prognosis prediction. For patients with chronic illness, the Barthel Index result can be affected by patients’ frailty12 13 and disability, which correlate with mortality.14 On the other hand, it could be the severity of the acute illness which is responsible for the admission. Although we did not discriminate the influence proportions of the Barthel index by acute or chronic illness, it affected the 7-day mortality, with the highest beta coefficient of 2.38 in the multivariable analysis. It might be more easily applied generally at initial admission.

The presence of underlying active cancer and metastatic status are both important predictors for in-hospital mortality due to their immune-compromised status.15 The effect of cancer on a patient’s outcome has been proven in critical care.16 17 However, whether patients have cancer with or without metastasis may provide little insight on the poor prognosis, and as high as 25% of terminal cancer patients receive vasopressors in the dying process.18 Therefore, hospitalists need to hold family meetings to explain and discuss treatment plans for shared decision making on cancer patients with high prediction scores.19

In contrast to a chronic illness such as cancer, triage on arrival to ED could be used as an initial summary index for acute status. The triage index is the five-level TTAS computerised system implemented nationally since 2010.20 The triage index includes changes in vital signs, organ failure and acute problems needing immediate treatment.21 Triage level 1 at the ED has been classified as an impact factor in the prediction score in the present study. In addition, admission diagnoses including sepsis and pneumonia are responsible for acute illness and associated with 7-day in-hospital mortality. Sepsis is a high-mortality syndrome caused by severe infection with or without organ failure. In-hospital mortality could be as high 17% for patients with sepsis and 26% for those with severe sepsis.22 Both factors (triage at the ED and admission diagnosis) represent acute changes for admitted patients. However, age was statistically significant only in the univariable analysis and not significant in the multivariable Cox analysis, possibly because its effect was erased by other comorbidity and performance statuses.

This study had several limitations. First, we did not record initial vital signs, laboratory results in the original study design of clinical analysis. However, we collected the triage level at ED arrival of each participant, which was associated with initial vital signs. Second, prehospital changes in the medical condition or Barthel Index were not recorded, so their roles in in-hospital mortality prediction require further study. In addition, patients enrolled in this study may have been more severely ill, and a higher proportion may have had cancer, because the present study was conducted in a tertiary referral centre. Third, we only enrolled patients admitted in the hospitalist ward and this study was performed in Taiwan. Therefore, our model may not be generalised to specialty ward and hospitals in other areas before validation. Fourth, the results of this study may be not applicable in hospitals with different triage scales.

Conclusion

The prediction score includes six factors: low Barthel Index Score, triage level 1 at the ED, admission diagnosis of sepsis, diagnosis of pneumonia, and presence of cancer and metastasis. It was developed to screen out the high-risk subgroup for 7-day mortality among patients admitted from the ED. Using the prediction score is important and convenient. A score of <4 could exclude the alarm status, whereas a score of ≥6 represents a high potential for short-term in-hospital mortality. Further validation is required before generalisation.