Prediction of fatal adverse prognosis in patients with fever-related diseases based on machine learning: A retrospective study.

BACKGROUND: Fever is the most common chief complaint of emergency patients. Early identification of patients at an increasing risk of death may avert adverse outcomes. The aim of this study was to establish an early prediction model of fatal adverse prognosis of fever patients by extracting key indicators using big data technology.
METHODS: A retrospective study of patients' data was conducted using the Emergency Rescue Database of Chinese People's Liberation Army General Hospital. Patients were divided into the fatal adverse prognosis group and the good prognosis group. The commonly used clinical indicators were compared. Recursive feature elimination (RFE) method was used to determine the optimal number of the included variables. In the training model, logistic regression, random forest, adaboost and bagging were selected. We also collected the emergency room data from December 2018 to December 2019 with the same inclusion and exclusion criterion. The performance of the model was evaluated by accuracy, F1-score, precision, sensitivity and the areas under receiver operator characteristic curves (ROC-AUC).
RESULTS: The accuracy of logistic regression, decision tree, adaboost and bagging was 0.951, 0.928, 0.924, and 0.924, F1-scores were 0.938, 0.933, 0.930, and 0.930, the precision was 0.943, 0.938, 0.937, and 0.937, ROC-AUC were 0.808, 0.738, 0.736, and 0.885, respectively. ROC-AUC of ten-fold cross-validation in logistic and bagging models were 0.80 and 0.87, respectively. The top six coefficients and odds ratio (OR) values of the variables in the Logistic regression were cardiac troponin T (CTnT) (coefficient=0.346, OR = 1.413), temperature (T) (coefficient=0.235, OR = 1.265), respiratory rate (RR) (coefficient= -0.206,OR = 0.814), serum kalium (K) (coefficient=0.137, OR = 1.146), pulse oxygen saturation (SPO2) (coefficient= -0.101, OR = 0.904), and albumin (ALB) (coefficient= -0.043, OR = 0.958). The weights of the top six variables in the bagging model were: CTnT, RR, lactate dehydrogenase, serum amylase, heartrate, and systolic blood pressure.
CONCLUSIONS: The main clinical indicators of concern included CTnT, RR, SPO2, T, ALB and K. The bagging model and logistic regression model had better diagnostic performance comprehesively. Those may be conducive to the early identification of critical patients with fever by physicians.

Introduction

Fever is the most common chief complaint of emergency patients and is an important pathophysiological process and common symptom of many febrile diseases.[ In recent years, major public health events, such as severe acute respiratory syndrome (SARS), which are mainly manifested by fever, have attracted worldwide attention.[ A fever may occur in sepsis and other infectious diseases, and may also be seen in many non-infectious diseases, such as malignancy, tissue ischaemia, cerebrovascular accident and autoimmune disease.[ Sometimes it is difficult to diagnose the cause of fever, such as the fever of unknown origin (FUO) and the patient's condition may deteriorate sharply. Early identification of patients at an increasing risk of death may avert adverse outcomes.

Because of the complexity of fever-related illnesses, no biomarker can definitely diagnose sepsis or predict its clinical outcome.[ General-purpose illness severity scoring systems such as the Acute Physiology and Chronic Health Evaluation (APACHE II) often contain too many complex items or are not specific to people with fever.[ With the continuous development of machine learning technology,[ a machine learning approach has outperformed existing clinical decision rules as well as traditional analytic techniques for predicting in-hospital mortality of emergency department (ED) patients with sepsis.[ This study, using big data analysis technology, aimed to explore the key factors associated with adverse prognosis of patients with febrile illness, establish an effective model to predict fatal adverse prognosis in patients with febrile disease, and provide technical support for auxiliary clinical diagnosis and treatment decision-making.

Methods

Ethical approval

This was a retrospective study of ED visits. Ethical approval as a retrospective study and data analysis was performed anonymously, this study was exempt from the ethical approval and informed consent from patients.

Study design

This study retrospectively analyzed the clinical data of 28,400 patients admitted to the emergency room from November 2014 to March 2018. Diagnostic criteria: fever was defined as axillary body temperature equal to or greater than 37.3°C. Inclusion criteria was fever (body temperature ≥ 37.3°C) and age ≥ l2 years. Exclusion criteria: 1. Age <12 years; 2. Patients who died shortly after admission (less than 4 hours after admission) and failed to complete laboratory examination. The included patients were divided into the fatal adverse prognosis group and the good prognosis group. Definition: Adverse prognosis group included the patients who experienced death or cardio-pulmonary resuscitation during emergency treatment, while good prognosis group included those who did not die during treatment in emergency room and did not undergo cardiopulmonary resuscitation, electric defibrillation, endotracheal intubation, tracheotomy and ventilator assisted respiration.

The number of patients with fever admitted to the emergency room was 6254. Among them 307 cases showed adverse prognosis outcomes while 5242 patients had good prognosis. All data elements for each ED visit were obtained from the Emergency Rescue Database of Chinese People's Liberation Army General Hospital. After the first measurement of body temperature greater than 37.2°C during the ED visit, sign values and test values were collected. But only the first set of data obtained or generated within 24 hours of the ED visit were used as prediction variables. Structured Query Language (SQL) queries were written to identify and abstract all demographic information (e.g., age and sex) and ED health status (e.g., vital signs and laboratory result values). Extraction and screening of variables indicators included vital signs, blood routine, blood biochemistry, coagulation function, and arterial blood gas score analysis, screening of key indicators, and completion of prediction model.

Data analysis

Data cleaning was made on the data obtained according to the inclusion and exclusion criteria. The variables that were excessively missing were removed. The individuals whose data showed obvious errors or missing situation were also removed. A baseline description analysis of the remaining variables was performed, where the normal distribution variables are expressed as mean ± standard deviation (SD); the non-normal distribution variables are expressed using the median (interquartile range). A differential hypothesis test was conducted on the variables. The t-test was used for the numerical variables that did not violate normality test and homogeneity test of variance, while the remaining numerical variables were tested by the Mann Whitney U test. With α=0.05 as the test level, the differences of the interested variables were discussed.

In terms of feature selection, recursive feature elimination (RFE) method was used to determine the optimal number of the included variables. The basic idea was that given a specified external learning algorithm, the prediction accuracy of all subsets of variable combinations were calculated through RFE, and number of the subset with the highest prediction accuracy was chosen as the optimal number. Then, the optimal number was used in RFE as the parameter to determine the entry predictor of the final model. In this research, the decision tree was used to select the predictors into the final model. Also, the Pearson correlation coefficient was used to analyze the variable correlation of the factors in model and the results were shown using a heat map. To explore the most important factors, we reduced the optimal number to a smaller one and repeated the above process. The co-existing factors were chosen for further discussion.

We compared the performance of logistic regression, random forest, ada-boost and bagging with the same predictors selected above. Accuracy, F1-score, precision, sensitivity and the areas under receiver operator characteristic curves (ROC-AUC) were used as criteria for judging model performance. The cohort was split as training and testing set at a ratio of 7:3. The training coefficient of Logistic regression was C = 0.01 and the penalty was set to L2. The base estimator of Bagging was decision tree classifier and the training loss criterion was set to entropy. A total of 1000 trainings were performed. The models with good performance were cross verified by 10 folds. For validation, we further collected the emergency room data from December 2018 to December 2019 with the same inclusion and exclusion criterion. The performance was also evaluated by accuracy, F1-score, precision, sensitivity and ROC-AUC. The descriptive baseline analysis and hypothesis testing were done in IBM SPSS Statistics for Windows (Version 19.0. Armonk, NY: IBM Corp., USA), and the rest of the process was done in Python 3.6.1 (https://www.python.org).

Results baseline analysis

The data of a total of 5549 fever patients were obtained, of which 307 patients had fatal poor outcome event and 5242 had no adverse outcome event. The data contained a total of 70 variables. But if certain variable was absent in more than 30% of the patients’ data, it was removed, so the remaining part contained 39 variables. After the deletion of the individuals with missing data, a total of 3474 people did not have adverse outcome events, and 208 patients had fatal adverse outcome events. In the baseline analysis, the statistical results of the resulting 39 variables are shown in Table 1. In the hypothesis test analysis, the variables with statistical differences are shown in Table 1.

Table 1

Baseline analysis results of 39 variables of 3682 patients with fever

Feature selection

After selecting the optimal number by the RFE method with decision tree model, we got the highest accuracy when there were 15 factors in model, and then 15 was taken as the coefficient of RFE to select the specific factors. The obtained 15 factors through these processes were: heartrate (HR), respiratory rate (RR), systolic blood pressure (SBP), diastolic blood pressure (DBP), pulse oxygen saturation (SPO2), temperature (T), creatine kinase myocardial isoenzyme (CK-MB), total bilirubin (TBIL), lactate dehydrogenase (LDH), serum amylase (AMY), serum lipase (LIP), cardiac troponin T (CTnT), aerum kalium(K), total protein (TP), and albumin (ALB). After that, the Pearson correlation test was done, with the details shown in Figure 1. The top 6 factors correlated with the results were: CTnT (0.260), LDH (0.210), SPO2 (-0.180), CKMB (0.180), HR (0.150), and T (0.150).

Figure 1

Heatmap of the selected factors into the final model.

Model performance

These factors were then put into logistic regression, decision tree, ada-boost and bagging models were also chosen to select the most suitable algorithm with best performance. In the training model, the accuracy of logistic regression, decision tree, adaboost and bagging was 0.951, 0.928, 0.924 and 0.924, F1-scores were 0.938, 0.933, 0.930 and 0.930, the precision was 0.943, 0.938, 0.937 and 0.937, the sensitivity was 0.571, 0.524, 0.524 and 0.762, and the ROC-AUC were 0.808, 0.738, 0.736 and 0.885, respectively (Table 2; Fig. 2). Ten-fold cross-validation was performed on the logistic and bagging models with better comprehensive performance, and their ROC-AUC were 0.80 and 0.87, respectively. In validation part, the decision tree model got the highest accuracy and F1-score, while the bagging model got the highest sensitivity and ROC-AUC.

Table 2

Key metrics results of logistic regression, decision tree, adaboost and bagging

Figure 2

Receiver operator characteristic (ROC) curves with area-under-the-curve (AUC) values for logistic regression, decision tree, adaboost and bagging.

The coefficients and OR values of the variables in the logistic regression are shown in Table 3. Among them, the top six were: CTnT (coef=0.346, OR = 1.413), T (coef=0.235, OR = 1.265), RR (coef=-0.206, OR = 0.814), K (coef=0.137, OR = 1.146), SPO2 (coef= –0.101, OR = 0.904) and ALB (coef= –0.043, OR = 0.958). The weights of the variables in the bagging model are shown in Table 3. Among them, the top six were: CTnT (0.221), RR (0.127), LDH (0.088), AMY (0.087), HR (0.058), and SBP (0.047).

Table 3

Coefficient and OR value of variables in logistic regression and bagging model

To explore the most important factors, we reduced the optimal number of RFE to 11 and repeated the above process. Similarly, the bagging model showed the best performance on sensitivity and ROC-AUC, while the logistic regression showed the best accuracy. The details are shown in Table 2. Ten-fold cross-validation was performed on the logistic and bagging models, and their ROC-AUC were 0.77 and 0.86, respectively. After the number of factors in the model was reduced to 11, the top six coefficients of the logistic model were RR (coef= –0.189, OR = 0.828), SPO2 (coef= –0.14, OR = 0.864), T (coef=0.191, OR = 1.210), Na (coef=0.115, OR = 1.22), CTnT (coef=0.378, OR = 1.460), and K (coef=0.20, OR = 1.227). The top 6 of the bagging model were CTnT (0.251), RR (0.145), P (0.088), ALB (0.711), RBC (0.070), and T (0.070). Also, in validation part, the decision tree model got the highest accuracy and F1-score, while the bagging model got the highest sensitivity and ROC-AUC. The details are shown in Table 2.

Discussion

Modern medicine often involves collecting large amounts of physiological data, laboratory resultsand imaging data into electronic records. The data, however, are complex and multidimensional. It is difficult to find subtle relationships between these data and clinical outcomes using traditional statistical techniques. In this study, the advantage of machine learning is to provide possible innovative solutions for clinical doubts, to find important indicators that may be ignored in treatment for clinicians, and to provide guidance for clinical decision-making to prevent adverse prognosis.

We selected 70 clinical variables associated with “fever” as candidate indicators. It can be seen that in the model with 15 impact factors (HR, RR, SBP, DBP, SPO2, T, CK-MB, TBIL, LDH, AMY, LIP, CTnT, K, TP and ALB) and in the model with 11 impact factors (RR, SPO2, T, Na, CL, CTnT, K, ALB, Ca, P and RBC), the co-existing influence factors were: RR, SPO2, T, CTnT, K and ALB. These factors could be the main clinical indicators of concern in this experiment, which should be given clinical attention.

Our results suggest that CTnT has the largest weight in the prediction model of adverse prognosis patients with fever, the serum levels of CK, CK-MB, CTnT and BNP in patients with fever adverse prognosis group were higher than those in patients with good prognosis group. Further logistic regression analysis showed that serum CK-MB and CTnT levels were independent risk factors for poor prognosis in patients with fever, which should be given clinical attention. Troponin T is one of the clinically recognized markers to reflect myocardial injury. When its level increases, it can reflect the increased severity of myocardial injury, with high specificity.[ Previous studies have suggested that myocardial injury, as reflected by elevated cardiac troponin levels in plasma, is common in patients with community-acquired pneumonia (CAP).[ The main reason for poor prognosis in patients with severe sepsis is that patients often have cardiac dysfunction. Once the cardiovascular system is damaged, the death rate of patients increases greatly.[

In this study, ALB was found to be a key factor in predictingthe death or poor prognosis in patients with fever. Hoeboer et al[ concluded that albumin rather than C-reactive protein may be valuable in predicting and monitoring the severity and course of acute respiratory distress syndrome in critically ill patients with or at risk for the syndrome after new onset fever. Therefore, active correction of hypoproteinemia in patients with fever may improve prognosis.

This study suggested that the vital signs of body temperature, HR, RR, SBP and DBP had important clinical value in predicting the fatal adverse prognosis in patients with fever, and the sum of their weights was 0.3186. It also showed that SPO2 was one of the independent protective factors for poor prognosis in patients with fever. When HR, RR and SBP increased in patients with fever, SPO2 should be detected in time, and should be improved by early non-invasive or invasive mechanical ventilation treatment, so as to reasonably shorten the mechanical ventilation time to reduce the mortality of patients. Single-factor analysis showed that serum sodium of adverse prognosis group was higher than that of good prognosis group, suggesting that for clinical patients with severe fever, timely fluid supplement should be given to correct hyper serum sodium caused by dehydration and other reasons.

In addition, this study suggested that serum K, LDH, AMY, TP and TBIL were independent factors affecting poor prognosis in patients with fever. In patients with fever, attention should be paid to the changes of the above indicators to save lives.

In the methodological part, when 15 variables were selected, the best results were obtained from the logistic regression model and the bagging model. Logistic regression had the highest accuracy (0.951), while the bagging model had the highest AUC score (0.885). Given the clinical application of the model, the clearer the focus on the indicators, the better it could help doctors make decisions. In addition, if more indicators are involved, it is not always possible to ensure that every factor can be obtained in a short time. Therefore, in the case that the performance of the model does not change much, we believe that the model with fewer factors is more conducive in the clinic. The model may be used to predict, verify and improve future clinical medical practices.

We need to admit that our study has two limitations. First, when we selected cohort, we did not separate people with diversified causes of fever into different groups. Now we cannot calculate the accuracy of different fever causes. This can serve as the direction for our future research. Second, as this was a retrospective study, there were many deficiencies in the data, and the data volume is not very ideal, we hope to have more data in the future.

In conculusion, big data analysis method was adopted to establish a scientific and objective prediction and evaluation model for adverse prognosis of patients with fever in this study. We found the main clinical indicators of concern, and the prediction model established had high diagnostic accuracy and reliability, which may be conducive to the early identification of critical patients with fever by physicians, thus improving the prognosis of patients with fever. The application of big data analysis combined with medical research is helpful to improve the diagnosis and treatment level of febrile critical diseases and the prevention and control of infectious diseases. Research on adverse event prediction model for critical patients with fever quantifies the recognition of critical diseases related to fever and provides a reference model for other similar clinical decision support studies.

Acknowledgements

None.

Funding

This study was supported by grants from The Big Data R&D Project of the People's Liberation Army General Hospital (No. 2017MBD-30), and Science and Technology Innovation Nursery Fund Project of the People's Liberation Army General Hospital (No. 17KMM50).

Conflicts of interest

Competing interests: The authors declare that they have no competing interests.

Author contributions

None.