Model-based Prediction of Critical Illness in Hospitalized Patients with COVID-19.

Background The prognosis of hospitalized patients with severe coronavirus disease 2019 (COVID-19) is difficult to predict, and the capacity of intensive care units was a limiting factor during the peak of the pandemic and is generally dependent on a country's clinical resources. Purpose To determine the value of chest radiographic findings together with patient history and laboratory markers at admission to predict critical illness in hospitalized patients with COVID-19. Materials and Methods In this retrospective study, which included patients from March 7, 2020, to April 24, 2020, a consecutive cohort of hospitalized patients with real-time reverse transcription polymerase chain reaction-confirmed COVID-19 from two large Dutch community hospitals was identified. After univariable analysis, a risk model to predict critical illness (ie, death and/or intensive care unit admission with invasive ventilation) was developed, using multivariable logistic regression including clinical, chest radiographic, and laboratory findings. Distribution and severity of lung involvement were visually assessed by using an eight-point scale (chest radiography score). Internal validation was performed by using bootstrapping. Performance is presented as an area under the receiver operating characteristic curve. Decision curve analysis was performed, and a risk calculator was derived. Results The cohort included 356 hospitalized patients (mean age, 69 years ± 12 [standard deviation]; 237 men) of whom 168 (47%) developed critical illness. The final risk model's variables included sex, chronic obstructive lung disease, symptom duration, neutrophil count, C-reactive protein level, lactate dehydrogenase level, distribution of lung disease, and chest radiography score at hospital presentation. The area under the receiver operating characteristic curve of the model was 0.77 (95% CI: 0.72, 0.81; P < .001). A risk calculator was derived for individual risk assessment: Dutch COVID-19 risk model. At an example threshold of 0.70, 71 of 356 patients would be predicted to develop critical illness, of which 59 (83%) would be true-positive results. Conclusion A risk model based on chest radiographic and laboratory findings obtained at admission was predictive of critical illness in hospitalized patients with coronavirus disease 2019. This risk calculator might be useful for triage of patients to the limited number of intensive care unit beds or facilities. © RSNA, 2020 Online supplemental material is available for this article.

Summary

Chest radiographic findings, together with patient history and laboratory values, predicted critical illness in hospitalized patients with COVID-19.

■ A model using baseline patient characteristics, laboratory markers, and chest radiography can predict short-term critical illness in hospitalized patients with COVID-19, with an internally validated AUC = 0.77.

■ At an example model risk threshold of 0.70, 71 of 356 patients would be predicted to develop critical illness of which 59 (83%) would be true-positives.

■ A risk calculator has been made available for download: Dutch COVID-19 risk model (https://docs.google.com/spreadsheets/d/1eFrdHxnOA-M_P-ijxnC2u30qk7IhMVV6YvHvJhrZ8Ws/edit#gid=0) (see Appendix E2).

Introduction

The worldwide spread of the novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), leading to coronavirus disease 2019 (COVID-19) caused more than 19 million confirmed cases and over 700,000 deaths up to August 9th, 2020 (1). The majority of patients with COVID-19 present without any symptoms or only mild respiratory symptoms, not requiring hospital admission. However, a proportion of patients are hospitalized because of systemic or severe respiratory symptoms (2-5) of which a subgroup may undergo fast clinical deterioration requiring invasive ventilation and intensive care unit (ICU) treatment, with lethal outcome in the worst-case scenario. This demand for ICU treatment exceeding the options of existing facilities lead to the mobilization of new ICUs, worldwide drastic actions of unprecedented social distancing and different degrees of lockdown to decrease the number of critically ill patients. A number of factors have been described that contribute to poor prognosis. Higher mortality rates occur with increasing age and presence of chronic comorbidities (6-8). A wide spectrum of laboratory values have been linked to poor prognosis and risk of ICU admission (2-4, 6, 7, 9-15).

Although some guidelines do not advise screening of patients suspected of COVID-19 with chest radiography or CT (16, 17), imaging remains important for the diagnosis and risk stratification at admission, as in other types of pneumonia (5, 18-20). In the previous severe acute respiratory syndrome epidemic in 2003 and Middle Eastern respiratory syndrome epidemic in 2012-2015, chest radiography findings were found to correlate with prognosis (21-25).

Publications exploring prognostic markers for COVID-19 thus far largely focused on laboratory findings alone, considered the presence of any abnormality on chest radiography to be a risk factor, or utilized the longitudinal development of disease on thoracic CT. The body of knowledge regarding the complementary information of clinical, laboratory and chest radiography findings is limited. Recently, a Chinese study was published including 1590 patients with COVID-19 of whom 131 (8%) became critically ill, and therefore needed ICU admission or died. The authors presented a predictive risk-score based on 10 variables, showing good discrimination (AUC = 0.88) (26).

The purpose of this study was to determine the prognostic value of the combined information of patient history, readily available laboratory markers, and chest radiography at admission to predict critical illness, defined as ICU admission for invasive ventilation and/or death, in hospitalized patients with known RT-PCT positive COVID-19.

Materials and Methods

Study population and data collection

In this retrospective cohort-study, inclusion took place in two large community teaching hospitals (Meander Medical Center, Amersfoort, the Netherlands; Isala Hospitals, Zwolle, the Netherlands)during the peak of the SARS-CoV-2 pandemic. The study was approved by the local institutional review boards of both hospitals, and written informed consent was waived (20-036, and 200435).

Using our hospital’s COVID-19 registry, all consecutive patients between March 7th and April 24th 2020 suspected of COVID-19 on admission at the emergency department were derived. Patients that did not have a positive real- time reverse transcription polymerase chain reaction (RT-PCR) proven COVID-19 or patients that were not hospitalized were excluded. First RT-PCR tests were taken within 24 hours of hospital admission. If the first test was negative but clinical suspicion remained subsequent tests were performed.

Other exclusion criteria were no chest radiography on admission, transferred patients with uncertain onset of symptoms, status after pneumonectomy, and children (<18 years).

Medical records were reviewed to collect information on patient gender, age, comorbidities (hypertension as diagnosed by a physician, diabetes (either type 1 or type 2 diabetes), history of any type of cancer (present or previous), chronic obstructive lung disease (i.e. COPD or asthma) and cardiac diseases (i.e. ischemic heart disease and/or pre-existent cardiomyopathy)), days of symptoms prior to admission, and temperature at admission (in Celsius). From the blood samples obtained at emergency ward presentation, the following markers were collected: white blood cell (WBC) count, lymphocyte count (x109/L), neutrophil granulocyte count (x109/L), C-reactive protein (CRP in mg/L), and lactate dehydrogenase (LDH in IU/L). Neutrophil-to-lymphocyte ratio (NLR) was calculated from the blood samples. Ferritin, procalcitonin, D-dimer and fibrinogen were not part of the standard laboratory work-up at admission in both centers and were not included. Subsequent laboratory tests were not taken into account.

Analysis of the chest radiographs

Chest radiographs obtained within 24 hours after emergency ward presentation were used for analysis. The collected radiographic features included a visual assessment of body habitus (evidently large vs. normal) and the distribution of lung abnormalities into predominantly central (i.e. perihilar), peripheral (i.e. attached to the pleural surface), or diffuse (i.e. both central and peripheral abnormalities). If other non-infectious extensive lung abnormalities, such as large tumors, extensive emphysema or fibrosis, were present in the image, these images were marked as images with pre-existing lung disease. Furthermore, the chest radiograph was divided into 4 zones: right upper lung, left upper lung, right lower lung, left lower lung, using simple anatomic landmarks (Figure 2). Then, the extent of abnormal lung parenchyma was visually scored as 0 = no involvement, 1 = mild/moderate involvement (estimated involvement 0-50% of lung parenchyma), and 2 = severe involvement (estimated involvement >50% of lung parenchyma) per zone resulting in a score between 0 and 8 (chest radiography score). Chest radiographs were independently analyzed by a chest radiologists (SS and CSP in hospital A, 5 and 25 years of experience, RvD in hospital B, 7 years of experience), blinded to laboratory values and patient outcome. A training session with representative cases was done to ensure a common understanding of the reading task. Image interpretations were done blinded for the clinical outcome, and no double readings were performed.

Figure 2:

Chest radiography scoring: in a 50-year-old patient with RT-PCR proven coronavirus disease 2019 who was hospitalized but not admitted to the ICU. Chest radiography scoring was as follows: right upper lung zone mild/moderate involvement (1 point), the right lower lung zone mild/moderate involvement (1 points), as well as for the left upper and lower lung zones mild/moderate involvement (both 1 point), resulting in a cumulative score of 4.

Outcome

Critical illness was defined by admission to the ICU for invasive ventilation and/or death. Patients admitted to the COVID-19 cohort wards without the need for mechanical ventilation and alive at follow-up are referred to as non-critically ill in this study. The outcome was determined by reviewing the patient’s records. Median follow-up time was 34 days (range, 12-53 days); no patients were lost to follow-up.

Statistical analysis

To achieve valid results for the chest radiography score and to reduce overfitting, a power analysis with 1-β = 0.8 (α=.05; 2-sided testing) resulted in a minimally required sample size of n=300. Continuous data are presented as means with standard deviations or medians with ranges as appropriate. Categorical data are presented as proportions. For cases with a missing value (i.e., CRP, LDH, lymphocytes, neutrophils, and days of symptoms, 21/356, 6%), data were imputed using multiple imputation (27). Prior to imputation, data was analyzed to ensure the assumption missingness at random was reasonable. Univariable analyses were performed by means of a Chi-square Test, Mann-Whitney U Test or Student T-Test as appropriate. Variables for the multivariable logistic regression were chosen based on univariable testing with P<.1 and clinical reasoning to prevent multiple testing.

Multivariable logistic regression (backward elimination with fractional polynomial transformation) was performed with P<.2 for the inclusion of covariables and P<.05 for fractional polynomial transformation of variables. Fractional polynomial transformation was chosen because of the exploratory character of the study (28). Discrimination performance was determined by the area under the receiver operating characteristic curve (AUC). Calibration was assessed graphically (29). Internal validation was done by non-parametric bootstrapping (k=2000) for the AUC (30).

A decision curve for our model was included to aid clinical decision making based on a risk threshold preference (31). A decision curve is a plot of standardized net benefit against the risk probability threshold. The net benefit describes the difference between the benefit of true positive calls (correct prediction of critical illness) and the harm of false positives, the latter being adjusted to how much a false positive outcome weighs compared to a true positive call (e.g., if the harms of a false positive outcome is considered to weigh twice as much the benefits of a true positive call, the odds ratio is 2:1, i.e., 2/1 = 2). The standardized net benefit is net benefit divided by the prevalence of the outcome (the proportion of patients with critical illness in this study) (32).

Analysis was conducted in IBM SPSS Statistics for Windows (Version 26.0; IBM, Armonk, NY) and R version 3.6.3 (R Foundation for Statistical Computing, Vienna, Australia). Results of the logistic regression are presented as adjusted odds ratios (ORs) with 95% confidence intervals (CIs). All tests were two-sided, a P-value < .05 was deemed statistically significant.

Results

Patient’s characteristics

A total of 356 patients were included in the study (Figure 1), with 121/356 (34%) patients from hospital A and 235/345 (66%) patients from hospital B. Between the patients from the two participating centers, no significant differences were found for laboratory values, chest radiography score, or duration of symptoms before admission (Table 1). Mean patient age was 69 years (±12) and 237/356 (67%) were male. Median duration of symptoms was 7 days (range, 0-35 days). A full summary of patient characteristics per hospital is given in Tables E1 and E2.

Figure 1:

Flowchart of patient inclusion. Patients without or with a negative RT-PCR, patients who were not hospitalized, and a small number of patients who did not receive a chest radiograph or were transferred from another hospital without a clear onset of symptoms were excluded. A total of 356 patients were eligible for this study.

Table 1.

Demographics and Clinical Findings per Disease Category.

Of all included patients 168/356 (47%) developed critical illness (ICU admission and/or death). Of those 24/168 (14%) died at the ICU, 73/168 (44%) died at a COVID-19 cohort ward, and 71/168 (42%) patients were admitted to ICU but were alive at follow-up.

For all patients, the median duration of symptoms at admission was 7 days (range 0-35 days). Median time-to-intubation from admission was 4 days (range, 0-18 days) and 8 days (range, 1-35 days) since the onset of symptoms. Median survival in the deceased population was 4 days (range, 0-30 days) after admission and 16 days (range, 4-39 days) since the onset of symptoms.

Univariable analysis

Patients who developed critical illness were significantly older (mean age, 70 years ±11 vs. 67 years ±13, p=.03), more often male (124/168 [73%] vs. 114/188 [61%], p=.01), and more often had chronic obstructive lung disease (40/168 [24%] vs. 29/188 [14%], p=.06) (Table 1).

Regarding chest radiographic findings, patients who developed critical illness more often had higher chest radiography scores (mean 4.4±1.9 vs. 3.3±1.8, p<.001) and bilateral involvement (144/168 [86%] vs. 137/188 [73%]; p=.006) at admission. Regarding the distribution of abnormalities on chest radiography, predominantly central or diffuse opacifications were correlated with the development of critical illness (p=.001) (Table 2).

Table 2:

Chest Radiographic Findings per Disease Category

Patients who developed critical illness more often had leukopenia (37/168 [22%] vs. 36/188 [19%], p=.004) and lymphopenia (84/168 [51%] vs. 63/188 [34%], p=.002) and a higher NLR (median 7.0 [range, 0.5-66.5] vs. 5.75 [range, 0.09-47.0], p<.001) at admission. Additionally, they exhibited higher CRP (mean 139.1±100.5 vs. 94.6±74.6, p=.001) and LDH (421.2±251.4 vs. 317.1±139.9, p=.001). No significant difference was found for the duration of symptoms (p=.15), but the proportion of patients with a symptom duration >7 days was slightly smaller in the critical group (64/168 [38%] vs. 87/188 [48%]; p=.08). No significant difference was found for body habitus on chest radiography (p=.12), temperature (p=.43), and the other comorbidities (Table 2).

Model development and internal validation

After backward elimination, the following variables remained in the model: male gender(adjusted OR 1.5, 95% CI 0.9-2.7, P=.10), obstructive lung disease (adjusted OR 1.9, 95% CI 1.0-3.5, P=.045), symptom duration > 7 days (adjusted OR 0.5, 95% CI 0.3-0.8, P=.003), (neutrophils/10)3 (adjusted OR 1.8, 95% CI 1.2-2.9, P=.01), (CRP/100)-2 (adjusted OR 0.98, 95% CI 0.95-0.997, P=.08), LDH/1000 (adjusted OR 8.4, 95% CI 1.7-49, P=.01), diffuse distribution (vs. peripheral) (adjusted OR 1.9, 95% CI 1.1-3.3, P=.03), central distribution (vs. peripheral) (adjusted OR 3.6, 95% CI 1.1-9.3, P=.01), and chest radiography score (adjusted OR 1.2 per point increasing, 95% CI 1.1-1.5, P=.01). There were no significant interactions between variables. A summary including a description of variables entered in the full model is given in Table 3. The AUC for this model was 0.77 (95% CI 0.72-0.81; P < .001). The model showed good calibration, with a slope of 0.974 (Fig E1 [Appendix E1]). The average AUCs of the models derived from 2000 bootstrap samples on the bootstrap and original samples were 0.78 (95% CI 0.78-0.78) and 0.75 (95% CI 0.75-0.76), respectively (optimism = 0.780 - 0.754 = 0.026).

Table 3.

Dutch COVID-19 Prognostic Model for Critical Illness in Known RT-PCR Positive and Hospitalized Patients.

Risk score and risk calculator

The risk score was developed using the regression coefficients of the logistic regression model (β) resulting in the following formula: probability = 1/(1+exp(-(intercept + gender × β + obstructive lung disease × β + > 7 days of symptoms × β + (neutrophils/10)^3 × β + (CRP/100)^-2 × β + (LDH/1000) × β + distribution × β + CXR score × β))). The risk calculator can be accessed on line (hyperlink: Dutch COVID-19 risk model) (see Appendix E2).

Decision curve analysis

The decision curve (Figure 4) shows that our model (red line) provides a superior net benefit for risk thresholds of 0.5 or higher, equivalent to a cost-benefit ratio of 1:1 or greater in favor of patients who do not develop critical illness. For example, the risk score threshold of 0.70 (cost-benefit-ratio 7:3) is at the 80th percentile of the population, meaning that 20% of the population will get a positive test (i.e. positive for development of critical illness). In our study this is 71/356 (20%) patients. Of these 59/71 (83%) would be true positives and 12/71 (17%) false positives. The calculated net benefit is: ((59/356 - (12/356 x 0.7/ (1-0.7))) = 0.087. The standardized net benefit is: 0.087 / (168/356) = 0.18. This calculation would be equivalent to, out of 100 patients analyzed, 18 of the critical patients would be correctly predicted without any incorrect predictions among severe patients. Note that the risk probability threshold should not be selected based on the highest net benefit, but rather on the clinical scenario (32).

Figure 4:

Decision curve for the Dutch COVID-19 risk model. The gray and black lines (horizontal) represent the scenarios where all or none of the hospitalized patients would be prospectively determined by the risk model, respectively. The red line demonstrates the net benefit of the risk model dependent at the chosen risk threshold. The accompanying thinner lines represent the 95% confidence intervals.

Receiver operating characteristics curve for the multivariable logistic regression model predicting critical illness in patients with COVID-19. The model including gender, clinical, laboratory, and imaging (chest radiography) parameters (n=356) reached an AUC of 0.77 for prediction of critical illness.

Examples of patients with coronavirus disease 2019 who did (i.e. required intubation and or died) and did not develop critical illness (i.e. did not need intubation and were discharged). A, Radiograph in an 80-year-old woman with 7 days of symptoms prior to emergency ward presentation. She had hypertension and chronic obstructive lung disease as comorbidity. Laboratory findings were as follows: neutrophil granulocytes 6.4x109/L, C-reactive protein 282 mg/L, lactate dehydrogenase 335 IU/L. On chest radiography she exhibited diffuse bilateral opacities and a chest radiography score of 4. The calculated risk score was 0.69. She developed critical illness and died 4 days after hospital admission. B, Radiograph in a 46-year-old woman with no comorbidities had 9 days of symptoms prior to emergency ward presentation. Laboratory findings were as follows: neutrophil granulocytes 3.8x109/L, C-reactive protein 29 mg/L, lactate dehydrogenase 429 IU/L. Chest radiography showed bilateral peripheral opacities and a chest radiography score of 5. The calculated risk score was 0.24. She was not admitted to the ICU and successfully discharged after 2 days of hospitalization.

Discussion

In this study we developed a risk model to predict short -term critical illness defined as ICU admission with mechanical ventilation and / or death in hospitalized patients with coronavirus disease 2019 (COVID-19). The model was based on clinical, laboratory, and radiographic findings obtained at admission in 356 patients and showed an AUC of 0.77 (95% CI 0.72-0.81; P <.001).

We show a complementary effect of clinical, laboratory, and specific chest radiography findings for the prediction of short-term critical illness in hospitalized patients with COVID-19. Two recently published retrospective studies explored the prognostic value of chest radiographic findings (26, 33). Toussie et al. included patients under 50 years old with confirmed COVID-19 (n=338) presenting at the emergency ward, thus a younger population with fewer comorbidities than in our study. The authors looked at the predictive value of chest radiography for various outcomes, including hospital admission, and did not include laboratory markers in their models. In the study by Liang et al., a clinical risk score was developed on a large cohort of hospitalized patients with COVID-19 (n=1590) recruited from a national Chinese database. Clinical parameters, laboratory findings, and presence of any abnormality on chest radiography were used to predict critical illness. Their population was also relatively healthy: the mean age was under 50 and only 8% (131/1590) of patients developed critical illness. This also explains why age, presence of severe symptoms (e.g., unconsciousness), or number of comorbidities were very strong predictors in their model and contributed to a high discriminatory predictive ability (AUC = 0.88). Our study population however was different and consisted exclusively of patients with severe disease who required hospital admission. Our population had a large proportion that developed critical illness (168/356; 47%). It is likely that the discriminatory ability of our model would improve when also applied to outpatients, since those patients would show less disease findings than our study population (26, 33). Furthermore, Liang et al. included only one measure of abnormality on chest radiography (yes vs. no), despite it is known that the extent of lung involvement in viral pneumonia visible on chest radiography or CT may predict a worsened outcome (21-23, 25, 33, 34). Moreover, we showed that the distribution of lung abnormalities on chest radiography provides additional prognostic information as well. Patients with diffuse or central lung abnormalities, had higher odds for critical illness in our risk model than only peripheral abnormalities.

Although we included several chest radiography characteristics and laboratory markers in our analysis, we aimed to keep our risk model simple so that it can be used in resource limited areas. We chose to use chest radiography instead of CT and the scoring of the extent of disease on chest radiography was a simple semi-quantitative visual assessment, since estimation of the precise pulmonary involvement on chest radiography is challenging, especially from portable bedside chest radiographs. Secondly, we did not include many of the commonly used laboratory markers such as ferritin, procalcitonin, D-dimer, fibrinogen (3, 4, 6, 7, 11, 26) in our analysis, since some laboratory tests are not widely available or are expensive.

Besides providing accuracy measures such as AUC, sensitivity, and specificity, we also performed decision curve analysis. This method has been increasingly used to assess diagnostic tests and prediction models. The advantage of decision curve analysis is that it incorporates the preference of the physicians, patients, and/or policy makers into analysis. It weighs the benefits and costs at a certain threshold, i.e., the minimum event risk probability which defines a positive test outcome. When ICU facilities are limited, one may prefer to predict critical illness with high specificity, i.e., a low false positive rate. We suggested a risk probability threshold of 0.70 using our model; this means that the early preparation for clinical deterioration would be justified if there were at least 2.3 true positives (0.70/[1–0.70]) per false positive. In our population, this would have resulted in 71 of 356 patients having a positive test (i.e., be predicted to develop critical illness) of which 59 (83%) would be true positives. This results in a standardized net benefit of 0.18. This concept is easiest understood by first considering the baseline where all patients are assumed to have a negative test, i.e., zero true positives and zero false positives. In comparison: By applying our model to 100 patients using the probability threshold of 0.70 for a positive test, an additional 18 true positives could be detected while keeping the number of false positives at zero. A more detailed explanation about decision curve analysis is provided in Appendix E3.

There are a number of limitations to this work. Our prediction model has not been externally validated, and is based on a retrospective cohort of only 356 patients from two hospitals. We included known predictive parameters of worse outcome in patients with COVID-19 at the time of inclusion of our study population. However, we were not able to incorporate all known clinical parameters and laboratory values such as, body mass index, history of tobacco use, oxygenation levels at admission or pulmonary function, D-dimer, ferritin, mainly because they were not available for all patients during the peak of the pandemic.

Chest radiographs were independently analyzed by radiologists from their respective centers using a scoring system designed for this study, and heterogeneity among radiologists was unaccounted for. However, no significant differences in chest radiography scoring were seen between centers with similar source populations, suggesting satisfactory inter-rater agreement.

Lastly, this study aimed to identify predictors and develop a risk model for critical illness in a setting where the knowledge regarding risk factors for critical illness in COVID-19 is still limited. The risk model shows good calibration and robustness upon internal validation (optimism = 0.026) and will therefore very likely perform well in a similar population. However, the performance may differ in other source populations.

In summary, we found that basic laboratory findings and a simple assessment of parenchymal involvement on a chest radiography acquired at hospital admission may provide complimentary information for short-term prognosis in hospitalized patients with coronavirus disease. We show that a simple model composed of gender, chronic obstructive lung disease, neutrophil granulocytes, C-reactive protein, lactate dehydrogenase, distribution of lung abnormalities and a chest radiography score were predictive of the need for mechanical ventilation/death among hospitalized patients with coronavirus disease. A risk calculator has been made available for download: Dutch COVID-19 risk model (Appendix E2).

Table 4:

Critical Illness Prediction Accuracy at Different Model Risk Thresholds in 356 Known RT-PCR Positive and Hospitalized Patients.