External Validation of 4C ISARIC Mortality Score in Critically ill COVID-19 Patients from Saudi Arabia.

Background: ISARIC mortality score is a risk stratification tool that helps predict the in-hospital mortality of COVID-19 patients. However, this tool was developed and validated in a British population, and thus, the external validation of this tool in local populations is important.
Objectives: External validation of the ISARIC mortality score in COVID-19 patients from a large Saudi Arabian intensive care unit (ICU).
Methods: This is a retrospective study that included all adult patients with COVID-19 admitted to the ICU of King Saud Medical City, Riyadh, Saudi Arabia, from March 2020 to June 2021. Patients who were pregnant or had pulmonary tuberculosis/human immunodeficiency virus were excluded along with patients with missing variables. Data were collected to calculate the ISARIC mortality score and then fitting receiver operator characteristic curve against patients' outcome.
Results: A total of 1493 critically ill COVID-19 patients were included. The mortality was 38%, the area under the curve of the score was 0.81 (95% confidence interval [CI]: 0.79-0.83, P < 0.001) and the cutoff value correctly classified 72.7% of the cohort. The cutoff value of >9 had sensitivity of 70.5% (95% CI: 66.6-74.3); specificity, 73.97% (95% CI: 71-76.8); positive predictive value, 62.4% (95% CI: 59.5-65.2) and negative predictive value, 80.2% (95% CI: 78.2-82.4).
Conclusion: The ISARIC score was found to have excellent predictive ability for mortality in critically ill COVID-19 patients in our Saudi Arabian cohort. A cutoff score of >9 was the optimal criterion. Copyright:

INTRODUCTION

The COVID-19 pandemic has had a significant impact worldwide, with >262 million cases reported as of December 1, 2021.[12345] The spectrum of clinical presentation of COVID-19 infection is ample, ranging from being asymptomatic, which accounts for the majority of cases, to mild cases requiring minimal support, up to critically ill cases requiring intensive care unit (ICU) admission and mechanical ventilation.[6] Life-threatening COVID-19 presentations are commonly characterized by acute respiratory distress syndrome, multiorgan failure, hyperinflammatory responses and thromboembolic manifestations, among others.[7] Therefore, it is comprehendible that the mortality rate in severe and critically ill COVID-19 patients is high,[8] particularly when complemented with poor prognostic factors such as comorbidities, elevated inflammatory biomarkers and old age.[37]

The variability in presentation necessitated the development of risk stratification tools that would allow early identification of COVID-19 patients at higher risk of mortality, using readily available objective criteria.[8910] Accordingly, Knight et al.[8] utilized The International Severe Acute Respiratory and emerging Infections Consortium (ISARIC) World Health Organization Clinical Characterization Protocol to develop and validate such a tool, the ISARIC 4C mortality score (hereafter referred to as ISARIC), which likely outperforms other risk stratification models.[11] The ISARIC mortality score utilizes variables that are readily available on hospitalization, thereby avoiding reliance on parameters such as radiological imaging or those that only become available after ICU admission.[1213] The model has been demonstrated to have a high discriminatory ability for in-hospital mortality and categorizes patients into four categories of severity with a uniformly increasing mortality risk [Supplementary Table 1].

Supplementary Table 1

Categories and risk of mortality of the 4C mortality score, in the original and current study

4C mortality score	Risk group	Risk of in-hospital mortality (%)
		Original study*	Current study
0-3	Low	1.2-1.7	2.3-7.3
4-8	Intermediate	9.1-9.9	20.8-28.9
9-14	High	31.4-34.9	43.9-52.7
≥15	Very high	61.5-66.2	76.6-86.8

*Source.[8] ISARIC – International Severe Acute Respiratory and emerging Infections Consortium; WHO – World Health Organization

However, the tool's development and validation were primarily done for the population in the United Kingdom. For generalizability, there is a need to validate the score within different populations. To the best of our knowledge, no study from Saudi Arabia has previously validated this tool, and thus, the current study was performed to validate the performance of the ISARIC score within the setting of a Saudi Arabian ICU.

METHODS

Study design, patients, and setting

This is a retrospective study that included all adults with COVID-19 admitted to the ICU of King Saud Medical City, Riyadh, Saudi Arabia, between January 1, 2020, and June 30, 2021; the first case was recruited on March 20, 2020. The study was approved by the hospital's institutional review board.

King Saud Medical City is a 1200-bed government hospital, with an ICU capacity that has been increased from 127 to 300 beds since the beginning of the COVID-19 pandemic. The ICU is closed, operated round the clock by intensivists and has a nurse:patient ratio of 1:1. For the pandemic, almost half of the beds were converted to single rooms, and all beds are fully equipped with capabilities of invasive and noninvasive monitoring and ventilation. The admission criteria of COVID-19–positive patients to the ICU were broad, as the center serves as a COVID-19 referral center of the region. COVID-19 was confirmed by nasopharyngeal reverse transcription–polymerase chain reaction (RT-PCR), and patients were admitted if they had any of the following: supplemental oxygen of any kind to maintain peripheral oxygen saturation (SpO2) ≥92%, mechanically ventilated, any level of consciousness disturbance or respiratory rate ≥28 per minute. The criteria were the same during both waves of COVID-19 and in line with the COVID-19 management recommendations of the Saudi Ministry of Health.[14]

Inclusion and exclusion criteria

All adult (≥18 years) patients with a nasopharyngeal RT-PCR–confirmed COVID-19 admitted to the ICU of our hospital were included in this study. Nasopharyngeal RT-PCR was repeated for all patients who previously tested positive either before ICU discharge or after 2 weeks of ICU admission, whichever occurred first.

Pregnant women as well as those known to have pulmonary tuberculosis (PTB) and human immunodeficiency virus (HIV) were excluded from this study, as common COVID-19 medications such tocilizumab or antiviral drugs may not be offered to these patients. After the completion of data collection, patients with missing variables that preclude the calculation of ISARIC score were also excluded.

Data management

For calculating the ISARIC score[8] [Supplementary Table 2], the following variables were collected from the electronic database of our ICU and/or manually from the patients’ medical records: age; gender; number of comorbidities; respiratory rate, SpO2 on room air, and Glasgow coma scale (GCS) at hospital admission; first available blood urea level (mmol/L); and C-reactive protein (CRP) (mg/L). In addition, ethnicity, smoking status, and nature of comorbidities were also noted. The collection of data was performed by a group of data collectors supervised by authors: IA, ZS, MA and WA. The electronic records of the hospital were used by the data collectors to gather demographic data, COVID-19 test and laboratory results along with all relevant dates such as admission and discharge, while data of the condition upon hospitalization (such as GCS and SpO2) were gathered manually from the patients’ files, and the aforementioned authors verified the same.

Supplementary Table 2

Calculation of the 4C mortality score

Variable	Score
Age group
<50	0
50-59	2
60-69	4
70-79	6
80 or more	7
Gender
Male	1
Female	0
Number of comorbidities
0	0
1	1
2 or more	2
Respiratory rate (breath/min)
<20	0
20-29	1
30 or more	2
Peripheral oxygen saturation on room air (%)
≥92	0
<92	2
GCS
15	0
<15	2
Blood urea (mmol/L)
<7	0
7-14	1
>14	3
CRP (mg/L)
<50	0
50-99	1
100 or more	2

Data are recorded upon hospital admission. GCS – Glasgow Coma Scale; CRP – C-reactive protein

Outcomes

The primary outcome of the study was the performance of ISARIC score in our settings by evaluating its discriminatory ability of survivors and deceased in the all-cause hospital mortality outcome. Secondary outcomes included comparison between subjects with score above and below the optimal criterion derived from the analysis with regard to mortality rate, ICU length of stay (LOS) and survival analysis.

Statistical methods

Statistical tests were performed using the STATA® statistical software (Release 16; StataCorp LLC, College Station, TX, USA). For descriptive purposes, the demographic and clinical characteristics of survivors and nonsurvivors are presented separately. Continuous variables are summarized as mean ± standard deviation and discrete variables as frequency and percentages. Continuous variables were compared by Student's t-test or Wilcoxon rank-sum test, as appropriate, while discrete variables were compared by Chi-square test or Fisher's exact test, as appropriate. Comparative results are presented with corresponding 95% confidence interval (CI) and P value.

The primary outcome was evaluated by fitting receiver operator characteristic (ROC) curve using known outcomes as reference variable and ISARIC score as classification variable, through 1000 bootstrap with replacements. The area under the curve (AUC) and P value are presented. Furthermore, the analysis identified the optimal criterion associated with Youden's index, which was used as a cutoff value for the ICU survival analysis, presented as Kaplan–Meier curve and log-rank P value, in addition to sensitivity, specificity, positive predictive values (PPV) and negative predictive value (NPV), and a 2 × 2 contingency table. All tests were two tailed, and P < 0.05 was considered statistically significant.

RESULTS

During the study period, there were 1749 ICU admissions of confirmed COVID-19 [Figure 1]. Of these, 32, 12, 5 and 2 patients were excluded because they were aged <18 years, were pregnant, had PTB and were HIV positive, respectively. Another 205 (11.7%) patients were excluded due to missing CRP values precluding score calculation. Accordingly, the study finally included 1493 patients.

Figure 1

Flow diagram of COVID-19 patients included in the study

Table 1 depicts the demographic and clinical characteristics of the cohort and comparison between survivors and nonsurvivors. The majority of the patients were Asians (42%) followed by Saudis (28.2%) and Middle Eastern (24.4%). There were 567 (38%) hospital deaths, with a mean age of 53.2 ± 14.1 years and the majority (76.3%) being males.

Table 1

Demographic and clinical characteristics of the cohort and comparison between survivors and nonsurvivors

Variable	Survivors (n=926), n (%)	Nonsurvivors (n=567), n (%)	95% CI of difference	P
Age (mean±SD)	51.1±13.6	56.8±14.4	−7.2-−4.3	<0.001
Males	708 (76.5)	431 (76)	−4-5.1	0.8
Ethnicity
Saudi	295 (31.9)	126 (22.2)	5-14.3	<0.001
Middle Eastern	241 (26)	124 (21.9)	−0.5-8.6	0.08
Asian	338 (36.5)	289 (51)	9.2-19.7	<0.001
African	33 (3.6)	12 (2.1)	−0.4-3.2	0.1
Other	4 (0.4)	2 (0.4)	−0.7-1	0.7
Unknown	15 (1.6)	14 (2.4)	−0.7-2.6	0.4
Number of comorbidities (mean±SD)	1.3±1.1	1.7±1.2	−0.5-−0.2	<0.001
Respiratory rate (mean±SD)	22.4±6.8	37.8±4.6	−16-−14.8	<0.001
Peripheral oxygen saturation (mean±SD)	92.9±3.3	91.6±3.4	0.96-1.7	<0.001
GCS (mean±SD)	14.7±0.8	14.5±0.9	0.1-0.3	<0.001
Blood urea (mean±SD)	6.7±2.9	14.7±4.8	−8.3-−7.6	<0.001
CRP (mean±SD)	51.8±17.1	112±12.9	−61.8-−58.6	<0.001
Smoker	458 (49.5)	308 (54.3)	−0.5-10.1	0.08
DM	458 (49.5)	357 (63)	8.2-18.7	<0.001
Hypertensive	443 (47.8)	337 (59.4)	6.3-16.8	<0.001
Asthma/COPD	37 (4)	37 (6.5)	0.1-5.1	0.04
CKD	139 (15)	96 (16.9)	−2-5.9	0.4
IHD	154 (16.6)	123 (21.7)	−0.8-11.6	0.09
ICU LOS (mean±SD)	8.1±7.2	14.3±10.2	−7-−5.3	<0.001
ISARIC score (mean±SD)	6.3±4.3	12.4±4.3	−6.5-−5.6	<0.001

GCS – Glasgow coma scale; CRP – C-reactive protein; ISARIC – International Severe Acute Respiratory and emerging Infections Consortium; COPD – Chronic obstructive pulmonary disease; ICU – Intensive care unit; LOS – Length of stay; SD – Standard deviation; CI – Confidence interval; CKD – Chronic kidney disease; IHD – Ischemic heart disease; DM – Diabetes mellitus

Survivors had significantly different values than nonsurvivors in all variables used in the calculation of ISARIC score, in the direction of a lower score, with the exception of gender distribution, and significantly lower prevalence of all recorded comorbidities, except for chronic kidney disease, ischemic heart disease and smoking. Only Saudis and Asians had significantly higher survival than mortality rates. The average ICU LOS of the cohort was 10.5 ± 9 days, being significantly shorter for survivors compared to nonsurvivors (8.1 ± 7.2 vs. 14.3 ± 10.2, 95% CI of difference: −7 to −5.3, P < 0.001). The cohort had a mean ISARIC score of 8.6 ± 5.3; survivors had significantly lower score than nonsurvivors (6.3 ± 4.3 vs. 12.4 ± 4.3, 95% CI of difference: −6.5 to −5.6; P < 0.001). Supplementary Figure 1 shows the numbers of patients, mortality rate and median (interquartile range) ISARIC score for different modes of supplemental oxygen.

Primary outcome

The fitted ROC curve of ISARIC score against known outcomes had AUC of 0.81 (95% CI: 0.79–0.83, P < 0.001) [Figure 2]. The associated criterion with Youden's index was a score of >9; accordingly, survival based on that cutoff value estimated by the Kaplan–Meier curve showed a median survival of 22 days with ISARIC scores ≤9 and 16 days with scores >9, and the associated log-rank test was highly significant [Figure 3]. The sensitivity and specificity with scores >9 were 70.5% (95% CI: 66.6–74.3) and 73.8% (95% CI: 71–76.8), respectively, the PPV and NPV was 62.4% (95% CI: 59.5–65.2) and 80.4% (95% CI: 78.2–82.4), respectively, and the cutoff value correctly classified 72.7% of the cohort [Supplementary Table 3].

Figure 2

Area under the curve of 4C ISARIC score discriminatory ability. Area under the curve = 0.81 (95% confidence interval: 0.79–0.83, P < 0.001)

Figure 3

Kaplan–Meier survival based on cutoff of >9. Significantly higher survival for patients with 4C ISARIC score <9

Supplementary Table 3

Contingency table of Youden’s index-associated criterion

ISARIC score >9	Actual
	Dead	Alive	Sum
Dead	400	241	641
Alive	167	685	852
Sum	567	926	1493

Sensitivity=70.5% (95% CI: 66.6-74.3), Specificity=73.97% (95% CI: 71-76.8), PPV=62.4% (95% CI: 59.5-65.2), NPV=80.2% (95% CI: 78.2-82.4), Correctly classified: 72.67%. CI – Confidence interval; ISARIC – International Severe Acute Respiratory and emerging Infections Consortium; PPV – Positive predictive value; NPV – Negative predictive value

Secondary outcomes

Dividing the enrolled patients into two groups based on ISARIC score of ≤9 and >9 showed significantly higher mortality in the >9 group compared to ≤9 (62.4% vs. 19.6%, 95% CI: 38%–47.4%; P < 0.001). Similarly, the group with a score of ≤9 had ICU LOS of 9.1 ± 8.2 days, which was significantly shorter than that of the group with score >9 (LOS: 12.3 ± 9.7 days) (95% CI: 2.3–4.1, P < 0.001) [Supplementary Table 4].

Supplementary Table 4

Comparison of patients by cut-off value of the 4C mortality score >9

Variable	Score ≤9 (n=852), n (%)	Score >9 (n=641), n (%)	95% CI of difference	P
Age (mean±SD)	46.8±11.6	61.8±12.4	60.8-62.7	<0.001
Males	628 (73.7)	511 (79.7)	1.6-10.3	0.008
Saudis	421 (49.4)	10 (1.6)	44.2-51.3	<0.001
DM	328 (38.5)	487 (76)	32.7-42.1	<0.001
HTN	269 (31.6)	511 (80)	43.8-52.7	<0.001
CKD	51 (6)	184 (28.7)	18.8-26.7	<0.001
Asthma/COPD	33 (3.9)	41 (6.4)	0.2-5	0.04
IHD	71 (8.3)	206 (32.1)	19.7-28	<0.001
Smoking	422 (49.5)	344 (53.7)	−1.02-9.4	0.1
ICU LOS (mean±SD)	9.1±8.2	12.3±9.7	2.3-4.1	<0.001
Hospital mortality	167 (19.6)	400 (62.4)	38-47.4	<0.001

DM – Diabetes mellitus; HTN – Hypertension; CKD – Chronic kidney disease; COPD – Chronic obstructive pulmonary disease; IHD – Ischemic heart disease; ICU – Intensive care unit; LOS – Length of stay; SD – Standard deviation; CI – Confidence interval

DISCUSSION

In the population of our study, the ISARIC score was found to have excellent[15] predictive ability for mortality risk, with an AUC of 0.81 (95% CI: 0.79–0.83, P < 0.001). This study excluded cases with missing data (11.2%) precluding calculation of ISARIC score, although modern imputation methods (such as multiple imputation) are likely to produce valid estimates; however, only when critical assumptions have been met,[16] which may have not been true for our data, they still carry the potential to cause bias.[17] Enrolled patients in our analysis were of multiple ethnicities, representing the general population of Saudi Arabia, and thus providing a good external validation cohort for the ISARIC score.

The AUC in our study was slightly higher than that reported by the original development and validation study (AUC = 0.786; 95% CI: 0.781–0.79). Although this is uncommon in external validation studies, higher AUCs were similarly reported by van Dam et al.[9] (AUC = 0.84; 95% CI: 0.79–0.88) and Wellbelove et al.[18] (AUC = 0.83; 95% CI: 0.71–0.95). Other studies have reported AUC values similar to the validation study[19] or lower.[10] This variation among studies – although minimal – utilizing the same prediction model may be a reflection of the variations in the studied populations, with regard to their demographic characteristics, clinical severity and sample size. For example, in our study, various ethnic groups were included, the age range was wide (from 18 to 104 years) and there was an unequal gender distribution (males: 76.3%). Furthermore, our cohort included only patients admitted to the ICU, which means that they presented a more critical picture of the disease, while the original study enrolled all hospital admissions. Regardless of those variations, all studies in the literature, including ours, found the discriminatory ability of ISARIC score to be acceptable or excellent.

Our external validation, similar to the original study, showed rising mortality rates across groups of severity [Supplementary Figure 1 and Supplementary Table 1], that is, a directly proportional relationship between mortality risk and increase in score. This reflects that the model performs optimally, especially when considering that the higher mortality rates were higher within all groups in our study compared with the original study.[8] In our analysis, the optimal cutoff value associated with Youden's index was a score >9, with this value correctly classifying 72.7% of the cohort.

The diagnostic parameters (sensitivity, specificity, PPV and NPV) of the cutoff of >9 in our model were considerably lower compared to the same cutoff value in the British study.[8] This could possibly be due to inclusion of only critically ill COVID-19 patients, which makes predicting outcome more difficult than mild cases, given the larger number of complications[7] that could be the indirect cause of death. This is also reflected in the fact that our study had a higher mortality rate compared with the study by Knight et al.[8] (38% vs. 32.2%, respectively). One of the most important diagnostic parameters is NPV, which indicates the probability of survival in patients with scores ≤9. In our model, NPV was 80.2% (95% CI: 78.2–82.4), which provides a reasonable risk probability to guide clinical decision making; this discrimination was supported by our finding of a significantly higher survival of patients with scores ≤9, shorter ICU LOS, and lower mortality.

Limitations

This study has several limitations. First, the inherent limitations of the retrospective study design are applicable. Second, this was a single-center study, reflecting the management in only our ICU, and thus, our results may not be extrapolated to other centers in Saudi Arabia or non-ICU patients. Third, not using imputation methods on missing data may have reduced our sample size and impacted our results. Fourth, despite working with a quite diverse population, our cohort lacked several ethnicities, and thus, our results may not be applicable in other regions of the world. Finally, the generalizability of our results may be limited to only critically ill patients, as non-ICU admissions were not included. Further multicenter studies with a larger sample size and including those with varied severities are required to validate the score in the larger Saudi population and possibly explore predictors of mortality in COVID-19 patients.

CONCLUSION

The ISARIC score was found to have excellent predictive ability for mortality in critically ill COVID-19 patients in our Saudi Arabian cohort. A cutoff score of >9 was the optimal criterion.

Ethical considerations

The study was approved by the institutional review board of King Saud Medical City, Riyadh (reference no.: H1R1-24-Feb21-04, dated: April 8, 2021). The requirement of patient consent was waived in view of the retrospective study design. The study followed the general principles outlined by the Declaration of Helsinki, 2013.

Data availability statement

The datasets generated during and/or analyzed during the current study are not publicly available, but are available from the corresponding author on reasonable request.

Peer review

This article was peer-reviewed by two independent and anonymous reviewers.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

Distribution of mortality rate by categories of 4C ISARIC score severity: (a) Mortality rates and number of patients (10s) in each category of supplemental oxygen. (b) Median (interquartile range) of ISARIC score for each category of supplemental oxygen. MV = Mechanical ventilation, NIV = Noninvasive ventilation, HFNC = High-flow nasal cannula, NRM = Nonrebreathing mask, NC = Nasal cannula