Validation of the PESI Scale to Predict in-Hospital Mortality in Patients with Pulmonary Thromboembolism Secondary to SARS CoV - 2 Infection.

OBJECTIVE: To evaluate the discriminative ability and the calibration of the Pulmonary Embolism Severity Index (PESI) to predict in-hospital mortality in patients with Pulmonary Embolism (PE) secondary to COVID 19 in two hospitals in Bogotá.
METHODS: External validation study of a prediction model based on a retrospective cohort of patients with PE secondary to COVID-19 treated at Hospital Universitario San Ignacio and Hospital universitario La Samaritana, between March 2020 and August 2021. Calibration of the scale was evaluated using the Hosmer-Lemeshow test and a calibration belt diagram. Discrimination ability was evaluated using a ROC curve.
RESULTS: 272 patients were included (median age 61.5 years, male 58.8%). PE was diagnosed in 45.6% of the patients at the time of admission. Of the remaining 54.4%, 95.9% received thromboprophylaxis until the time of diagnosis.17.6% of the patients died. Regarding calibration, the scale systematically underestimates risk in all classes of PESI. For class I, the ratio of observed/expected events was 4.4 vs 0.8%, class II 4.8 vs 1.8%, class III 15.2 vs 4.2%, class IV 14.3 vs 5.9% and class V 46.7 vs 5.8%. The calibration test rejected the adequate calibration hypothesis (p < 0.001). The discriminatory ability was adequate (AUC = 0.7128, 95% CI 0.63-0.79).
CONCLUSIONS: The PESI scale in patients with PE secondary to COVID 19 underestimates the risk of in-hospital mortality, while maintaining adequate discrimination. It is suggested not to use the PESI scale until it is recalibrated in this context.

Introduction

As of January 2022, 715 inhabitants per million died because of coronavirus disease 19 (COVID-19) worldwide and 2595 per million died in Colombia. Although mortality has been mainly associated with the development of Acute Respiratory Distress Syndrome (ARDS), there are also multiple different pathophysiological alterations in patients with COVID-19.

Patients with COVID-19 can have pulmonary embolism (PE) secondary to an increase in the systemic inflammatory response generated by an imbalance between procoagulant and anticoagulant factors. Between 15% and 30% of patients with COVID-19.[5-7] develop PE. Also, PE is associated with increased mortality, up to 45%.[8,9]

International PE clinical practice guidelines recommend the use of the PESI (Pulmonary Embolism Severity Index) to stratify mortality risk. PESI scale has been validated to predict 30-day mortality in patients with PE,[11-13] which allows to guide patient management according to mortality risk. However, at the moment there is no consensus about using PESI in patients with PE secondary to COVID-19.[14,15] There are multiple concerns about PESI underestimating mortality risk in this context and this hypothesis has not been evaluated yet.

The purpose of this study is to validate the PESI scale estimation of in-hospital mortality risk in patients with PE secondary to COVID-19, to evaluate PESI scale ability to discriminate between high and low risk patients and to evaluate the relationship between observed and predicted in-hospital mortality by the PESI scale, in two hospitals in Bogotá, Colombia.

Methods

Study Design and Participants

This study is a validation of a prediction model based on a retrospective cohort. The databases of the institutional anticoagulation registry of Hospital Universitario San Ignacio and the COVID-19 registry of Hospital Universitario La Samaritana were considered, which include the patients managed between March 6, 2020, and August 31, 2021. The inclusion criteria were hospitalized patients older than 18 years with a confirmed diagnosis of PE and COVID-19. Patients with missing information on medical records and patients referred to another health facility were excluded. This determination was made to secure information about mortality outcome. The institutional research ethics committee of both institutions approved the study.

Sociodemographic data was systematically collected during patient care using standard recording practices in the database described. Variables related to comorbidities, laboratories, thromboprophylaxis, use of in-hospital anticoagulation, days of hospitalization, hospitalization in general ward or Intensive Care Unit (ICU), oxygen supply system, in-hospital death and hemorrhagic complications were obtained from the mentioned databases and complemented with revision of medical records when needed. Standard thromboprophylaxis was defined as a Low Molecular Weight Heparin (LMWH) in standard dosis, generally enoxaparin 40 mg/day subcutaneous (SC). Adjusted thromboprophylaxis was defined as any different dose, according to individualized decisions of the treating physician based on thrombotic risk, weight or kidney failure.

Regarding the PESI scale, the version validated in Colombia was used. Data necessary to calculate PESI scale (Age, sex, cancer, heart failure, chronic obstructive pulmonary disease, heart rate >110 bpm, blood pressure, systolic < 100 mm Hg, respiratory rate > 30 rpm, Temperature < 36 centigrade degrees, altered mental status and arterial oxygen saturation (SaO2) < 90%) was collected when PE diagnosis was made.

The diagnosis of PE was operatively defined by detection of thrombi in pulmonary arteries in a computed tomography pulmonary angiogram (CTPA), contrast-enhanced chest computed tomography or ventilation/perfusion lung scan. COVID 19 infection was confirmed with RT PCR or SARS-CoV 2 antigen positivity. Finally, major bleeding was defined as fatal bleeding, bleeding in a critical area or organ or bleeding with a decrease in hemoglobin ≥ 2 g/dl, according to the guidelines of the International Society of Thrombosis and Haemostasis.

Statistical Analysis

For description of qualitative variables, absolute and relative frequencies were used. Central tendency and dispersion measures were calculated for quantitative variables. Mean and standard deviation were described for variables with normal distribution and median and interquartile range for variables with non-normal distribution. Normal distribution was evaluated using the Kolmogórov-Smirnov test at a significance level of 5% (p < 0.05).

Subsequently, the PESI scale was validated by analyzing its calibration and discrimination capacity. Calibration was evaluated comparing the number of observed and predicted in-hospital mortality events within each of the five risk classes proposed by PESI scale. Expected proportion of In-hospital mortality was obtained from the original study of Aujesky (16), in the derivation sample, for each PESI class. The expected events were calculated by multiplying the expected proportions by the number of patients in each class.

Proper calibration hypothesis was evaluated according to the Hosmer-Lemeshow statistical test. A graphical analysis was performed through a calibration belt diagram. This methodology allowed visual inspection to determine whether the observed frequencies differ significantly from the expected probabilities. Also, this methodology allowed to localize the direction of the miscalibration deviation, 80% and 95% confidence interval levels were used. Finally, a calibration test was obtained to assess if the deviation from the bisector (45-degree line of perfect fit) was significant.

The discriminatory ability (ability of the scoring system to differentiate between patients with different outcomes) was evaluated with a Receiver operating Characteristic (ROC) curve. An area under the curve (AUC) greater than 0.7 was considered appropriate.

Statistical analysis was performed with the statistical program STATA (Stata Statistical Software: Release 16. College Station, TX: StataCorp LLC).

Results

Table 1 summarizes the demographic and clinical characteristics of the 272 patients with PE secondary to COVID-19. Median age was 61.5 years, with a higher prevalence of males. Hypertension was the main associated comorbidity. It is noteworthy in the laboratory reports that 78.9% of the patients had D-dimer levels greater than 1000 ng/ml. In 45.6%, PE diagnosis was made at the time of admission; within the 54.4% remaining, 95.9% had received thromboprophylaxis at the time of diagnosis. Low molecular weight heparin (LMWH) was the most frequent anticoagulant.

Table 1.

Sociodemographic Characteristics of COVID-19 and PE patients

Variable	n = 272
Age, years, median (IQR)	61.5 (50.5-71)
Male, n (%)	160 (58.8)
Comorbidities n (%)
Hypertension	93 (34.2)
Coronary disease	10 (3.7)
Chronic heart failure	8 (2.9)
Diabetes Mellitus	30 (11.0)
CKD*	3 (1.1)
COPD	10 (3.7)
Autoimmune disease	6 (2.2)
Cancer	26 (8.5)
Laboratories report, n (%)	170 (62.7)
LDH > 350 U/L	186 (68.4)
CRP > 10 mg/dl	130 (47.7)
LAC < 1000	14 (5.3)
D-dimer, ng/ml	44 (15.8)
<500	84 (31.0)
500 − 1000	130 (47.9)
1000-5000	124 (45.6)
>5000	142 (95.9)
PE arrival diagnosis, n (%)	140 (98.6)
Pharmacologic thromboprophylaxis, n (%)**	2 (1.4)
Thromboprophylaxis dose, n (%) **
Standard
Adjusted
In-hospital anticoagulant, n (%)
LMWH	271 (99.6)
UFH	1 (0.4)
Hospitalization length, median (IQR)	11 (5-19.5)

Abbreviations: PE, Pulmonary embolism: SD, standard deviation; IQR, interquartile range; CKD: Chronic Kidney Disease: COPD: Chronic Obstructive Pulmonary Disease; LDH, Lactate dehydrogenase; CRP, C-Reactive protein; LAC, Lymphocyte absolute count; LMWH, Low Molecular Weight Heparin; UFH; unfractionated heparin; U/L, units per liter; mg/dl, milligram per deciliter; ng/ml, nanogram per milliliter. * CKD ≥ stage3 **Calculated according to patients with thromboprophylaxis indication

Regarding the outcomes of interest17.6% died, 34.9% required ICU,28.7% required mechanical ventilation and 16.9% required high-flow oxygen supply systems (non-rebreather mask or high-flow nasal cannula), 7.7% did not require O2 supply during hospitalization. Finally, 8.8% presented hemorrhagic complications, half of them characterized as major bleeding and the other half as non-major bleeding.

The evaluation of the calibration of the PESI scale showed that the observed in-hospital mortality was higher than the expected mortality (17.7 vs 5.8%). This result was consistent within each class of the PESI scale (Hosmer-Lemeshow 2.02, p = 0.364) (Table 2). The calibration belt diagram demonstrates an inadequate calibration of the model for all risk ranges with confidence intervals of 80% and 95%. Similarly, the calibration test rejected the hypothesis of adequate calibration (p < 0.001) (Figure 1).

Table 2.

Calibration of PESI Scale, Observed Versus Expected Events of in Hospital Mortality in Patients with COVID 19 and Pulmonary Embolism

Class PESI	n	% total	Expected events	Observed events	Expected proportion	Observed proportion
1	23	8.46	0.2	1	0.8	4.4
2	42	15.44	0.8	2	1.8	4.8
3	99	36.40	4.2	15	4.2	15.2
4	63	23.16	3.7	9	5.9	14.3
5	45	16.54	7.1	21	15.8	46.7
Total	272	100	16.0	48	5.8	17.7

Expected proportion of In-hospital mortality was obtained from the original study of Aujesky (16), in the derivation sample. The expected events were calculated by multiplying the expected proportions by the number of patients in each class.

*Hosmer-Lemeshow = 2.02 p = 0.364.

Figure 1.

Calibration of PESI scale for predicting in-hospital mortality in patients with COVID 19 and pulmonary embolism. Calibration belt showing deviations from the bisector (45° line of perfect fit) at the 80% (inner belt: light grey area) and 95% (outer belt: dark grey area) confidence levels.

PESI scale discriminatory ability was acceptable with an area under the ROC curve of 0.71 (95% CI 0.63, 0.79) (Figure 2), similar to results reported by the original scale.

Figure 2.

Discriminatory ability of PESI scale to predict in hospital mortality in patients with COVID 19 and pulmonary embolism

Discussion

This is the first study that formally evaluates the performance of the PESI scale to predict in-hospital mortality in patients with PE secondary to COVID 19. Our results show that the calibration of the scale is inadequate since it systematically underestimates the probability of death at all PESI classes. The ratio of observed events/expected events is close to 3:1. However, PESI maintains its acceptable ability to discriminate patients with lower or higher risk.

Studies have evaluated the usefulness of the PESI scale in the context of PE secondary to COVID-19. Miró et al showed a higher mortality in patients with PE secondary to COVID-19 compared to PE not secondary to COVID-19 (12.8 vs 5.3%, p < 0.001). Their results suggest an underestimation of in-hospital mortality with different scores on the sPESI scale (simplified PESI) in patients with PE secondary to COVID-19 compared to PE not caused by COVID-19 (sPESI 0 p = 0.003, sPESI 1 p = 0.028, sPESI, 2 p = 0.012, sPESI>2 p = 0.055). Additionally, Xu et al. showed that in patients with PE secondary to COVID-19, the PESI scale apparently adequately discriminated in-hospital mortality when comparing high-intermediate risk patients (PESI class III, IV and V) compared to low-risk patients (PESI class I and II) (27 vs 6%, p = 0.007). However, these studies did not formally assess the discrimination and calibration of the PESI scale in this context. Kagan et al studied a modified sPESI scale to determine the need for ICU admission in patients with COVID-19. It used the variables: age ≤ 65 years, history of cancer, cardiopulmonary diseases, heart rate < 110 bpm, systolic blood pressure ≥ 100 mm Hg and O2 saturation ≥ 90%. They report an area under the ROC curve of 0.95, sensitivity 84.6% and specificity 94.6% for a sPESI score > 2. However, their study did not include patients with PE. Additionally, sPESI recommendations have been made without having studied its discriminative capacity or calibration.[21-24] Therefore, this is the first study to evaluate the discriminative capacity and calibration of the PESI scale in patients with PE secondary to COVID-19.

Our results show an adequate discrimination ability (area under ROC curve 0.7377), similar to AUC reported in the original PESI validation study (0.79 CI95% 0.65-0.93) and in the Colombian validation study (AUC 0 0.78 CI95% 0.6-0.96).

The main strength of the PESI scale is its ability to identify patients with low mortality risk (class I and II).[10,25] However, our results show an increased mortality rate in these classes (4.4 and 4.8%, respectively). Therefore, in the context of PE secondary to COVID-19, it is possible that PESI classes I and II need in-hospital surveillance, rather than early hospital discharge and outpatient follow-up as current guidelines recommend.[10,25] Regarding PESI classes III to V, the in-hospital mortality observed in these patients is greater than 15%. Hence, it would be advisable to hospitalize these patients in an Intermediate Care Unit or ICU.

The sociodemographic, clinical and laboratory characteristics of these patients are similar to other reported studies of patients with COVID-19 and PE. A meta-analysis that included more than 7178 patients with COVID-19 and PE showed a median age of 60.4 years with a prevalence of men between 58% and 80%. They reported a prevalence of PE in hospitalized patients between 1.6 and 62.5% and in the ICU between 4.2% and 75%. Regarding comorbidities, the main associated comorbidity was hypertension (47%-85%), followed by diabetes (4%-26%) and cancer (3%-20%), similar to our study. This data is also reported in other studies.[7,26] When comparing laboratory reports, other studies have also shown elevated levels of lactate dehydrogenase (LDH), CRP, and D-dimer in patients with PE. Compared to patients with COVID-19 without PE, there are statistically significant differences in D-dimer and LDH levels.[7,27]

It is concerning that nearly half of the patients who presented PE secondary to COVID-19 were receiving thromboprophylaxis at the time of PE diagnosis. Another meta-analysis showed a similar result, they reported 38% of patients hospitalized in the ICU presented thromboembolic events under thromboprophylaxis treatment. One hypothesis is the possible resistance to heparin treatment in patients with severe COVID-19 manifestation. It is also notable the high proportion of bleeding complications in our study (8.8%). Other studies report an incidence of clinically significant bleeding or major bleeding between 6 and 14.7%. Additionally, they report an association between major bleeding and the administration of therapeutic anticoagulation compared to prophylactic, ECMO treatment and dialysis treatment.[30,31]

Among the strengths of this study, it stands out that we used different methodologies to evaluate the calibration of the scale obtaining consistent results. Additionally, the external validity of our results is strong since we included patients from two different university hospitals.

One of the main limitations of this study is its retrospective nature which could lead to misclassification bias. The underreporting of comorbidities (given the self-report by patients) or the underreporting of altered consciousness state in medical records are possible biases in this study. Additionally, the relatively small number of events limits the accuracy of our estimates, mainly in low-risk classes. Therefore, prospective validation studies with a larger number of patients are required to confirm our results and to recalibrate the PESI scale in this context.

In conclusion, our study suggests that PESI scale in patients with PE secondary to COVID-19 underestimates the risk of in-hospital mortality while maintaining adequate discrimination. Therefore, PESI scale utilization in this context is not suggested until it has been recalibrated. When used, we suggest careful interpretation, bearing in mind in-hospital mortality underestimation in patients with PE secondary to COVID-19.