External validation of the IMPROVE-DD risk assessment model for venous thromboembolism among inpatients with COVID-19.

There is a need to discriminate which COVID-19 inpatients are at higher risk for venous thromboembolism (VTE) to inform prophylaxis strategies. The IMPROVE-DD VTE risk assessment model (RAM) has previously demonstrated good discrimination in non-COVID populations. We aimed to externally validate the IMPROVE-DD VTE RAM in medical patients hospitalized with COVID-19. This retrospective cohort study evaluated the IMPROVE-DD VTE RAM in adult patients with COVID-19 admitted to one of thirteen Northwell Health hospitals in the New York metropolitan area between March 1, 2020 and April 27, 2020. VTE was defined as new-onset symptomatic deep venous thrombosis or pulmonary embolism. To assess the predictive value of the RAM, the receiver operating characteristic (ROC) curve was plotted and the area under the curve (AUC) was calculated. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated. Of 9407 patients who met study criteria, 274 patients developed VTE with a prevalence of 2.91%. The VTE rate was 0.41% for IMPROVE-DD score 0-1 (low risk), 1.21% for score 2-3 (moderate risk), and 5.30% for score ≥ 4 (high risk). Approximately 45.7% of patients were classified as high VTE risk, 33.3% moderate risk, and 21.0% low risk. Discrimination of low versus moderate-high VTE risk demonstrated sensitivity 0.971, specificity 0.215, PPV 0.036, and NPV 0.996. ROC AUC was 0.703. In this external validation study, the IMPROVE-DD VTE RAM demonstrated very good discrimination to identify hospitalized COVID-19 patients at low, moderate, and high VTE risk.

Highlights

In this external validation study, the IMPROVE-DD VTE RAM demonstrated a very good degree of discrimination to identify hospitalized COVID-19 patients at low, moderate, and high risk of developing VTE.

Using the IMPROVE-DD VTE RAM in hospitalized COVID-19 patients, 45.7% of patients were classified as high VTE risk, 33.3% moderate risk, and 21.0% low risk.

This validation of the IMPROVE-DD VTE RAM in COVID-19 inpatients will assist healthcare providers in individualizing thromboprophylaxis strategies based on VTE risk category, and thus, has the potential to decrease VTE-associated morbidity and mortality in this medically ill population.

Introduction

Venous thromboembolism (VTE), including deep vein thrombosis (DVT) and pulmonary embolism (PE), is very common in acutely ill medical patients with coronavirus disease-2019 (COVID-19), especially in patients with elevated D-dimer (Dd). Early data in critically ill patients suggested extraordinarily high rates of VTE [1]. More recent data in medical ward patients show VTE rates up to 6.2% [2], which is still substantially higher than historical rates of VTE in hospitalized non-COVID medical populations. Indeed, VTE risk in hospitalized COVID-19 patients remains a serious concern [2, 3]. Antithrombotic guidelines recommend a universal thromboprophylaxis strategy for COVID-19 inpatients, and suggest intermediate-dose anticoagulation may be beneficial in subsets of high-risk patients [4, 5]. Randomized trials assessing escalated or treatment dose anticoagulants for thromboprophylaxis are ongoing [6]. Effective stratification of patients and identification of those at high risk of VTE remains challenging.

The International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) risk assessment model (RAM) has been extensively validated and shown excellent discrimination between low- and at-risk hospitalized medical patients [7]. The IMPROVE VTE RAM assigns 1 to 3 points to seven VTE risk factors: age > 60, previous VTE, known thrombophilia, current lower-limb paralysis, active cancer, immobilization, and intensive care (ICU)/cardiac care unit (CCU) stay. An 8-factor model (IMPROVE-DD) incorporating Dd > 2 times the upper limit of normal (ULN) enhances area under the curve (AUC) for the receiver operating characteristic curve (ROC) by 0.06 [8]. Noting the association of elevated Dd with increased risk of thrombosis and poor outcomes in our COVID-19 inpatient population [8, 9] we sought to externally validate the IMPROVE-DD VTE RAM in a hospitalized, COVID-19 medical population.

Methods

This retrospective cohort study included medical patients aged 18 years or older who were admitted to 1 of 13 Northwell Health hospitals in the New York metropolitan area between March 1, 2020 and April 27, 2020, and who were confirmed to be COVID-19 positive by polymerase chain reaction. Patients were excluded if they were on the obstetrics service, experienced a VTE event within 8 h of admission, experienced a length of stay less than 8 h, or if key variables were missing. Data and outcomes were extracted from Sunrise Clinical Manager electronic health record (EHR) (Allscripts, Chicago, IL) and tracked until April 30, 2020. This study was approved by the Institutional Review Board of Northwell Health, and the need for consent of individual patients was waived.

VTE was defined as new-onset symptomatic DVT or PE, as diagnosed by imaging studies performed by the Radiology Department or by point-of-care lower extremity ultrasound. Events were manually adjudicated by two attending radiologists.

Points were assigned to patient characteristics corresponding to the variables of the IMPROVE-DD RAM [8]. Previous VTE, thrombophilia, and cancer were extracted from corresponding ICD-9 or ICD-10 codes. In light of standard room isolation precautions for COVID-19, all patients were considered relatively immobile and assigned 1 point for this factor. For the risk factor of ICU stay, a proxy of admission to a named ICU, use of mechanical ventilation, or administration of vasopressors was used. Considering common use of paralytic agents in our system, mechanical ventilation was also used as a proxy for lower limb paralysis. For Dd value, 2 points were assigned if the maximum Dd value throughout the hospitalization or prior to VTE event was ≥ 2 times the upper limit of normal per local laboratory.

To assess the predictive value of the RAM, the ROC curve was plotted, and AUC was calculated. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated using standard methods, using the observed VTE prevalence in the calculation of PPV and NPV. Analyses were performed with SAS version 9.4 (SAS institute, Cary, North Carolina).

To assess the predictive value of the RAM, the ROC curve was plotted, and area under the curve (AUC) was calculated. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated using standard methods, using the observed VTE prevalence in the calculation of PPV and NPV. Further exploratory analyses were done comparing the AUC for both the IMPROVE-DD RAM and the original 7-factor IMPROVE RAM without D-dimers [10]. The curves were compared using the method of DeLong, DeLong and Clarke-Pearson [11]. Upon finding a significant difference, a comparison using a Bonferroni adjustment was used, such that p < 0.0083 was considered significant. Analyses were performed with SAS version 9.4 (SAS institute, Cary, North Carolina).

Results

Of 9407 patients that met study criteria, 274 patients developed VTE, with a prevalence of 2.91%. The VTE rate was 0.41% for IMPROVE-DD score 0–1 (low risk), 1.21% for score 2–3 (moderate risk), and 5.30% for score ≥ 4 (high risk) (Table 1). Approximately 45.7% of patients were classified as high VTE risk, 33.3% as moderate risk, and 21.0% as low risk.

Table 1

Observed VTE Events among COVID-19 Inpatients, based on IMPROVE-DD VTE RAM Score Thresholds

	VTE events
	VTE	No VTE	Total
IMPROVE-DD
0–1, Low risk	8 (0.41%)	1967 (99.59%)	1975 (21.00%)
2–3, Moderate risk	38 (1.21%)	3091 (98.79%)	3129 (33.26%)
4–12, High risk	228 (5.30%)	4075 (94.70%)	4303 (45.74%)
Total	274 (2.91%)	9133 (97.09%)	9407 (100.0%)

The IMPROVE-DD RAM discrimination of low versus moderate-high VTE risk demonstrated a sensitivity of 0.971, specificity of 0.215, PPV of 0.036, and NPV of 0.996. The AUC of the ROC was calculated to be 0.703 (Fig. 1). The IMPROVE RAM without D-dimers discrimination of low versus moderate-high VTE risk demonstrated a sensitivity of 0.839, specificity of 0.292, PPV of 0.034, and NPV of 0.984. The AUC of the ROC was calculated to be 0.635. The 6.8% difference of the AUC (delta AUC) comparing the IMPROVE-DD and IMPROVE RAMs was significant (p < 0.0001).

Fig. 1

Logistic regression with receiver operating characteristic curve for inpatient VTE for IMPROVE-DD score

Discussion

With an AUC of ROC of 0.703, the IMPROVE-DD VTE RAM demonstrated very good model discrimination and negative predictive value to predict the risk of VTE in this cohort of COVID-19 inpatients, similar to the discrimination seen in prior validations of the original IMPROVE and IMPROVE-DD RAMs in non-COVID cohorts [7]. In comparison with the original 7-factor IMPROVE RAM, addition of elevated D-dimers with the IMPROVE-DD RAM improved model discrimination of VTE in COVID-19 inpatients by 6.8%, a difference that was statistically significant. The IMPROVE-DD was also recently validated in the hospitalized COVID-19 population, which achieved an AUC of 0.702, though the study was limited by the absence of thrombophilia data [12]. In comparison, our cohort included 123 (1.31%) patients with thrombophilia and achieved similar AUC. This confirms the expectation of good model discrimination with the addition of risk factors that are strongly weighted, yet relatively uncommon, such as inherited thrombophilia.

As additional data on thromboprophylaxis in COVID-19 continues to surface, international guidelines suggest a need for a risk-adapted approach to thromboprophylaxis [3-5]. Notably, while 21.0% of our study population was identified as low VTE risk, nearly half (45.5%) of patients were identified as high VTE risk. These patients represent a subgroup of hospitalized COVID-19 patients who may potentially benefit from increased intensity thromboprophylaxis [3]. Though this approach is not currently supported by guidelines, high quality randomized trials data is forthcoming [6]. Conversely, our data suggests potential harm from a universal thromboprophylaxis strategy in approximately 20% of COVID-19 inpatients identified as low VTE risk by our validation study.

As a retrospective study, our validation has several limitations. Although VTE events were adjudicated by two attending radiologists, the VTE rate may be underreported as a result of avoidance of confirmatory studies due to concerns of virus exposure. We relied on surrogate markers for lower limb paralysis and immobility. In addition, the highest Dd value during hospital stay was used, which may alter the accuracy of the RAM when used to stratify risk on admission.

Conclusion

This large, external validation study of the IMPROVE-DD VTE RAM demonstrated very good discrimination to identify hospitalized COVID-19 patients at low, moderate, or high risk of VTE. The use of this model led to the stratification of 45.7% of patients as high VTE risk, 33.3% moderate risk, and 21.0% low risk, which can be used to create individualized, risk-adapted strategies for VTE thromboprophylaxis in hospitalized patients with COVID-19.