The data-collection on adverse effects of anti-HIV drugs (D:A:D) model for predicting cardiovascular events: External validation in a diverse cohort of people living with HIV.

OBJECTIVES: Little is known about the external validity of the Data-collection on Adverse Effects of Anti-HIV Drugs (D:A:D) model for predicting cardiovascular disease (CVD) risk among people living with HIV (PLWH). We aimed to evaluate the performance of the updated D:A:D model for 5-year CVD risk in a diverse group of PLWH engaged in HIV care.
METHODS: We used data from an institutional HIV registry, which includes PLWH engaged in care at a safety-net HIV clinic. Eligible individuals had a baseline clinical encounter between 1 January 2013 and 31 December 2014, with follow-up through to 31 December 2019. We estimated 5-year predicted risks of CVD as a function of the prognostic index and baseline survival of the D:A:D model, which were used to assess model discrimination (C-index), calibration and net benefit.
RESULTS: Our evaluable population comprised 1029 PLWH, of whom 30% were female, 50% were non-Hispanic black, and median age was 45 years. The C-index was 0.70 [95% confidence limits (CL): 0.64-0.75]. The predicted 5-year CVD risk was 3.0% and the observed 5-year risk was 8.9% (expected/observed ratio = 0.33, 95% CL: 0.26-0.54). The model had a greater net benefit than treating all or treating none at a risk threshold of 10%.
CONCLUSIONS: The D:A:D model was miscalibrated for CVD risk among PLWH engaged in HIV care at an urban safety-net HIV clinic, which may be related to differences in case-mix and baseline CVD risk. Nevertheless, the HIV D:A:D model may be useful for decisions about CVD intervention for high-risk patients.

INTRODUCTION

People living with HIV (PLWH) have an increased risk of cardiovascular disease (CVD) compared with the general population [1, 2], which is attributable to ageing , inflammation, and antiretroviral therapy use [1, 2, 3, 4, 5, 6]. HIV infection may also be an important risk factor for CVD [7, 8, 9]. Despite the higher overall risk of CVD among PLWH, individual risk of CVD may vary substantially [10]. Knowledge of the individual risk of CVD events among PLWH could facilitate clinical decision‐making related to managing comorbidities, prescribing specific medications and recommending other preventive interventions such as lifestyle changes [11]. The American Heart Association includes the Data‐collection on Adverse Effects of Anti‐HIV Drugs (D:A:D) model as an option for assessing atherosclerotic CVD risk among PLWH [12]. The D:A:D model was developed using data from HIV cohorts across several countries and includes routinely collected CVD predictors with additional predictors unique to PLWH [13]. The model was recently updated for easier implementation [14].

Few studies [11, 15] have evaluated the external validity of the updated D:A:D model. Limited evidence suggests that the D:A:D model performs better than the Framingham CVD risk model for PLWH, but the D:A:D model still underestimates CVD risk [14]. In addition, questions remain about the external validity of the D:A:D model considering that non‐white PLWH were under‐represented [14] and some PLWH with different CVD risk profiles were excluded from the data used for assessment of model performance in a US cohort [11]. Consequently, the validity of the D:A:D model is largely unknown in HIV care settings with diverse PLWH [11, 15]. Inaccurate predictions could result in decisions that compromise patient outcomes [16], which emphasizes the need for external validation before a model is implemented in practice [17, 18]. Therefore, we aimed to evaluate the performance of the updated D:A:D model for 5‐year CVD risk in a diverse group of PLWH engaged in HIV care.

METHODS

Setting

Our study population was derived from people living with HIV engaged in care at JPS Health Network (JPS), a large, urban safety‐net health system in North Texas. JPS is the primary source of care for socioeconomically disadvantaged individuals. The network comprises a 578‐bed academic teaching hospital, with over 40 satellite clinics including a comprehensive HIV clinic which is partially supported by funding from the national Ryan White programme [19].

Study population

We identified eligible PLWH from the JPS HIV Care and Outcomes Registry (HIVCOR), which is a longitudinal registry that includes PLWH aged ≥ 18 years who received HIV care at JPS between 2013 and 2019. The registry contains data on patient demographics, clinic visits, medications, laboratory results, and mortality based on linkage with the Centers for Disease Control and Prevention (CDC) National Death Index database [20]. Individuals eligible for our study had at least one clinical encounter between 1 January 2013 and 31 December 2014, which allowed for at least 5 years of follow‐up. For consistency with the D:A:D protocol [14], PLWH were included in the study cohort at the first time point when information on all required predictors was accrued. We excluded data for PLWH who had documented evidence of a prior CVD event or PLWH without data on one or more diagnostic determinants for CVD risk as defined in the reduced D:A:D model [14].

Variables

The outcome of interest was a composite variable of cardiovascular disease events consisting of myocardial infarction (fatal or nonfatal), stroke, invasive coronary artery procedures (coronary artery bypass or angioplasty) and death from coronary heart disease events within 5 years of initial eligibility. The CVD events were identified in HIVCOR using a combination of International Classification of Diseases (ICD), Current Procedural Terminology (CPT) and Healthcare Common Procedure Coding System (HCPCS) codes (Table S1). As specified in the reduced D:A:D model, our model predictors included age at baseline (years), sex (male vs. female), diabetes status (diabetic vs. not diabetic), family history of CVD (yes vs. no), smoking status (current, former and never smoker), total cholesterol (mmol/L), HDL cholesterol (mmol/L), systolic blood pressure (mmHg) and CD4 lymphocyte count (cells/μL) [14]. These predictors were measured within the 2‐year eligibility period at various clinical encounters from 2013 to 2014, with baseline defined as the first time point at which all required predictors had been accrued. We used natural logarithms for all continuous variables and log2 for CD4 count, consistent with D:A:D protocol [14].

Data analysis

We estimated the predicted 5‐year risk of CVD, where follow‐up started at baseline for each patient and ended at the first time of occurrence of any of the following: CVD event of interest, last known clinical encounter, death or 31 December 2019 (end of study period). We first computed the prognostic index by applying coefficients and centred values for the predictors from the original model to our population, where famgp is family history of CVD, currsmk is smoking status (current vs. never), exsmk is smoking status (former vs. never), ln refers to loge, and ln2 represents log2. We subsequently used the prognostic index as a function of baseline survival to estimate predicted risks using the following formula, where PI is the prognostic index. Only complete cases were included in the analysis (i.e. no missing values for predictors).

Our evaluable population excluded individuals with missing values for any of the predictors. We did not pursue multiple imputation because of overlapping missing predictors and limited auxiliary predictors to specify multiple imputation models that could reasonably justify the missing at random assumption [21]. Nevertheless, we explored whether the evaluable population was systematically different from the non‐evaluable population based on information from predictors without missing values. We described the distribution of demographic characteristics and CVD incidence for evaluable and non‐evaluable populations. The patterns of missing values precluded describing other predictors.

Discrimination and calibration

We estimated Harrell's concordance index (C‐index), which is a measure of model discrimination in our population in the context of right‐censored data [22]. The C‐index ranges between 0.5 (equivalent to randomness) and 1.0 (perfect discrimination) and represents discrimination across the full duration of follow‐up rather than at a specific time [22]. In addition, we estimated slope of the prognostic index, which is a measure of the spread of predicted risks. A slope < 1.0 implies that discrimination in the validation population is lower than in the original population. We assessed model calibration based on calibration in the large and graphical assessment. Calibration in the large is a measure of systematic under‐ or overestimation of model calibration based on comparing the expected 5‐year predicted risk for the entire population (i.e. average risk over all individuals) with the observed 5‐year risk based on the Kaplan–Meier estimate (i.e. 1 – Kaplan–Meier survival estimate). An expected‐to‐observed ratio of 1.0 is interpreted as perfect calibration in the large. Lastly, we plotted calibration curves based on expected and observed 5‐year risks of CVD, where expected and observed risks were grouped according to suggested cut‐points for the D:A:D model (< 1.0%, 1.0–5.0%, 5.0–10% and > 10%) [14], and graphically evaluated calibration across the range of risks [22].

Net benefit

We assessed net benefit of the D:A:D model using decision curve analysis, which graphically represents the clinical utility of using a model for decision‐making across a range of possible decision thresholds compared with treat‐none or treat‐all approaches for CVD prevention [23, 24, 25]. The net benefit (NB) is the sum of the number of true positives (TP; individuals with CVD events for whom preventive interventions should be considered) minus a weighted number of false‐positive (FP) classifications (individuals without CVD events for whom preventive interventions should not be considered): NB = (TP/n) − (FP/n) × (p/(1 − p)), where n is the total sample size and p is the relative weight of the harm of unnecessary intervention vs. the benefit of intervention to prevent a CVD event [23, 24, 25]. The weight p is defined as the threshold probability that defines at‐risk patients who need intervention to prevent CVD. The HIV D:A:D model is recommended for use at a decision threshold of 10% (i.e. > 10% suggests high probability of CVD event within 5 years). Consequently, we qualitatively evaluated the net benefit of the HIV D:A:D model at 10% compared with preventive interventions for all patients or preventive interventions for no patients.

RESULTS

We identified 2359 eligible PLWH, of whom 1330 were excluded because of missing values for any of the predictors. Our evaluable population thus comprised 1029 PLWH. Table 1 summarizes baseline characteristics of our evaluable population by CVD status. The median age of our evaluable population was 45 years, 70% were male, and 50% were non‐Hispanic black. Family history of CVD was reported by 31% of the population, 12% were diagnosed with diabetes, and 41% were current smokers. We observed 78 CVD events during the 5‐year follow‐up including 38 individuals with myocardial infarction, 30 with stroke, six with invasive coronary procedures and four deaths from CVD. Table S2 summarizes the distribution of demographic characteristics of evaluable and non‐evaluable PLWH. We observed modest differences in the distribution of age, gender and racial/ethnic characteristics, and marked differences in insurance status, between evaluable and non‐evaluable PLWH. CVD risk at year 5 was modestly ( ∼2.0%) higher for non‐evaluable than for evaluable PLWH.

TABLE 1

Characteristics of people living with HIV and engaged in care at an urban safety‐net HIV clinic

Characteristics	No CVD (n = 951)	Incident CVD (n = 78)
Age (years) [median (IQR)]	45 (35, 52)	51 (43, 56)
Gender [n (%)]
Male	665 (70)	52 (67)
Female	286 (30)	26 (33)
Race/ethnicity [n (%)]
Non‐Hispanic white	317 (33)	23 (30)
Non‐Hispanic black	470 (49)	44 (56)
Hispanic	131 (14)	11 (14)
Other	33 (3.5)	‐
Diabetes [n (%)]
Yes	104 (11)	24 (31)
No	847 (89)	54 (69)
Family history of CVD [n (%)]
Yes	280 (29)	39 (50)
No	671 (71)	39 (50)
Smoking [n (%)]
Current	395 (42)	31 (40)
Former	186 (20)	17 (22)
Never	370 (39)	30 (35)
Systolic blood pressure (mmHg) [median (IQR)]	125 (115–137)	131 (119–148)
Total cholesterol (mmol/L) [median (IQR)]	4.22 (3.54–4.94)	4.16 (3.55–4.89)
HDL cholesterol (mmol/L) [median (IQR)]	1.11 (0.91–1.37)	1.09 (0.85–1.45)
CD4 count (cells/μL) [median (IQR)]	482 (292–695)	429 (211–694)

Abbreviations: CVD, cardiovascular disease; IQR, interquartile range.

Table 2 summarizes discrimination and calibration in the large. The C‐index for the D:A:D model in our population was 0.70 [95% confidence limits (CL): 0.64–0.75] for 5‐year risk prediction. The slope of the prognostic index was 0.71 (95% CL: 0.47–0.93). The expected 5‐year CVD risk was 3.0% based on the D:A:D model, whereas the observed 5‐year CVD risk was 8.9% (expected/observed ratio = 0.33, 95% CL: 0.26–0.54). Figure 1 illustrates the calibration plot for 5‐year predicted and observed risk of CVD for our study population. This plot illustrates systematic underprediction of CVD risk by the D:A:D model for the risk groups at pre‐specified cut‐points of 1%, 5% and 10%.

TABLE 2

Discrimination and calibration in the large of the D:A:D model for predicting 5‐year cardiovascular disease risk among people living with HIV at an urban safety‐net HIV clinic

Performance measures	Estimates (95% confidence limits)
Harrell's C‐index	0.70 (0.64–0.75)
Expected risk	3.0%
Observed risk	8.9%
Expected/observed ratio	0.33 (0.26–0.54)

Abbreviation: C‐index, Harrell's concordance index.

FIG 1

Calibration plot for 5‐year cardiovascular disease (CVD) risk groups for people living with HIV based on the Data‐collection on Adverse Effects of Anti‐HIV Drugs (D:A:D) model.

Figure 2 illustrates net benefit of the D:A:D model compared with two possible strategies (treat none or treat all) for treatment decisions about CVD prevention. Treatment decisions based on the D:A:D model would have marginally greater benefit than treating everyone or treating none if the pre‐specified risk threshold for CVD was 10%. Risk thresholds < 8% or > 23% would not have greater net benefit than treating all or none.

FIG 2

Net benefit analysis of the Data‐collection on Adverse Effects of Anti‐HIV Drugs (D:A:D) model for predicting 5‐year risk of cardiovascular disease among people living with HIV and engaged in care at an urban safety‐net HIV clinic.

DISCUSSION

Our results suggest that the reduced D:A:D model is better than randomly ranking individuals’ risk of incident CVD events among PLWH and engaged in care at an urban safety‐net HIV clinic. More importantly, the D:A:D model severely underpredicts 5‐year CVD risk in this population. Despite suboptimal discrimination and calibration, treatment decisions for CVD prevention based on the D:A:D model may have greater net benefit compared with treating all or treating no one in the population, if based on the suggested 10% threshold.

Similar to a prior evaluation of the D:A:D model [11], missing values for several predictors resulted in exclusion of a substantial number of otherwise eligible individuals from our analysis. We explored whether characteristics of excluded individuals differed from characteristics of evaluable individuals (Table S2). The distribution of demographic characteristics was modestly different between evaluable and non‐evaluable PLWH. In addition, 5‐year CVD incidence was ~2.0% higher among non‐evaluable PLWH. Given that high baseline CVD risk among evaluable PLWH may be a key explanation for poor performance of the model in our population, similarly high CVD risk among non‐evaluable PLWH suggests that model performance would not have improved if all PLWH had evaluable data.

We matched our outcome definitions as closely as possible to the outcome definitions used in the original CVD reporting system for the D:A:D study [26, 27, 28, 29], but our results may be sensitive to misclassification because we used routinely collected data rather than a protocol for primary data collection as in the D:A:D study. Specificity of CVD classification is high in electronic health records, but sensitivity may vary [30]. The consequence would be underestimated CVD incidence, which would further compromise model performance. In addition, data limitations precluded evaluating the performance of the full D:A:D model, which includes antiretroviral regimens as predictors. Nevertheless, the full and reduced models had similar performance in the original D:A:D cohort [14].

Few studies aimed to validate the original or updated D:A:D model for HIV populations [15, 31, 32], and we identified only one prior study in the United States [11]. Our findings are consistent with a prior assessment of the D:A:D model in the HIV Outpatient Study (HOPS) [11]. Thompson‐Paul et al. [11] reported a C‐index of 0.72 and an expected‐to‐observed ratio of 0.44 for 5‐year CVD risk using the reduced D:A:D model, but the authors did not report an evaluation of net benefit. We observed comparable CVD incidence in our population (8.9% vs. 8.5% in HOPS) and our population had similar distributions of some characteristics as the HOPS population (e.g. proportion of current smokers), but the median age of our population was ~3 years older than the HOPS population and our population included 50% non‐Hispanic black PLWH compared with 34% in HOPS [11]. More importantly, both our population and the HOPS population had notably different characteristics and CVD incidence than the population used to develop the D:A:D model [14], which may partially explain suboptimal performance when transported outside the original population. Differences in case‐mix may be particularly pronounced between the D:A:D population and our population (Table S3), which comprises socioeconomically disadvantaged individuals with a high prevalence of CVD risk factors [33, 34, 35]. For example, our population had a higher prevalence of diabetes and family history of CVD at baseline compared with the D:A:D population [14].

Despite severe miscalibration of the D:A:D model, our net benefit analysis suggests that the model may be clinically useful in our population for decisions regarding CVD prevention for high‐risk PLWH at the suggested 10% risk threshold. This finding emphasizes the distinction between model performance and clinical utility [23, 24, 25]. Model performance measures such as discrimination and calibration are insufficient to provide insights into the value of a prediction model for clinical decisions [23, 24, 25]. Net benefit analysis using decision curves can facilitate interpretation about the benefits and harms of a model [23, 24, 25]. Nevertheless, real‐world implementation of the D:A:D model requires further consideration.

The D:A:D model is designed for use at the first time point at which information on all predictors has accrued [14], but this definition is problematic because predictors should be measured at the time of intended use [36]. The duration required to accrue information may vary substantially between patients and settings, and time‐dependent predictor values may change by the time all the information is accrued. This issue is particularly important in safety‐net health systems that provide care for socioeconomically disadvantaged populations [33, 37]. These populations experience multiple barriers to care such as being uninsured or under‐insured, which create challenges for consistent follow‐up including laboratory testing [38]. Consequently, intended moments of use of the model must be clearly identified. For example, milestones in HIV care that may be amenable to using the D:A:D model include encounters when laboratory results may be available such as initial linkage to care, re‐engagement in care or routine follow‐up visits. These milestones may be opportunities for patient–provider discussions about CVD prevention approaches such as lifestyle modification (e.g. smoking cessation) or medication for individuals classified as high risk [12].

In summary, our findings suggest that the reduced D:A:D model is miscalibrated for prediction of 5‐year CVD risk among PLWH engaged in HIV care at an urban safety‐net HIV clinic, but the model may have more benefit than harm for decisions regarding CVD prevention at a risk threshold of 10% compared with treating all or none. Future studies should consider direct comparisons of CVD risk prediction models to identify the model with greatest net benefit rather than comparing with treating all or none. Meanwhile, our findings combined with findings from another population in the United States [11] may inform deliberations about implementing the D:A:D model by comparing population characteristics from these studies with the local population of interest. Nevertheless, further clarity is needed about the intended moment of use of the model to optimize care. A universally accurate risk prediction model is improbable considering multiple sources of heterogeneity between populations [39, 40, 41]. Rather than developing new models to address miscalibration of the D:A:D model, the D:A:D model or other CVD risk prediction models may need to be updated for improved performance in local settings. For example, models could be recalibrated for more accurate predictions and possibly greater net benefit in the population of interest or extended by adding predictors [42, 43, 44]. HIV‐specific predictors such as CD4/CD8 ratio could be candidates for model extension given reported associations with CVD risk [45, 46], but evidence is accumulating that post‐baseline interventions (e.g. initiation of statin therapy or other interventions) may have more profound impact on model performance between populations [47, 48, 49]. Consequently, post‐baseline interventions require further consideration in future studies.

CONFLICT OF INTERESTS

The authors declare there are no conflicts of interest.

AUTHOR CONTRIBUTIONS

IA: conceptualization, methodology, investigation, validation, visualization, writing (original draft), project administration. AMA: conceptualization, validation, investigation, writing (reviewing and editing), supervision. MJC: methodology, software, data curation, formal analysis, visualization, writing (reviewing and editing). FL and MJ: resources, validation, writing (reviewing and editing). RPO: conceptualization, methodology, validation, formal analysis, writing (reviewing and editing), supervision.

Table S1‐S3

Click here for additional data file.