Agreement Between Cardiovascular Disease Risk Scores in Resource-Limited Settings: Evidence from 5 Peruvian Sites.

It is unclear how well currently available risk scores predict cardiovascular disease (CVD) risk in low-income and middle-income countries. We aim to compare the American College of Cardiology/American Heart Association (ACC/AHA) Pooled Cohort risk equations (ACC/AHA model) with 6 other CVD risk tools to assess the concordance of predicted CVD risk in a random sample from 5 geographically diverse Peruvian populations. We used data from 2 Peruvian, age and sex-matched, population-based studies across 5 geographical sites. The ACC/AHA model were compared with 6 other CVD risk prediction tools: laboratory Framingham risk score for CVD, non-laboratory Framingham risk score for CVD, Reynolds risk score, systematic coronary risk evaluation, World Health Organization risk charts, and the Lancet chronic diseases risk charts. Main outcome was in agreement with predicted CVD risk using Lin's concordance correlation coefficient. Two thousand one hundred and eighty-three subjects, mean age 54.3 (SD ± 5.6) years, were included in the analysis. Overall, we found poor agreement between different scores when compared with ACC/AHA model. When each of the risk scores was used with cut-offs specified in guidelines, ACC/AHA model depicted the highest proportion of people at high CVD risk predicted at 10 years, with a prevalence of 29.0% (95% confidence interval, 26.9-31.0%), whereas prevalence with World Health Organization risk charts was 0.6% (95% confidence interval, 0.2-8.6%). In conclusion, poor concordance between current CVD risk scores demonstrates the uncertainty of choosing any of them for public health and clinical interventions in Latin American populations. There is a need to improve the evidence base of risk scores for CVD in low-income and middle-income countries.

Increasing rates of cardiovascular diseases (CVD) in low-income and middle-income countries (LMIC), and occurring at younger ages than seen in high-income countries, urge the need for action.[1-3] Many public health interventions have arisen to prevent CVD in LMIC, including policies to reducing tobacco usage and salt intake. CVD prevention at the individual level remains necessary and cost-effective.[4,5] The World Health Organization (WHO) recommends using CVD risk assessment tools to direct evidence-based medication, including aspirin and statin therapy, for primary prevention of asymptomatic individuals in relation to any given level of CVD risk.[6] In addition, recent evidence augments these recommendations by showing that blood pressure lowering protects patients according to their CVD risk status. Because larger benefits were observed among groups with greater baseline CVD risk, risk equations appear well positioned to also inform blood pressure-lowering treatment decisions.[7]

However, to date, no CVD risk assessment tool has been developed using longitudinal data from LMIC. The likelihood of this occurring in the near term is very small. The most recent comprehensive systematic pooling of worldwide longitudinal cardiometabolic data, which analyzed 96 cohorts totaling 1.8 million subjects,[8] looked prospectively to the relationship and burden of 2 important outcomes, ie, coronary heart disease (CHD) and stroke. Yet, out of the 97 studies, only one was from the South American region (Brazil).[8] In this context, existing CVD risk assessment tools do fulfill a role on informing day-to-day clinical practice in most LMIC. However, frontline clinical workers and policy makers in LMIC settings do not have sufficient guidance on which tool to use better.

Although the Framingham Heart Study resulted in the first cardiovascular risk prediction model, the Framingham risk score (FRS), additional risk scores have been developed accounting for novel CVD risk factors and different populations.[9] Based on the FRS, the third Adult Treatment Panel incorporated risk stratification in determining eligibility for statin therapy.[10] However, the FRS risk assessment tool used in the third Adult Treatment Panel focuses in predicting 10-year risk of coronary events, defined as nonfatal myocardial infarction (MI) and CHD-related mortality. In 2014, the American College of Cardiology/American Heart Association (ACC/AHA) introduced the Pooled Cohort risk equations (ACC/AHA model),[11] which used data from several community-based US-specific cohort studies to develop new sex-specific and race-specific equations to predict 10-year risk for atherosclerotic CVD events, including nonfatal MI, CHD-related mortality, and stroke.[12] The ACC/AHA model has been validated in a separate USA cohort and demonstrated good calibration for the population it was designed.[13] However, when compared with a European cohort, these models showed poor calibration and moderate to good discrimination.[14] It is unclear how well the currently available risk scores assess or predict CVD risk in LMIC and in Latin American settings. With the exception of the FRS, most of CVD risk assessment tools lack external validation in LMIC.[9]

We aimed to compare the ACC/AHA model with 6 other CVD risk tools to assess the concordance of predicted CVD risk in a random sample from 5 geographically diverse Peruvian populations. Moreover, to estimate the clinical impact of adopting the use of each risk assessment tool in Peru, we compared the proportion of individuals classified as high CVD risk according to the established cut-offs for each risk assessment tool.

METHODS

We used secondary cross-sectional data from 2 Peruvian population-based epidemiological studies: the PERU MIGRANT Study[15] and the CRONICAS cohort study.[16] The former, n = 989, described the CVD risk profiles in 3 Peruvian populations: Ayacucho (rural, highlands), Lima (urban, sea level), and rural-to-urban migrants in Lima. The CRONICAS cohort study has been designed as a prospective study of cardiopulmonary risk factors in over 3000 individuals in 4 Peruvian populations: Lima (highly urban, sea level), Tumbes (semiurban, sea level), rural, and urban Puno (highlands). In this study, we used the baseline data from CRONICAS. Both studies used a structured questionnaire and collected anthropometric and laboratory data[15,16] required for CVD risk estimation (Table 1).

TABLE 1.

Features of Cardiovascular Risk Prediction Tools

CVD Risk Prediction Tools

We compared the ACC/AHA model with 6 other CVD risk prediction tools: laboratory Framingham risk score (FRS-lipids),[17] non-laboratory Framingham risk score using body mass index (FRS-non-lab),[17] Reynolds risk score (RRS),[19,20] systematic coronary risk evaluation (SCORE),[21] WHO’s Risk Chart for the Americas Region (WHO/International Society of Hypertension),[18] and the risk chart developed by the Lancet chronic diseases group.[22] The last 2 risk models provide a chart-based categorization of high risk and were included in our analyses because they were created specifically for use in developing countries.

Statistical Analysis

Participants with known CVD, MI, or stroke were excluded from our analysis. To enable comparisons across scores, we restricted our analysis to participants within 45–65 years age range.

Continuous and categorical 10-year CVD risks were calculated, using the risk algorithms or charts, where available. Further details about each risk prediction tools included in this study are shown in Table 1. We assessed agreement in pair-to-pair analyses against the ACC/AHA model. For those scores where risk equations were available, we calculated predicted risk as a continuous variable and used Lin’s concordance correlation coefficient (CCC).[23] CCC quantifies the agreement ranging from −1 to 1, with perfect agreement at 1. CCC has the following classification according to strength of agreement (theoretical): >0.99 almost perfect, 0.95–0.99 substantial, 0.90–0.95 moderate, and <0.90 poor.[24] Empirical cut-offs for delimiting optimal concordance, where CCC values obtained after contrasting different versions of the same CVD risk scores, ie, FRS-Lipids versus FRS-non-lab; SCORE-1 versus SCORE-2, and SCORE-3 versus SCORE-4, are shown separately in Supplemental Tables 1 and 2 (Supplemental Digital Content, http://links.lww.com/HPC/A198).

We calculated percentages of people at high CVD risk for every score, using the established cut-off points in their original reports: ACC/AHA model ≥7.5%, FRS ≥20%, RRS ≥20%, Lancet Chronic Diseases ≥15%, SCORE ≥5%, and WHO/International Society of Hypertension ≥20%. Furthermore, a subsequent analysis was conducted using a similar definition of high CVD risk at 20% across all tools for visual assessment on their percentages of people classified as high risk. Additional agreement evaluation between dichotomous variables of CVD risk was performed using Kappa index (Supplemental Table 3, Supplemental Digital Content, http://links.lww.com/HPC/A198).

STATA 12 software (STATA Corp, College Station, TX, USA) was used for analysis. In all estimations 95% confidence intervals were calculated.

RESULTS

Participant’s Characteristics

The combined datasets included a total of 4604 participants. The following were excluded: 184 participants with prior episodes of MI or stroke and 2165 participants outside of the scores’ age range. To avoid individual-level data duplication, 72 records from the PERU MIGRANT study were also removed, as these subjects also took part in the CRONICAS Cohort Study. Therefore, data from a total of 2183 participants, mean age 54.3 (SD ± 5.6) years, were included in the analysis.

Table 2 shows study populations’ characteristics. Across sites, most of the participants reported a monthly family income of $196 USD or less, except in Tumbes and urban Puno where the majority of participants reported a monthly income between $197 and $535 USD. Both rural sites, Ayacucho and rural Puno, had the lowest body mass index and highest mean high-density lipoprotein cholesterol. Relative to the other study sites, mean glucose was much higher in Tumbes and lowest in Ayacucho, and these 2 sites had higher mean systolic blood pressures.

TABLE 2.

Participant’s Sociodemographic and Cardiovascular Risk Profiles by Study Site

Agreement Between Risk Scoring Tools

Agreement between CVD risks in continuous format (Table 3) was evaluated using CCC. Overall, when scores were compared with ACC/AHA model, in the total sample as well as specific sites, we found poor CCC agreement values. The highest agreement was observed in comparisons between ACC/AHA model and FRS-non-lab (40%) and FRS-Lipid scores (44%). The lowest agreement was observed between ACC/AHA model and SCORE-3 and SCORE-4, with 14% and 17%, respectively.

TABLE 3.

Agreement Between CVD Risk Prediction Tools, Calculated as Continuous Risk

Agreement between risk scores also differed in magnitude by study sites. The highest agreement values were observed in rural Ayacucho, 74% between ACC/AHA model and FRS-non-lab, and in rural Puno, 77% between ACC/AHA model and RRS.

Prevalence of High CVD Risk as Per Recommended Guidelines

Figure 1 shows the prevalence of high CVD risk status in the pooled dataset as per recommendations by each risk assessment tool’s original publication. ACC/AHA model depicted the highest proportion of people at high CVD risk predicted at 10 years, with a prevalence of 29.0% (95% confidence interval, 26.9–31.0%). When analyzed by specific study sites, ACC/AHA model also had the highest prevalence of individuals at high CVD risk relative to other scores (Supplemental Fig. 1, Supplemental Digital Content, http://links.lww.com/HPC/A198). Supplemental Table 3 (Supplemental Digital Content, http://links.lww.com/HPC/A198) shows Kappa for agreement between categories of predicted high risk by scores, with findings indicating poor agreement.

FIGURE 1.

Proportion of high-risk individuals using guidelines’ recommended cut-off levels: ACC/AHA model, 29.0% [95% confidence interval (CI), 26.9–31.0%]; FRS-non-lab, 15.1% (95% CI, 13.4–16.8%); FRS-lipids, 13.7% (95% CI, 12.1–15.3%); RRS, 1.6% (95% CI, 1.1–2.2); WHO/ISH, 0.6% (95% CI, 0.2–8.6%); SCORE-1, 7.5% (95% CI, 6.4–8.7%); SCORE-2, 8.9% (95% CI, 7.6–10.2%); SCORE-3, 1.5% (95% CI, 0.9–2.1); SCORE-4, 1.8% (95% CI, 1.2–2.4); Lancet Chronic Diseases, 1.1% (95% CI, 0.6–1.5).

Using a similar definition of high risk at 20% across all tools, FRS yielded the highest overall prevalence of high-risk status (Fig. 2). This pattern was the same across subgroup populations, except in Ayacucho where high-risk status was much higher with ACC/AHA model (Supplemental Fig. 2, Supplemental Digital Content, http://links.lww.com/HPC/A198).

FIGURE 2.

Proportion of high-risk individuals using a uniform 20% high-risk cut-off level: ACC/AHA model, 6.6% (95% CI, 5.5–7.7%); FRS-non-lab, 15.1% (95% CI, 13.4–16.8%); FRS-lipids, 13.7% (95% CI, 12.1–15.3%); RRS, 1.6% (95% CI, 1.1–2.2); WHO/ISH, 0.6% (95% CI, 0.2–8.6%); SCORE-1, 0.2% (95% CI, 0–0.3%); SCORE-2, 0.3% (95% CI, 0.1–0.5%); SCORE-3, 0%; SCORE-4, 0%.

DISCUSSION

This study compares 6 CVD risk assessment tools in a pooled dataset with different Peruvian populations, representing low-income communities in diverse geographical settings, ie, urban and rural, coastal and mountainous areas. Poor agreement was found for interscore agreement evaluating CVD risk in 10 years. If these risk assessment tools were to be applied following their respective guidelines there would be wide variation in the proportion of groups classified as high risk for CVD. A remarkable high number of people would be eligible to initiate pharmacological therapy according to our results. Our study provides insights about the applicability of CVD high-risk recommendations for LMIC contexts by using real data across a diversity of settings, including poor overlap between risk assessment tools, which signals to major discrepancies that merit careful attention.

Clinical and preventive usefulness can be drawn from our results. We found poor agreement between the newest ACC/AHA model and other available tools. Moreover, there is huge variation between the proportions of the total population classified as high-risk by each risk assessment tool, potentially impairing recommendations for CVD prevention at the individual level. These findings limit the utility of risk prediction scores for public health and health systems planning, particularly in resource-limited settings. In these countries, further economic studies may be needed to assess cost-effectiveness of risk prediction as primary prevention.[25]

Variations observed in risk-score agreement between different Peruvian sub-populations may be explained by differences in baseline predictors’ profiles. The 2 rural sites in our study, Puno and Ayacucho, had different proportions of high risk as determined by ACC/AHA model. These discrepancies were also appreciated in a discordance between which individuals were classified as high risk by each risk assessment tool. Given the rise of noncommunicable diseases in LMIC, further calibration and validation of these tools are needed in such settings, and longitudinal studies could help to clear this cloudy panorama.[26]

In line with Krumholz et al’s[27] reflections, our study also raises substantial concerns about the generalizability of the risk equations and also raises the question of whether a threshold for treatment, which is admittedly arbitrary and is imbued with values about what level of risk is worth treating, is relevant to all countries. Given WHO’s recommendations of using risk assessment scores to determine eligibility for evidence-based therapy, including statin therapy, it is necessary to have accurate and precise CVD risk assessment tools. The relevance of CVD risk assessment is compounded by increasing interest in polypills—low-dose combinations of blood-pressure lowering, cholesterol lowering, and antiplatelet medications—as a form of risk reduction for individuals with globally increased high risk rather than interventions tailored to specific risk factor control.[28]

Using hypothetical data applied to 25 different risk calculators, Allan et al[29] also found poor agreement, in the order of 67%, similar to our study, highlighting the need to calibrate CVD risk assessment models to every population when applying clinical guidelines for pharmacological therapy. Current risk-scoring strategies are limited to USA and European populations,[30] and the INTERHEART modifiable risk score, originated from multi-country cross-sectional data, is restricted to the prediction of MI.[31] Although FRS has been validated for diverse US ethnic groups,[32] it has also been demonstrated that FRS is not stable in predicting risk, overestimates and underestimates, in non-USA groups.[26,33-36] The ACC/AHA model has been tested against US cohorts showing discrepancies.[13,37,38] Similarly, the ACC/AHA model was shown to yield poor calibration against FRS-lipids and SCORE in the Rotterdam cohort study.[14] Although our pooled dataset does not have 10-year follow-up to externally validate any of the risk assessment models, the variability in risk prediction demonstrated in our study highlights the need to externally validate CVD risk assessment models in LMIC.

Populations from different geographical and epidemiological profiles enrich the value of our study, including rural, urban, lowland and high altitude sites, much common across Latin America. Some limitations deserve consideration. Not all tool’s equations were available, thus chart-based scores did not provide risk assessments as a continuous variable, yet in our analysis we included the most commonly used risk scores. Another limitation that affects our comparability arises from differences in the definitions used in risk scores’ predictors and outcomes. For example, WHO and SCORE address CVD-related mortality, whereas the ACC/AHA model focuses on fatal and nonfatal atherosclerotic CVD; however, we would have expected a higher proportion of overlapping between tools because mortality is included in ACC/AHA model. Also, we are cognizant that the comparison of yields using different cut-offs, as per current recommended guidelines, is limited. The reason for doing this assessment is that most healthcare providers will not necessarily be aware of the technical and modeling details behind a “10-year high-risk” label that guidelines tend to “have a paternalistic tone [and] tell physicians what should be done”[27] and, in the absence of locally available guidelines, they will likely drive practitioners’ prescription practices, misassembling level of risk with the consequent overtreatment of people with low level of risk. Therefore, our work signals salient differences and poor agreements that in turns calls for more education around the usage of risk prediction tools in LMIC together with the need to advance the development of LMIC-specific validation efforts.

In summary, we have shown poor agreement when available scores were compared with the newly released ACC/AHA model. We have also shown that in Peruvian population, a high proportion of individuals would be classified into high CVD risk category. These findings emphasize the uncertainty of choosing any of these tools, into both clinical and public health fields, in LMIC realities. Our work signals to a major and urgent need to improve the evidence base for the development and appropriate use of appropriate risk scores for CVD in LMIC.

ACKNOWLEDGMENTS

The authors are indebted to all participants who kindly agreed to participate in the study. Special thanks to all field teams for their commitment and hard work, especially to Lilia Cabrera, Rosa Salirrosas, Viterbo Aybar, Sergio Mimbela, and David Danz for their leadership in each of the study sites, as well as Marco Varela for data coordination. Special thanks to Tracy Wolbach and Patricia Davis (Westat) for their great support provided as part of the NHLBI Trainee Seed Grant Award experience. CRONICAS Cohort Study Group—Cardiovascular disease: Antonio Bernabé-Ortiz, Juan P. Casas, George Davey Smith, Shah Ebrahim, Héctor H. García, Robert H. Gilman, Luis Huicho, Germán Málaga, J. Jaime Miranda, Víctor M. Montori, Liam Smeeth; Chronic obstructive pulmonary disease: William Checkley, Gregory B. Diette, Robert H. Gilman, Luis Huicho, Fabiola León-Velarde, María Rivera, Robert A. Wise; Training and Capacity Building: William Checkley, Héctor H. García, Robert H. Gilman, J. Jaime Miranda, Katherine Sacksteder. PERU MIGRANT Study—Antonio Bernabé-Ortiz, Lilia Cabrera, Héctor H. García, Robert H. Gilman, J. Jaime Miranda, Julio A. Poterico, Renato Quispe, Candice Romero, Juan F. Sánchez, Liam Smeeth.

AUTHOR CONTRIBUTION

J.J.M. conceived the idea. J.C.B.A., F.P., J.A.P., T.P., and J.J.M. further developed the idea and obtained funding for its secondary analysis. J.C.B.A. led the statistical analysis. J.C.B.A., R.Q., F.P., J.A.P., G.A.V., M.B., and T.P., as part of a trainee-led team, wrote the initial drafts of this manuscript. J.J.M., R.H.G., W.C., and L.S. conceived, designed, and supervised the overall fieldwork studies. J.J.M., A.B.O., and W.C. coordinated and supervised fieldwork activities in Ayacucho, Lima, Tumbes, and Puno. All authors participated in writing of manuscript, provided important intellectual content, and gave their final approval of the version submitted for publication.

DISCLOSURES

The authors declare that they have no competing interests.