Systematic Review of Validity Assessments of Framingham Risk Score Results in Health Economic Modelling of Lipid-Modifying Therapies in Europe.

BACKGROUND: The Framingham Risk Score is used both in the clinical setting and in health economic analyses to predict the risk for future coronary heart disease events. Based on an American population, the Framingham Risk Score has been criticised for potential overestimation of risk in European populations.
OBJECTIVE: We investigated whether the use of the Framingham Risk Score actually was validated in health economic studies that modelled the effects of lipid-lowering treatment with statins on coronary heart disease events in European populations.
METHODS: In this systematic literature review of all relevant published studies in English (literature searched September 2016 in PubMed, EMBASE and SCOPUS), 99 studies were identified and 22 were screened in full text, 18 of which were included. Key data were extracted and synthesised narratively.
RESULTS: The only type of validation identified was a comparison against coronary heart disease risk data from one primary preventive and one secondary preventive clinical investigation, and from observational population data in one study. Taken together, those three studies reported an overall satisfactory accuracy in the results obtained by Framingham Risk Score predictions, but the Framingham Risk Score tended to underestimate non-fatal myocardial infarctions. In five studies, potential issues in applying the Framingham Risk Score on a European population were not addressed.
CONCLUSION: Further studies are needed to ascertain that the Framingham Risk Score can accurately predict cardiovascular outcome in health economic modelling studies on lipid-lowering therapy in European populations. Future modelling studies using the Framingham Risk Score would benefit from validating the results against other data.

Key Points

Introduction

Prevention of cardiovascular (CV) disease is based on identifying individuals at risk, most often by assessing the prevalence of risk factors for CV disease. Numerous risk factors have been implicated in the development of atherosclerosis and subsequent coronary heart disease (CHD) and all-cause CV disease. However, relatively few risk factors such as age, sex, blood pressure, dyslipidaemia, smoking, and diabetes mellitus provide sufficient information to form the basis of most CV risk prediction models. Such risk prediction models usually estimate the risk of a CHD event or CV mortality occurring within the next 5–10 years.

The Framingham Heart Study is a long-term cohort study on the incidence and effects of CV disease. The study started in 1948 in Framingham, MA, USA, with a closed cohort of adult residents, and is based on data from every second year routine health examinations. The Framingham Heart Study has previously been described in detail [1]. Based on the Framingham Heart Study, the Framingham Risk Score (FRS) prediction model for incident CV disease (CHD, ischaemic stroke, occlusive peripheral artery disease, and congestive heart failure) was presented in 1976, and included the variables sex, age, systolic blood pressure, total cholesterol, diabetes, current smoking status, and left ventricular hypertrophy by electrocardiography [2]. Subsequent results from the Framingham Heart Study including larger data sets, extended follow-up, and the influence of cholesterol subfractions have provided further updated FRS prediction models for incident stroke, heart failure and certain other CV diseases [3-6], survival following CV events [4, 7], and risk equations for primary CHD morbidity and mortality [8-10], and for secondary (subsequent) CHD events [10].

The FRS version often currently recommended for use in clinical practice for 10-year risk estimates of incident myocardial infarction or CHD mortality in subjects with no prior CV disease or diabetes includes sex, age, total cholesterol, high-density lipoprotein-cholesterol, systolic blood pressure, treatment for hypertension, and current smoking status [11, 12]. However, given its simple points-based system [11] and electronic access to continuous risk functions [12], it is not widely used in health economic modelling studies of new treatments, where other FRS equations are preferred (see Table 1).

Table 1

Summary of the included studies

Author	Year published	Population studied	Type of study	Treatment	Use of Framingham risk score	Framingham Risk Score version used	Validated against data	Limitations discussed	Level of validation
Martens et al. [27]	1989	Dutch	Cost-effectiveness	Simvastatin vs. cholestyramine	To assess probabilities of CHD mortality	1987 [3]	No	Yes	2
Martens et al. [28]	1990	Dutch	Cost-effectiveness	Simvastatin vs. cholestyramine	To assess probabilities of CHD mortality	1987 [3]	No	No	1
Hjalte et al. [29]	1992	Swedish	Cost-effectiveness	Simvastatin vs. cholestyramine	To estimate lifetime risk of CHD mortality	1987 [3]	No	No	1
Morris [30]	1997	Scottish	Model-type comparison	Pravastatin vs. placebo	Comparison to results from the WOSCOPS study for CHD morbidity and mortality	1987 [3], 1991 [8]	Yes	Yes	3
Muls et al. [31]	1998	Belgian	Cost-effectiveness	Pravastatin vs. placebo	To project the risk for mortality 10 years post-myocardial infarction	1988 [7]	No	Yes	2
Marang-van de Mheen et al. [32]	2002	Dutch	Cost-effectiveness	Screening and statin treatment of FH	To obtain CHD-specific mortality risk under various treatment strategies	1991 [4]	No	Yes	2
Roze et al. [33]	2006	British	Economic impact/events	Adding nicotinic acid to statin treatment	Risk estimates of the cumulative incidence of CHD events	2000 [9]	No	No	1
Roze et al. [34]	2006	Scandinavian	Cost-effectiveness	Adding lipid-modifying drug treatment to statin treatment	To evaluate lifetime clinical and economic outcomes	2000 [9]	Yes	Yes	3
Davies et al. [35]	2006	British	Cost-effectiveness	Treatment with different statins	To estimate primary risk for CV disease, and secondary risk for CHD	1991 [4], 2000 [9]	No	Yes	2
Roze et al. [36]	2007	French	Cost-effectiveness	Adding nicotinic acid to statin treatment	Risk estimates of the cumulative incidence of CHD events	2000 [9]	No	Yes	2
Franco et al. [37]	2007	Dutch	Cost-effectiveness	Statin treatment	To model effects on CHD outcomes, stroke and mortality	1991 [4]	No	Yes	2
Alonso et al. [38]	2008	Spanish	Cost-effectiveness	Atorvastatin vs. routine clinical practice in FH	To estimate CV mortality	2000 [9]	No	Yes	2
Stollenwerk et al. [39]	2009	German	Cost-effectiveness	CHD risk screening and statin use vs. no screening	To calculate lifetime CHD risk	1991 [8]	Yes	Yes	3
Soini et al. [40]	2010	Finnish	Cost-effectiveness	Treatment with different statins	To estimate the annual probability for recurrent CHD events	1991 [4], 2000 [9]	No	Yes	2
De Smedt et al. [41]	2012	8 European countries	Cost-effectiveness	Statin treatment	To estimate the risk for subsequent CHD, stroke and chronic heart failure	1991 [5], 1999 [6], 2000 [9]	No	Yes	2
Fragoulakis et al. [42]	2012	Greek	Economic impact	Treatment with different statins	To estimate probability of non-fatal acute MI or stroke; and transition probabilities from acute MI or stroke to death	2002 [11, 12]	No	No	1
Ohsfeldt et al. [43]	2012	Swedish	Cost-effectiveness	Rosuvastin vs. no active treatment	To model effects on CV disease outcomes and mortality	1998 [10]	No	No	1
Liew et al. [44]	2012	Belgian	Economic impact/events	Atorvastatin vs. simvastatin	To estimate incident non-fatal or fatal CHD or stroke	1991 [4]	No	Yes	2

CHD coronary heart disease, CV cardiovascular, FH familial hypercholesterolemia, MI myocardial infarction, WOSCOPS West of Scotland Coronary Prevention Study

The studies were allocated to one of three categories according to the level of validation of the use of the FRS to predict future CV events, as follows. Category 1 contains studies without any validation of the use of FRS to predict future CV events in a European population. The authors applied the FRS without discussing potential shortcomings. Category 2 contains studies where the use of the FRS to predict future CV events on a European population was in some way discussed but not validated, indicating awareness about potential shortcomings and its implications. Category 2 also includes studies, which assessed the FRS outcome to predict future CV events in a sensitivity analysis. In category 3, the results of the FRS to predict future CV events were both discussed and validated. The authors performed some type of validation, such as a comparison to another risk score or to relevant epidemiological or clinical data on the CV outcome

When using the FRS (where a 10-year risk for a CHD event is calculated) in a clinical setting, it is important to recognise that age carries a considerable weight. This results in patients of high age to be candidates for preventive advice and treatment, such as a smoking cessation programme or starting lipid-lowering drug therapy with a statin treatment even if they lack other risk factors. This is in contrast to less common risk algorithms where lifetime risk is assessed [13]. There are other risk assessment tools to calculate the 10-year risk of CV disease outcomes, such as ASSIGN, PROCAM, Q-RISK and SCORE [13-16]. They may all fall short in motivating lifestyle changes in younger people with a low 10-year risk but a high prevalence of risk factors [13].

Modelling the cost effectiveness of new treatments for CHD requires the ability to estimate the number of incident events, as well as incremental changes in event frequency related to intervention, to estimate the potential benefit of the new treatment. One way of estimating such a benefit is to use a risk prediction tool, such as the FRS, to predict CHD morbidity and mortality. One advantage with the FRS is that it offers publicly accessible risk equations, making it possible for modellers to use a validated approach for the estimation of future CHD risk and to model the effect of interventions on the risk factors included in the FRS. However, there have been concerns about the applicability of the FRS when modelling CHD events in European populations, where the prevalence of risk factors is different, as compared with the original Framingham Heart Study cohort. One argument is that the FRS overestimates the absolute risk of CHD in specific low-risk populations [17] with an overestimation of the mortality in CHD in European populations of around 50% [15, 18]. Other studies, however, have reported that the FRS has a high predicative ability also in populations perceived to be quite different from the Framingham Heart Study cohort [19]. This difference warrants further validation of the results of the FRS when used in health economic modelling.

Although new risk-prediction models have been developed in an effort to overcome the purposed shortcomings of the FRS and to better reflect national European population characteristics (e.g. ASSIGN, PROCAM, QRISK and SCORE) [13-16], the FRS continues to be used in recent European health economic models. Lipid-lowering drug therapy is a timely topic in CV prevention as recent large randomised controlled studies indicate that new classes of lipid-lowering therapy will complement statins with significant reductions in CV events [20, 21]. Thus, the objective of this systematic review was to investigate whether health economic studies on the effects of lipid-lowering treatment with statins on CV events in European populations have validated the ability of the FRS to predict future CV events.

Methods

Literature Search

A literature search was undertaken in September 2016 in EMBASE and PubMed databases to identify health economic modelling studies published in English using the FRS to predict future CV events. Additional searches in SCOPUS did not yield further results. Inclusion criteria for the studies were (1) to be a health economic modelling study with assessment of CV outcomes, (2) to be performed in a European setting, and (3) to have a focus on lipid-lowering statin treatment. The focus on lipid-lowering statin treatment was used to limit the scope of the review to an area of high importance. No time restrictions were included. The literature search was based on search facets: #1 for lipid-modifying therapies, #2 for European countries, #3 for validation of study results, #4 for the Framingham Heart Study and the FRS, and #5 for health economics. The search facet #3 for methodological search terms was not applied to the final results of the EMBASE search because it excluded too many studies. The search facets were combined to find relevant studies. The search strategy is described in detail in the Appendix (see Electronic Supplementary Material). Reference lists in relevant review articles were also screened for potential eligible studies. This review is reported with alignment to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement, where applicable [22].

Study Selection and Data Extraction

We identified a total of 113 studies, as presented in Fig. 1. After excluding duplicates and conference abstracts, 99 studies remained and the study abstracts was screened to ascertain the suitability for inclusion. Of the 99 studies, 77 were excluded because they did not fulfil the inclusion criteria; mainly because the study design was not health economic modelling or that the investigated treatment was not lipid-lowering statins. A total of 22 studies were assessed in full text, four of which were subsequently excluded, leaving a total of 18 studies in the final review. Of the four excluded studies, one study was a pharmacoeconomic review [23], one did not use the FRS directly to predict the future risk of CV disease events [24], one study analysed antihypertensive treatment [25], and one was excluded because of a non-European study population [26]. The study selection process was based on the PRISMA statement [22].

Fig. 1

Flow diagram of selected studies. The study selection process was based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement [21]

The contents of all studies was assessed independently by both investigators, who extracted information about study type, the use of FRS, study outcome and its validation, potential weaknesses of the study design and the validation of FRS ability to predict future CV events. All extracted information was summarised in an extraction grid to generate an overview of the information. Initial disagreements were resolved by consensus. All included studies were allocated to one of three categories according to the level of validation of the use of the FRS to predict future CV events, as follows. Category 1 contains studies without any validation of the use of the FRS to predict future CV events in a European population. The authors applied the FRS without discussing potential shortcomings. Category 2 contains studies where the use of the FRS to predict future CV events on a European population was in some way discussed but not validated, indicating awareness about its potential shortcomings and implications. Category 2 also includes studies, which assessed the FRS outcome to predict future CV events in a sensitivity analysis. In category 3, the results of the FRS to predict future CV events were both discussed and validated. The authors performed some type of validation, such as a comparison to another risk score or to relevant epidemiological or clinical data on CV outcome.

Results

All 18 studies included are summarised in Table 1. A Markov model for economic evaluation was reported in approximately half of the studies (e.g. [3, 33–36, 39–42, 44]). In five of the included studies, the authors had applied the FRS without discussing potential shortcomings. Another ten studies indicated awareness about potential shortcomings of the FRS to predict future CV events on a European population. The type of economic evaluation model (i.e. Markov models or other discrete events models) did not seem to relate to comments by the authors on potential shortcomings of the FRS. Of note, one Finnish study reported that their model structure had been validated previously, but provided no details on the validation of the FRS outcome [40]. Another study adjusted their sensitivity analysis to account for the FRS not being validated in a French population [36]. The authors found that lowering the transition probability for the risk of CV disease by 50% resulted in a 100% increase (i.e. worsening) of the incremental cost-effectiveness ratio [36]. Davies and co-workers reported [35] that they applied a risk prediction for FRS in a British population by use of a previously published risk adjustment method to decrease the overestimation of CHD risk [18], thereby making the results obtained by the FRS more adapted to the British population. One study that included patients from a number of European countries reported that altering the FRS affected the incremental cost-effectiveness ratio more than any of the other analysed parameters in their sensitivity analysis [41].

Only three of the included studies [30, 34, 39] validated the risk predictions obtained by the FRS against other risk estimates, such as epidemiological or clinical databases on CHD events (Table 2). One British study concluded that the risk for CHD derived by the FRS underestimated the risks of non-fatal acute myocardial infarctions, while the FRS predicted the risk of death from CHD with high accuracy [30]. Another study reported that the predicted risk by the FRS for both non-fatal and fatal CHD events was in good agreement with the observed incidence in the secondary preventive setting of patients with established CHD [34]. A German study found that the predicted risk for all CHD events derived by the FRS was higher than the reported incidence of non-fatal myocardial infarction and CHD death, while the predicted risk for CHD derived by the FRS in subjects with no previous myocardial infarction was below the reported incidence of non-fatal myocardial infarctions and CHD death [39].

Table 2

Studies where validation of the results obtained by the Framingham Risk Score (FRS) were performed against clinical or epidemiological data

Author	Year published	Type of validation
Morris [30]	1997	Risks for CHD morbidity and CHD mortality derived from the FRS were validated against data obtained from a randomised placebo-controlled outcome trial on statins in primary prevention (WOSCOPS)
Roze et al. [34]	2006	Lifetime clinical (CHD) and economic outcomes derived from the FRS were validated against European data from a randomised placebo-controlled outcome trial on statins in secondary prevention in patients with CHD (4S)
Stollenwerk et al. [39]	2009	Lifetime CHD risk derived from the FRS was validated against empirical data from large population surveys (MONICA Augsburg study)

CHD coronary heart disease, WOSCOPS West of Scotland Coronary Prevention Study, 4S Scandinavian Simvastatin Survival Study

Discussion

This systematic review of the validation of CV event rates obtained from FRS equations in health economic modelling of lipid-lowering therapy in European populations showed that only three studies [30, 34, 39] included validation of the FRS event rate to other epidemiological risk assessments. When the FRS outcome was validated against outcome data on CV risk, the results of one British study [30] suggested that the FRS performs well in predicting CHD mortality, while underestimating the risks of non-fatal myocardial infarctions. Additionally, results from a secondary preventive setting in the Scandinavian countries [34] suggested that the FRS could predict fatal (and recurrent non-fatal) CHD events well. However, both studies [30, 34] compared predicted the incidence from FRS with data from another study, which can be viewed as a rather crude method for validation. Finally, one study validated the outcome predicted by the FRS to observational population data in Germany [39]. The predicted risk for all CHD events derived by the FRS was higher than the observed incidence of non-fatal myocardial infarction and CHD fatality, while the predicted risk for CHD in subjects with no previous myocardial infarction was below the observed incidence of non-fatal myocardial infarctions and CHD death. The results [39], which were to be anticipated, suggested that the FRS performs well in predicting CHD events.

Three studies changed key parameters of the model, including the FRS-derived risk, in sensitivity analyses [35, 36, 41]. Another seven studies mentioned potential issues in applying the FRS on a European population, whereas five studies did not attempt validation of their results with regard to the risk algorithm used, as further presented in Table 1. Sensitivity analyses show that cost-effectiveness models for lipid-lowering therapy seem to be highly sensitive to the FRS assumptions [35, 36, 41]. A sensitivity analysis is a test of the model inputs and its effects on the model outcome, thereby not a method for validating the FRS outcome from the model.

Although the use of regression-based risk estimations such as the FRS is a valid method to predict CV events, the use of the FRS is often criticised for providing an incorrect event rate in populations different from the Framingham cohort [15, 18]. However, based on the findings in three studies [30, 34, 39], the FRS may still be a valid option when conducting health economic studies in Europe on CV outcome with lipid-lowering drug treatment. The FRS could also provide health economic modellers with a prediction of absolute risk in a person with specific characteristics to be used in health economic models. However, this is somewhat different to what the FRS was intended for and could be criticised, thus highlighting the need for validating results when using risk scores intended for clinical practice in health economic models. When extrapolating FRS risk for more than the intended 10 years, the issue of lifetime risk of CV disease subjects comes into play [11, 13] and makes validation of the FRS results important. Thus, caution is warranted when extrapolating FRS results beyond 10 years. Prediction issues may arise if epidemiological data and clinical outcome data are collected for one geographical population in one specific time period, but then used to assess CV risk in another geographical and/or temporal setting [31].

The applicability of FRS data to non-US populations has been partly addressed in previous studies. When comparing estimated CV event rates using risk equations based on US populations with observed incidence rates in European countries, CV incidence in Europe was only one half of those estimated from the US-based risk equations [15, 18]. Another study reported that FRS or PROCAM risk functions should not be used to estimate the absolute CHD risk of middle-aged men without history of CHD living in Northern Ireland or in France because of a clear overestimation in both cases [45]. However, one study demonstrated that FRS-derived CV risk accurately predicted CV event incidence in the PROCAM study in Germany [19]. Thus, we suggest that future studies would benefit from including a validation of the FRS outcome, especially when used on European populations. Validations could be performed using national healthcare data and performing separate validation analyses, or comparing the results from the model with those of published studies on similar interventions [46]. Another way to validate outcome models is to use the Assessment of the Validation Status of Health Economic decision models (AdViSHE). In AdViSHE, modellers report in a systematic way both validation and outcomes [47]. Such efforts would aid future decisions on the use of the FRS and its outcomes in health economic models in European populations.

This study on outcomes in health economic models in European populations has important limitations to be considered, and the results must be interpreted accordingly. We included studies reported in English only and eligible reports in other languages may have been excluded. There was no formal assessment of the methodological quality of the included studies. Several FRS equations are available and were used in the studies, and the results may not be interchangeable between different FRS risk prediction models. The European population is not uniform and the results may not be valid for all European countries or regions. However, the studies with validation of the FRS event rate to other epidemiological risk assessments represent three different countries.

Conclusions

In this systematic review, only three health economic modelling studies of lipid-lowering therapy in European populations used the FRS and also validated their results. Those results suggest rather good CV event predictions. Future modelling studies on lipid-modifying therapies would benefit from validating the FRS prediction of CV outcome, to confirm or reject the validity of using the FRS in modelling studies on European populations.

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 120 kb)

The Framingham Risk Score is used in health economic analyses to predict the risk for future coronary heart disease events. However, based on an American population, the Framingham Risk Score has been criticised for potential overestimation of cardiovascular risk in European populations.

Only three studies have validated the use of the Framingham Risk Score in health economic studies modelling the effects of lipid-lowering treatment on coronary heart disease events in European populations. Those studies reported an overall satisfactory accuracy in the risk predictions by the Framingham Risk Score, but the risk for non-fatal myocardial infarctions tended to be underestimated.

Further studies should ascertain that the Framingham Risk Score could accurately predict cardiovascular outcome in health economic modelling studies in European populations. Future modelling studies using the Framingham Risk Score should validate results against other data.