Using Published Health Utilities in Cost-Utility Analyses: Discrepancies and Issues in Cardiovascular Disease.

BACKGROUND: Health utilities are commonly used as quality weights to calculate quality-adjusted life years in cost-utility analysis (CUA). However, if published health utilities are not properly used, the credibility of CUA could be affected.
OBJECTIVES: To identify discrepancies in using published health utilities in CUAs for cardiovascular disease (CVD).
METHODS: CVD CUAs in the Tufts Cost-Effectiveness Analysis Registry that reported health utilities were included in the analysis. References cited for health utilities in these CUAs were reviewed to identify the original health utility studies. The description and value of health utilities used in the CUA were compared with those reported in the original utility studies. Logistic regression was used to identify the factors that can predict the discrepancy.
RESULTS: A total of 585 eligible CUAs published between 1977 and 2016 were identified and reviewed. Of these studies, 74.5% were published between 2007 and 2016. 442 CUAs that used a total of 2235 health utilities published in 203 original utility studies were included for the comparison. As compared with those utilities originally reported, only 596 (26.7%) health utilities had the same description and value, whereas 991 health utilities (44.3%) differed in both description and value. Of 1290 health utilities with a different description, 69.1% were due to different severity or disease. No explanation or justification was provided for 1171 (87.4%) of 1340 health utilities with different value.
CONCLUSIONS: There are concerning discrepancies in using published health utilities for CVD CUAs. Given the important role health utilities play in CUAs, authors of CUAs should always refer to the original studies for health utilities and be transparent about how published health utilities are selected and incorporated into CUAs.

Cardiovascular disease (CVD) is a class of diseases that involve the heart and blood vessels. CVD includes coronary heart disease, congenital heart disease, rheumatic heart disease, and other conditions.[1-3] Globally, 470.8 million patients were living with CVD in 2016, a substantial increase by 26.7% from 2006. CVD has been the leading cause of noncommunicable disease deaths worldwide. In 2016, it was estimated that 17.9 million people died from CVD, accounting for 44% of noncommunicable disease deaths. Among these deaths, approximately 75% occurred in low- and middle-income countries.[3,7]

The financial and humanistic burden of CVD is enormous. It imposes a significant impact on the health-related quality of life (HRQoL) of patients.[8,9] Globally, CVD cost $863 billion in 2010. Direct health care costs and productivity loss accounted for 55% and 45% of the total cost, respectively. In 2016, the total cost of CVD was $555 billion in the United States, and this figure is estimated to reach $1100 billion by 2035.

Allocating scarce health care resource has been an ongoing challenge to the sustainability of health care systems. Economic evaluations consider both costs and patient-reported outcomes and have been increasingly used as an important tool to inform allocation decisions. Cost-utility analysis (CUA), a type of economic evaluation that uses the quality-adjusted life-year (QALY) as the outcome measure, is widely recommended[12-15] and commonly used to assess the cost-effectiveness of CVD treatments.[16-19]

Health utility measures the cardinal preference of a health state on the scale anchored at 0 for dead and 1 for full health, with a negative value for states perceived as worse than dead. Health utilities could be measured directly by using preference elicitation techniques or indirectly administering utility-based instruments to target respondents. Health utilities are used as quality-of-life weights to calculate QALYs in CUA and therefore play an important role in determining the cost-effectiveness of a treatment. Studies show that health utilities are one of the most sensitive variables in CVD CUA.[22-24] A common practice in conducting model-based CUA is to identify and choose health utilities from published sources. However, little is known about how published health utilities have been used in CUAs. Therefore, the objective of this study was to identify discrepancies and issues related to health utilities used in the CVD CUAs as compared with those reported in the referenced original health utility studies.

Methods

Identification of CUAs in CVD

Tufts Cost-Effectiveness Analysis Registry (CEAR) was used to identify published CUAs for CVD. CEAR is a registry database that includes more than 6000 cost-effectiveness analyses (CEA) published since 1976. All CEAs were identified in MEDLINE and summarized in the registry according to a previously formalized review protocol. Diseases are divided into 21 chapters corresponding to ICD-10 classification codes in CEAR, and all data are updated regularly. We focused on the disease of the circulatory system chapter (ICD-10 classification code I00-02, I05-I09, I10-I15, I20-I25, I26-I28, I30-I52) indicated as the primary disease to identify CVD in the registry using the search term of DisChapter1. CEAs that reported information on health utilities were eligible. CEAs not published in English or without reporting health states and utility values were excluded (e.g., only report QALYs). We reviewed all references cited for health utilities in the included CUAs to identify the original health utility studies where these utilities were first reported.

Sources of Health Utilities in the CUAs

For the purpose of this study, we define original health utility study as a study that estimates and reports health utilities (including the description and utility value of health states) based on either primary data collection or analysis of existing data. We group the sources of health utilities in the CUAs into 6 categories: 1) the CUA itself contained an original health utility study (e.g., CUA alongside a clinical trial that also administered direct elicitation methods such as time tradeoff or preference-based instruments such as EQ-5D), 2) published original health utility studies, 3) published nonhealth utility studies, 4) unpublished studies, 5) expert opinion or clinical judgment, and 6) no identifiable source. To do this, we retrieved and reviewed the full text of all references cited for health utilities in CUA, if available. If these references were the original health utility studies, we labeled them as a first-level citation. If the reference was not the original health utility studies but instead, for example, a CUA, we then retrieved and reviewed the references cited for health utilities in that CUA. If those references were the original health utility studies, we labeled them as a second-level citation, and so forth, until no such study could be identified.

Data Extraction

We designed a data extraction form to record and compare the health utilities between CUAs and referenced original health utility studies. The form was pilot tested by extracting the data of 10 randomly selected CUAs in the registry and referenced original health utility studies. The final form records key characteristics of the CUA, including author information, published year, study type, perspective, time horizon, intervention, and probabilistic sensitivity analysis and of the original health utility studies, including the citation level, publication year, and the utility measurement methods. The description and value of each health state in the CUA and the original health utility studies were recorded as well.

Comparison between CUA and Original Health Utility Studies

The comparison focused on the health utilities that came from identifiable published original health utility studies. We compared the description and the utility value of the health state used in the CUA with that reported in the original studies. The results of the comparison fall into 1 of 3 groups: 1) The utility value in the CUA is the same as in the original study, but the two differ in the state description. Furthermore, we documented the discrepancy in the health state description in terms of disease (e.g., myocardial infarction [MI] in CUA v. stroke in the original utility study), severity (e.g., mild angina in CUA v. severe angina in the original utility study), duration (e.g., first year after MI in CUA v. 6 wk after MI in the original utility study), comorbidity (e.g., resistant hypertension with angina in CUA v. diabetes with angina in the original utility study), patient population, and treatments. 2) The health state description is the same, but the utility value differs between the CUA and the original study. We retrieved any justification for the change in utility value or ascertained whether disutility was mistakenly used as utility. and 3) Both description and value were different.

To identify the factors that might have significantly influenced the discrepancy, we performed logistic regressions using dichotomized variables for discrepancy as the response variable with selected explanatory variables for CUA, including country, sponsor (industry v. others), author affiliations (academia v. others), year of publication (before or after 2007), perspective (public health care payer v. others), quality score (range: 2 [lowest] to 7 [highest]), citation level for utilities, and utility elicitation methods (indirect v. direct methods). A 2-tailed P < 0.05 was considered statistically significant. All analyses were performed using STATA14MP (StataCorp LLC, College Station, TX, USA).

Results

Characteristics of CUAs for CVD

As shown in Figure 1, we identified 741 CEAs from the CEAR using the ICD-10 classification codes for the circulatory system. Of these studies, we excluded 156 (147 did not report any information on health utilities, 2 were not published in English, and the full texts of 7 were not available). Of the 585 CUAs reviewed, 442 used 2235 health utilities published in 203 original studies, with a median number of 4 (range, 1–36) health utilities used in the CUAs. Of the 2235 health utilities, 1753 (78.4%) were identified through first-level citation, 425 (19.0%) through second-level citation, and 57 (2.6%) through third-level or higher citation (Figure 1).

Figure 1

Identification of cost-utility analysis and referenced original health utility studies. CUA, cost-utility analysis; CEA, cost-effectiveness analysis, CVD, cardiovascular diseases.

Of the 585 CUAs reviewed, 524 (89.6%) were model based and 61 (10.4%) trial based. As shown in Table 1, 361 (61.7%) and 145 (24.8%) were conducted from health care payer and societal perspectives, respectively. The health care payer’s perspective increased from 53 (35.6%, prior to 2007) to 308 (70.6%, 2007 to 2016), while the societal perspective decreased from 58 (38.9%) to 87 (20.0%) during the same time periods. Despite CVD’s long-term impact on patient health,[1,7] only 323 (55.2%) CUAs used the lifetime horizon. Most of the CUAs (n = 466, 79.6%) assessed medications, invasive surgery, or medical procedures. There were 103 CUAs mistakenly using published nonhealth utility studies and 54 without providing the source for health utilities (Table 1).

Table 1

Characteristics of Cost-Utility Analyses (CUAs) for Cardiovascular Disease Identified (n = 585)

	1977–2006 (n = 149)	2007–2016 (n = 436)	1977–2016 (n = 585)
CUA study type
Model-based study	133 (89.3%)	391 (89.7%)	524 (89.6%)
Trial-based study	16 (10.7%)	45 (10.3%)	61 (10.4%)
Disease code/type
I00-I02: Acute rheumatic fever	0 (0%)	2 (0.5%)	2 (0.3%)
I05-I09: Chronic rheumatic heart diseases	0 (0%)	1 (0.2%)	1 (0.2%)
I10-I15: Hypertensive diseases	17 (11.4%)	33 (7.6%)	50 (8.5%)
I20-I25: Ischemic heart diseases	57 (38.3%)	140 (32.1%)	197 (33.7%)
I26-I28: Pulmonary heart disease anddiseases of pulmonary circulation	4 (2.7%)	3 (0.7%)	7 (1.2%)
I30-I52: Other forms of heart disease a	71 (47.7%)	257 (58.9%)	328 (56.1%)
Perspective
Health care payer	53 (35.6%)	308 (70.6%)	361 (61.7%)
Society	58 (38.9%)	87 (20.0%)	145 (24.8%)
Others b	1 (0.7%)	6 (1.4%)	7 (1.2%)
Not stated/unclear	37 (24.8%)	35 (8.0%)	72 (12.3%)
Time horizon
Lifetime	81 (54.4%)	242 (55.5%)	323 (55.2%)
Nonlifetime	47 (31.5%)	176 (40.4%)	223 (38.1%)
Not stated/unclear	21 (14.1%)	18 (4.1%)	39 (6.7%)
Intervention types
Pharmaceutical products	96 (64.4%)	299 (68.6%)	395 (67.5%)
Surgery/medical procedure	22 (14.8%)	49 (11.2%)	71 (12.1%)
Health care delivery/education	12 (8.1%)	49 (11.2%)	61 (10.4%)
Diagnostic/screening	17 (11.4%)	31 (7.1%)	48 (8.2%)
Other	2 (1.3%)	8 (1.8%)	10 (1.7%)
Sources of health utilities c
Published original health utility studies	104 (67.5%)	338 (61.6%)	442 (62.9%)
CUA contained a health utility study	26 (16.9%)	71 (12.9%)	97 (13.8%)
Published non-health utility study	7 (4.5%)	96 (17.5%)	103 (14.7%)
Unpublished study	0 (0%)	1 (0.2%)	1 (0.1%)
Expert opinion/clinical judgement	0 (0%)	6 (1.1%)	6 (0.9%)
No source provided	17 (11.0%)	37 (6.7%)	54 (7.7%)

Cardiac arrest, heart failure, atrial fibrillation, acute pericarditis, endocarditis, nonrheumatic valve disorders, myocarditis, cardiomyopathy.

The hospital or patient perspective.

Health utilities used in a CUA might come from multiple sources, so the sum of the CUAs is larger than 585.

Methods Used to Derive Health Utilities

Figure 2 shows the distribution of methods used to derive 2235 health utilities. The EQ-5D was the most frequently used (1127 health utilities, 50.4%), followed by time tradeoff (750 health utilities, 33.6%).

Figure 2

Utility measurement methods reported in the original health utility studies (n = 2235 health utilities). Other indirect methods include the Health Utility Index, Quality of Well-Being, and 15-Dimension Questionnaire. Other direct methods include the rating scale and visual analogue scale. TTO, time tradeoff; SG, standard gamble; SF-6D, Short Form–6 Dimension Questionnaire.

Discrepancies in Using Published Health Utilities between CUA and the Original Utility Study

As shown in Table 2, of 2235 health utilities used in CUAs, 596 (26.7%) had the same description and value as in the original health utility studies. This percentage decreased slightly from 28.2% before 2007 to 26.4% after 2007. There were 299 (13.4%) health utilities with the same value but different descriptions between the CUAs and original studies, decreasing from 18.4% before 2007 to 12.4% after 2007. There were 349 (15.6%) health utilities with the same description but different values, increasing from 9.2% before 2007 to 16.8% after 2007. In contrast, there were 991 (44.3%) health utilities with different descriptions and values from those originally reported. Figure 3 shows these numbers by year.

Table 2

Comparison of Health Utilities Used in the Cost-Utility Analyses (CUAs) with Those Originally Reported (n = 2235)

Item	1977–2006	2007–2016	All
All health utilities	n = 347	n = 1888	n = 2235
Same description and value	98 (28.2%)	498 (26.4%)	596 (26.7%)
Different description only	64 (18.4%)	235 (12.4%)	299 (13.4%)
Different value only	32 (9.2%)	317 (16.8%)	349 (15.6%)
Different description and value	153 (44.1%)	838 (44.4%)	991 (44.3%)
Different health state description	n = 217	n = 1073	n = 1290
Different severity	104 (47.9%)	511 (47.6%)	615 (47.7%)
Different disease	63 (29.0%)	214 (19.9%)	277 (21.5%)
Different intervention	14 (6.5%)	128 (11.9%)	142 (11.0%)
Different disease duration	18 (8.3%)	98 (9.1%)	116 (9.0%)
Different age group or sex	9 (4.1%)	84 (7.8%)	93 (7.2%)
Different comorbidity	9 (4.1%)	38 (3.5%)	47 (3.6%)
Different health utility value	n = 185	n = 1155	n = 1340
No explanation	164 (88.6%)	1007 (87.2%)	1171 (87.4%)
Adjustment by CUA authors	20 (10.8%)	146 (12.6%)	166 (12.4%)
Disutility used as utility	1 (0.5%)	2 (0.2%)	3 (0.2%)

Figure 3

Comparing the description and value of health utilities between cost-utility analyses (CUAs) and original health utility studies by year.

We further documented the details of the discrepancies. Of 1290 health utilities with different description (i.e., 299 with a different description only plus 991 with a different description and value), 615 (47.7%) were due to different severity. Different disease accounted for 21.5% of the difference, decreasing from 29.0% before 2007 to 19.9% after 2007. Other reasons for the difference included intervention (11.0%), disease duration (9.0%), age/sex (7.2%), and comorbidity (3.6%; Table 2).

Of 1340 health utilities with a different value (including 349 with a different value only and 991 with a different description and value), 166 (12.4%) were due to adjustments made by the CUA authors. However, there was no explanation provided for 1171 (87.4%) health utilities. Less than a half of values used in CVD CUAs were larger than the original value.

Table 3 presents the results of the logistic regression. CUAs conducted in the United States (odds ratio [OR] 0.720; 95% confidence interval [CI] 0.578, 0.897; P = 0.003) and with first-level citation for health utilities (OR 0.298; 95% CI 0.221, 0.403; P < 0.001) more likely used the same health state description and utility value as in the original utility studies. Similarly, CUAs conducted in the United States (OR 0.734; 95% CI 0.602, 0.894; P = 0.002) and with first-level citation for health utility (OR 0.280; 95% CI 0.220, 0.357; P < 0.001) more likely used the same health utility value, whereas CUAs sponsored by industry (OR 1.460; 95% CI 1.203, 1.774; P < 0.001) were more likely to use utility values different from the originally reported. CUAs published after 2007 (OR 0.723; 95% CI 0.556, 0.940; P = 0.016) and with first-level citation for health utilities (OR 0.579; 95% CI 0.466, 0.719; P < 0.001) were more likely to use the same health state description as in the original utility studies.

Table 3

Logistic Regressions with Study Characteristics on the Discrepancy in Health Utilities between Cost-Utility Analyses (CUAs) and Original Health Utility Studies

	Response Variable (v. Others)
	Different Value or Description		Different Value Only		Different Description Only
Explanatory Variable	OR	95% CI	OR	95% CI	OR	95% CI
Industry sponsor	1.149	[0.926, 1.426]	1.460***	[1.203, 1.774]	1.153	[0.955, 1.391]
Academic affiliation	1.228	[0.895, 1.683]	1.060	[0.789, 1.426]	1.318	[0.996, 1.744]
Country: United States	0.720**	[0.578, 0.897]	0.734**	[0.602, 0.894]	0.854	[0.702, 1.039]
Health care payer perspective	0.952	[0.759, 1.195]	0.851	[0.690, 1.050]	1.116	[0.912, 1.366]
CUA quality rating score (>5)	0.939	[0.757, 1.165]	1.194	[0.983, 1.449]	0.856	[0.709, 1.033]
CUA publication year (after 2007)	1.015	[0.761, 1.354]	1.122	[0.859, 1.465]	0.723*	[0.556, 0.940]
First-level citation	0.298***	[0.221, 0.403]	0.280***	[0.220, 0.357]	0.579***	[0.466, 0.719]
Indirect methods	0.903	[0.734, 1.112]	1.160	[0.958, 1.406]	0.955	[0.791, 1.153]
Intercept	7.701***	[4.464, 13.275]	3.240***	[2.013, 5.219]	2.321***	[1.479, 3.643]
Pseudo R2	0.038		0.053		0.016
Goodness-of-fit test	P=0.000		P=0.000		P=0.000

P < 0.05, **P < 0.01, ***P < 0.001.

Discussion

This registry-based analysis found that published studies have been a major source of health utilities for CVD CUAs, especially for model-based analyses. The main approach to measuring health utilities in CVD shifted from direct methods before 2007 to indirect methods after 2007, with the EQ-5D being the dominant utility-based instrument. However, the observed discrepancies in using published health utilities in CUAs for CVD remain concerning.

There was an increase in the incorrect use of non–health utility studies as the source of health utilities. For example, SF-36 summary scores or the EQ-5D visual analogue scale scores were used as health utilities.[26,27] The proportion of CUAs that did not provide any reference for health utilities, although decreasing over time, is still nonnegligible. The references cited for health utilities that were not original health utility studies were found in about 20% of CUAs. Simply referring to another CUA as a source of health utilities, for example, is a poor practice and more likely to lead to discrepancy. When using published health utilities, authors of CUAs should always identify and review the original studies to determine how the health utilities were measured and reported before using them in the CUA.

In less than one-third of the published health utilities, both description and value used in the CUAs were the same as those originally reported. Many health utilities differed in description, value, or both. For some health utilities, a slight difference in description might be reasonable. For instance, the health state for atrial fibrillation patients with warfarin therapy in CUA might be considered similar to the health state for atrial fibrillation patients with aspirin therapy in the original health utility study, because the utility decrement was mainly associated with the act of taking a pill daily for CVD prevention rather than the side effect of warfarin or aspirin. However, the use of the health utility for different disease severity (e.g., mild angina in the CUA and severe angina in the original health utility study) could be problematic.

With a growing body of literature on health utilities related to CVD,[8,29] systematically searching and identifying potentially relevant health utilities for CUA is recommended. When multiple health utilities are identified, CUA authors might have to choose or pool the utilities to, for instance, reflect the characteristics of the target health status and patient population in CUA. Explicit justification is needed for the purpose of transparency and appropriateness.[30-32] However, a concerning finding is that the vast majority of those that differed in utility values from the originally reported did not provide any explanation for the difference. This issue has not been improved over time.

There are a few limitations to our analyses. Our analysis was based only on the CUAs included in the CEAR. The registry may not include all published CUAs, because it searches MEDLINE only. Second, we focused on comparing health utilities between CUAs and referenced original health utility studies. We did not assess the quality of the original health utility studies, nor did we assess the appropriateness of any explanation or justification associated with the use published health utilities in CUAs.

Conclusions

There are concerning issues and discrepancies in using published health utilities in CUAs for CVD. Given the important role health utilities play in CUAs, we should always identify the original source of health utilities and be transparent about how published health utilities are incorporated in CUAs.