Outcome selection and role of patient reported outcomes in contemporary cardiovascular trials: systematic review.

OBJECTIVES: To systematically assess the type of outcomes selected and the prevalence of patient reported outcomes in contemporary cardiovascular trials and to quantify any misuse or underuse of patient reported outcomes using a specially developed tool that would allow estimation of the relevance of such outcomes to clinical decision making.
DESIGN: Systematic review. DATA SOURCES: Medline and Embase. STUDY SELECTION: Randomised controlled trials of the treatment for or prevention of cardiovascular disease published in 10 leading general medical and cardiology journals from January 2005 to December 2008.
RESULTS: Primary outcomes were patient important (death, morbidity, or patient reported outcomes) in only 93 of 413 trials (23%, SE 2%), whereas another 92 (22%, SE 2%) combined these outcomes with other less important ones into a composite. Sixty five trials (16%; SE 2%) used at least one instrument to measure patient reported outcomes, mostly in trials where such information would have been important or crucial for clinical decision making (52 trials). Patient reported outcomes were judged to be of little incremental value to a large number of, mostly explanatory, cardiovascular trials (152 trials). However, many trials in which patient reported outcomes would have been important or crucial for clinical decision making did not report such outcomes (122 of 174 trials, 70%). These included several trials that primarily aimed to improve symptoms or functional status, trials that tested interventions with a considerable potential for causing harm (mainly bleeding) that were not meaningfully measured, and trials with composite outcomes that were dominated by outcomes of questionable importance to patients.
CONCLUSIONS: Despite a continued rise in the reporting of patient reported outcomes with no evidence for their misuse in more recent cardiovascular trials, they seem to be still underused once their relevance to clinical decision making has been taken into account. This was largely explained by inappropriate use of composite outcomes and inadequate measurement of harms.

Introduction

The ultimate goal of clinical trials is to identify interventions to benefit future patients. The selection of best variables for measuring such benefits or harms is complex, however, and has been open to debate.1 2 Perceived from the patients’ perspective, medical interventions should either prevent premature death or improve health related quality of life. Instruments for patient reported outcomes allow measurement of the effect of interventions from the patients’ perspective.3 Such instruments can be particularly useful when interventions aim to improve symptoms and functional status (for example, the walking impairment questionnaire4) or health related quality of life in a wider sense (for example the chronic heart failure questionnaire5).

Earlier studies have suggested that patient reported outcomes might have been underused in randomised trials in general6 7and more so in cardiovascular trials.6 These studies were, however, published several years ago when suitable instruments for measuring patient reported outcomes were limited and only a few researchers had the necessary expertise for using them. More importantly, no information is available on the relevance of the use of patient reported outcomes to contemporary cardiovascular trials. It may be that such instruments are less relevant to many cardiovascular trials, where traditionally the focus has been on major clinical outcomes such as myocardial infarction and stroke rather than the relief of symptoms. Furthermore, many cardiovascular trials are concerned with biological effects of interventions, without any intentions on changing clinical practice. For example, when investigators seek to understand the effect of high dose folic acid on coronary vasodilation8 or to explore the pharmacodynamics and pharmacokinetics of different dosages of an antiplatelet agent,9 the measurement of patient reported outcomes is expectedly of little value. Therefore, without assessment of the relevance of patient reported outcomes to individual trials and their impact on clinical decision making, no reliable conclusions can be made about their misuse or underuse.

We assessed the types of outcomes selected and the prevalence of patient reported outcomes in recent cardiovascular trials. To quantify any current misuse or underuse of patient reported outcomes we further sought to develop a tool that would allow estimation of the relevance of such outcomes to clinical decision making and applied it to a set of cardiovascular trials.

Methods

We searched Medline and Embase for articles with any of the subject terms “cardiovascular diseases” or “cardiovascular agents” published between January 2005 and December 2008 in any of the following leading general medical or cardiology journals: New England Journal of Medicine, Lancet, JAMA, Annals of Internal Medicine, PLoS Medicine, BMJ, Circulation, Journal of the American College of Cardiology, Circulation Research, and European Heart Journal (see web extra appendix 1 for the detailed search strategy).

Review methods and data extraction

Figure 1 summarises the retrieval and selection process for studies. The combined searches yielded 2374 reports, of which 1376 were excluded on the basis of titles and abstracts. The remaining 998 reports were allocated a unique random number with the use of a random sequence generator.10 We assessed in full text the first 500 articles according to this random list for their eligiblity, and extracted relevant data from eligible trials. Of these, 87 did not meet the eligibility criteria. We used an electronic data extraction form to capture pivotal information from each trial. We categorised reported outcomes and individual components of any composite outcomes into patient important outcomes (death, major morbidity, patient reported outcome) or intermediate outcomes—that is, those not immediately or directly important to patients, such as biochemical, physiological, or surrogate outcomes. The categorisation of outcome was based on previously published ranking systems,11 12 with more emphasis on the methods of outcome measurement (see also web extra appendix 2). From the trials evaluated, we assessed those that were reporting patient reported outcomes to see whether a previously validated instrument was used or whether psychometric properties were reported or referenced in the article. All data were extracted in duplicate. The results and disagreements were discussed until consensus was reached.

Fig 1 Search retrieval and study selection process

Development and validation of a tool for ranking relevance of patient reported outcomes

Several articles described seven pivotal features of trials that would need to be considered when making decisions about the appropriateness of patient reported outcomes in clinical trials.3 7 13 14 15 16 17 18 19 20 We used the proposed framework from these studies to devise a tool that would allow a systematic and reliable ranking of the relevance of patient reported outcomes in cardiovascular trials into five categories (crucial, important, potentially relevant, irrelevant, and uncertain) by the level of importance of patient reported outcomes to clinical decision making. With the lack of any gold standards, we sought expert opinion to reach consensus about the validity of the tool. An initial version of the tool, together with a set of instructions for its use, was sent to nine experts who were identified on the basis of previous publications in the area of outcome measurement and health services research. After incorporation of the comments received from four experts, two reviewers (KR and AM) independently applied the modified and face validated tool to 100 randomly selected articles. The experience gained from the initial data extraction was used to inform the final tool (fig 2).

Fig 2 Flow chart for assessment of relevance of patient reported outcomes in clinical trials. Percentages only apply to trials that used composite outcomes

Web extra appendix 2 provides a detailed description of the tool. Briefly, a patient reported outcome was considered irrelevant in studies that primarily sought to test a hypothesis useful for understanding the differences between intervention strategies, without any claims on changing clinical practice. Studies that aimed to help make decisions about alternative strategies were categorised as pragmatic trials. If the main goal of such trials was to improve symptoms, functional status, or quality of life, we considered the patient reported outcome to be crucial. Otherwise, ranking of the relevance of patient reported outcomes depended on several other trial features (see fig 2 and web extra appendix 2).

Prevalence of patient reported outcomes over time

One study assessed the prevalence of quality of life in cardiovascular trials and showed that between 1980 and 1997, 2.1% of such trials reported quality of life and that the rate increased over time (from 0.34% in 1980 to 3.6% in 1997).6 We replicated and updated that reported search and extended the timeframe to 2009 to assess the effect of time on reporting of quality of life in cardiovascular trials.

Statistical analyses

We used descriptive statistics to characterise the included trials. Univariate analyses (χ2 test, Fisher’s exact test, and Mantel-Haenszel test for trend) explored associations of trial characteristics and type of outcome (patient important: yes or no) and measurement of patient reported outcomes. In the first 50 reviewed trials the prevalence for use of patient reported outcomes was 14%. We therefore estimated that data from about 400 trials would provide sufficient information for assessment of associations between trial features and type of outcomes selected. To allow for several ineligible trials and duplicate reports, we set a target of 500 articles to be reviewed in full text.

Excluding the first 100 trials that were evaluated in the pilot phase for development of the ranking tool, we used information from the next set of randomly selected trials, independently extracted by two reviewers (KR and AM), to estimate the chance corrected inter-rater reliability к. We weighted the outcomes to account for the closeness of different categories, as shown in the matrix (see web extra appendix 3). Assessment of inter-rater reliability based on the final 330 reports showed a substantial chance corrected inter-rater agreement between the two investigators: weighted к=0.79; quadratic weighted к=0.77. All analyses were carried out using STATA v 9.1.

Results

Type of reported outcomes

In 93 trials (23%, SE 2%) the primary outcome was patient important (death, major morbidity, patient reported outcome), whereas another 92 trials (22%, SE 2%) combined these with intermediate outcomes of less or no importance to patients (table 1). In 52 of these 92 trials (57%) less than half of the events were attributable to patient important outcomes—that is, the composite outcome was dominated by intermediate outcomes such as physiological, biochemical, or surrogates. All other trials (228 (55%), SE 2%) used intermediate outcomes as their primary outcome. Patient reported outcomes were reported in 65 of the 413 trials assessed (16%, SE 2%) but were the primary outcome in only two trials and a coprimary outcome in another four trials.

Table 1

 Features of 413 trials included overall, by type of primary outcome and by presence or absence of patient reported outcomes. Values are numbers (percentages) unless stated otherwise

Feature	All trials		Primary outcomes dominated by patient important outcomes	P value*		Outcomes included patient reported outcomes	P value†
Total	413		133 (32)			65 (16)
Type of primary outcome:
Patient important	93		93 (100)	—		18 (19)	0.39
Biochemical or surrogate	228		0 (0)			36 (16)
Mixed	92		40 (44)			11 (12)
Year of publication:
2005	117		40 (34)	0.7		16 (14)	0.91
2006	102		33 (32)			20 (20)
2007	100		29 (29)			16 (16)
2008	94		31 (33)			13(14)
Trial design:
Parallel	335		116 (35)	0.002		55 (16)	0.83
Crossover	26		1 (4)			3(12)
Other	52		16 (31)			7 (13)
Blinding methods:
Double blind	206		65 (32)	0.92		26 (13)	0.26
Partially blinded	68		22 (32)			11 (16)
Open	132		43 (33)			27 (20)
Unspecified	7		3 (43)			1 (14)
No of participants:
<101	115		4 (3)	<0.001		20 (17)	0.20
101-1000	175		43 (25)			31 (18)
≥1001	123		86 (70)			14 (11)
Study coordinating region:
North America	170		31 (18)	0.02		31 (18)	0.49
Europe	206		30 (15)			30 (15)
Rest of world	37		4 (11)			4 (11)
No of centres:
1	145		15 (10)	<0.001		20 (14)	0.45
2-10	66		16 (24)			11 (17)
>11	202		102 (51)			34 (17)
Funding:
For profit	185		64 (35)	0.001		27 (15)	0.61
Not for profit	121		41 (34)			22 (18)
Mixed	54		23 (43)			10 (19)
Not reported	53		5 (9)			6 (11)
Disease or condition under investigation:
Acute coronary syndromes	53		23 (43)	0.005		4 (8)	<0.001
Other coronary artery disease	110		20 (18)			7 (6)
Heart failure	55		14 (25)			28 (51)
Arrhythmia	15		5 (33)			3 (20)
Other cardiac disease	20		8 (40)			0 (0)
Other vascular conditions	75		31 (41)			16 (21)
Healthy or at risk of cardiovascular disease	85		32 (38)			7 (8)
Type of intervention:
Drug	250		86 (34)	0.06		34 (14)	<0.001
Drug v placebo	141		45 (32)			22 (16)
Drug v active drug	81		33 (41)			4 (5)
Drug v other	28		8 (29)			8 (29)
Coronary intervention or devices	62		14 (23)			4 (6)
Non-coronary intervention	34		12 (35)			6 (18)
Lifestyle or nutrition	31		5 (16)			10 (32)
Other interventions	36		16 (44)			11 (31)

*For comparison against trials where primary outcomes were not dominated by patient important outcomes.

†For comparison against studies that had not included patient reported outcomes in their report.

Table 1 shows the association between trial characteristics, the type of reported outcomes, and the presence or absence of patient reported outcomes. Contrasting the significant association between the type of primary outcome and several features of the trials, there was no evidence that the trials reporting patient reported outcomes were significantly different from those not reporting patient reported outcomes, based on the type of primary end point selected, year of publication, trial design, blinding methods, number of participants, number of centres, study coordinating region, or the type of funding. Strong evidence was found that studies reporting patient reported outcomes differed by the condition under investigation and the type of intervention. In particular, studies that included patients with heart failure, arrhythmia (for example, atrial fibrillation) or “other vascular conditions” (for example, pulmonary arterial hypertension, peripheral arterial disease, stroke) were more likely to use patient reported outcomes than those that included mainly people with coronary artery disease or when interventions were aimed at primary prevention of cardiovascular conditions. Studies that assessed the effect of lifestyle interventions, nutrition, and those with “other interventions,” such as reminder systems or screening tests, were also more likely to use patient reported outcomes, whereas studies of coronary intervention or other procedures were less likely to do so.

Quality and types of reported instruments for measuring patient reported outcomes

Of the 65 trials reporting patient reported outcomes, only seven (11%) used ad hoc measures (that is, no previously validated instrument used or psychometric properties reported or referenced) or proxies for assessment of patients’ health status, and in another three (5%) published information was insufficient to ascertain the properties of the instruments used. The other 55 trials (85%) used at least one validated instrument (seven trials reported three different instruments and 16 reported two instruments). The most commonly used disease specific instruments were the Minnesota living with heart failure questionnaire and the Seattle angina questionnaire, which were used in 18 and five different studies, respectively. The most commonly used generic instruments were the short-form 36 health survey and the EuroQol 5-D, which were used in nine and five studies, respectively.

Prevalence of reporting on patient reported outcomes over time

The replicated and updated search from a previous study showed that the proportion of cardiovascular trials that also mentioned quality of life has increased more than threefold since 1997 (fig 3). By 2009, about 14% of trials mentioned quality of life. However, these crude figures should be interpreted with caution as they were only based on a simple bibliographic search, which may not reflect actual use of patient reported outcomes.

Fig 3 Prevalence of cardiovascular trials that also mentioned “quality of life”

Relevance of reporting on patient reported outcomes

Table 2 shows the assessed level of relevance of patient reported outcomes overall and by the status of reporting of such outcomes to individual trials. Patient reported outcomes were found to be crucial for informed clinical decision making in 93 trials (23%). These were mostly (n=57) trials that had set out chiefly to improve patients’ wellbeing or quality of life. For example, the Canadian Cardiac Randomized Evaluation of Antidepressant and Psychotherapy Efficacy (CREATE) trial investigated the effect of citalopram and interpersonal psychotherapy on depression in people with coronary artery disease (and had used two validated scoring tools for the assessment of depression).21 A few other studies in the category of “patient reported outcome crucial” were trials that aimed to change clinical practice, but their outcomes were dominated by ones of questionable importance to the patients. At the same time these trials failed to measure adequately the effect of harms or had included patients who had chronic symptoms for whom improvement in quality of life would have been highly desirable. For example, the Plasma Brain natriuretic Peptide-Guided Therapy to Improve Outcome in Heart Failure: The STARS-BNP Multicenter Study trial assessed the effect of plasma brain natriuretic peptide guided therapy on death and admission to hospital in patients with chronic heart failure.22 However, deaths due to heart failure accounted for only about 15% of the composite outcome, with the remainder being admissions to hospital. Overall, 37 of the 93 trials in which patient reported outcomes would have been crucial reported such outcomes.

Table 2

 Comparison of reporting and relevance of patient reported outcomes (PROs) to clinical decision making. Values are numbers (percentages) unless stated otherwise

Level of relevance to clinical decision making	All trials	Trials not reporting PROs	Trials reporting PROs	P value for trend*
Crucial	93 (23)	56 (60)	37 (40)	<0.001
Important	81 (20)	66 (81)	15(19)
Potentially relevant	59 (14)	56 (95)	3 (5)
Irrelevant	93 (23)	91 (98)	2 (2)
Uncertain	87 (21)	79 (91)	8 (9)

*Trend test excludes trials with uncertain level of relevance.

Patient reported outcomes were judged to be important in 81 of the trials (20%), of which 15 reported such outcomes. The majority of these were superiority trials that had used mostly patient important outcomes but had not adequately measured harms or had included a patient population with chronic symptoms. Studies that did not measure harm adequately were mostly trials of interventions known to be associated with a substantial bleeding risk. For example, the value of a decrease in blood haemoglobin levels, which was commonly used as a criterion for major bleeding, is not obvious from the patient’s perspective. The combination of such harms with potential benefits of the intervention has been used in an attempt to estimate the net clinical benefit.23 However, such combined effects without any valuation by the patients have made the interpretation of the results even more challenging.24

In another 59 trials (14%) patient reported outcomes were judged to be potentially relevant. These were trials that aimed to inform clinical decision making beyond the relief of symptoms or improvements in quality of life, had mainly used patient important outcomes as the primary end point, and had measured harm adequately (if expected or possible). In this setting, patient reported outcomes were thought to be only potentially relevant, mainly for estimation of the preference based outcomes of the intervention and the calculation of quality adjusted life years as part of economic evaluation.

Patient reported outcomes were judged to be irrelevant in 93 trials (23%). These were explanatory trials that did not intend to inform clinical decision making directly. For example, one trial investigated the effect of angiotensin receptor blockers on blood kinin levels to explore possible mechanisms of action of this type of drugs.25 Only two trials in this category used patient reported outcomes, giving little evidence for the possibility of misuse of patient reported outcomes in circumstances where such information would be of limited value.

The relevance of patient reported outcomes was ranked as uncertain in 87 trials (21%). These trials were primarily limited by the lack of patient important outcomes to address a clinically relevant question. In such trials, judgment on the incremental value of patient reported outcomes would have been highly hypothetical. For example, in a trial that aimed to assess the efficacy of reperfusion therapy 12 hours after the onset of symptoms in patients with acute myocardial infarction but used infarct size as its primary end point,26 assessment of the level of relevance of patient reported outcomes would have been secondary because to make an informed decision on patient care such a trial would in the first instance be expected to use patient important outcomes before the incremental value of patient reported outcomes could be evaluated.

Discussion

Patient reported outcomes were reported in 16% of contemporary cardiovascular trials published in major cardiology and general medical journals during 2005-8. Reported patient reported outcomes were mostly validated instruments and mostly used in settings where such information would have been highly relevant to clinical decision making. Although patient reported outcomes were judged to be of little value to a large proportion of, mostly explanatory, cardiovascular trials, still more than two thirds of trials in which patient reported outcomes were judged to be important or crucial for clinical decision making failed to report such outcomes. In addition, composite outcomes in many trials were often dominated by outcomes that were not immediately important to patients.

Three major areas were identified in this study where the measurement of patient reported outcomes was underused but could have made a valuable contribution to interpretation of the study findings. Firstly, many trials that were primarily concerned with relief of symptoms and improvement in quality of life failed to use patient reported outcomes, and even when they did these were rarely the primary outcome (contributing to the lack of impact of such findings on informing clinical practice27).

Secondly, shortcomings in reporting of patient reported outcomes were also observed in trials that assessed benefits of an intervention in light of considerable harms caused by the same intervention. While it could be argued that most interventions are likely to have multiple effects, desired and undesired, expected and unexpected, it would be inappropriate to demand measurement of patient reported outcomes in all trials. Standard safety monitoring of trials (despite all its limitations) and crude reporting of events might be sufficient for clinical decision making in circumstances where the spectrum of harms is reasonably well known and clinically measurable.28 However, issues arise when potential harms are substantial and their objective evaluation is not directly relevant to patients. One common area identified in this study was the assessment of the risk of bleeding in trials aiming to reduce the risk of cardiovascular events. The investigators used a diversity of methods for measurement and classification of bleeding, with some of these being purely based on objective laboratory measures with little relevance to patients. This bears the risk of invalidating comparisons between the different outcome components within the same study, as well as between studies reporting similar outcomes.29 A few investigators have attempted to estimate the net clinical effect by combining harms and benefits into a single composite outcome. However, such an approach would assume that detection methods for different types of outcomes are equally sensitive and that different types of events have a similar clinical weight. Since such conditions cannot easily be met in practice,30 and in the absence of valid and meaningful scoring systems for measurement of such harms, patient reported outcomes could offer an alternative method for assessment of the impact of such effects on patients. Preference based instruments for patient reported outcomes (that is, instruments that provide relative values for different health states23), would allow an estimate of the weighted average of benefits and harms from the patient’s perspective and could provide a common denominator for different types of outcomes.

Thirdly, the use of patient reported outcomes could be improved in the group of trials using composite end points that incorporate outcomes of different importance to the patients. This can be particularly problematic when outcomes that are less important to the patients dominate the results or when treatment effects are discordant and qualitatively different across the individual components. The current study shows that in over half of the studies with mixed primary outcomes, the findings were dominated by the intermediate outcomes of questionable relevance to patients. A generic preference based instrument for patient reported outcomes would have the ability to evaluate the overall effect of such outcomes from the patient’s perspective.31 Serial measurements would also circumvent problems with time based analyses that commonly count the first occurrence of events only and ignore recurrences, even if these are of greater importance to patients.

Concerns have been raised that widespread recommendations from funding agencies and regulatory bodies for the use of patient reported outcomes in clinical trials may lead to inappropriate use of such outcomes. In this study only 5% of trials in which patient reported outcomes were ranked as irrelevant reported such outcomes, therefore dismissing any concerns for substantial misuse. However, this may be an underestimate because our protocol included high quality medical journals only and the trials may not be representative of all cardiovascular trials. In addition, the prevalence of reporting patient reported outcomes was based on the published information. Some investigators may have measured but not reported such outcomes.27

Strengths and limitations of the review

To the best of our knowledge, this is the first study that has devised a tool for the reliable and transparent assessment of the relevance of patient reported outcomes to cardiovascular trials. The tool was validated by experts, applied to a large set of cardiovascular trials with a high inter-rater reliability, and may guide researchers in the future at the design stage of trials. Although a certain degree of judgment was inevitable in categorising trials, discrepancies between independent assessors were small and easily resolved through discussion, with no material effect on the overall findings. The focus of the current study was the assessment of patient reported outcomes as study outcome measures. We acknowledge that patient reported outcomes can play an important part at other stages of clinical trials, such as during recruitment to identify potentially eligible participants or at baseline for categorisation of participants into different risk groups.32 We also acknowledge that patient reported outcomes will not be the panacea for all existing challenges of outcome selection in clinical trials. Lack of suitable instruments and the somewhat sophisticated requirements for analysis and interpretation of results33 may have contributed to their underuse. Inadequate blinding of study participants may also prevent investigators from using patient reported outcomes, as such outcomes are more prone to bias than more objective outcomes. However, we found no evidence that the type of blinding correlated with use of patient reported outcomes. Conduct of trials is time consuming and many of the assessed trials may have been initiated a few years before publication, when guidelines on use of patient reported outcomes in trials were unavailable. Other feasibility issues such as the need for larger sample sizes and longer duration of follow-up may also be difficult to overcome. However, the study size or number of participants was not associated with the use of patient reported outcomes in the current study. This study was not designed to investigate the underlying causes of underuse of patient reported outcomes. Qualitative studies may be useful to help understand the investigators’ approach towards outcome selection and to explain why selected outcomes are often inadequate.

Conclusions and policy implications

Despite the continued rise in reporting of patient reported outcomes in cardiovascular trials, such outcomes were still underused in many trials, even after taking account of their relevance to individual trials. The present study reiterates the need for selection of more patient focused outcomes and highlights some of the problems associated with composite outcomes and assessment of harms in cardiovascular trials. In such areas, more widespread use of patient reported outcomes may be a valuable complement to the traditional clinical outcomes. Involvement of patients and experts in outcomes at the design stage of clinical trials may help in the process of selection of outcomes that are relevant and meaningful to those who potentially will be affected by them.

Patient reported outcomes assess the effect of trial interventions from patients’ perspectives and are particularly useful when interventions aim to improve symptoms or functional status

Studies report a low prevalence of patient reported outcomes particularly in cardiovascular trials, perhaps because many cardiovascular interventions aim to affect mortality and major morbidity

Little information is available on the appropriateness of the use of patient reported outcomes in contemporary cardiovascular trials

The prevalence of reporting patient reported outcomes in cardiovascular trials published in high quality medical and cardiology journals has increased to 16%, with no evidence of misuse

About two thirds of trials in which patient reported outcomes were judged to be important or crucial for clinical decision making still failed to report such outcomes

This was largely due to inappropriate use of composite outcomes and inadequate measurement of harms