Prognostic effect size of cardiovascular biomarkers in datasets from observational studies versus randomised trials: meta-epidemiology study.

OBJECTIVE: To compare the reported effect sizes of cardiovascular biomarkers in datasets from observational studies with those in datasets from randomised controlled trials.
DESIGN: Review of meta-analyses. STUDY SELECTION: Meta-analyses of emerging cardiovascular biomarkers (not part of the Framingham risk score) that included datasets from at least one observational study and at least one randomised controlled trial were identified through Medline (last update, January 2011). DATA EXTRACTION: Study-specific risk ratios were extracted from all identified meta-analyses and synthesised with random effects for (a) all studies, and (b) separately for observational and for randomised controlled trial populations for comparison.
RESULTS: 31 eligible meta-analyses were identified. For seven major biomarkers (C reactive protein, non-HDL cholesterol, lipoprotein(a), post-load glucose, fibrinogen, B-type natriuretic peptide, and troponins), the prognostic effect was significantly stronger in datasets from observational studies than in datasets from randomised controlled trials. For five of the biomarkers the effect was less than half as strong in the randomised controlled trial datasets. Across all 31 meta-analyses, on average datasets from observational studies suggested larger prognostic effects than those from randomised controlled trials; from a random effects meta-analysis, the estimated average difference in the effect size was 24% (95% CI 7% to 40%) of the overall biomarker effect.
CONCLUSIONS: Cardiovascular biomarkers often have less promising results in the evidence derived from randomised controlled trials than from observational studies.

Introduction

A plethora of novel biomarkers are being examined as predictors of cardiovascular outcomes.1 2 However, concerns have been raised on the reported effect sizes and on their claims of improved prediction, as several biases may inflate the observed associations.3 4 The population samples used to test these prognostic associations come primarily from traditional observational epidemiological studies, including cohort and case-control studies. However, some evidence may be available also from datasets of participants of randomised controlled trials. Randomised controlled trials would still be analysed as observational datasets in this scenario, but would they show the same results as analyses from traditional epidemiological studies?

Epidemiological studies may differ from randomised controlled trials in several ways, including the risk of confounding, the extent of susceptibility to publication and selective reporting biases,5 and the characteristics of enrolled participants. Because of such differences, observational studies often give larger estimates for the size of treatment benefits of interventions.6 7 8 9 However, it is unknown whether differences exist between the two types of study design when biomarker effects, rather than treatment effects, are assessed (and, if so, in which direction). This is important, since the eventual clinical use of these biomarkers depends on whether the prognostic effects are large enough to make some meaningful impact on risk classification and decision making for treatment or preventive interventions.

We aimed here to assess a comprehensive sample of published meta-analyses of biomarkers in relation to cardiovascular outcomes and compare the reported effect sizes in datasets from observational studies and from randomised controlled trials.

Methods

Literature search and eligibility of articles

We assembled meta-analyses that examined any emerging biomarker, defined as any biological parameter10 other than those included in the Framingham risk score,11 in relation to cardiovascular disease, coronary heart disease, or cardiovascular mortality. Biomarkers were eligible regardless of whether they were derived from blood, urine, tissue, or imaging. We excluded meta-analyses of single common genetic variants, since these markers are known to have very limited prognostic ability when examined in isolation,12 but multi-gene scores were eligible.

We used three different approaches to collect a comprehensive sample of biomarker meta-analyses indexed in Medline (no year restriction, last update 15 January 2011). First, we used the algorithm “(Biological marker[MeSH terms]) AND (cardiovascular OR coronary [Title/Abstract])” limited to meta-analyses, English language, and human studies. Second, we performed targeted Medline searches for meta-analyses of 71 additional specific emerging biomarkers included in recent comprehensive reviews10 13 14 with the same algorithm but using each biomarker’s name instead of the generic “Biological marker” that may not index some otherwise eligible biomarkers (see web extra table A on bmj.com for full list of biomarkers searched). We first perused the title and abstract of each of these citations, and potentially eligible articles were then retrieved for perusal in full text. Finally, we identified all meta-analyses of individual participant data published by major consortia operating in the specialty (Emerging Risk Factor Collaboration, Fibrinogen Studies Collaboration, Ankle Brachial Index Collaboration).

Articles were eligible if they included at least one meta-analysis examining the association between an eligible biomarker with an eligible outcome and containing data from at least one dataset from an observational study and one from a randomised controlled trial, which was subsequently analysed as an observational study. We included studies regardless of the baseline characteristics (clinical setting) of the examined populations. If an article presented separate meta-analyses on more than one eligible biomarker or outcome or on participants with different clinical settings, these meta-analyses were kept separate. Finally, meta-analyses were eligible regardless of whether the included studies used adjustment for some covariates or score (such as the Framingham risk score) or tested for association in unadjusted analyses. When we identified more than one meta-analysis examining the same biomarker and same outcome on the same clinical setting, we kept only the most recent one with eligible data. We accepted meta-analyses regardless of whether they were meta-analyses of the published literature or of individual participant data.

Data extraction

Data extraction was performed independently by two investigators, and discrepancies were solved by discussion. From each eligible meta-analysis, we recorded the first author, journal, year of publication, number of studies in the meta-analysis, the biomarker examined, risk factors or score used for covariate adjustment, and outcome examined. We extracted the study-specific estimates of relative risk (risk ratio, odds ratio, hazard ratio, or incident risk ratio) for each biomarker and outcome. If this information was not given in sufficient numerical detail in the meta-analysis, we sought to complement it by extracting information from the meta-analysis forest plot using software for image digitisation (Engauge Digitizer 4.1). Whenever data were provided with different adjustments in each study, we preferred estimates that adjusted for the Framingham risk score or the model with Framingham risk score variables; if none of these options was available, we preferred the model with the larger number of adjusting factors.

We also recorded for each study that was included in the meta-analysis whether the data came from a randomised controlled trial or observational study; if the latter, we also specified whether it was a prospective or retrospective study. We verified the original study design by checking the original publication of each study cited in each meta-analysis. We also extracted information on whether both treatment arms of each randomised controlled trial population were included in the meta-analysis. For randomised controlled trials, which included all treatments arms, we further extracted information on whether the trial has previously reported a significant difference (P<0.05) between treatment arms for cardiovascular disease outcomes. We also categorised biomarkers according to whether they were recommended (class I or II recommendation) by leading guideline authorities15 16 17 18 19 20 21 22 23 24 for use in the clinical setting where the meta-analysis pertained.

To perform sensitivity analyses, for all eligible meta-analyses of individual participant data, we conducted Medline searches to identify the most recent meta-analysis of published literature for the same biomarker, outcome, and setting. We then extracted the relevant data on relative risk estimates for each newly identified meta-analysis as outlined above.

Statistical analysis

For each meta-analysis, we synthesised the reported relative risks of each dataset with random effects for all datasets and separately for datasets from observational studies and those from randomised controlled trials. Random effects calculations were implemented with the inverse variance approach, considering the sum of the within-study plus between-study variance.25 All summary relative risks were coined so as to be ≥1.00 (if the summary relative risk in the original meta-analysis was <1.00, all the datasets’ relative risks were inversed (for example, 0.50 became 2.00)). Prediction intervals were calculated for the summary relative risks for datasets from observational studies and from randomised controlled trials.26 We also calculated the relative relative risk within each meta-analysis as the ratio of the summary relative risk in datasets from observational studies to the summary relative risk in datasets from randomised controlled trials. A relative relative risk >1 means that datasets from observational studies show a stronger effect for the biomarker than datasets from randomised controlled trials.

The relative relative risk is difficult to compare across different meta-analyses, because each biomarker risk may have been expressed for a different contrast (such as comparison of extreme tertiles or quintiles, or per one standard deviation of a continuous measurement, etc) and different relative risk metric. To standardise the relative relative risks across meta-analyses, we also calculated the ratio of the log of the relative relative risks (logRRR) to the summary log of the relative risk (logRR) of each meta-analysis. As an example, the lnRR and lnRRR would be 10 times higher if the results were expressed as per 100 mg/dl of biomarker instead of per 10 mg/dl, whereas their ratio (called the design difference) remains the same. The design difference represents the difference between datasets from observational studies and from randomised controlled trials as a proportion of the summary effect of each meta-analysis. Thus, if the logRRR is 0.2 and the summary logRR is 0.5, the design difference is 0.2/0.5=40%—that is, the difference in the effect between datasets from observational studies and from randomised controlled trials has 40% of the size of the overall effect of the biomarker that is estimated when all studies are considered. We combined design differences across meta-analyses to obtain a summary design difference and its variance according to random and fixed effects calculations and calculated the 95% prediction interval for the summary design difference.25 26 27

We performed analyses calculating the design difference according to type of observational design (prospective v retrospective); type of meta-analysis (published literature v individual participant data); type of randomised controlled trial (those with significant difference between treatment arms and the biomarker analysed in the combined population of both treatment arms v those with no significant treatment effect or included only placebo arms in the biomarker analysis); statistical significance of the biomarker (those with P<0.05 for the summary relative risk including all datasets v those not statistically significant); and whether the biomarker was recommended for clinical use.

In sensitivity analyses, we also calculated the design difference in (a) meta-analyses of individual participant data of biomarkers for which we could identify a published meta-analysis of published literature for the same clinical setting and outcome, (b) meta-analyses of individual participant data limited only to studies that were also included in the respective matched meta-analyses of published literature, and (c) the matched meta-analyses of published literature.

Heterogeneity was quantified with Cochran’s Q test and the I2 metric,25 28 and 95% confidence intervals for I2 were calculated.29 I2 has a scale of 0–100%, and values >75% suggest very large between-study heterogeneity. Analyses were performed in STATA 10.1 (STATA Corp., College Station, TX). P values are two tailed.

Results

Eligible data

Overall, we searched 295 articles (see web extra fig A on bmj.com), and 21 articles corresponding to 31 meta-analyses were deemed eligible.30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 This is recent literature, and 11 articles appeared in 2009–10 alone. The eligible meta-analyses pertained to a wide range of biomarkers used in different clinical settings, with a predominance of general population (primary prevention) settings (table 1). All meta-analyses examined coronary heart disease as the outcome of interest, except for five, which examined cardiovascular disease and cardiovascular mortality.34 37 46 47 48 Sixteen of the 31 meta-analyses (52%) corresponded to situations where guidelines already recommended the clinical use of the respective biomarker in that particular setting. Seven meta-analyses were of individual participant data, and 24 analysed published literature. Each meta-analysis included a median (interquartile range) of 11 (6–20) datasets from observational studies and 2 (1–3) from randomised controlled trials. The total number of events (number of participants with the outcome of interest) ranged from 4 to 26 459 (median 3867 (interquartile range 1415–7316), data not available for two meta-analyses). The total number of events was larger in datasets from observational studies than in those from randomised controlled trials in 21 (72%) of the 29 meta-analyses with information on number of events. In 48% (14/29) of the meta-analyses the average number of events per dataset was larger for observational studies than for randomised controlled trials.

Table 1

 Characteristics of the 31 eligible meta-analyses that examined biomarkers for cardiovascular risk and included data from at least one observational study (OS) and one randomised controlled trial (RCT)

Analysis ID, reference, and biomarker examined	Clinical setting	Recommended for clinical use in guidelines	Type of meta-analysis	No of studies			Total No of events		RCT names
				OS	RCT		OS	RCT
139, selenium	No history of CVD	No	Published literature	11	2		1934	302	PHS, NITL
237, C reactive protein	Vascular surgery patients†	Yes18 20 23	Published literature	1	1		1	3	AAART
334, serum albumin	Haemodialysis patients‡	No	Published literature	7	1		NA	NA	HEMO Study
433, C reactive protein	No history of CVD	Yes15-17 23	Individual participant data	26	5		9171	1170	AFTCAPS, WHS, MRFIT, PROSPER, WOSCOPS
535, triglycerides	No history of CVD	Yes15-17	Individual participant data	61	7		10 341	2444	AFTCAPS, WHS, MRFIT, PROSPER, WOSCOPS, ALLHAT, LEADER
635, non-HDL cholesterol	No history of CVD	Yes17	Individual participant data	61	7		10341	2444	AFTCAPS, WHS, MRFIT, PROSPER, WOSCOPS, ALLHAT, LEADER
736, Lp(a) lipoprotein	No history of CVD	Yes15 22	Individual participant data	25	5		8285	1036	AFTCAPS, WHS, MRFIT, WOSCOPS,* PHS
832, lipoprotein associated phospholipase A2 activity	General	Yes15	Individual participant data	10	2		1798	2165	PROSPER, HPS
932, lipoprotein associated phospholipase A2 mass	General	Yes15	Individual participant data	9	3		1821	2540	PROSPER, HPS, WOSCOPS
1045, serum amyloid A	No history of CVD	No	Published literature	1	1		506	85	WHS
1145, serum amyloid A	Pre-existing CVD	No	Published literature	1	1		75	391	CARE trial
1245, serum albumin	No history of CVD	No	Published literature	6	1		3320	450	MRFIT†
1345, leucocyte count	No history of CVD	No	Published literature	12	2		5402	562	WOSCOPS, HHS
1445, leucocyte count	Pre-existing CVD	No	Published literature	3	2		222	1043	CDS,* PAR
1544, Chlamydia pneumoniae IgG titre	General	No	Published literature	12	3		2639	530	PHS, WHS, HHS
1643, homocysteine	General	Yes16 17	Published literature	27	2		NA	NA	PHS,‡ MRFIT†
1742, D-dimer	No history of CVD	No	Published literature	4	1		998	296	PHS
1810, apolipoprotein B	General	No	Published literature	16	3		5442	891	AFTCAPS, PHS, WHS
1940, apolipoprotein A I	General	No	Published literature	18	3		5429	891	AFTCAPS, PHS, WHS
2040, apolipoprotein B:A I ratio	General	Yes17	Published literature	5	2		3085	645	AFTCAPS, WHS
2138, fasting insulin	General	No	Published literature	13	1		26 251	208	MRFIT
2231, apolipoprotein(a) isoforms	General	No	Published literature	29	1		7187	195	MRFIT
2330, post-load glucose	No history of diabetes	Yes21	Published literature	14	1		11 851	801	PHS
2441, fibrinogen	General	No	Individual participant data	29	2		5319	1799	Throm Prev, WOSCOPS
2550, tissue plasminogen activator antigen	General	No	Published literature	6	1		1438	231	PHS
2646, night time ambulatory blood pressure	High blood pressure	Yes24	Published literature	6	2		922	337	Syst-Eur trial,* ASCOT BPLA
2746, daytime ambulatory blood pressure	High blood pressure	Yes24	Published literature	6	2		922	337	Syst-Eur trial,* ASCOT BPLA
2847, B-type natriuretic peptide	General population or stable CVD	Yes20	Published literature	30	10		4005	6620	Steno-2, LIFE, AASK, FRISC II, PROVE IT TIMI-22, PROGRESS, LIPID, PEACE, HOPE, HPS
2948, coronary artery calcium	General population	Yes15	Published literature	11	2		1729	174	INSIGHT, St Francis Heart Study*
3049, troponin T/I	Acute coronary syndrome patients (non-ST elevation)	Yes18	Published literature	11	5		132	140	GUSTO-IIa, TIMI-IIIb, TRIM, TIMI-11a, CAPTURE*
3149, troponin T/I	Acute coronary syndrome patients (ST elevation)	Yes18	Published literature	1	2		8	1322	GUSTO-IIa, GUSTO-III

CVD=cardiovascular disease, HDL=high density lipoprotein, NA=data not available.

Names of trials: WOSCOPS=West of Scotland Coronary Prevention Study, PHS=Physicians Health Study, NITL=Nutrition Intervention Trial in Linxian, AAART=Abdominal aortic aneurysm repair trial, HEMO=Hemodialysis study, WHS=Women’s Health Study, CARE trial=Cholesterol and Recurrent Events trial, MRFIT=Multiple Risk Factor Intervention Trial, CDS=Coronary Drug Study, PAR=Persantin-Aspirin Reinfarction Study, HHS=Helsinki Heart Study, AFTCAPS=Air Force/Texas Coronary Atherosclerosis Prevention Study, Throm Prev=Thrombosis Prevention, PROSPER=Prospective Study of Pravastatin in the Elderly at Risk, ALLHAT=Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial, LEADER=Lower Extremity Arterial Disease Event Reduction Trial, HPS=Heart Protection Study, LIFE=Losartan Intervention For Endpoint, AASK=African American Study of Kidney Disease and Hypertension, FRISC II=Fragmin and fast Revascularisation during InStability in Coronary artery disease, LIPID=Long-term Intervention with Pravastatin in Ischemic Disease, PROGRESS=Perindopril Protection Against Recurrent Stroke Study, PEACE=Prevention of Events with Angiotensin Converting Enzyme Inhibition, HOPE=Heart Outcomes Prevention Evaluation, INSIGHT=International Nifedipine Study Intervention as Goal for Hypertension Therapy, GUSTO=Global Utilization of Streptokinase and Tissue Plasminogen Activator for Occluded Coronary Arteries, TIMI=Thrombolysis in Myocardial Ischemia Trial, TRIM=Thrombin Inhibition in Myocardial ischemia, CAPTURE=c7E3 Fab Antiplatelet Therapy in Unstable Refractory Angina, Syst-Eur trial=Systolic Hypertension in Europe, ASCOT-BPLA=Anglo-Scandinavian cardiac outcomes trial-blood pressure lowering arm.

*Only placebo arm analysed.

†MRFIT analysed as 2 different cohorts.

‡PHS analysed as 4 different cohorts.

Biomarker effects

By random effects calculations, 27 of the 31 meta-analyses showed significant (P<0.05) associations of the examined biomarker with the outcome of interest. The summary relative risks were adjusted for different variables in each meta-analysis, ranging from age and sex to multivariable models with a range of cardiovascular disease risk factors. None reported adjustment for the Framingham risk score. Estimates of between-study heterogeneity were high for many meta-analyses, and often it remained high even within the subgroups of datasets from observational studies and those from randomised controlled trials (table 2 and web extra tables B and C).

Table 2

 Biomarker effects in 31 meta-analyses that examined biomarkers for cardiovascular risk and included data from at least one observational study and one randomised controlled trial. Values are summary relative risk (RR) estimates by random effects, and corresponding heterogeneity (I2) values, for each meta-analysis and separately for datasets from the observational studies and from randomised controlled trials

Analysis ID and biomarker examined	Contrast of comparison	Meta-analysis			Observational studies			Randomised controlled trials
		RR (95% CI)	I2 (%)		RR (95% CI)	I2 (%)		RR (95% CI)	I2 (%)
1, selenium	Lowest v highest concentration*	1.19 (1.00 to 1.41)	18		1.18 (0.99 to 1.41)	15		1.10 (0.49 to 2.44)	60
2, C reactive protein	≤3 mg/L v >3 mg/L	2.47 (0.34 to 18.08)	0		8.11 (0.31 to 211.8)	NA		1.22 (0.10 to 15.06)	NA
3, serum albumin	Per unit decrease	1.15 (1.04 to 1.27)	65		1.14 (1.03 to 1.27)	69		1.37 (0.93 to 2.04)	NA
4, C reactive protein	Per 1 SD increase	1.37 (1.27 to 1.48)	26		1.44 (1.32 to 1.56)	19		1.17 (1.03 to 1.34)	0
5, triglycerides	Per 1 SD decrease	1.01 (0.95 to 1.06)	34		1.01 (0.94 to 1.08)	37		1.03 (0.95 to 1.11)	1
6, non-HDL cholesterol	Per 1 SD increase	1.49 (1.39 to 1.60)	73		1.54 (1.42 to 1.67)	73		1.27 (1.14 to 1.42)	32
7, Lp(a) lipoprotein	Per 1 SD increase	1.13 (1.09 to 1.18)	49		1.15 (1.10 to 1.21)	50		1.03 (0.96 to 1.11)	0
8, lipoprotein associated phospholipase A2 activity	Per 1 SD increase	1.09 (1.04 to 1.15)	21		1.07 (0.99 to 1.14)	13		1.13 (1.08 to 1.19)	0
9, lipoprotein associated phospholipase A2 mass	Per 1 SD increase	1.11 (1.06 to 1.16)	28		1.09 (1.02 to 1.17)	31		1.13 (1.07 to 1.19)	33
10, serum amyloid A	Top v bottom third	1.58 (0.99 to 2.51)	0		1.65 (1.00 to 2.71)	NA		1.23 (0.35 to 4.28)	NA
11, serum amyloid A	Top v bottom third	1.50 (1.02 to 2.22)	0		1.32 (0.36 to 4.83)	NA		1.52 (1.01 to 2.29)	NA
12, serum albumin	Top v bottom third	1.55 (1.38 to 1.75)	0		1.54 (1.36 to 1.75)	1		1.69 (1.02 to 2.82)	19
13, leucocyte count	Top v bottom third	1.44 (1.31 to 1.58)	38		1.44 (1.30 to 1.56)	43		1.53 (0.91 to 2.60)	36
14, leucocyte count	Top v bottom third	1.53 (1.25 to 1.88)	42		1.59 (1.17 to 2.15)	27		1.55 (1.03 to 2.35)	72
15, Chlamydia pneumoniae IgG titre	Top v bottom third	1.15 (1.02 to 1.31)	0		1.15 (1.00 to 1.31)	0		1.22 (0.82 to 1.82)	16
16, homocysteine	Per unit increase	2.17 (1.84 to 2.56)	71		2.26 (1.88 to 2.73)	74		1.84 (1.28 to 2.64)	44
17, D-dimer	Top v bottom third	1.92 (1.36 to 2.69)	57		2.16 (1.51 to 3.08)	47		1.27 (0.80 to 2.01)	NA
18, apolipoprotein B	Top v bottom third	1.98 (1.65 to 2.38)	80		2.03 (1.64 to 2.51)	83		1.77 (1.36 to 2.30)	30
19, apolipoprotein A I	Bottom v top third	1.61 (1.42 to 1.82)	51		1.58 (1.37 to 1.82)	54		1.72 (1.35 to 2.19)	32
20, apolipoprotein B:A I ratio	Top v bottom third	1.84 (1.54 to 2.21)	75		1.76 (1.42 to 2.17)	79		2.17 (1.71 to 2.76)	0
21, fasting insulin	Top v bottom third	1.20 (0.97 to 1.49)	72		1.24 (0.98 to 1.57)	73		0.96 (0.67 to 1.36)	NA
22, apolipoprotein(a) isoforms	Smaller v larger isoforms	2.08 (1.67 to 2.59)	85		2.08 (1.66 to 2.62)	85		2.01 (1.20 to 3.36)	NA
23, post-load glucose	Per unit increase	1.06 (1.03 to 1.08)	67		1.06 (1.04 to 1.09)	63		1.02 (0.99 to 1.05)	NA
24, fibrinogen	Per unit increase	2.64 (2.27 to 3.07)	76		2.74 (2.40 to 3.13)	58		1.60 (1.43 to 1.78)	0
25, tissue plasminogen activator antigen	Top v bottom third	1.50 (1.07 to 2.11)	72		1.56 (1.03 to 2.36)	77		1.40 (0.92 to 2.12)	NA
26, night time ambulatory blood pressure	Per 10 mm Hg increase	1.20 (1.14 to 1.26)	25		1.17 (1.10 to 1.25)	22		1.27 (1.18 to 1.38)	0
27, daytime ambulatory blood pressure	Per 10 mm Hg increase	1.14 (1.04 to 1.25)	63		1.15 (1.05 to 1.26)	47		1.11 (0.81 to 1.51)	89
28, B-type natriuretic peptide	Top v bottom third	2.82 (2.40 to 3.32)	83		3.05 (2.48 to 3.77)	81		2.15 (1.78 to 2.61)	61
29, coronary artery calcium	Presence v absence	6.28 (4.63 to 8.53)	59		6.12 (4.36 to 8.59)	64		7.51 (3.92 to 14.38)	0
30, troponin T/I	Positive v negative	6.28 (3.80 to 10.36)	67		10.05 (4.89 to 20.66)	65		3.30 (2.05 to 5.31)	42
31, troponin T/I	Positive v negative	2.86 (2.36 to 3.48)	0		3.38 (0.77 to 14.90)	NA		2.85 (2.35 to 3.47)	0

HDL=high density lipoprotein, NA=not applicable (single study).

*Definition of highest and lowest category not given.

See web extra table D on bmj.com for related prediction intervals26 of relative risks for observational studies and randomised controlled trials.

Comparison of effects in datasets from observational studies versus those from randomised controlled trials

The summary estimates in datasets from observational studies and from randomised controlled trials were in the same direction for all comparisons apart from fasting insulin (table 2). Related prediction intervals for the summary relative risks are shown in web extra table D. For 15 meta-analyses the summary relative risk was nominally significant in both types of datasets, for eight it was nominally significant only in datasets from observational studies, and in three it was nominally significant only in datasets from randomised controlled trials.

In 19 of the 31 meta-analyses the relative relative risk estimate was >1 (suggesting a stronger effect in datasets from observational studies than in those from randomised controlled trials, design difference >0). For seven meta-analyses, the effect was significantly stronger in datasets from observational studies than those from randomised controlled trials: this included meta-analyses which examined C reactive protein, non-HDL cholesterol, lipoprotein(a), post-load glucose, fibrinogen, B-type natriuretic peptide, and troponins. For five of those biomarkers, the effect was less than half as strong in the randomised controlled trial datasets; these differences would probably be considered not only statistically significant but also clinically important. There was no meta-analysis with a significantly stronger effect in datasets from randomised controlled trials than in datasets from observational studies.

The random effects summary design difference across all 31 meta-analyses was 24% (95% confidence interval 7% to 40%, P=0.005), showing that the difference in the effect size between datasets from observational studies versus those from randomised controlled trials amounted to 24% of the overall effect of the biomarker (figure). The associated 95% prediction interval of the design difference was −29% to 76%. This means that, typically, observational studies are expected to show larger or even much larger effects than randomised controlled trials, but exceptions can exist where larger effects are seen in the randomised controlled trials. There was modest heterogeneity between meta-analyses in the design difference (estimated I2=39%), and fixed effects estimates were slightly larger (33% (23% to 43%)) (table 3). A funnel plot of the design differences and their standard errors is shown in web extra figure B and reveals a non-significant trend (Egger’s P value=0.09) for larger design differences when there is more evidence.

Results from 31 eligible meta-analyses that examined biomarkers for cardiovascular risk and included data from at least one observational study and one randomised controlled trial: comparison of effect sizes in datasets from observational studies v those from randomised controlled trials populations

Table 3

 Summary design difference* estimates of the 31 meta-analyses that examined biomarkers for cardiovascular risk and included data from at least one observational study (OS) and one randomised controlled trial (RCT)

Comparison	No of meta‑analyses	Design difference (95% CI)		I2 (95 % CI)	P value (Q)
		Random effects	Fixed effects
All data	31	0.24 (0.07 to 0.40)	0.33 (0.23 to 0.43)	39 (6 to 60)	0.015
Design contrast:
RCT v prospective OS	31	0.23 (0.07 to 0.40)	0.33 (0.23 to 0.43)	37 (2 to 59)	0.023
RCT v retrospective OS	9	0.34 (−0.02 to 0.70)	0.34 (−0.02 to 0.70)	0 (0 to 65)	0.457
Type of meta-analysis:
Of individual participant data	7	0.42 (0.14 to 0.70)	0.51 (0.37 to 0.65)	50 (0 to 79)	0.064
Of published literature	24	0.15 (−0.01 to 0.31)	0.16 (0.02 to 0.30)	10 (0 to 44)	0.320
Type of RCT:
Effective intervention†	21	0.14 (−0.08 to 0.37)	0.34 (0.22 to 0.46)	52 (20 to 71)	0.003
Other	23	0.31 (0.07 to 0.54)	0.27 (0.12 to 0.42)	42 (4 to 64)	0.02
Significant biomarker effect:
Yes	27	0.22 (0.05 to 0.39)	0.33 (0.23 to 0.43)	45 (14 to 65)	0.006
No	4	1.19 (−0.48 to 2.86)	1.19 (−0.48 to 2.86)	0 (0 to 85)	0.901
Biomarker recommended for clinical use:
Yes	16	0.22 (−0.01 to 0.45)	0.26 (0.12 to 0.39)	55 (20 to 74)	0.005
No	15	0.43 (0.28 to 0.58)	0.43 (0.28 to 0.58)	0 (0 to 54)	0.502

*The difference in biomarker effects between datasets from observational studies and from randomised controlled trials as a proportion of the summary effect of each meta-analysis.

†Refers to trials showing significant difference between treatment arms for cardiovascular disease end points and populations of both arms were considered in analysis of the biomarker.

Of the five trials that contributed biomarker data to more than five different meta-analyses, the three trials that targeted high risk populations (with high levels of low density lipoprotein cholesterol in the West of Scotland Coronary Prevention Study, high cardiovascular disease risk in the Multiple Risk Factor Intervention Trial, and postmenopausal women in the Women’s Health Study) quite consistently found smaller effect size estimates than the meta-analysis summary (5/7 times, 7/8 times, and 7/9 times respectively), whereas this trend was not seen in the two trials that enrolled healthy populations (5/9 times in the Physicians Health Study, 4/7 in the Air Force/Texas Coronary Atherosclerosis Prevention Study).

Subgroup and sensitivity analyses

Based on random effects calculations, the design difference did not differ beyond chance when analyses were performed according to type of observational study, type of meta-analysis, type of randomised controlled trial, statistical significance of the biomarker, and whether the biomarker was recommended for clinical practice (table 3). There were trends for stronger design differences in meta-analyses of individual participant data versus those of published literature and in non-significant versus significant biomarkers; the contrast between meta-analyses of individual participant data and those of published literature was nominally statistically significant with fixed effects calculations (table 3).

In sensitivity analysis, we compared the overall design difference of four biomarkers (C reactive protein, Lp(a) lipoprotein, lipoprotein associated phospholipase A2 mass, and fibrinogen) examined in meta-analyses of individual participant data of our sample and the corresponding design difference of these four biomarkers in data from meta-analyses of published literature identified through additional literature searches. The random effects summary design difference was similar in the meta-analyses of individual participant data (53% (21% to 84%)), in data from the corresponding meta-analyses of published literature (69% (33% to 105%)), and in the meta-analyses of individual participant data limited only to studies included both in meta-analyses of individual participant data and in those of published literature (55% (22% to 89%)) (see web extra tables E and F for tabulated data).

Discussion

Principal findings

In this empirical evaluation of 31 meta-analyses examining the association of a wide range of cardiovascular biomarkers, effect sizes were on average stronger in datasets derived from observational studies than in datasets from randomised controlled trials. The average difference in effect size amounted to about a quarter or a third of the estimated overall effect of the biomarker based on all data. For seven biomarkers, six of which are recommended for wide clinical use by major guidelines,15 16 17 18 19 20 21 22 23 24 the prognostic effect sizes were significantly stronger in datasets from observational studies than in those from randomised controlled trials.

New cardiovascular biomarkers are continuously proposed.1 2 Several of them have received great attention in the medical literature, and multiple meta-analyses and individual data consortia thereof have reported consistent associations with cardiovascular disease, raising hopes for improved cardiovascular prediction over and above what traditional markers and scores such as the Framingham risk score achieve.2 33 51 52 53 Accurate estimates of the prognostic ability of these markers are important for their clinical translation, and deviant results with different study designs raise some concern.

Possible explanations

There are different possible interpretations for these discrepancies between observational studies and randomised controlled trials. Firstly, publication bias and selective reporting is well documented,53 54 55 56 and this applies also to prognostic analyses.55 57 Such biases may inflate the size of biomarker associations. Another evaluation of meta-analyses of biomarkers has shown that the largest studies almost always show smaller effect size estimates than the most highly cited, smaller studies of biomarkers.58 Epidemiological studies and their analyses may suffer differently from publication and other selective reporting biases than analyses of randomised controlled trial data.5 Large randomised controlled trials (those that are often also used for the assessment of biomarkers with major outcomes) are highly visible and unlikely to remain unpublished. Moreover, the analysis of biomarker associations is not a primary goal of these trials, and there may be less zeal in obtaining and reporting specific results for these markers in analyses of randomised controlled trials than in observational studies. On the other hand, in some observational studies, in contrast to randomised controlled trial studies, the biomarker analysis may be the primary aim—that is, the study may have been designed for that aim and therefore may suffer less from publication bias.

The estimated design difference tended to be higher, if anything, in meta-analyses of individual participant data than in meta-analyses of published literature. Groups that have undertaken meta-analyses of individual participant data, such as the Emerging Risk Factors Collaboration (ERFC), have performed a massive task bringing together multiple studies by harmonising and standardising their data, and problems of selective reporting might be partially amended in these analyses.59 If so, the gap between effect sizes in data from observational versus randomised controlled trial designs might be expected to close in meta-analyses of individual participant data. However, we found that the observed design difference was of similar magnitude in meta-analyses of individual participant data and in the corresponding meta-analyses of published literature for the same biomarker, outcome, and clinical setting.

One possible explanation is that in these meta-analyses of individual participant data each biomarker is studied in a selected subset of all potentially eligible datasets. For example, the meta-analyses by ERFC included data from 12–68 studies for each biomarker even though there are currently over 100 studies included in this collaboration.59 60 61 In fact, biomarkers examined in large consortia are among the most studied and “hottest” biomarkers in the medical literature—such as C reactive protein, fibrinogen, and lipid related markers. Selection forces might be affecting some of those particular biomarkers to a greater extent and may not be fully remedied even with meticulous standardisation of data contributed to a retrospective consortium such as ERFC.62

An alternative explanation may be that some randomised controlled trials may have more restrictive inclusion criteria than observational studies. Some observational studies enrol participants from the general population without any restrictions. This may lead to a more limited range of risk profiles across randomised controlled trial participants and less prominent prognostic effects for biomarkers. However, the typical randomised controlled trials included here are studies that also target either the general population or all patients without documented cardiovascular disease.

One may also speculate whether the design difference might be a result of higher measurement error of biomarkers or outcomes in randomised controlled trials than in observational studies. However, most of the biomarkers examined here, including almost all those that showed significant differences between the two study designs, are routinely measured with standardised, highly reproducible assays. As for outcome misclassification, this is also unlikely to be greater in randomised controlled trials than in observational studies, since randomised controlled trials (especially the generally high profile trials analysed here) are usually sufficiently meticulous for the ascertainment of such major outcomes.

Potential limitations of study

Some limitations should be acknowledged. Although we examined a large number of emerging biomarkers through comprehensive and complementary search strategies, there are still biomarkers that do not have available meta-analyses or have not been examined in any randomised controlled trial populations. It is unclear whether our results can be extended to any tested biomarker, including those that have only one or a few studies reported on them. However, the set of biomarkers that we examined includes many that have a strong presence in the literature, and most of them are either routinely used or in the process of entering clinical practice. Finally, we concentrated on the effect size of biomarkers and not on other important measures of predictive ability such as discrimination, calibration, and reclassification. However, these additional useful metrics are rarely reviewed in meta-analyses of biomarkers and are not reported by most studies.3

Conclusions

When the results of different types of studies disagree, it is useful to consider whether a biomarker would still be useful for clinical practice, given the different levels of estimated predictive performance. For example, C reactive protein and Lp(a) lipoprotein have been promoted for routine clinical practice. The European Atherosclerosis Society recently endorsed routine measurement of Lp(a) lipoprotein in individuals at moderate or high risk of cardiovascular disease and suggested that Lp(a) lipoprotein concentrations <50 mg/dL should be a treatment priority.15 22 However, in the data that we examined Lp(a) lipoprotein had markedly stronger prognostic effect size in datasets from observational studies than from randomised controlled trials—in the latter, the effect was practically null. Similarly, C reactive protein has been recommended by several guidelines for routine use,15 16 17 23 but again the prognostic effect was small in randomised controlled trials. If one considered only data from randomised controlled trials, probably neither Lp(a) lipoprotein nor C reactive protein would be considered good biomarkers, whereas data from observational studies suggest the opposite.

Given that biases are difficult to control for after publication of data, prospective solutions need to be considered. One option is to register available study populations, either from observational or from randomised controlled trial studies, for which the quality of the outcome and covariate data are acceptable and biospecimens are available, and then ensure that all these populations are assessed for each emerging biomarker of interest, not just those datasets that investigators are interested in or have reported on for that specific biomarker. Registration has been used for randomised trials to decrease biases in the evidence that determines the use of medical interventions.63 64 Biomarkers can sometimes be costly, especially if they are used for screening and primary prevention purposes in wide segments of the population. Prognostic estimates derived from all-inclusive consortia of registered studies may be more reliable for understanding the true potential of these biomarkers.

Several cardiovascular disease biomarkers are proposed, but there is uncertainty about their effects and their ability to improve prediction

It is unknown whether the observed effect sizes differ between datasets from observational studies versus those from randomised controlled trials

Across all 31 meta-analyses included in this study, datasets from observational studies suggested, on average, larger prognostic effects than those from randomised controlled trials

For seven major biomarkers the prognostic effect was significantly stronger in datasets from observational studies than in datasets from randomised controlled trials

Cardiovascular biomarkers often have less promising results in the evidence derived from randomised controlled trial populations than from observational studies