Bias in animal studies of estrogen effects on cardiovascular disease: A systematic review and meta-analysis.

BACKGROUND: Randomized controlled trials on menopausal hormone therapy in humans have not confirmed the benefit of estrogens on cardiovascular disease found in animal studies. Flawed methodology or publication bias in animal studies may explain the dicrepancy.
OBJECTIVES: The aim of this study was to investigate whether publication of the randomized controlled trials Heart and Estrogen/Progestin Replacement Study and Women's Health Initiative influenced study authors' assessment of research findings (confirmation bias) as well as to investigate publication bias and small-study effects in animal studies of estrogen effects on atherosclerosis.
METHODS: The data source for this study was PubMed from inception to 2018. We selected animal studies with cardiovascular outcomes comparing 17-β-estradiol, its natural metabolites, or conjugated equine estrogen with control. Qualitative data were extracted on authors' conclusions about estrogen effects on cardiovascular disease, as well as quantitative data for atherosclerosis outcomes. Fixed- and random-effects meta-analyses were used. Publication bias/small-study effects were assessed using funnel plots and Egger's regression. Trim-and-fill plots and extrapolation from Egger's regression were used to adjust for publication bias. The main outcomes and measures were the primary study authors' interpretation of their own results, and estrogen effects on cardiovascular disease in general before and after publication of the Women's health Initiative study (2003). The effects of estrogens on atherosclerosis were measured as standardized mean difference between intervention and control.
RESULTS: Of 1925 studies retrieved, 360 were eligible for analyses. Study-specific statements concluded that estrogens were protective against cardiovascular disease in 75% of studies before 2003 and 78% after, but the percentage of general statements about estrogens being cardioprotective changed from 70% to 40%. Meta-analyses showed less atherosclerosis in estrogen-treated animals. Extremely skewed funnel plots and P < .01 in Egger's regression suggested publication bias and/or exaggerated effects in small studies, which was more pronounced after 2002. There was substantial heterogeneity of effects (I 2 = 68%-86%) overall and in all subgroups except cynomolgus monkeys (I 2 = 9%), the only animal subgroup without clear bias. Adjusting for publication bias, overall estimates of estrogen effects on atherosclerosis were close to null effect.
CONCLUSIONS: We found substantial evidence of publication bias but not of confirmation bias. Publication bias and flawed small studies may partly explain why findings differ between animal studies and clinical trials on the effect of estrogens on cardiovascular disease.

Studies on animals show different effects of estrogens on cardiovascular disease than in humans.

We reviewed studies of estrogen‐treated animals with cardiovascular outcomes.

We found strong indications of publication bias in these animal studies.

This may partly explain the difference in estrogen effects between animal studies and clinical trials.

INTRODUCTION

Before 2002, menopausal hormonal therapy (MHT) was recommended for prevention of coronary heart disease, based on the results of observational studies, in particular the Nurses’ Health Study. , , The observational studies showed that women using MHT had reduced risk of coronary heart disease compared with women not using MHT, and they were largely corroborated by animal studies that confirmed the positive effects of estrogen on the vascular bed. , , ,

Controversy ensued in 1998 as the first randomized clinical trial (RCT) on MHT, the Heart and Estrogen/Progestin Replacement Study (HERS), failed to show that MHT protected against coronary heart disease. In 2002 and 2004, the Women’s Health Initiative (WHI) studies , confirmed the results from HERS. Moreover, the latter two were discontinued due to excess risk of cardiovascular disease in the MHT group. The WHI findings prompted a sharp decline in MHT use and left researchers struggling to explain why results from RCTs differed from observational studies and laboratory studies. Some researchers continued to argue that MHT protected against coronary heart disease, especially when it was started shortly after menopause, while others put more confidence in the RCTs and argued that MHT had no or very little effect on coronary heart disease. , , Additional publications from the WHI study and the Nurses’ Health Study have shed further light on the difference between RCTs and observational studies, although a unifying conclusion has not been reached. ,

One of the main arguments against the HERS and WHI results was that most laboratory studies on atherosclerosis and vascular disease showed beneficial results of estrogens. , Interventions showing promising results in animals have often proven ineffective or even harmful in subsequent clinical trials. , , Possible mechanisms include that inherent differences between species may impair generalizability to humans , and that many animal studies lack methodological rigor. , , Failure to publish negative results may further contribute to a flawed body of evidence. , , ,

In our study, we investigated two hypotheses regarding the diverging results in animal studies and RCTs on estrogens and cardiovascular disease. First, we hypothesized that confirmation bias may have led authors of animal studies to interpret findings on estrogen effects differently before and after the WHI study was published. We hence investigated if study conclusions were in alignment with the predominant scientific understanding at the time of publication. Second, we hypothesized the presence of publication bias, meaning that studies finding negative or neutral effects of estrogens remain unpublished.

METHODS

Literature search

We searched PubMed from inception through January 17, 2018, using relevant free‐text and MeSH terms for applicable interventions combined with terms for cardiovascular outcomes. We applied a previously published filter for retrieving animal studies. The search strategy is detailed in Appendix A in the Supporting Information. Records were deduplicated using EndNote X8.2 and the deduplication algorithm in the online literature screening tool Covidence.

Study selection

We included primary reports of in vivo animal studies providing data on the effect of estrogens on any cardiovascular outcome, restricting interventions to 17‐β‐estradiol and its natural metabolites or conjugated equine estrogen, administered by any route. Eligible studies must either have a time‐series design or an intervention and a control group. The two groups should differ only by the presence or absence of estrogen administration. We applied no restrictions regarding animal species, sex, or age.

Two authors (CFB and PR) screened all abstracts retrieved in the search independently and in duplicate using Covidence and scrutinized records deemed relevant in the primary screening for eligibility in full‐text. Disagreements were solved by discussion. For full‐text reports not available through our institution, we retrieved paper copies through the medical library of University of Oslo and its collaborating institutions. We used Google Translate for assessing reports in languages that neither of the screeners knew (ie, Chinese and Japanese).

Outcomes and data extraction

Using a standardized data extraction sheet, two authors (CFB and PR) independently extracted data from the included studies for the prespecified outcomes described below. Disagreements were resolved by discussion. The extraction sheet was piloted on a small subset of the data set and did not need any adjustments.

From each included article, we extracted qualitative information about whether authors made general statements about the effect of exogenous estrogens on cardiovascular risk, not limited to the scope of their study. General statements could be either extrapolations to other populations (including effect of MHT in humans), references to “well‐known” or established effects on cardiovascular risk, explicit acknowledgment that evidence across studies is contradictory or inconclusive, or explicit assessments about the probable net direction of effect of estrogens across studies. If the estrogen‐treated animals were included as a positive control in multiarm trials, we would consider this an implicit statement about beneficial cardiovascular effects. In addition, we also extracted qualitative information about the authors’ statements about the effect of estrogens within their own study (study specific statements), ie, the authors’ interpretation of their own results. All statements were categorized as follows: “protective” (exogenous estrogens reduce cardiovascular risk), “detrimental” (exogenous estrogens increase cardiovascular risk), “neutral” (exogenous estrogens do not affect cardiovascular risk), “ambivalent” (exogenous estrogens have both protective and harmful effects on cardiovascular disease) or “unclear” (we could not identify a clear conclusion about the effects of estrogen on cardiovascular risk).

For analysis of treatment effects in the studies and publication bias, we assessed whether each study contained information for a continuous outcome measuring the development of atherosclerosis. We chose this outcome because piloting indicated this to be the most commonly reported outcome, and one of the hypothesized effects of MHT before the HERS and WHI studies were published was that it reduced the risk of coronary heart disease in postmenopausal women. Where data on atherosclerosis were reported for both intervention and control groups with appropriate measures of uncertainty, we extracted (i) the mean value for the outcome in each group; (ii) either the standard deviation, standard error of the mean, or confidence interval (CI) for the mean in each group (in order of preference); and (iii) the number of animals in each group. We extracted data for only one outcome per study.

As many studies report on multiple similar outcomes, we had prespecified a heuristic to choose between outcomes that were otherwise deemed equally relevant to the development of atherosclerosis:

We primarily extracted data for the outcome presented as the main or most important analysis by study authors.

If this was unclear, we extracted the first eligible outcome for which extractable data were presented in the text (either inline or in a table).

If numeric data were not presented in writing, we extracted data for approach 1 or 2 by reading values off graphs where available.

If data for several doses of estrogens were available for one study, we extracted data for the median dose, resorting to the lowest dose above the median if an even number of doses was tested.

If sufficient information was not identified in the report, the study was not included in the quantitative analysis of treatment effects. Eligibility for inclusion in this analysis was assessed independently by two authors (CFB and PR), and disagreements were resolved by discussion. Numerical accuracy of extracted data points was scrutinized by the reviewers together, and disagreements were resolved by joint reexamination of the study.

Statistical analysis

For both the authors’ general statements about estrogen effects on cardiovascular risk and their statements about their own results, we computed the proportions of studies in each of the five categories. Studies were stratified by whether they were published in 2002 or earlier, or in 2003 or later. We chose this cutoff point because we believed studies published in 2003 or later could reasonably be expected to take the results of the WHI study into account. We also dichotomized the outcome by binning statements into positive and nonpositive, that is, whether estrogens were regarded as protective versus all other categories, and recomputed the proportions. For the two‐by‐two tables produced by dichotomizing both time of publication and assessments about outcome, we performed a test to investigate whether author statements about estrogen effects changed after publication of the first WHI study.

As a supplementary post hoc analysis, we computed Cramer’s V for multilevel categorical data to assess how general statements correlated with study‐specific conclusions, with bootstrap CIs.

For the quantitative analysis of the effect of exogenous estrogens on the development of atherosclerosis, we performed meta‐analyses applying two alternative models: one fixed‐effect and one random‐effects model (DerSimonian‐Laird method). Studies were weighted with a generic inverse variance approach. The random‐effects model is generally more conservative in terms of precision if there is large heterogeneity in treatment effects across studies. However, as it attributes comparatively more weight to smaller studies, it may be more sensitive to bias if small‐study effects (ie, bias causing exaggerated effects due to lack of methodological rigor in small studies ) are present. As we were investigating effects across species where one cannot expect one common fixed effect, we report the random‐effects model as our primary model and give additional fixed‐effects estimates in the appendix. As post hoc analyses, we additionally performed subgroup analyses per animal species and by general and study‐specific statements about estrogen effects (dichotomized as protective vs all other groups).

Atherosclerosis outcomes are reported on a plethora of scales and using different approaches to quantify atherosclerosis. We therefore used the standardized mean difference (SMD) as the meta‐analytic outcome measure, applying the Hedges’ g estimator for the standardization. When standard deviations of treatment group means were not found in study reports, we computed these values from the corresponding standard errors or confidence intervals and the group sizes. In a single study, a SMD of 1.0 means that the mean amount of atherosclerosis in the intervention group is shifted by one standard deviation compared to the control group, on whichever scale was used in the study.

We computed I 2 as a measure of statistical heterogeneity in all meta‐analyses, and P values for the significance of heterogeneity. An I 2 of 0% indicates no heterogeneity. Higher numbers mean that variation in treatment effects across studies is less likely to be caused by sampling error alone. Values between 50% and 100% are commonly interpreted as indicative of important study‐level differences, such as differing populations, impacting between‐study variation of effect estimates.

To test for small‐study effects and publication bias, we visually inspected funnel plots and applied Egger’s regression test, , considering a P < 0.1 as indicative of bias. If probable bias was detected, we performed trim‐and‐fill analyses, a nonparametric method imputing effects of presumed missing studies to reduce the skewness of the funnel plot and obtain adjusted treatment effect estimates. We also obtained adjusted meta‐analytic estimates from the Egger analysis, which corresponds to a weighted regression of study effects against their standard errors. The regression line intercept at zero standard variation may be construed as an estimate of the underlying mean effect if publication bias were not present. , We performed separate analyses for publication bias in all subgroups with ≥10 studies; applying such tests with too few studies is not advisable both due to lack of power and the risk of spurious findings. Tests for small‐study effects and publication bias may be at risk of type I errors when SMD is used as the outcome measure, particularly if groups are small. We hence performed a sensitivity analysis using a normalized mean difference (NMD) as an alternative outcome measure in these tests. To obtain NMDs, individual study outcomes were normalized by computing the raw mean difference between groups as a fraction of the control group mean, that is, as percentage reduction or increase in atherosclerosis compared to control.

We used R version 3.6.0 with the libraries meta, metafor, and ggplot2 for all analyses and plots.

Search strategies, inclusion and exclusion criteria, outcomes, methods for data extraction and analyses were pre‐planned in a protocol available from the authors upon request. The analyses for adjusting meta‐analytic estimates for publication bias and one subgroup analyses (by animal species) were conducted post‐hoc in addition to the pre‐specified analyses.

RESULTS

Study conclusions on cardiovascular disease from animal studies before and after the WHI trial

A total of 1925 studies were retrieved in the initial search; 360 were included for the qualitative analysis of report conclusions, of which 135 also had data available for quantitative synthesis (included and excluded studies are listed in Appendices B and C in the Supporting Information). Study flow is illustrated in Figure 1.

FIGURE 1

Study flow for the inclusion and exclusion of studies

Of the 360 studies included, 9 were published from 1950 to 1959, 9 from 1960 to 1969, 8 from 1970 to 1979, 18 from 1980 to 1989, 79 from 1990 to 1999, 152 from 2000 to 2009, and 85 from 2010 to 2018. The number of published studies stratified per decade is shown in Figure S1, where also the years of the publication of the Nurses’ Health Study (1991) and the WHI study (2002) are marked.

The proportions of study‐specific statements concluding that estrogen was effective, detrimental or had a neutral effect in preventing cardiovascular disease, as well as those being ambivalent or unclear about whether it was effective, are shown in Figure 2. Of the 360 studies included, 276 (77%; 95% CI, 72%‐81%) concluded that estrogens had a protective effect against cardiovascular diseases in their experiments. We found no evidence that researchers concluded differently about their own findings before or after publication of the first WHI study (75%; 95% CI, 67%‐81%), “protective” before WHI compared with 78% (95% CI, 71%‐83%) after WHI, right panel Figure 2). The test when dichotomizing into protective versus all other categories showed no indication of a difference (P = .62) in the proportion of study findings reported as positive. However, the authors’ extrapolations to general statements about the properties of exogenous estrogens changed markedly: 70% (95% CI, 63%‐76%) stated it to be cardioprotective before WHI, while 40% (95% CI, 34%‐48%) considered it protective after. In Table 1, we show a cross‐tabulation of general statements with study‐specific statements for the entire period and for the periods before and after WHI. The correlation between study‐specific statements and general statements was strong for the period before WHI (Cramer’s V, 0.47; 95% CI, 0.34‐0.61), whereas it dropped in the period after WHI (Cramer’s V, 0.25; 95% CI, 0.15‐0.42). For the entire period, there was moderate to strong correlation between statements (Cramer’s V, 0.37; 95% CI, 0.26‐0.48).

FIGURE 2

Study authors’ statements about whether estrogen therapy protects against cardiovascular disease or not. The left panel shows general statements made by authors in their reports on the effect of estrogens on cardiovascular risk. The right panel shows how authors concluded about their own study findings in published study reports. The absolute numbers of reports in each category are given above the bars. WHI, Women’s Health Initiative

TABLE 1

Authors’ general statements on the effect of menopausal hormone therapy on cardiovascular risk compared with study‐specific statements on the effect of estrogens in their own experiments

	Study‐specific statements
	Ambivalent	Detrimental	Neutral	Protective	Unclear	Total
General statements, n (%)
All studies
Ambivalent	15 (4.2)	12 (3.3)	9 (2.5)	83 (23.1)	4 (1.1)	123 (34.2%)
Detrimental	0 (0.0)	7 (1.9)	0 (0.0)	1 (0.3)	0 (0.0)	8 (2.2)
Protective	8 (2.2)	0 (0.0)	11 (3.1)	174 (48.3)	5 (1.4)	198 (55.0)
Unclear	3 (0.8)	0 (0.0)	4 (1.1)	18 (5.0)	6 (1.7)	31 (8.6)
Total	26 (7.2)	19 (5.3)	24 (6.7)	276 (76.7)	15 (4.2)	360 (100.0)
Before WHI (1950‐2002)
Ambivalent	6 (3.4)	4 (2.2)	3 (1.7)	24 (13.4)	1 (0.6)	38 (21.2)
Detrimental	0 (0.0)	5 (2.8)	0 (0.0)	0 (0.0)	0 (0.0)	5 (2.8)
Protective	7 (3.9)	0 (0.0)	9 (5.0)	106 (59.2)	3 (1.7)	125 (69.8)
Unclear	2 (1.1)	0 (0.0)	1 (0.6)	5 (2.8)	3 (1.7)	11 (6.1)
Total	15 (8.4)	9 (5.0)	13 (7.3)	135 (75.4)	7 (3.9)	179 (100.0)
After WHI (2003)
Ambivalent	9 (5.0)	8 (4.4)	6 (3.3)	59 (32.6)	3 (1.7)	85 (47.0)
Detrimental	0 (0.0)	2 (1.1)	0 (0.0)	1 (0.6)	0 (0.0)	3 (1.7)
Protective	1 (0.6)	0 (0.0)	2 (1.1)	68 (37.6)	2 (1.1)	73 (40.3)
Unclear	1 (0.6)	0 (0.0)	3 (1.7)	13 (7.2)	3 (1.7)	20 (11.0)
Total	11 (6.1)	10 (5.5)	11 (6.1)	141 (77.9)	8 (4.4)	181 (100.0)

WHI, Women’s Health Initiative.

Effect of estrogens on atherosclerosis in animal studies

Results from the meta‐analyses of the effect of exogenous estrogens on the development of atherosclerosis in animal studies are summarized in Figure 3 (full forest plots including individual study effects and fixed‐effect model estimates are provided in Appendices D and G in the Supporting Information). Overall, a reduction in atherosclerosis was detected (SMD in the random‐effects model −1.25; 95% CI −1.42 to −1.07). The SMD was −0.99 (95% CI, −1.21 to −0.78 in the random‐effects model) before 2003 versus −1.63 (95% CI, −1.91 to −1.36 in the random‐effects model) after 2003. Thus, the effect of estrogens appeared to be weaker before 2003 compared with after 2003 (P < .001). There was substantial heterogeneity of treatment effects, both overall (I 2 = 76%), before WHI (I 2 = 77%) and after WHI (I 2 = 68%) (all P < .01 for significant heterogeneity) (Figure 3).

FIGURE 3

Pooled effect estimates of exogenous estrogen on the development of atherosclerosis in animal models. Horizontal lines represent 95% confidence intervals (CI). Negative standardized mean differences (SMD; left side of vertical dotted line) mean less atherosclerosis in the estrogen group. The trim‐and‐fill analyses and estimates adjusted using Egger regressions provide estimates corrected for the impact of publication bias and exaggerated effects in small studies. WHI, Women’s Health Initiative

Treatment effects of estrogens for different species

Due to the large heterogeneity of treatment effects, we performed an additional subgroup analysis to investigate whether treatment effect differed by animal species (Figure 4, full forest plot and fixed‐effects model estimates in Appendices D and G in the Supporting Information). Effects did vary across species (P < .01). Seven of the subgroups were largely uninformative, containing only one to three studies. Moreover, six of these comprised <25 animals.

FIGURE 4

Subgroup analysis of treatment effect by species. Horizontal lines represent 95% confidence intervals (CI). Negative standardized mean differences (SMD; left side of vertical dotted line) mean less atherosclerosis in the estrogen group. I 2 is a measure of statistical heterogeneity; higher numbers mean more heterogeneity (not applicable for subgroups with only one study)

Notably, we found consistent effect estimates without significant heterogeneity (I 2 = 9%; P = .34) in studies on cynomolgus monkeys, whereas within‐group heterogeneity remained high for studies on all other species with more than one study (I 2 = 68%–86%; P < .01 for all). The estimate for cynomolgus monkeys suggests a small protective effect of estrogens on atherosclerosis is present in this group (SMD, −0.45; 95% CI, −0.58 to −0.32 in random‐effects model).

Treatment effects according to general and study‐specific statements about estrogen effects

There were no difference in standardized mean effects between studies where general statements indicated that estrogen had protective effects and where they did not (SMD in random‐effects model, −1.29; 95% CI, −1.50 to −1.09 vs −1.16; 95% CI, −1.48 to −0.85, Appendix D in the Supporting Information). A comparison between the study‐specific conclusion of protective effects with all other effects found that studies concluding with protective effects from their own results indeed found less atherosclerosis than studies concluding with other effects (SMD in random‐effects model, −1.45; 95% CI, −1.64 to −1.26 vs −0.22; 95% CI, −0.60 to 0.16; Appendix D in the Supporting Information).

Analysis of publication bias and small‐study effects

The analyses for small‐study effects strongly indicated publication bias in the overall body of animal studies (P < 10−12 in Egger’s test), as well as in other subgroups with 10 studies or more (before WHI: P < 10−4, after WHI: P < 10−8, mice: P < 0.01, rabbits: P < 10−7) except cynomolgus monkeys (P = .7) and the small group of rat studies (P = .97). The tests may also indicate the presence of exaggerated effect estimates in small studies. The funnel plots in Figure 5 show pronounced skewness, with a clear sparsity of studies on the lower right side of the plots for the total cohort (upper left) and all subgroup analyses except for cynomolgus monkeys (lower left). Many studies fall outside the area of expected variability in effects, most of them showing large effects. For rabbits, however, skewness was not significant in the sensitivity analysis using NMD outcomes (see Figure S2).

FIGURE 5

Funnel plots for the effect of exogenous estrogens on the development of atherosclerosis. If meta‐analytic estimates (vertical dotted lines) are unbiased, an even spread of studies is expected within the triangular areas representing the treatment effect variability anticipated to arise by chance across studies. Larger studies with smaller standard errors appear toward the top. Regression lines from Egger’s test are shown whenever significant skewness was detected. The regression line is not drawn for the cynomolgus monkey studies as no signs of publication bias were present in this group. WHI, Women’s Health Initiative

An approximation of treatment effect estimates corrected for bias may be obtained by observing where the Egger regression lines intersect the x axis at the top of the plots in Figure 5. Notably, all intersect near null effect. Applying this method, the effects of estrogens shift from protective to slightly detrimental (estimates shown in Figure 4). The trim‐and‐fill analyses imputing the effect of unpublished studies also shifted the SMDs considerably toward null effect for the main analyses (Figure 4). Plots of the trim‐and‐fill analyses are given in Appendices D and E in the Supporting Information, and results for Egger regression sensitivity analyses using NMD outcomes are given in Appendix F.

DISCUSSION

Our study confirmed that most authors of animal studies conclude that the results of their experiments show that estrogens are beneficial for cardiovascular disease. We investigated two explanations for the discrepancies between results from animal studies and RCTs, that is, confirmation bias and publication bias. The results showed that publication bias is a possible explanation, probably in combination with exaggerated effects in small studies.

To our knowledge, this is the first study to systematically appraise animal studies assessing the effects of estrogen on cardiovascular diseases. Small animal studies have shown to be prone to lack of rigor in other areas. , , , As most identified studies were small, we believe that the heterogeneity observed is mainly caused by exaggerated effects in small studies. , Laboratory studies have an inherent tendency for bias since experimental conditions can be adapted to what is assumed to be the “correct” result. Whereas the scientific community and regulatory authorities have invested large resources to improve the conduct and reporting of clinical studies, less attention has been devoted to the basic sciences. Preclinical researchers may consider that a negative result is a botched experiment, although it could in fact, if results are published, constitute a valuable scientific contribution.

We hypothesized that confirmation bias within the predominant scientific paradigm might have affected the study authors’ conclusions about estrogenic effects on cardiovascular risk to become less positive after the publication of the first WHI study. Somewhat surprisingly, we found that the studies published after WHI showed stronger cardioprotective effects of estrogens than before, with even more pronounced signs of publication bias. This was despite the fact that study authors seemed more cautious in their general statements regarding protective effects of estrogens on cardiovascular diseases after WHI. It is difficult to explain this observation. One possibility is that it was harder to publish ambivalent or neutral results compared with protective effects, even if such results were more in line with the WHI results.

Publication bias tests must be interpreted with caution in the presence of large heterogeneity, , and also when SMDs are used to pool studies. The strength of the correlation between standard errors (a proxy for study size) and treatment effects in the overall Egger regression analysis in our study is, however, considerable. The correlation is also consistent across all subgroup analyses but one. We hence find it improbable that our main findings of publication bias and probable small study effects are spurious or significantly impacted by other sources of heterogeneity. It is, however, important to emphasize that our study does not rule out a true effect in animal models, and a plausible effect is present in the studies of cynomolgus monkeys, without signs of publication bias. A sensitivity analysis using NMD also failed to demonstrate significant skewness for rabbit studies, indicating that findings for this subgroup are less certain.

I 2 was high in all animal studies, except cynomolgus monkeys, suggesting large heterogeneity between study results. Possible explanations include variation in outcome measures or differences in design that is difficult to quantify, “nonobserved confounders.” Publication bias and small‐study effects will also contribute to a high I 2. It can sometimes be difficult to separate true heterogeneity from such effects. In the current article, however, with the skewed funnel‐plots and the association between study size and effect suggested by Egger’s test, it is likely that publication bias causes a large part of the I 2.

Many obstacles preclude generalization of findings from animals to humans. The models may inherently lack external validity, extrapolation to humans may be wrong due to differences between species, or the clinical studies may have shortcomings. However, a Cochrane review from 2015 found that most RCTs were of good methodological quality, and we believe major bias in the clinical study results to be less likely.

Whether the populations included in the early RCTs on MHT were the ideal target population has been debated. , , , , Indeed, subgroup analyses in the Cochrane review provide moderate‐quality evidence that MHT started within 10 years of menopause may reduce the overall risk of death (relative risk [RR], 0.70; 95% CI, 0.52‐0.95) and cardiovascular death (RR, 0.52; 95% CI, 0.29‐0.96), at the expense of an increased stroke rate (RR, 1.37; 95% CI, 0.80‐2.34). Considering the relatively low baseline risk of these events in the younger postmenopausal cohort, this still translates to small differences in absolute risk. Based on five studies , , , , with a median follow‐up of 7.1 years (range, 3.4‐10.1), the review authors suggest approximately six fewer deaths and four more strokes per 1000 women treated with estrogen for the duration of follow‐up. Moreover, the estimates for effect on mortality were significantly impacted by the inclusion of one study deemed to be at high risk of bias.

Nevertheless, an effect in humans may be present. Considering our analyses indicating that treatment effects in published animal studies probably often provide biased and exaggerated estimates, the real discrepancy between findings in animals and humans does not seem inexplicably large. In fact, our analyses adjusted for possible publication bias suggest that the effect of estrogens on cardiovascular diseases may be small or close to null in animals as well.

LIMITATIONS

The literature search was pragmatically limited to one database; hence, we may have missed some studies not indexed there. Given the large number of studies identified, we believe this would not impact significantly on our overall conclusions. Furthermore, our analysis of the study authors’ conclusions is inherently dependent on the reviewers’ discretion and interpretations, and hence at risk of misclassification bias. Although all assessments were performed in duplicate to mitigate this risk, we cannot be sure that we have not occasionally misinterpreted the study authors.

Our methods for assessing the impact of presumed unpublished studies on the meta‐analytic estimates inherently involve extrapolating beyond the study data at hand. We do not know the true missing values, and the statistical corrections cannot be taken as providing certain adjusted estimates. However, the consistency of the findings indicates that the failure to publish negative or neutral studies decisively impacts the overall body of evidence from animal studies.

For Chinese and Japanese language studies, we relied on Google translate for abstracting data. Although a manual translation would arguably have been better, a recent study indicates this tool may now have reached sufficient accuracy for abstracting data for systematic reviews.

CONCLUSIONS

Most publications on estrogen‐treated animals interpret their own results as a beneficial effect of estrogens on cardiovascular outcomes. Most publications on estrogen‐treated animals also contain general statements about the protective effects of estrogens on cardiovascular disease, although such statements became rarer after the publication of the WHI study at the same time as ambivalent general statements became more frequent. However, the change in general statements about estrogens and cardiovascular disease did not influence the authors’ interpretations of their results. The majority of studies of estrogen on atherosclerosis in animals showed indirect signs of large publication bias favoring results that showed less atherosclerosis in estrogen‐treated animals. An important exception were the studies on cynomolgus monkeys. Our results indicate that the difference between animal studies and human studies on the effect of estrogens on cardiovascular disease is perhaps less than previously assumed.

RELATIONSHIP DISCLOSURE

CFB and PR report no conflicts of interest. AEAD reports no conflicts of interests involving the work under consideration for publication but has the following conflicts of interests outside the submitted work: honoraria or payments from Pfizer, Bristol‐Myers Squibb, Novartis, Bayer, and MSD, as well as research support from Pfizer Norway.

AUTHOR CONTRIBUTIONS

CFB performed the literature search, screened the abstracts, extracted the data, did the statistical analyses, made the first article draft, and approved the submitted manuscript. PR performed the literature search, screened the abstracts, extracted the data, wrote the article, and approved the submitted manuscript. AEAD had the idea for the study, did the literature search, wrote the article, and approved the submitted manuscript.

Supplementary Material

Click here for additional data file.