Methodological Rigor and Temporal Trends of Cardiovascular Medicine Meta-Analyses in Highest-Impact Journals.

Background Well-conducted meta-analyses are considered to be at the top of the evidence-based hierarchy pyramid, with an expansion of these publications within the cardiovascular research arena. There are limited data evaluating the trends and quality of such publications. The objective of this study was to evaluate the methodological rigor and temporal trends of cardiovascular medicine-related meta-analyses published in the highest impact journals. Methods and Results Using the Medline database, we retrieved cardiovascular medicine-related systematic reviews and meta-analyses published in The New England Journal of Medicine, The Lancet, Journal of the American Medical Association, The British Medical Journal, Annals of Internal Medicine, Circulation, European Heart Journal, and Journal of American College of Cardiology between January 1, 2012 and December 31, 2018. Among 6406 original investigations published during the study period, meta-analyses represented 422 (6.6%) articles, with an annual decline in the proportion of published meta-analyses (8.7% in 2012 versus 4.6% in 2018, Ptrend=0.002). A substantial number of studies failed to incorporate elements of Preferred Reporting Items for Systematic Reviews and Meta-Analyses or Meta-Analysis of Observational Studies in Epidemiology guidelines (51.9%) and only a minority of studies (10.4%) were registered in PROSPERO (International Prospective Register of Systematic Reviews). Fewer manuscripts failed to incorporate the Preferred Reporting Items for Systematic Reviews and Meta-Analyses or Meta-Analysis of Observational Studies in Epidemiology elements over time (60.2% in 2012 versus 40.0% in 2018, Ptrend<0.001) whereas the number of meta-analyses registered at PROSPERO has increased (2.4% in 2013 versus 17.5% in 2018, Ptrend<0.001). Conclusions The proportion of cardiovascular medicine-related meta-analyses published in the highest impact journals has declined over time. Although there is an increasing trend in compliance with quality-based guidelines, the overall compliance remains low.

Meta‐Analysis of Observational Studies in Epidemiology

Preferred Reporting Items for Systematic Reviews and Meta‐Analyses

randomized controlled trial

What Is New?

The number of cardiovascular medicine‐related systematic reviews and meta‐analyses published in the highest impact journals has decreased over time.

The overall compliance with the quality standards for systematic reviews and meta‐analyses remains suboptimal.

What Are the Clinical Implications?

Raising the awareness of these limitations and better adherence to respective guidelines would allow publication of only high‐quality systematic reviews and meta‐analyses.

Well‐conducted and rigorous systematic reviews and meta‐analyses are considered to be the highest level of evidence and are positioned at the top of the evidence‐based pyramid hierarchy. This is accomplished by identifying and combining relevant data to increase power, evaluating the risk of bias, identifying sources of variation, and rating the overall quality of existing evidence. , The output of such studies carries the potential to augment existing data from randomized controlled trials (RCTs) and observational studies. Since their inception in the 1970s, there has been a tremendous upsurge in the number of systematic reviews and meta‐analyses published each year. , , Several factors have contributed to this trend such as the technological advancements resulting in the ease of access to available databases and simplified algorithm‐based software to efficiently screen, synthesize, and accurately analyze large sets of data, as well as the rise in the production of well‐conducted RCTs and the rapid evolution of the field. This expansion in systematic reviews and meta‐analyses has led to a dramatic increase in duplication of studies with identical subjects as well as fabrication of many disorganized, nonguideline adherent, and suboptimal‐quality studies. , , Despite this global observation, there is a lack of objective data to study the trends in quality, proportion, and guideline concordance of the existing systematic reviews and meta‐analyses within the realm of cardiovascular medicine. The objectives of this analysis were to evaluate trends of cardiovascular medicine‐related systematic reviews and meta‐analyses published in highest impact journals with regards to their methodological quality, relative proportion to original investigations, and adherence to recommended guidelines for conducting systematic reviews and meta‐analyses.

Methods

The authors declare that all supporting data are available within the article and the online supplementary files. This study was exempted from institutional review board because this is a study‐level analysis.

Search Strategy and Study Selection Criteria

Systematic reviews and meta‐analyses published in the 8 highest Web of Science Impact Factor journals (according to Journal Citation Reports 2018: category “General Internal Medicine” and “Cardiology”) were included. The following journals were included: The New England Journal of Medicine, The Lancet, Journal of the American Medical Association, The British Medical Journal, Annals of Internal Medicine, Circulation, European Heart Journal, and Journal of American College of Cardiology. , The Medline database was used to identify meta‐analyses and systematic reviews published between January 1, 2012 and December 31, 2018. Systematic reviews that did not report any quantitative data synthesis were excluded. The retrieved articles were manually screened based on the title and/or abstract to identify cardiovascular medicine‐related publications, which were the focus of this analysis. For each journal, we also identified the number of original investigation publications over the same period. Three investigators (D.M., A.T., and A.S.) worked independently to extract the data. In case of discrepancies, the full text was reviewed by an independent investigator (I.E.) along with the previous author to reach a consensus.

Data Extraction and Assessment of Methodological Quality

For each cardiovascular medicine‐related article, we identified the following variables: year of publication, registration in PROSPERO (International Prospective Register of Systematic Reviews), change in the study protocol, type of meta‐analysis (pairwise versus network), type of studies included (RCTs, observational trials, or both), data level (patient level versus study level), focus of the article, funding source (industry versus nonindustry), and access (open versus closed access). Furthermore, full‐length articles were reviewed to assess the methodological quality of the systematic reviews and meta‐analyses based on the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) and Meta‐Analysis of Observational Studies in Epidemiology (MOOSE) guidelines (Table S1). , The quality of systematic reviews and meta‐analyses was judged on the following factors: mention of “meta‐analysis or systematic review” in the title, inclusion of a PRISMA flow diagram, assessment of risk bias, use of Grading of Recommendations Assessment, Development and Evaluation (GRADE) assessment, assessment of publication bias if >10 studies, and assessment of heterogeneity as well as exploration via subgroup, sensitivity, or metaregression analyses if the degree of statistical heterogeneity was high. Heterogeneity was defined as being high if I2 statistic was >50%. The overall assessment of these variables for each included meta‐analysis is reported in Table S2.

Statistical Analysis

We assessed the proportion of cardiovascular medicine‐related systematic reviews and meta‐analyses in relation to the overall number of original investigations published in the same journal during the same publication year. After stratifying the publications by the year of publication, we computed the percentage of systematic reviews and meta‐analyses published as compared with original investigations. Subsequently, we assessed the temporal trend in the ratio of published cardiovascular medicine‐related systematic reviews and meta‐analyses to original investigations. The distribution of the aforementioned variables was evaluated for all cardiovascular medicine‐related systematic reviews and meta‐analyses. Chi‐square test for linear trend was used to evaluate current temporal trends. The rates were expressed as a percentage and all P values were double sided. All categorical variables were reported as raw frequencies and overall percentages. All 2‐sided P values were considered significant at the α=0.05 level. We used SAS version 9.1.3 (SAS Institute, Inc., Cary, NC) and Stata version 14 (StataCorp, College Station, TX) for all analyses.

Results

Included Meta‐Analyses

The search identified 6406 original investigations published during the study period in the journals reviewed, with meta‐analyses representing 422 (6.6%) articles. The ratio of original investigations to systematic reviews and meta‐analyses ratio was ≈14:1. The characteristics of the systematic reviews and meta‐analyses are displayed in Table 1. Notably, a large proportion of studies were pairwise meta‐analyses (91.7%), exclusively included RCTs (50.9%), and were nonindustry funded (50.5%). Patient‐level meta‐analyses comprised only a small proportion (16.6%) of all systemic reviews and meta‐analyses.

Table 1

Overall Characteristics of Cardiovascular Medicine−Related Systematic Reviews and Meta‐Analyses

Cardiovascular medicine−related systematic reviews and meta‐analyses	Studies (n=422)
Journal, n (%)
The New England Journal of Medicine	1 (0.2)
The Lancet	43 (10.2)
Journal of American Medical Association	20 (4.7)
British Medical Journal	71 (16.8)
Annals of Internal Medicine	44 (10.4)
Circulation	72 (17.1)
European Heart Journal	76 (18.0)
Journal of American College of Cardiology	95 (22.5)
Type of meta‐analysis, n (%)
Pairwise	387 (91.7)
Network	35 (8.3)
Type of studies included, n (%)
Randomized controlled trials	215 (50.9)
Observational studies	120 (28.4)
Both	87 (20.6)
Level of meta‐analysis, n (%)
Patient level	70 (16.6)
Study level	352 (83.4)
Focus of study, n (%)
Therapeutic	239 (56.6)
Epidemiology	89 (21.1)
Diagnostics	60 (14.2)
Other	49 (11.6)
Source of funding, n (%)
None	144 (34.1)
Industry	65 (15.4)
Nonindustry	213 (50.5)
Open access publication, n (%)	237 (56.2)

Temporal Trends

During the study period, there was a decreasing trend in the number of published systematic reviews and meta‐analyses (68 studies in 2012 versus 40 studies in 2018, P trend=0.006). Furthermore, we observed a decreasing trend in the proportion of cardiovascular medicine‐related systematic reviews and meta‐analyses to original investigations (8.7% in 2012 versus 4.6% in 2018, P trend=0.002) (Figure 1). There was a decline in the proportion of systematic reviews and meta‐analyses that did not follow the elements of PRISMA or MOOSE guidelines (60.2% in 2012 versus 40.0% in 2018, P trend<0.001) (Figure 2). The proportion of studies registered in PROSPERO demonstrated an increasing trend over time (2.4% in 2013 [PROSPERO was launched in late 2011] versus 17.5% in 2018, P trend<0.001) (Figure 3). Finally, there were no significant temporal change in the number of industry‐funded studies (P trend=0.16), systematic reviews and meta‐analyses consisting of only RCTs (P trend=0.92), and number of patient‐level meta‐analyses (P trend=0.43).

Figure 1

Temporal trend in proportion of published systematic reviews and meta‐analyses and their methodological rigor.

A, Proportion of cardiovascular related systematic reviews and meta‐analyses to original investigations were calculated and stratified by the year of publication. Trend analysis was conducted to depict P‐trend. B, Frequency of methodological rigor was calculated based on compliance with various elements of PRISMA and MOOSE guidelines as well as PROSPERO registration. GRADE indicates Grading of Recommendations Assessment, Development and Evaluation; MOOSE, Meta‐Analysis of Observational Studies in Epidemiology; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta‐Analyses; and PROSPERO, International Prospective Register of Systematic Reviews.

Figure 2

Temporal trend among systematic reviews and meta‐analyses failing to comply with PRISMA or MOOSE guideline elements.

Proportion of studies that did not incorporate PRISMA or MOOSE guideline elements were calculated and stratified by the year of publication. Trend analysis was conducted to depict P‐trend. MOOSE indicates Meta‐Analysis of Observational Studies in Epidemiology; and PRISMA, Preferred Reporting Items for Systematic Reviews and Meta‐Analyses.

Figure 3

Temporal trend in the proportion of published cardiovascular medicine‐related systematic reviews and meta‐analyses with PROSPERO registration.

Proportion of studies with PROSPERO registration were calculated and stratified by the year of publication. PROSPERO indicates International Prospective Register of Systematic Reviews.

Assessment of Meta‐Analyses Quality

Table 2 demonstrates the compliance of studies with guidelines for conducting high‐quality systematic reviews and meta‐analyses. A substantial proportion (51.9%) of all systematic reviews and meta‐analyses failed to incorporate MOOSE or PRISMA guidelines. The majority of systematic reviews and meta‐analyses published did not include all PRISMA elements: flow diagram (66.6%), risk assessment of bias (53.1%), assessment of heterogeneity (83.4%), exploration via subgroup analyses or sensitivity analyses in cases of high heterogeneity (78.7%), and assessment of publication bias (54.3%). Registration in PROSPERO (10.4%) and use of GRADE assessment (7.1%) was incorporated by a small proportion of all systematic reviews and meta‐analyses (Figure 1).

Table 2

Compliance with Guidelines for Conducting High‐Quality Systematic Reviews and Meta‐Analyses

Guidelines‐based quality metrics	Relevance and definition	Compliant studies, n (%)
PROSPERO (International Prospective Register of Systematic Reviews)
Registration	PROSPERO is an international database of prospectively registered systematic reviews in health care. A permanent record of essential details of study protocol is maintained. Studies should be registered in PROSPERO at the protocol/inception stage and details should be entered to avoid unplanned duplication and to enable comparison between published methods and planned protocol. A lack of registration in PROSPERO may discredit a study because of the potential of subject duplication or modification of methods during data analysis stage.	44 (10.4%)
Change in PROSPERO protocol	A change in PROSPERO protocol refers to divergence from predefined methods or assessment of variables. A change in PROSPERO protocol without reasonable explanation may discredit a study.	3 (6.8%)
Preferred Reporting Items for Systematic Reviews and Meta‐Analyses elements
Title indicates “Meta‐analysis” or “Systematic Review”	Title should clearly identify the study as a systematic review, meta‐analysis, or both. Without proper identification and transparency, the study may be perceived as an original investigation by the readership on a quick glance. A lack of proper title may result in misperceptions regarding the robustness of conclusions disseminated by the study.	343 (81.3)
Flow diagram provided	A flow diagram should be provided with clear identification of methods used to acquire the study sample, procedure for screening studies, assessment of study eligibility, and the number of studies included in the analysis. The number of studies excluded and the reasons for exclusions should also be highlighted at each stage. Without such transparency displayed in a flow diagram, a systematic review or meta‐analysis may lose credibility in their methodology of cohort creation.	281 (66.6)
Risk of bias assessed	The risk of bias should be assessed in individual studies and across all the included studies. The investigators should specify assessment of such risk, whether this was performed at the study level or outcome level, and how it should be incorporated during data‐interpretation of the cumulative evidence. Without a thorough assessment of risk bias, the results reported by cumulative systematic review or meta‐analysis may lose credibility.	224 (53.1)
Use of GRADE assessment	GRADE assessment developed by the Grading of Recommendation, Assessment, Development and Evaluation (GRADE) working group aims to grade quality of evidence and strength of recommendation presented by healthcare outcomes. This approach allows for a consistent method of assessing the quality of evidence for predefined outcomes across several studies. The quality of evidence is rated as high, moderate, low, or very low. All inclusive, this allows for a transparent judgment of the quality of evidence provided by the studies included in the systematic review or meta‐analyses. Without GRADE assessment, the quality of evidence included in the systematic review or meta‐analysis remains questionable and thereby decreasing the robustness of the conclusions drawn.	30 (7.1)
Assessment of heterogeneity	Heterogeneity refers to the variation in evaluated outcomes among the included studies. I2 statistic is often used to assess variation across studies with higher I2 indicative of higher variance in results that is attributable to heterogeneity in the studies included. Without assessment of heterogeneity, the estimation of the combined effect of studies included in the meta‐analysis or systematic review is often discredited.	352 (83.4)
Exploration via subgroup analysis, sensitivity analysis, or meta‐regression (if heterogeneity was high)	If heterogeneity across the studies included is high, further exploratory analyses (such as subgroup analysis, sensitivity analysis, or metaregression) must be performed to validate the results displayed by the systematic review or meta‐analysis. Without a well‐conducted exploratory analysis in the setting of high heterogeneity, the conclusions drawn by the study may not be validated.	277 (78.7)*
Assessment of publication bias (if >10 studies)	Publication bias refers to the higher likelihood of a study being published based on not only the quality of methodology, but also the hypothesis tested, significance, and directionality of results presented. This may result in studies supporting their hypothesis being published more often and faster as compared with studies refuting their hypothesis. Without evaluation of publication bias, the conclusions drawn by the systematic review or meta‐analysis ought to be interpreted with caution.	216 (54.3) †

277/352 (No of studies with further exploration in cases of high heterogeneity / # of studies with assessment of heterogeneity).

216/398 (No of studies with assessment of publication bias / # of studies with analysis of >10 studies).

Discussion

In this comprehensive analysis of cardiovascular medicine‐related systematic reviews and meta‐analyses, we demonstrated that approximately 1 in 14 original investigations published in the 8 highest impact journals between 2012 and 2018 were meta‐analyses. In addition, the proportion of systematic reviews and meta‐analyses to original investigations has significantly decreased over time in these journals. Although there has been a significant rise in the proportion of systematic reviews and meta‐analyses registered in PROSPERO, the overall percentage of systematic reviews and meta‐analyses registered in PROSPERO remains small (~10%). Similarly, although we observed an increasing temporal trend in the incorporation of elements from PRISMA or MOOSE guidelines, a greater proportion of cardiovascular systematic reviews and meta‐analyses did not comply with all domains of PRISMA or MOOSE guidelines.

In light of recent technological advancements including ease of access to large publicly accessible databases and software with capabilities to screen and analyze prior publications, together with the rapidly expanding base of RCTs, there has been a high level of interest in generating systematic reviews and meta‐analyses by various investigator groups. Accordingly, the clinical research community has criticized and voiced mixed feelings toward the quality of this burgeoning expansion of systematic reviews and meta‐analyses. , , , In this analysis, which focused only on the highest impact journals as defined by impact factor, we observed a decreasing trend in published systematic reviews and meta‐analyses. This trend observed is in stark contrast to the previously reported uptrend in other specialties. , This might be attributed to several factors. Editors and reviewers of these highest impact journals may curtail the number of systematic reviews and meta‐analyses that successfully pass the peer‐review process. Moreover, evidence‐based medicine in the cardiovascular community has been primarily driven by large RCTs and high‐quality observational studies. The fact that systematic reviews and meta‐analyses mostly confirm the findings of these high‐quality original investigations by providing more refined estimates to the effect size may prohibit their uptake in these high‐impact journals. Finally, a potential redundancy in data and suboptimal compliance with quality standards may predispose the authors of non‐high‐quality systematic reviews and meta‐analyses to avoid submission to these high‐impact journals. ,

This study demonstrated that a significant proportion of cardiovascular medicine‐related systematic reviews and meta‐analyses published in high‐impact journals did not adequately fulfill guideline recommended elements of publishing high‐quality systematic reviews and meta‐analyses (ie, PRISMA or MOOSE guidelines). Compliance with elements such as the use of GRADE assessment, incorporation of a flow diagram, and assessment of risk bias and publication bias remained particularly low. Our findings, which highlight the suboptimal compliance with PRISMA and MOOSE guidelines, are in line with what has been previously demonstrated in other clinical areas of research. , , , These investigations have reported a compliance rate of as low as 50% with PRISMA and MOOSE guidelines among systematic reviews and meta‐analyses, a finding that is consistent with our results despite the high‐impact nature of our selected journals. We also observed that only a minority of the included meta‐analyses reported the GRADE assessment. The steady dissemination of these quality standards and guidelines can be attributed to the increasing temporal trend in the incorporation of these guideline elements as demonstrated in our analysis. , However, the continued suboptimal rates of compliance may be in part attributed to the lack of adoption of such guidelines by investigator groups. Additionally, lack of awareness by investigator groups and relying on methodology noted in prior high‐impact journal publications rather than seeking out specific PRISMA and MOOSE guideline checklists may also propagate the manufacturing of suboptimal quality systematic reviews and meta‐analyses.

An overall increase in the temporal trend of PROSPERO registration was also demonstrated by our analyses. Launched in 2011, the primary aim of PROSPERO is to provide transparency about ongoing systematic reviews. , Registration in PROSPERO serves to avoid unnecessary duplication of similar analyses, minimize reporting bias, and hold authors accountable for reporting all prespecified end points. , Therefore, registration in PROSPERO protects systematic reviews against biases that may otherwise dilute and discredit the reported results. A large‐scale analysis by Page et al demonstrated a significant increase in registration of systematic reviews in PROSPERO between 2011 and 2017. The authors demonstrated a roughly 10‐fold increase in PROSPERO registration. Our results pertaining to cardiovascular medicine‐related systematic reviews are in line with such observations. Despite this, it is noteworthy that among the overall published systematic reviews and meta‐analyses, only 1 in 10 were registered in PROSPERO. Hence, although there has been an uptrend over time, the compliance with PROSPERO registration remains poor. Future investigations evaluating barriers to PROSPERO registration are needed to fully understand the reasons behind our findings. Identifying such obstacles would help improve PROSPERO registration rates and thereby provide credible and high‐quality systematic reviews to the research community.

This study has some limitations that are worth noting. First, additional information regarding overlap among systematic reviews and meta‐analyses topics could not be ascertained in our study, but other studies have shown that there are concerns related to overlapping cardiovascular‐related meta‐analyses. , We were unable to assess whether authorship guidelines of the included journals have evolved over time with regard to requiring adherence to qualitative guidelines and whether this has contributed to the observed trend in adherence to MOOSE and PRISMA guidelines. The results noted in our study may not be representative of noncardiovascular‐related systematic reviews and meta‐analyses or generalizable to lower impact cardiovascular journals. Finally, we did not have to access to the submission rates of meta‐analyses among the included journals, which might contribute to the reduction in the number of published meta‐analyses among the included journals.

Conclusions

The number of cardiovascular medicine‐related systematic reviews and meta‐analyses published in the highest impact journals has decreased over time. The proportion of these systematic reviews and meta‐analyses to original articles published in such journals has also experienced a downtrend. The overall compliance with the quality standards for systematic reviews and meta‐analyses remains suboptimal. Similarly, although PROSPERO registration for systematic reviews has increased over time, only 1 in 10 cardiovascular medicine‐related systematic reviews are registered in PROSPERO. High‐quality systematic reviews and meta‐analyses serve as an important constituent of the wide range of methodological studies. Raising the awareness of these limitations and better adherence to respective guidelines would allow publication of only high‐quality systematic reviews and meta‐analyses.

Sources of Funding

Open Access funding was provided by the Qatar National Library.

Disclosures

Dr. Virani has disclosures unrelated to the article including Department of Veterans Affairs, World Heart Federation, Tahir and Jooma Family, Honorarium: American College of Cardiology (Associate Editor for Innovations, acc.org), Steering Committee Member: PALM (Patient and Provider Assessment of Lipid Management) registry at Duke Clinical Research Institute (no financial remuneration). Dr. Mamas reports grants from Biosensors, grants from Terumo, grants from Abbott, personal fees from Daiichi Sankyo, and personal fees from Pfizer outside the submitted work. Dr. Masri reports grants and personal fees from Pfizer, grants and personal fees from Akcea, grants from Ultromics, personal fees from Ionis, personal fees from Cytokinetics, personal fees from Alnylam, and personal fees from Eidos outside the submitted work. Dr. Elgendy has disclosures unrelated to this article content including receiving research grants from Caladrius Biosciences, Inc. The remaining authors have no disclosures to report.

Tables S1–S2

Click here for additional data file.