Methodological and Clinical Heterogeneity and Extraction Errors in Meta-Analyses of Catheter Ablation for Atrial Fibrillation in Heart Failure.

Background Meta-analyses are expected to follow a standardized process, and thus, they have become highly formulaic, although there is little evidence that such regimentation yields high-quality results. Methods and Results This article describes the results of a critical examination of 14 published meta-analyses of catheter ablation for atrial fibrillation in heart failure that were based on a nearly identical core set of 4 to 6 primary trials. Methodological issues included (1) the neglect of primary data or the failure to report any primary data; (2) the inaccurate recording of the number of randomized patients; (3) the lack of attention to data missingness or baseline imbalances; (4) the failure to contact investigators of primary trials for additional data; (5) the incorrect extraction of data, the misidentification of events, and the assignment of events to the wrong treatment groups; (6) the calculation of summary estimates based on demonstrably heterogenous data, methods of differing reliability, or unrelated end points; and (7) the development of conclusions based on sparse numbers of events or overly reliant on the results of 1 dominant trial. Conclusions These findings reinforce existing concerns about the methodological validity of meta-analyses and their current status in the hierarchy of medical evidence, and they raise new questions about the process by which meta-analyses undergo peer review by medical journals.

Clinical Perspective

What Is New?

A critical examination was performed of 14 published meta‐analyses of catheter ablation for atrial fibrillation in heart failure that were based on a nearly identical core set of 4 to 6 primary trials.

There were significant methodological issues in the meta‐analyses, including (1) neglect of primary data; (2) inaccurate recording of the number of randomized patients; (3) lack of attention to data missingness or baseline imbalances; (4) failure to contact investigators for additional data; (5) incorrect extraction of data, the misidentification of events, and the assignment of events to the wrong treatment groups; (6) calculation of summary estimates based on demonstrably heterogenous data, methods of differing reliability, or unrelated end points; and (7) conclusions based on sparse numbers of events or reliant on the results of 1 dominant trial.

What Are the Clinical Implications?

These findings reinforce existing concerns about the methodological validity of meta‐analyses and their current status in the hierarchy of medical evidence, and they raise new questions about the process by which meta‐analyses undergo peer review by medical journals.

A meta‐analysis is an observational study in which the authors formulate summary estimates by collating the results of studies that have addressed the same research question. Despite their well‐known limitations, systematic reviews with a meta‐analysis are considered by some to constitute the highest level of medical evidence.1 The number of published meta‐analyses has soared,2 possibly because these studies can be done quickly following a simple literature review and using readily available software that requires minimal statistical expertise. Some editors favor the publication of meta‐analyses, believing that they have a positive effect on the impact factor of their journals,2, 3 especially if they report a benefit of the intervention.4

Many journals require that meta‐analyses be carried out using a standardized process and that the results be presented using regulated format.5, 6 As a result, the conduct of meta‐analyses has become highly formulaic; such regimentation is believed to yield high‐quality results.7 Yet, there have been few opportunities to test the operational validity of this assumption; checks of quality control often determine only if the stated requirements have been fulfilled.8 Although it is common for standardized meta‐analyses that focus on the same research question to report different estimates of a treatment effect, such differences are generally related to the fact that meta‐analyses on the same topic typically analyze different data sets. Little is known about variations in the performance of meta‐analyses when they are based on the same data sets, even though it is well‐known that authors of meta‐analyses may make errors in the extraction of data points from the original trials.9, 10

In the past 2 years, an unusually large number of meta‐analyses have focused on the effects of catheter ablation for the treatment of atrial fibrillation in patients with chronic heart failure. The sudden burst of numerous systematic reviews on the same subject over an exceptionally short period of time in the absence of any new trials affords an opportunity to compare the conduct and validity of systematic reviews and meta‐analyses that relied on the same group of studies.

Methods

All data that form the basis of the examinations in this article were published in the original manuscripts, except as indicated in the text. The details of the examination are provided in the Table. There have been 23 meta‐analyses of catheter ablation for atrial fibrillation in patients with chronic heart failure.11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33 In 5 instances, the authors of the meta‐analyses generated estimates by commingling data from randomized trials and observational studies,11, 12, 13, 14, 15 and another 4 reports16, 17, 18, 19 were based on only 3 small trials. These 9 articles are not considered further in the present examination. Of the remaining 14 systematic reviews and meta‐analyses, 12 relied on the same core data set of 6 trials (Table).34, 35, 36, 37, 38, 39 The 6 trials randomly assigned a total of 816 patients with chronic heart failure and atrial fibrillation to catheter ablation or to pharmacological interventions to achieve rate or rhythm control. Although several reports included a seventh randomized trial of catheter ablation,40 that study did not evaluate a control group of patients who received only pharmacological treatments for their atrial fibrillation and was not included by many authors. Of note, only 3 of the 14 meta‐analyses23, 24, 33 had been registered in PROSPERO (International Prospective Register of Systematic Reviews).

Table 1

Features of Meta‐Analyses of Catheter Ablation for Atrial Fibrillation in Patients With Chronic Heart Failure

Publication	Data Extraction and Analysis of Ejection Fraction	Data Extraction and Analysis of Hospitalizations and Deaths
Chen (2018)20	Did not include primary EF data by CMR in 2 trials34, 37	Not done for heart failure trials; did not obtain HFH data in 1 trial35
Ahn (2018)21	Did not include primary CMR data on EF in 1 trial34	Did not obtain mortality or HFH data in any of the 4 trials34, 35, 36, 37
Khan (2018)22	Not analyzed by the authors	Incorrect number of randomized patients in 2 trials36, 39; did not obtain mortality or hospitalization data in 4 trials34, 35, 36, 37; mortality effect no longer significant when 1 trial39 removed from analysis; commingled trials of rate and rhythm control
Kheiri (2018)23	Did not include >200 patients with EF data and relied on imputed EF data in 1 trial39; commingling of data using reliable and unreliable methods	Incorrect number of randomized patients in 4 trials34, 36, 37, 39; HFH data not obtained or extracted incorrectly in 3 trials34, 35, 38; commingled trials of rate and rhythm control
Elgendy (2018)24	No information on number of included patients or values of extracted data	Incorrect number of randomized patients in 3 trials36, 37, 39; mortality or HFH data not obtained or extracted correctly in 3 trials34, 35, 38; commingled trials of rate and rhythm control
Briceño (2018)25	Incorrect number of patients with paired data in all 6 trials; did not include >200 patients with paired EF data in one trial39; commingling of data using reliable and unreliable methods	Incorrect number of randomized patients in 2 trials36, 39; mortality data not obtained in 2 trials34, 35; commingled trials of rate and rhythm control
Ma (2018)26	Incorrect number of patients with paired data in all 6 trials; did not include >200 patients with paired EF data in 1 trial39; commingling of data using reliable and unreliable methods	Incorrect number of randomized patients in 3 trials36, 37, 39; HFH data misidentified in 1 trial38; no reduction in HFH and mortality when comparator was rate control
Smer (2018)27	Did not include primary EF data by CMR in 2 trials34, 37	Incorrect number of randomized patients in 4 trials34, 36, 37, 39; mortality and/or HFH data not extracted correctly in 2 trials34, 38; no reduction in HFH and mortality when comparator was rate control
Virk (2018)28	Commingled data using reliable and unreliable methods	Analysis restricted to 3 trials with ≥1‐year follow‐up; incorrect number of randomized patients in 1 trial39; commingled trials of rate and rhythm control
Turagam (2018)29	Did not include primary EF data by CMR in 2 trials34, 37; did not include >200 patients with paired EF data in 1 trial39	Incorrect number of randomized patients in 4 trials34, 36, 37, 39; mortality data not obtained in 2 trials34, 37; commingled trials of rate and rhythm control
Malik (2018)30	No information on number of included patients or values of extracted data	No information on number of included patients or number of major events extracted from individual trials; commingled trials of rate and rhythm control
AlTurki (2019)31	Did not include primary EF data by CMR in 2 trials34, 37; did not include >200 patients with paired EF data in 1 trial39	Incorrect number of randomized patients in 3 trials36, 37, 39; mortality or HFH events were misidentified or not extracted correctly in 3 trials34, 35, 38; no reduction in HFH and mortality when comparator was rate control
Moschonas (2018)32	No information on number of included patients or extraction of primary EF data; commingling of data using reliable and unreliable methods	Incorrect number of randomized patients in 2 trials36, 39; no information on extraction of primary data on major events; commingling of disparate reasons for hospitalization; commingled trials of rate and rhythm control
Agasthi (2019)33	Commingled data using reliable and unreliable methods	No information on extraction of primary data on major events; commingling of different reasons for hospitalization; commingled trials of rate and rhythm control

Chen et al20 and Ahn et al21 included only 4 of the core set of 6 primary trials. CMR indicates cardiac magnetic resonance; EF, ejection fraction; HFH, heart failure hospitalization.

The core set of 14 systematic reviews specified 3 principal end points of interest, that is, left ventricular ejection fraction, hospitalizations for heart failure, and all‐cause mortality. Although several meta‐analyses also reported summary estimates of the effect of catheter ablation on exercise tolerance and quality of life, these measures were not standardized across the 6 component trials. Furthermore, each of the 6 trials was carried out in an open‐label manner without blinding to the treatment assignment; knowledge of the treatment assignment can bias assessments of symptoms and functional capacity. For these reasons, these end points were not the primary focus of any meta‐analysis, and they are not evaluated in this article.

All 14 systematic reviews stated that the authors had complied with established standards for the conduct of a meta‐analysis, including a description of the search process, quality assessment, heterogeneity, and publication bias. However, each of the meta‐analyses generally yielded different summary estimates for the end points of interest, even though they relied on the same information. To explore the reasons for these differences, the data points reported in each meta‐analysis were compared with the original data that were collected in each of the 6 primary clinical trials.

Results and Findings

The authors of the 14 systematic reviews and meta‐analyses made numerous errors in obtaining, extracting, and analyzing the data on ejection fraction, hospitalizations for heart failure, and mortality from the 6 original trials (Table).

Reliability and Heterogeneity of the Methods Used to Evaluate Changes in Left Ventricular Ejection Fraction

The most commonly assessed end point in the 6 original trials was the change in left ventricular ejection fraction over a prespecified duration of follow‐up. Three of the largest trials36, 38, 39 (enrolling 80% of the patients) relied on 2‐dimensional echocardiography, but this method is not reliable in patients with atrial fibrillation.41 The possibility of observer bias may be particularly strong if posttreatment images in sinus rhythm are paired with pretreatment images captured during atrial fibrillation. In recognition of this methodological limitation, 3 trials were specifically designed to assess ejection fraction by reliable methods,34, 35, 37 and 2 elected to use cardiac magnetic resonance imaging as their preferred technique.34, 37 However, when the authors of the 14 meta‐analyses extracted data on ejection fraction, treatment effects recorded by magnetic resonance imaging were frequently omitted,20, 21, 27, 29, 32 even when they were the primary end point of the trial (Table). All meta‐analyses commingled data derived by echocardiography and magnetic resonance imaging, despite the differences in methodological reliability. Interestingly, many authors noted that summary estimates of the effect of treatment on ejection fraction were characterized by striking heterogeneity, which should have limited the ability of the authors of the meta‐analyses to provide a reliable summary estimate of a treatment effect; yet all meta‐analyses reported such estimates.

Concerns About the Commingling of Dissimilar Comparator Groups in the Assessment of Surrogate and Clinical End Points

Of the 6 trials, 2 trials used rhythm control as the comparator group, whereas 4 trials used rate control as the comparator. The rate‐control trials typically enrolled patients with long‐standing permanent atrial fibrillation who were treated with atrioventricular nodal blocking drugs. The rhythm‐control trials often enrolled patients with atrial fibrillation of shorter duration (<1 year), who received amiodarone or other membrane‐active drugs. These 2 control groups are inherently different; furthermore, the use of membrane‐active drugs can adversely affect cardiac performance or cause hemodynamically important bradyarrhythmias and thereby diminish patient survival.42, 43 Nevertheless, the majority of the meta‐analyses commingled different control groups when formulating their summary estimates of a treatment effect. In a few instances where the authors of the meta‐analyses analyzed the 2 different types of comparator groups separately, they concluded (based on sparse data) that choice of the comparator had an important influence on the presence and magnitude of the treatment benefit on the risk of death or hospitalization for heart failure (Table). A benefit of catheter ablation was observed only in trials where the comparator group received potentially cardiotoxic membrane‐active antiarrhythmic drugs.

Concerns About Incomplete Acquisition and Incorrect Extraction of Information on the Occurrence of Deaths and Hospitalizations in the Original Trials

Four of the primary trials,34, 35, 36, 37 which focused on structural or functional end points measured after a short duration of follow‐up, did not report the number of patients who died or were hospitalized for heart failure for the planned duration of the trial. For these trials, the authors of the meta‐analyses did not generally make a concerted effort to fully account for the clinical course of each randomized patient. Unfortunately, when no information about outcomes was provided, some authors of the meta‐analyses27 assumed that no events actually occurred; that is, the authors imputed a value of zero events. In 3 meta‐analyses,26, 27, 29 6 hospitalizations for heart failure were appropriately recorded for a trial35 that did not describe any of these occurrences in the original report. It is standard practice for the authors of a meta‐analysis to directly contact the investigators of a trial to obtain unpublished information, and the principal investigator of the trial35 made this information freely available. However, the authors of the majority of meta‐analyses made no effort to reach out to the investigators of the primary trials to obtain complete follow‐up data.

Interestingly, in 2 meta‐analyses,24, 32 the deaths that were reported in the primary trials were extracted but assigned to the wrong treatment groups. Similarly, in 6 meta‐analyses23, 24, 26, 27, 32, 33 the authors extracted the data on all‐cause hospitalizations in 1 trial,38 but in their analyses, they assumed (incorrectly) that all of the hospital admissions were related to worsening heart failure, whereas many were actually related to a recurrence of atrial fibrillation. One meta‐analysis31 commingled disparate data on cause‐specific and all‐cause hospitalizations from different trials intentionally, and in several meta‐analyses, the number of deaths and hospitalizations that were used in the calculations was not provided.24, 30, 32, 33

Incorrect Recording of Number of Participating Patients and Lack of Attention to Missingness of Data

The integrity of a randomized trial is highly dependent on retaining all randomized patients in the analysis of each measure of interest. However, investigators may inappropriately remove patients from an analysis following randomization, and, additionally, they may fail to ensure that all randomized patients are followed for the full duration of the trial and undergo all planned assessments. Even though the lack of follow‐up is unavoidable in certain circumstances, such missingness creates meaningful opportunities for bias.44

None of the authors of the 14 meta‐analyses addressed the issue of missingness of data. When extracting information on the number of patients, the authors of meta‐analyses often described the number who were analyzed rather than the number who were randomized; several meta‐analyses did not provide any information on the number of patients who had been included in the summary estimates.24, 30, 32, 33 In the analysis of morbidity and mortality, the 14 meta‐analyses did not make note of ≈40 to 50 patients who were randomized in 4 of the original trials (Table).34, 36, 37, 39 Furthermore, the studies typically did not present data on the number of patients who were lost to follow‐up, even when these may have been differentially distributed across treatment groups.45

Similarly, in extracting information on the number of patients who contributed data on ejection fraction, the authors of several meta‐analyses cited the number of patients with baseline data, rather than the number with paired data; this decision influenced the appropriate assignment of weights to different trials. Approximately 25% of randomized patients in the 6 trials were not included in the paired comparisons of ejection fraction; in some studies, paired data in >200 patients were not included, even when they were available (Table). Interestingly, because of concerns about incomplete data, 1 large trial39 imputed values for missing patients, and 1 meta‐analysis23 relied on imputed (rather than actual) data.

Lack of Attention to Baseline Imbalances and Sparseness of Data

Although a meta‐analysis of randomized controlled trials is inherently an observational study, its validity depends on the integrity of the randomization process in the component trials that form the basis of the systematic review. When the trials are small, it is possible for important baseline imbalances to occur by chance alone. In 2 of the 6 trials in the core data set, meaningful differences between the treatment groups were apparent at the time of randomization and were specifically noted by the investigators.34, 39 However, none of the authors of the meta‐analyses made note of these imbalances.

The findings of a meta‐analysis are fragile if they are dependent on the analysis of a sparse number of events.45 In the treatment of chronic heart failure, findings in randomized trials that are based on the analysis of fewer than 100 events are likely not to be replicated in larger trials, whereas reproducibility is high if a trial data set relies on >300 events.46 It is therefore noteworthy that all of the meta‐analyses of catheter ablation for atrial fibrillation in heart failure collected fewer than 100 deaths across all trials combined; yet, the authors reached unqualified conclusions about the effects of the intervention on mortality, despite the sparseness of data.

The treatment effects reported in meta‐analyses may be highly dependent on the results of a single trial. Although authors may add a few events that occurred in other trials, when there are only 1 or 2 dominant studies, the meta‐analysis essentially replicates their findings, and it is not reasonable for the authors to leave an impression that the totality of available data across numerous studies are directionally concordant. Typically, researchers who perform meta‐analyses are aware of this fragility and the need to carry out sensitivity analyses to evaluate the robustness of their data; a common approach is to repeat the analysis after 1 or 2 dominant studies have been removed. However, in the case of the meta‐analyses of catheter ablation, the researchers usually did not perform any sensitivity analyses of mortality, and if such analyses were performed, the researchers usually did not carry out a stepwise exclusion of 1 study at a time.

Discussion

An exceptionally large number of meta‐analyses have been published on the effects of catheter ablation for atrial fibrillation in patients with chronic heart failure and a reduced ejection fraction. Although the researchers stated that they adhered to accepted standards for performing a systematic review, there were a remarkably large number of discrepancies across the reports, and in particular, most of the meta‐analyses were characterized by errors in the acquisition and extraction of data from the original trials. Mistakes in data collection were far more common and influential than technical issues related to the choice of trials or statistical methodologies used in the meta‐analyses.5, 6 Similar concerns about the accuracy of data extraction have been raised by others.9, 10

Even though the number of published meta‐analyses has soared,2 it has been argued that the incremental knowledge provided by most meta‐analyses is marginal.47, 48 Meta‐analyses are particularly unlikely to be helpful when they are based on a small number of events or when the main findings are driven almost entirely by the results of 1 trial. Under such conditions, the meta‐analysis provides no information that was not already available and is subject to the same methodological limitations as in the original studies. In the case of catheter ablation for atrial fibrillation in heart failure, it is noteworthy that the ratio of the number of meta‐analyses to the number of randomized trials contributing to these meta‐analyses is nearly 4, that is, 23 meta‐analyses and 6 core trials.

In the case of catheter ablation, all meta‐analyses combined data across trials that evaluated an extraordinarily diverse group of patients. The duration of atrial fibrillation varied widely across studies, as did the clinical features of patients who would be expected to influence the response to treatment, that is, etiology of heart failure or left atrial size. Interestingly, the trial34 that enrolled the patients with the most advanced heart failure and the longest duration of atrial fibrillation noted the lowest rate of procedural success with catheter ablation and failed to show improvements in exercise tolerance, quality of life, or the primary end point of ejection fraction; a large proportion of patients who underwent ablation in this trial experienced serious adverse events. In contrast, one trial that reported the most reliable benefit of ablation on ejection fraction largely evaluated patients who had circulating levels of natriuretic peptides that were in the expected range for older patients with atrial fibrillation without left ventricular dysfunction.37 Combining data across such diverse studies obscures (rather than elucidates) differences in the response to treatment that are clinically meaningful.

Current standards for the performance of systemic reviews provide few recommendations as to how data extraction should be performed. The lack of such guidance is a serious deficiency, which needs to be addressed by the scientific community. Furthermore, the findings of the current examination about the quality of data extraction raise important concerns about the process by which meta‐analyses undergo peer review by medical journals.

There are no easy remedies for the deficiencies identified in the current examination. Many have recommended that a meta‐analysis be prospectively registered in PROSPERO, and interestingly, such registration was uncommon in the current set of publications; yet such listing serves as a notification rather than an assurance of quality, and authors seeking to perform a meta‐analysis may not necessarily be discouraged to discover the existence of many similar efforts. Clearly, the quality issues identified in this article can best be discovered during the review process, but most meta‐analyses are reviewed by clinical investigators who may have little methodological expertise and are unable to spend the enormous amount of time that would be required to examine each of the original trials to confirm the accuracy of the data extraction process or to detect biases in the way that the authors gathered the information from the primary sources. Perhaps journal editors should routinely add a prominent statement to every published meta‐analysis that the article has not been checked for extraction or methodological errors—unless the editors have implemented a process to ensure that such a review has actually occurred.

The validity of any meta‐analyses is entirely dependent on the quality of the data that are used to formulate summary estimates. When the first step in the process is flawed (because of deficiencies in the data extraction process or in the combining of data from heterogenous methods or populations), the combining of data in a meta‐analysis is unlikely to result in a useful contribution to medical research.

Disclosures

Dr Packer has recently consulted for Abbott, Actavis, Akcea, Amgen, AstraZeneca, Bayer, Boehringer Ingelheim, Cardiorentis, Daiichi Sankyo, Gilead, Johnson & Johnson, NovoNordisk, Pfizer, Relypsa, Sanofi, Synthetic Biologics, and Theravance.