The attractiveness of network meta-analysis: a comprehensive systematic and narrative review.

Network meta-analysis provides a global estimate of comparative treatment effectiveness combining both direct and indirect evidence. In the past decade, the medical literature has witnessed a rapid increase in the possibility to combine evidence from different treatment comparisons. This opportunity is attractive for clinicians since their major concern is to identify the single best available treatment. In addition, despite the sudden increase of publications concerning network meta-analysis, only a limited number focus on methodological and statistical aspects, and many issues remain unclear. The aim of our work was to explore and emphasize the potential attractiveness of network meta-analyses. We performed a systematic and narrative review (last updated on April 15, 2014) in order to assess the scholarly diffusion of network meta-analyses. The following data were collected: author identification, year and journal of publication, PubMed index, number of treatments and studies included, characteristics of network configuration, nature of primary outcome, clinical indication, type of intervention investigated and medical area. Since 2003 there has been an exponential increase in the number of published network meta-analyses. Out of 340 articles included according to our selection criteria, encompassing 248 treatment networks, cardiovascular and pulmonary diseases were the most prevalent topics, with an average of 5 treatments being compared stemming from an average of 10 controlled trials. In conclusion, network meta-analyses are becoming increasingly attractive as they offer a comprehensive framework for decision-making. Whether they will also contribute to improvements in patient outlook remains to be proven.

Introduction

Evidence-based medicine has become known to conscientiously exploit the current best available evidence to make decisions about the patient care [1]. This involves evaluating the quality of the clinical data by critically assessing methodologies reported in publications. Moreover, it requires integrating both clinical expertise and patient values [2].

Meta-analyses of randomized controlled trials are considered, or close to, the top of the hierarchy of evidence [3], being considered as the most internally valid clinical proof. In fact, meta-analysis is a validated method to cumulate and summarize knowledge by increasing the number of patient’s data used and thus the effective statistical power. However, researchers must be fully aware of the limitations and the critical issues in performing meta-analysis [4]. The major drawback is the possibility to evaluate only pairwise comparisons. Unluckily, head-to-head comparisons are not always available in the literature or they are not sufficient to answer a specific clinical question. Network meta-analysis can overcome this limit. This is done by providing a global estimate of efficacy or safety of multiple experimental treatments, that have not before been directly compared with adequate precision, or at all. Network meta-analysis incorporates both direct and indirect effects, stemming from the entire set of evidence. Furthermore, on the basis of valid statistical inference methods, it allows to rank the treatments investigated in order to identify which is the best or worst among them [5].

The idea underlying the network meta-analysis approach is relatively recent. Bucher ( 1997 ) [6] and Hasselblad ( 1998 ) [7] first suggested the use of indirect comparisons when direct comparisons were unavailable, generalizing the meta-analytic methods. Lumley ( 2002 ) [8] proposed the term “network meta-analysis” and the application of linear mixed model approach to cope with multiple treatments. Moreover, Lu and Ades ( 2004 ) [9] conceived an alternative Bayesian approach to perform network meta-analyses for multi-arm studies implementing the Markov Chain Monte Carlo algorithm.

This method of simultaneously comparing all available healthcare interventions is very attractive to clinicians because it can respond to their major concern, namely which treatment is the best or the worst among several alternative ones. However, this technique must be properly mastered before putting it in practice because it is not free from limitations or caveats. Despite the fact that the assumptions and critical points concerning standard pairwise meta-analyses have already been widely discussed and understood [4], the underpinnings and specifics of network meta-analysis may be perceived as more intricate and obscure, potentially leading to misinterpretation [10].

The aim of this work is to recognize and highlight the attractiveness of network meta-analyses, although acknowledging their complex issues and limitations. Accordingly, we investigated the dissemination of network meta-analyses in the clinical literature of discussing the principal areas of application and their general descriptive characteristics.

In the past ten years, the medical literature has witnessed a rapid increase in the possibility to combine evidence coming from a set of treatment comparisons. This can be achieved by performing a network meta-analysis using either a frequentist or a Bayesian approach. Network meta-analysis provides a global estimate of treatment effects for a set of multiple interventions, combining direct and indirect evidences and is particularly useful when pairwise comparisons are not available in the literature [5]. Different treatment effects are analyzed by statistical inference methods and models, as apposed to a weighted average of trial specific effects in the classical approach.

All methodological and statistical issues related to performing a network meta-analysis must be known and handled to avoid bias that can compromise the validity of the analysis. Guidelines and checklists have been developed to assess the quality of reports and to perform systematic reviews, including pairwise and network meta-analyses in the context of decision making [11,12,13,14,15]. It is important to underline that these techniques are potentially much more accurate if they combine (i) studies that are sufficiently homogeneous to be grouped, (ii) interventions and study designs that are sufficiently similar in the target populations and in their effect on the outcomes and (iii) direct and indirect evidences that are sufficiently consistent [15].

Despite the sudden increase in publications concerning network meta-analyses [16, 17], only a restricted part is focused on methodological and statistical aspects [5,6,7,8,9, 14,15, 18,19,20,21,22,23,24,25,26,27,28,29,30]. Furthermore, many issues are still unclear [10, 16, 17, 31,32,33,34,35].

One of the critical points in carrying out a network meta-analysis is related to the evaluation of assumptions. There are three principal sources of variation concerning the modified effect: within-study, between-study and between-comparison variability [5, 11]. Focusing on a single clinical study, within-study heterogeneity is caused by differences in patient features and may occur in trials without accurate eligibility criteria. Quality assessment of included studies [11, 36] may be useful to select those with low risk of bias and to confirm the validity of the network meta-analysis results. Between-study heterogeneity takes place when there are systematic differences in treatment effects across trials. These can be attributed to specific study characteristics, such as differences in choice of outcomes, inclusion criteria, follow-up duration or methods for event adjudication. This variability may be taken into account using a random effect model, adjusting for pre-specified study-level characteristics or planning an appropriate subgroup analysis. In a network meta-analysis, an additional source of variability may however affect the global estimate of treatment effects on outcome. This is due to the different effect of study design, namely the set of treatments compared in a trial (for example the AB, AC, BC, or ABC comparisons). In other words, the network is inconsistent (presence of between-comparison heterogeneity) if the distribution of effect modifiers varies among different designs. This happens, for example, when the treatment effect difference between groups A and B is dissimilar in studies with an AB design when compared to studies which evaluate together the groups A, B and C (ABC design). In this case, it is appropriate to evaluate this source of variation, properly adjusting the statistical model. The modified effect due to between-study or between-comparison variations may be easily investigated adjusting the inferential model for the proper covariates.

Furthermore, the validity of network meta-analysis results may be affected by divergence between the direct and indirect estimate. The indirect estimate, for example BC, for the true difference effect between B and C can be obtained from the direct estimates of A versus B and A versus C, and then suitably compared with the direct estimate stemming from a traditional pairwise meta-analysis [5, 21,22,23, 25, 28,29,30].

Methods

We performed a literature survey to investigate the dissemination of network meta-analyses in the biomedical and clinical setting.

We searched MEDLINE/PubMed for systematic reviews in which any possible approach to perform a network meta-analysis was applied without control for primary studies design. Literature searches were last updated on April 15th, 2014 and included the following search string: (network[tiab] OR ((mixed[tiab] OR multiple[tiab] OR indirect[tiab]) AND (treatment*[tiab] OR comparison*[tiab])) AND meta-analysis[tiab]) NOT (animal[mh] NOT human[mh]). No language restriction was enforced.

References obtained from database and literature searches were first independently examined by two reviewers to identify reviews in which a network meta-analysis or indirect comparison were explicitly used in the comparison of different healthcare interventions according to the articles titles or abstracts. Divergences were resolved by consensus and then, if potentially pertinent, the reference was retrieved as a complete article. After examining full publications, we excluded reports in which: (i) any indirect comparison had not been done, (ii) the work was methodological or descriptive, (iii) the article was a comment, a letter or an editorial style review or (iv) the articles were protocols of a network meta-analysis.

From the included reports, the following data were extracted and collected in a spreadsheet file: author identification, year and journal of publication, PubMed ID, number of treatments and studies included, characteristics of network configuration and nature of primary outcome. Furthermore we selected network meta-analyses with at least four treatments, one closed loop and a dichotomous primary outcome. From these, we extracted data on clinical indications, type of investigated interventions and medical area. We summarized the data extracted in tables and we narratively described our survey.

Results

Our search strategy identified 2,952 unique publications, the titles and abstracts of which were screened for inclusion (Figure 1).

Review profile.

Figures 2 and 3 show, respectively, the time-based distribution of citations generated by the MEDLINE/PubMed search string and of shortlisted studies, highlighting their exponential increase over the years. Eventually, the full text of 340 articles was retrieved, yielding 248 treatment networks which met the inclusion criteria, as some articles provided more than one network meta-analysis set.

Time based distribution of the 2,952 hints generated by the following MEDLINE/PubMed search string: (network[tiab] OR ((mixed[tiab] OR multiple[tiab] OR indirect[tiab]) AND (treatment*[tiab] OR comparison*[tiab])) AND meta-analysis[tiab]) NOT (animal[mh] NOT human[mh]).

Time based distribution of the 248 networks meta-analyses published in MEDLINE/PubMed.

The list of the 92 major exclusions is available in Supplemental table S1.

Click here for additional data file.

The 248 networks meta-analyses were published between 2003 and 2014. The median number of investigated treatments was 5 (1st quartile-3rd quartile: 3-8; minimum-maximum: 2-120) while the median of individual studies included in each network meta-analysis was 10 (1st quartile-3rd quartile: 21-43; minimum-maximum: 2-267). Table 1 shows that more than half of the network meta-analyses analyzed 4-10 treatments (54%) and most of them included 6-15 studies (38%) (Table 1).

Number of treatments and studies included in the 248 networks meta-analysis published in MEDLINE/PubMed (update April 15th, 2014).

Supplemental table S2 reports the descriptive characteristics of the 71 network meta-analyses with at least four treatments, two closed loop and a dichotomous primary outcome.

Click here for additional data file.

Most of them were performed in the cardiovascular setting (25 of 71, 35%) followed by the endocrinology and metabolic disorder setting (9 of 71, 13%), then psychiatry, pulmonology (chronic obstructive pulmonary disease), neurology (6 of 71, 8.5%) and gastrointestinal disease (5 of 71, 7.0%). The types of intervention most frequently analyzed were: coronary stents (8 of 71, 11.3%),antihypertensive drugs (7 of 71, 9.9%), bronchodilator drugs (5 of 71, 7.0%), antidepressant drugs (4 of 71, 5.6%), chemotherapy or radiotherapy (4 of 71, 5.6%), and statins (3 of 71, 4.2%).

Discussion

In the present paper we highlight the increasing diffusion of network meta-analyses in the biomedical literature over the last two decades, and particularly in the last few years. Differently from pairwise meta-analysis, network meta-analysis is focused not only on a single comparison but also on a set of treatments that leads to multiple contrast assessments. As a result, the modified effect of treatments on the outcome may vary internally and across both studies and comparisons.

Conceptual and technical issues concerning network meta-analysis need to be studied and well mastered before carrying out an analysis with this powerful statistical tool. Particular attention to the sources of variation is required in order to avoid invalid conclusions.

Accordingly, network meta-analysis results need to be placed and interpreted in the context of the specific network investigated, and to look only at the target population selected, with pre-specified inclusion criteria, and at the set of treatments brought out [37].

The review we carried out showed that the network meta-analysis was mainly applied to the cardiovascular and pulmonary settings. This is in keeping with some of the peculiar inherent features of these disciplines (e.g. abundance of randomized trials focusing on clinically relevant dichotomous endpoints [e.g. death]) or may have to do with the fact that network meta-analysis pioneers had previously focused (and had continued to focus) on, respectively, cardiovascular disease and pneumology [38,39,40]. However, it is clear that the interest in this type of research synthesis tool is becoming widespread among all medical disciplines, from dermatology to odontology.

It is further true that this specific pattern of uptake of network meta-analysis may be due to the established hierarchy in worldwide causes of morbidity and mortality. Indeed, the World Health Organization (WHO) reported that the most common causes of death were ischemic heart disease, stroke, lower respiratory infections and chronic obstructive lung disease [41]. Accordingly, it is not surprising to see such attention to cardiopulmonary topics among reviewers, journals, and readers.

Given these premises, and stemming from our experience in this field and the comprehensive appraisal of available network meta-analyses, we believe we may offer some succinct guidance to those interested in performing or understanding correctly a network meta-analysis (Figure 4) [37].

Succinct algorithm for conducting a network meta-analysis.

The founding stone of any network meta-analysis is the hypothesis. Then, the design of the review should be explicitly defined and the project should be registered online, whenever possible. Finally, search, selection, abstraction and appraisal follow suite and can be conducted in a fashion similar to pairwise meta-analysis standards.

Notwithstanding the specific analytical subtleties, network meta-analysis can be performed with a single-step approach (including the pairwise analyses in the network ones). However we favor a two-step approach (with pairwise followed by network analyses), which is easier to understand and interpret, and also diminishes the risk of scaring or confusing the reader with a ‘black box’ effect. Accordingly, a pairwise meta-analysis should be conducted, computing effect estimates, appraising heterogeneity and inconsistency, evaluating small study effects, and, if deemed appropriate, subgroup and meta-regression analyses. The network meta-analysis phase can follow smoothly the pairwise one, with estimation of indirect and network effects, appraisal of consistency between direct and indirect estimates, analysis for small study effects, and, if deemed appropriate, subgroup and meta-regression analyses. Reporting then concludes the pairwise and network meta-analytic efforts.

Other applications of network meta-analysis can also be envisioned, such as in umbrella reviews or meta-epidemiologic studies, or for cost-effectiveness analysis, but were beyond the scope of our own systematic and narrative review [42,43]. Our selective search limited to MEDLINE/PubMed translates into adequate internal validity but possibly weak external validity. In addition, we did not include articles published after April 2014, and thus our estimates for the 2014 output are merely informed guesses. Another limitation of the present work is the lack of formal appraisal of review validity (for instance with the AMSTAR tool) or comparative analysis of scholarly citations [44,45]. These goals were also beyond our scope and will surely be interesting in future research efforts.

The final caveat is that network meta-analysis to date remains a very elegant analytical exercise for evidence synthesis, but it is unclear whether any such work can truly, either directly or indirectly, lead to improve patient care and outcomes. Whether this can or should be tested at all also remains uncertain, but would be a very important issue to address. Indeed, even strenuous supporters of meta-analysis frankly acknowledge that a plethora of overlapping meta-analyses with heterogeneous findings may eventually confound and paralyze readers and decision makers.

Conclusion

In conclusion, network meta-analyses are becoming increasingly attractive as they offer a comprehensive framework for decision-making. Whether they will also contribute to improvements in patient outlook remains to be proven.