The effects of excluding treatments from network meta-analyses: survey.

OBJECTIVE: To examine whether the exclusion of individual treatment comparators, including placebo/no treatment, affects the results of network meta-analysis.
DESIGN: Survey of networks with individual trial data. DATA SOURCES: PubMed and communication with authors of network meta-analyses. STUDY SELECTION AND METHODS: We included networks that had five or more treatments, contained at least two closed loops, had at least twice as many studies as treatments, and had trial level data available. Investigators abstracted information about study design, participants, outcomes, network geometry, and the exclusion of eligible treatments.
RESULTS: Among 18 eligible networks involving 757 randomised controlled trials with 750 possible treatment comparisons, 11 had upfront decided not to consider all treatment comparators and only 10 included placebo/no treatment nodes. In 7/18 networks, there was at least one node whose removal caused a more than 1.10-fold average relative change in the estimated treatments effects, and switches in the top three treatments were observed in 9/18 networks. Removal of placebo/no treatment caused large relative changes of the treatment effects (average change 1.16-3.10-fold) for four of the 10 networks that had originally included placebo/no treatment nodes. Exclusion of current uncommonly used drugs resulted in substantial changes of the treatment effects (average 1.21-fold) in one of three networks on systemic treatments for advanced malignancies.
CONCLUSION: Excluding treatments in network meta-analyses sometimes can have important effects on their results and can diminish the usefulness of the research to clinicians if important comparisons are missing.

Introduction

Network meta-analysis (also called multiple or mixed treatment comparison meta-analysis, MTC) permits the evaluation of the comparative effectiveness of multiple interventions.1 2 This approach has an inherent appeal for clinicians and decision makers as new or existing interventions must be placed within the context of all available evidence.3 4 5 6 Often, those undertaking an MTC will selectively choose interventions to include in the analysis. For example, some MTCs exclude placebo or no treatment from consideration because it is sometimes believed that placebo trials vary over time or are set in favourable conditions to appease regulatory authorities.7 Other MTCs may include only the treatments available in particular settings (for example, a specific country), only those of perceived dose relevance, or (often in the case of industry submissions to health technology assessment bodies) only specific competing treatments.8 To obtain empirical evidence on whether these choices make a difference in the results such as treatment effect estimates and treatment rankings, we examined a sample of complex networks and reanalysed their data after excluding specific treatment nodes.

Methods

Eligibility criteria and retrieval of data from existing networks

We considered networks that had five or more treatments, contained at least two closed loops, had at least twice as many studies as nodes, and had individual trial level data or estimates available. The eligibility criteria aimed to generate a sample of networks that had many treatments and studies and sufficient data to explore the impact of exclusions. We used a systematic literature search that has been published previously that identified potentially eligible networks.9 We also attempted to contact study authors for missing individual data at trial level. We included an additional network from an MTC conducted by our team, where we had direct access to the primary data at trial level. In studies that considered more than one outcome using MTCs, we favoured the efficacy outcome over safety outcomes.

Data abstraction

For each eligible network with available trial level data, we recorded whether the eligibility criteria excluded specific types of active or inactive/control (placebo, no treatment, best supportive care) treatment comparators, and the rationale for such exclusions. We recorded for each network the number of studies, treatments, and loops; the geometry of the network (the distribution of treatments and comparisons thereof in each network); the condition being treated; the primary outcome measure and the statistical effect measure used; and the range of node connectivity (the number of direct comparisons connected to each node). The supplementary figure displays the concepts of loops and connectivity.

Statistical analysis

Regardless of the analyses chosen in the original publications, we analysed each network using random effects Bayesian MTCs with uninformed priors, the most common analytical approach used for network meta-analysis.8 Details on code and specific analysis are available from the authors.

For each network we analysed the complete available data (full model) and also performed analyses excluding one or multiple treatment nodes—that is, disregarding in the calculations data from trials where the excluded nodes were comparators. Firstly, we investigated the effect of excluding the treatment node with the largest expected impact from each network. We used the Brier score to identify the treatment node with the largest expected impact on results.10 The Brier score is the average of the squared differences between the log ratios (odds, relative risk or hazard) estimated with the full treatment network data versus the treatment network data where one or more treatment nodes are excluded. Secondly, we investigated the effect of excluding other single treatment nodes that could be classified as active interventions (that is, not placebo/no treatment). Thirdly, we investigated the impact of excluding placebo/no treatment from the treatment network. Lastly, we focused on selected examples of situation specific exclusions that were chosen owing to perceived relevance for clinical practise or decision making. Specifically, for the network of thrombolytic therapy we excluded data from trials involving anisoylated plasminogen streptokinase activated complex (ASPAC) and urokinase because these treatments were not available in the United Kingdom and one previous UK based network meta-analysis had excluded them11; and for the networks of breast, colorectal, and ovarian cancer, we excluded nodes of treatment regimens that are currently uncommonly used (all those deemed miscellaneous older agents alone or in combination for breast cancer; fluorouracil monotherapy and older drug monotherapy for colorectal cancer; platinum monotherapy and non-platinum/non-taxane drug regimens for ovarian cancer).

We used odds ratios for dichotomous outcomes, rate ratios for Poisson outcomes, and hazard ratios for survival outcomes. From each network we estimated treatment effects for each treatment comparison and the probability of each treatment being the best treatment.12

To evaluate the impact of excluding treatment nodes we calculated the relative change in the estimated treatment effects of the remaining treatments; changes in the top three ranked treatments, according to the probability of being best; and changes in the probability of being the best treatment for the top treatment of the full model. Relative changes in the estimated treatment effects of remaining nodes are always expressed as a value greater than 1.00—for example, if a specific odds ratio is 0.80 in the full network and 0.88 in the network with one node excluded, then the relative change is 0.88/0.80=1.10-fold; if these odds ratios are 0.80 and 0.60, then the relative change is 0.80/0.60=1.25-fold. Given that most treatment effects are relatively small or modest,13 relative changes of 1.10-fold can be considered large and of 1.20-fold can be considered substantial. Choice of reference group does not affect the values of the fold change. As most networks have a natural choice for the reference group we chose to keep this node as a reference group to best mimic changes that would be observed by researchers. For each network with excluded nodes, we noted the maximum and geometric average of the relative change.

Analyses were conducted using WinBUGS version 1.4 (Medical Research Council Biostatistics Unit, Cambridge) and R version 2.15.1 (www.r-project.org/).

Results

Analysed networks

The search identified 890 relevant abstracts, of which 276 were assessed as potentially eligible and their full articles were screened. After adding an eligible network from our own available data, 18 networks with individual trial data met the inclusion criteria,14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 involving 757 randomised controlled trials with 750 possible treatment comparisons and 1 036 701 patients. Table 1 shows the characteristics of the individual networks. Networks ranged in size from 11 studies on five treatments to 128 studies on 22 treatments (figure).

Table 1

 Study characteristics

Network	Type of treatments	Condition/outcome	Consideration/exclusion of specific treatment comparators	No of studies (nodes)	No of loops	Effect measure	Connectivity (min/max)†
Ara 200914	Statins	Hypercholesterolaemia/discontinuation due to adverse events	Only trials involving high dose statins or low dose statins	12 (5)	5	Odds ratio	2/4
Bansback 200915	Biologicals	Moderate to severe plaque psoriasis	All options considered	22 (8)	2	Relative risk*	1/3
Brown 200616	Pharmacological interventions	NSAID induced gastrointestinal toxicity/serious gastrointestinal complications	Only trials involving specific strategies were considered	40 (6)	6	Relative risk*	2/4
Cipriani 200917	Antidepressants	Unipolar major depression in adults/response to treatment	Only trials involving second generation antidepressants were considered (not placebo)	111 (12)	70	Odds ratio	3/11
Cooper 200618	Anticoagulants and antiplatelet therapy	Non-rheumatic atrial fibrillation/ischaemic stroke prevention	All options considered	19 (9)	11	Rate ratio	1/7
Dias 201019	Thrombolytic drugs and angioplasty	Acute myocardial infarction/death	All options considered (previous analysis excluded two treatments not licensed in the UK)	50 (9)	11	Odds ratio	1/7
Elliot 200720	Antihypertensive drugs	Hypertension, high risk patients/patients who developed diabetes	All options considered	22 (6)	16	Odds ratio	4/5
Golfinopoulos 200721	Chemotherapy regimens	Colorectal cancer	Excluded best supportive care, radiotherapy, hormonal therapy, and gene therapy	40 (11)	7	Hazard ratio	1/7
Imamura 201022	Non-surgical	Stress urinary incontinence/cure	Only non-surgical options	38 (13)	26	Odds ratio	1/7
Kyrgiou 200623	Chemotherapy regimens	Ovarian cancer	Excluded best supportive care, radiotherapy, hormonal therapy, and gene therapy	61 (9)	10	Hazard ratio	1/6
Lam 200824	Implantable defibrillator	Left ventricular dysfunction	All options considered	11 (5)	3	Odds ratio	2/4
Mauri 200825	Chemotherapy regimens	Breast cancer	Excluded best supportive care, radiotherapy, hormonal therapy, and gene therapy	128 (22)	27	Hazard ratio	1/10
Mills 201126	COPD drugs	COPD/exacerbations	All options considered	25 (10)	7	Rate ratio	1/6
Mills 201227	Nicotine replacement therapy	Smoking addiction/cessation	All options considered	100 (6)	4	Odds ratio	1/4
Psaty 199728	Antihypertensive therapy	Coronary heart disease/fatal and non-fatal events	Only the six most commonly used antihypertensives	28 (7)	10	Relative risk*	1/5
Sciarretta 201129	Antihypertensive therapy	Heart failure/prevention of heart failure	Excluded trials before 1997	26 (8)	13	Odds ratio	1/6
Thijs 200830	Antiplatelets	Transient ischaemic attack or stroke	Excluded oral anticoagulants, trifusal, cilostazol, dipyridamole alone, and glycoprotein llb/lla inhibitors	24 (5)	3	Odds ratio	2/4
Wang 201031	Central venous catheters	Catheter related infections	Excluded self preparing coated catheters	43 (9)	4	Odds ratio	1/4

NSAID=non-steroidal anti-inflammatory drug; COPD=chronic obstructive pulmonary disease.

*Our analyses use odds ratios to avoid issues of non-transitivity with risk ratios.

†Connectivity reports the smallest and largest number of connections in each network.

Networks included in analysis. The line thickness is proportional to the number of trials comparing the treatments along the edge. Nodes leading to changes in ranks among the top ranked treatments are in large font

Exclusion of treatment nodes in analysed networks

Table 1 shows the eligibility criteria that had been set by the original authors of these meta-analyses and that resulted in exclusion of specific treatment comparators from the network. Eleven of the 18 networks upfront decided not to consider all treatment comparators. One network mentioned that a previous network meta-analysis had excluded data on two treatments (ASPAC and urokinase for thrombolysis) that were not licensed in the United Kingdom, but the full data were available in the current analysis. Several networks deliberately excluded placebos or no treatment options (for example, best supportive care for cancers). Several other networks focused on specific types of treatments that they considered to be of clinical interest and made statements that other treatments were not considered at all, because they were of different class, old (for example, trials published before 1997), or to be considered in a separate review.

Changes after removal of treatment node with largest expected impact

When the node with the largest Brier score was removed (table 2) the average relative change of treatments effects exceeded 1.10-fold in seven of the 18 networks.14 16 18 22 27 29 31 The largest observed relative change exceeded 1.10-fold (maximum 3.64-fold) in all but four networks. In three networks some non-significant effects became significant, and in 12 networks the opposite change was seen. Switches in the ranking of the top three treatments were observed in nine of the 18 networks.16 17 18 21 22 25 27 29 31 Table 3 shows a worked example of the calculations and results for a single node removal in the Cooper 2006 network.18

Table 2

 Nodes leading to greatest change in estimated effects and average fold changes in estimated treatment effects when single treatment nodes are removed

Network	Node leading to largest Brier score*	Connectivity of removed node	Removal of largest expected impact node			Removal of other active intervention nodes
			Average fold change	Largest fold change		Average fold change	Largest fold change
Change in top three ranked treatments:
Brown 200616	Cox 2 preferentials (20%)	3 (max 4)	1.83	3.13		1.38	4.40
Cipriani 200917	Fluoxetine (0%)	11 (max)	1.06	1.18		1.03	1.16
Cooper 200618	Fixed low dose warfarin+aspirin (0%)	4 (max 7)	1.25	2.65		1.06	1.60
Golfinopoulos 200721	Fluorouracil (0%)	5 (max 7)†	1.03	1.05		1.01	1.09
Imamura 201022	Pelvic floor muscle training+biofeedback (5%)	6 (max 7)	1.22	1.40		1.07	2.18
Mauri 200825	Standard dose anthracycline (0%)	10 (max)	1.05	1.12		1.01	1.09
Mills 2012 (6 mth)27	Low dose nicotine replacement therapy (0%)	4 (max)	1.12	1.39		1.01	1.03
Sciarretta 201129	Calcium channel blockers (1%)	6 (max)	1.27	1.53		1.04	1.31
Wang 201031	Chlorhexidine and silver sulfadiazine (0%)	4 (max)	1.27	4.64		1.01	1.08
No changes in top three ranked treatments:
Ara 200914	Simvastatin 40 mg (21%)	4 (max)	1.13	1.19		1.04	1.11
Bansback 200915	Adalimumab (0%)	2 (max 3)	1.08	1.32		1.01	1.02
Dias 201019	Accelerated alteplase (0%)	7 (max)	1.08	1.16		1.03	1.07
Elliot 200720	Angiotensin converting enzyme inhibitors (22%)	4 (max 5)	1.08	1.12		1.02	1.06
Kyrgiou 200623	Platinum monotherapy (0%)	6 (max)	1.08	1.25		1.02	1.13
Lam 200824	Implantable cardioverter defibrillator (9%)	3 (max 4)†	1.03	1.07		1.01	1.04
Mills 201126	Inhaled corticosteroids (0%)	3 (max 6)	1.01	1.02		1.00	1.01
Psaty 199728	Diuretics (10%)	5 (max)	1.06	1.14		1.03	1.08
Thijs 200830	Aspirin (0%)	4 (max)	1.03	1.04		1.01	1.03

*Probability of being best treatment.

†Most connected node that was removed.

Table 3

 Example of fold change calculation and change in rank probabilities for Cooper 200618 network

Treatment	Full model			Reduced model		Fold change*
	Relative rate v placebo (95% CI)	Probability (%)		Relative rate v placebo (95% CI)	Probability (%)
Warfarin:
Adjusted standard dose	0.37 (0.26 to 0.53)	3		0.41 (0.27 to 0.7)	2	1.11
Adjusted low dose	0.34 (0.19 to 0.58)	13		0.34 (0.19 to 0.60)	41	1.00
Fixed low dose	0.76 (0.3 to 1.76)	1		0.93 (0.36 to 2.66)	1	1.22
Aspirin	0.62 (0.43 to 0.86)	0		0.66 (0.47 to 1.01)	0	1.06
Fixed low dose warfarin+aspirin	0.98 (0.6 to 1.67)	0		Removed	Removed	NA
Ximelagatran	0.34 (0.18 to 0.62)	14		0.37 (0.21 to 0.83)	13	1.09
Alternate day aspirin	0.17 (0.01 to 1.15)	66		0.45 (0.03 to 18.48)	38	2.65
Indobufen	0.46 (0.19 to 1.14)	5		0.54 (0.23 to 1.57)	6	1.17

NA=not applicable.

*Average fold change reported in table 2 is averaged on log scale. The average of these fold changes is 1.33, slightly higher than 1.25.

The most influential nodes were typically highly connected. In 10 of the 18 networks they actually represented the most connected node. However, the most influential nodes were never the top ranked treatment of the networks and almost always (15/18) had a 10% or less probability of being the best treatment (0% probability in 11/18 networks). Thus standards of care, placebo, or other treatments that may easily be overlooked represent nodes of which the exclusion could be influential to the network results.

Changes of removal after other active intervention nodes

Removal of the remaining, less influential nodes caused minor changes (average relative change ≤1.03-fold) in 13 of the 18 networks. None the less, seven networks contained a node the exclusion of which caused a relative change greater than 1.10-fold. Of the nine networks with changes in the top ranks by the exclusion of the most influential node, three had an additional node whose removal affected the top ranks.16 18 29

Changes after placebo node removals or context specific node removals

Table 4 presents the results after removal of placebo/no treatment nodes or after removal of context specific sets of treatment nodes. Placebos and no treatment nodes were removed from 10 networks that had originally included such nodes. This resulted in important changes in the results of four of these 10 networks,16 18 22 27 with average relative changes ranging from 1.16-fold to 3.10-fold, whereas in the other six networks the average relative changes were minor (1.01-1.03-fold). In one network, exclusion of the placebo node resulted in a switch in position between the first and second best treatment. In the other cases, the ranking of the top three treatments did not change, but modest to large changes were seen in the probability of being the best treatment, such as in Mills 2011 in which the probability of high dose nicotine replacement being the best treatment went from 58% to 90% (table 4).26

Table 4

 Average fold changes in treatment effects and impact on treatment ranks and probability of best treatment with clinically relevant removals

Network	Nodes deleted	Average fold change	Largest fold change	Change in top three treatment ranks	Change in probability of best treatment
					Full model	Reduced model
Changes after removal of placebo/no treatment nodes:
Ara 200914	Placebo	1.03	1.05	None	Rosuvastatin (53%)	Rosuvastatin (56%)
Brown 200616	Placebo	3.10	4.44	None	Cox 2 inhibitors (41%)	Cox 2 inhibitors (43%)
Cooper 200618	Placebo	1.16	1.50	None	Alternate day aspirin (66%)	Alternate day aspirin (79%)
Elliot 200720	Placebo	1.02	1.06	None	Angiotensin receptor blocker (77%)	Angiotensin receptor blocker (83%)
Imamura 201022	No treatment	1.21	1.77	None	Pelvic floor muscle training+biofeedback+bladder training (61%)	Pelvic floor muscle training+biofeedback+BT (56%)
Mills 201126	Placebo	1.01	1.02	None	Roflumilast+long acting muscarinic antagonists (49%)	Roflumilast+long acting muscarinic antagonists (56%)
Mills 201227	Placebo	1.22	1.40	None	High dose nicotine replacement therapy (58%)	High dose nicotine replacement therapy (90%)
Psaty 199728	Placebo	1.01	1.03	None	Angiotensin converting enzyme inhibitors (55%)	Angiotensin converting enzyme inhibitors (43%)
Sciarretta 201129	Placebo	1.03	1.06	Top two switch	β blockers (29%)	Angiotensin converting enzyme inhibitors (32%)
Thijs 200830	Placebo	1.01	1.03	None	Aspirin+dipyridamole (80%)	Aspirin+dipyridamol (70%)
Changes after removal of situation specific relevant treatment nodes
Dias 201019	ASPAC and urokinase	1.04	1.06	None	Percutaneous transluminal coronary angioplasty (83%)	Percutaneous transluminal coronary angioplasty (96%)
Golfinopoulos 200721	Monotherapies: fluorouracil, oxaliplatin, irinotecan	1.06	1.18	None	Fluorouracil and leucovorin+irinotecan+bevacizumab (63%)	Fluorouracil and leucovorin+irinotecan+bevacizumab (48%)
Kyrgiou 200623	Platinum monotherapy and non-platinum/non-taxane drugs	1.21	1.68	None	Platinum+taxane based combination (ip) (92%)	Platinum+taxane based combination (ip) (89%)
Mauri 200825	All drugs deemed miscellaneous	1.05	1.17	3rd becomes 1st	Novel non-taxane agents+taxanes (24%)	Taxanes (combination regimen) (21%)

APSAC=anisoylated plasminogen streptokinase activated complex; ip=including at least one agent given intraperitoneally.

Removal of APSAC and urokinase from the network on thrombolysis and percutaneous intervention did not result in major changes. Removal of uncommonly used treatment regimens from the networks of treatments for malignancies resulted in a large change in estimated effects (on average by 1.21-fold) for ovarian cancer and small changes for colorectal and breast cancer. However, in the case of advanced breast cancer, these removals switched the ranks of novel non-taxane agents+taxanes and taxanes (combination regimen), the first and third ranked treatments, respectively.

Discussion

In this metaepidemiological study involving 18 large network meta-analyses, we found that it is common for networks to exclude trials and specific comparators based on widely diverse criteria. Furthermore, we documented that exclusions occasionally can have an important impact on the results. The comparators that had the largest potential to change the results were typically those used in many trials; and they were unlikely to be the best treatments. We also explored the impact of exclusion of each other active treatment and placebo/no treatment as well as the impact of excluding several treatments based on scenarios that may be clinically relevant. Our results show, moreover, that some of these exclusions could affect substantially the estimated treatment effects and occasionally even affect the ranking of the top treatments.

These findings seem particularly relevant to network meta-analysis where analysts choose only to evaluate certain newer treatments or where they have chosen to exclude well established interventions or placebo, or both from a network. Readers of network meta-analysis should examine whether a network represents all available interventions. Excluding treatments from networks may reduce the applicability and usefulness of a multiple or mixed treatment comparison (MTC) if comparisons of interest to doctors are not considered. A substantial literature already exists on traditional pairwise meta-analyses, showing how eligibility criteria may result in differences in the results and conclusions of meta-analyses on the same topic.32 33 34 35 36 As meta-analyses have become popular, most of the topics of interest are addressed by two or more published meta-analyses and these may differ in their eligibility criteria.37 Some examples have started to accumulate where independent network analyses on the same topic may reach different conclusions.38 A key consideration may be which treatment nodes are considered eligible for analysis. For example, on evaluating the relative merits of second generation antidepressants, an MTC that did not consider placebo comparisons yielded different rankings from one that included only placebo controlled comparisons and one that considered both types of comparisons.17 39 40 41 We should also note that although treatment rankings and probabilities are arguably easy to interpret, their interpretability is limited by the fact that they are driven predominantly by the estimated effect sizes, and that standard errors play an unduly small role in determining their position.32 Readers are advised to observe the estimated effects first and use the rankings only as a supplementary measure.

Limitations of this study

There are some caveats to our analysis. Firstly, we excluded simple networks such as star networks (that cannot be analysed in an MTC)42 and poorly connected networks. Less well connected networks with less total evidence are likely to be more “fragile” when nodes are removed—that is, the results may change even more from what we documented here for well connected networks. Secondly, we applied a uniform approach—a random effects analysis using non-informative priors—to our reanalysis of each published network meta-analysis. Different analytical choices may introduce some further variability in the results. Thirdly, some of the examined networks had already excluded specific trials and treatments upfront, and it was not practical for us to try to retrieve the excluded trials and evaluate the impact of their inclusion. Conversely, we focused on the potential impact that further exclusions would have had on the results. Fourthly, we did not question the clustering of treatment regimens into specific nodes by the original meta-analysts. However, in some MTCs there may be some ambiguity on whether slightly or modestly different regimens (for example, different doses or schedules for administering drugs or drugs belonging to the same class) should be treated as a single node or separate nodes. This could introduce some additional variability in the results, depending on what choices are made. For example, in the MTC of systemic treatment for colorectal cancer,21 242 trials were identified but only 37 could be used, because the others were deemed to compare regimens that belonged to the same treatment node and thus would not be distinguishable. In all, given these reasons our estimates of the potential impact on the results of choices pertaining to the geometry of the network are probably conservative. To avoid potential bias, network meta-analysts should consider the inclusion of all relevant interventions for the condition of interest.

We should acknowledge that the choice of including all possible interventions that have ever been evaluated in randomised controlled trials on a topic can be daunting. Sometimes specific exclusions are clearly justifiable, and many of these exclusions may not have any substantial impact on the estimated treatment effects of the remaining “relevant” interventions. However, exclusions of potential nodes and decisions about eligibility criteria need to be carefully justified—for example, it may be argued that alternative medicine or non-mainstream comparators should not be included in the same network as standard interventions. It may even be reasonable to perform sensitivity analyses to examine the impact of the removal of specific nodes. Moreover, our findings indicate that the largest impact on the results occurs when well connected nodes are removed. It seems reasonable to advocate that the most evaluated treatments available for a condition should be considered necessary to include for a network to be valid. Typically, this includes older established standards of care, placebo, and well-tested interventions. In particular, some industry sponsored meta-analyses increasingly focus on target competitor agents that compete for market share.43 However, the exclusion of other interventions in a network can importantly affect the results.

Network meta-analysis is an increasingly popular method that allows the comparative effectiveness of multiple treatments to be evaluated

Examples have started to accumulate where network analyses on the same topic have reached different conclusions based on the exclusion of treatment nodes

It is common for networks to exclude trials and specific comparators based on widely diverse criteria

Excluding treatments from a network meta-analysis can importantly change effect estimates and the probability rankings of being the best treatment

Well connected treatments that are unlikely to be the best treatment are the most likely to be influential