Methodological quality of meta-analyses on treatments for chronic obstructive pulmonary disease: a cross-sectional study using the AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool.

BACKGROUND: Meta-analysis (MA) of randomised trials is considered to be one of the best approaches for summarising high-quality evidence on the efficacy and safety of treatments. However, methodological flaws in MAs can reduce the validity of conclusions, subsequently impairing the quality of decision making. AIMS: To assess the methodological quality of MAs on COPD treatments.
METHODS: A cross-sectional study on MAs of COPD trials. MAs published during 2000-2013 were sampled from the Cochrane Database of Systematic Reviews and Database of Abstracts of Reviews of Effect. Methodological quality was assessed using the validated AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool.
RESULTS: Seventy-nine MAs were sampled. Only 18% considered the scientific quality of primary studies when formulating conclusions and 49% used appropriate meta-analytic methods to combine findings. The problems were particularly acute among MAs on pharmacological treatments. In 48% of MAs the authors did not report conflict of interest. Fifty-eight percent reported harmful effects of treatment. Publication bias was not assessed in 65% of MAs, and only 10% had searched non-English databases.
CONCLUSIONS: The methodological quality of the included MAs was disappointing. Consideration of scientific quality when formulating conclusions should be made explicit. Future MAs should improve on reporting conflict of interest and harm, assessment of publication bias, prevention of language bias and use of appropriate meta-analytic methods.

Introduction

Chronic obstructive pulmonary disease (COPD) is a chronic respiratory disease distinguished by airway limitation.[1,2] COPD is estimated to rank fifth worldwide in disease burden,[3] and it is estimated to increase in the coming decades as a consequence of increasing exposure to environmental risk factors and an ageing population.[2,3] More than 5% of the population is affected with the disease and it is associated with high mortality and morbidity.[4] Because of the high prevalence and chronicity, COPD patients rely heavily on medical interventions. For stable COPD, a wide variety of pharmacological treatments are used.[3] Non-pharmacological and rehabilitation strategies are also commonly used to improve the functional capacity of COPD patients.[3]

There are numerous treatments available for COPD and it is important for clinicians to make a decision on their use on the basis of up-to-date clinical evidence.[5] Meta-analysis (MA) of randomised trials is considered to be one of the best approaches for summarising high-quality evidence on the efficacy and safety of various therapeutic options.[6] Although these MAs are published continually as trustworthy sources of evidence, little attention is paid to their methodological quality. Methodological flaws in the conduct of MAs could cause biased conclusions. For example, an incomprehensive literature search could lead to failure in locating unpublished data. This omission may cause changes in the direction of effect for a particular treatment.[7,8] Other methodological shortcomings that can lead to biased results include the lack of independent audit during data extraction, inappropriate combination of heterogeneous study results, and failure to consider risk of bias among primary studies. As biased results from MAs can mislead clinical practice,[9] it is important to assess the methodological quality of MAs on COPD treatment. Results will provide an overview of MA quality in the field of COPD, and accordingly suggestions for future methodological improvement can be made.

AMSTAR (Assessing the Methodological Quality of Systematic Reviews) is a freely accessible validated tool for assessing the methodological quality of MAs.[10,11] The full version can be found in the AMSTAR official website.[10] Validation studies[9,11] have shown that AMSTAR is a reliable critical appraisal tool with good agreement, construct validity and feasibility. The AMSTAR is adopted by a number of research and health technology assessment groups, such as the Canadian Agency for Drugs and Technologies in Health and The Cochrane Effective Practice and Organization of Care Group.[12]

Using a cross-sectional study design, this study aimed to (1) describe the bibliographical characteristics of MAs on COPD therapies, (2) evaluate the methodological quality of MAs on COPD treatments by using AMSTAR and (3) examine the association between bibliographical characteristics and methodological quality.

Materials and Methods

Sampling of meta-analyses

We sampled MAs by searching the Cochrane Database of Systematic Reviews (CDSR) and Database of Abstracts of Reviews of Effect (DARE) using the keyword ‘COPD’. CDSR enabled us to locate Cochrane reviews, whereas DARE is an extensive, weekly updated systematic review database that indexes non-Cochrane MA. The use of these databases allowed us to assemble a representative sample of MAs for appraisal.

Inclusion and exclusion criteria

MAs on COPD treatments published between January 2000 and April 2013 were eligible. Narrative reviews, systematic reviews with no meta-analyses, network meta-analyses and systematic reviews on respiratory diseases other than COPD (e.g., asthma, emphysema, chronic bronchitis, bronchiectasis and so on) were excluded. Two authors independently screened the titles, abstracts and full texts of the retrieved literature to assess their eligibility. Discrepancies were discussed and solved by discussion and consensus among the authors.

Extraction of bibliographical characteristics and critical appraisal

Bibliographical characteristics were assessed with a pre-designed questionnaire. All 11 items of the AMSTAR instrument were used for appraising methodological quality.[11] Judgements were given as either ‘yes’ or ‘no’ for items 1, 2 and 5–11. For items 3 and 4, an option from ‘yes’, ‘cannot answer or not reported’ or ‘no’ was chosen. No aggregate score was calculated, as the original version of AMSTAR did not provide any recommendation on using this approach[10] and there is no empirical evidence to advise on the appropriate weighting for each item when calculating a total score.[9] A detailed operational guide for AMSTAR can be found in Supplementary Material 1.

Data analysis

The associations between bibliographical characteristics and scoring on AMSTAR items were analysed using multivariate logistic regression or multi-nominal logistic regression depending on the scaling of the dependent variable. Hosmer and Lemeshow tests were performed to evaluate the model fitting for all logistic regressions. All statistical analyses were conducted with SPSS 18.0, with a two-tailed significance level of 0.01 to avoid multiple test bias.

Results

MA searching and selection

The search strategy identified a total of 137 systematic reviews (73 from CDSR and 64 from DARE). No duplicates were found. After assessing the titles, abstracts and full texts, a total of 79 MAs met our inclusion criteria and were included in this study (Figure 1). A full list of included MAs can be found in Supplementary Material 2.

Figure 1

Sampling of meta-analyses on COPD treatment: flow chart.

Bibliographical characteristics and methodological quality

The median year of publications was 2010 (n=44, 55.7%). There were more MAs published in the field from 2010 onwards. Fifty (63.3%) MAs had an impact factor of >5. Detailed results for bibliographical characteristics and methodological quality are shown in Tables 1 and 2.

Table 1

Bibliographical characteristics of 79 included meta-analyses on COPD treatments

Bibliographical characteristics	Results
Cochrane review	47 (59.5%)
An update of a previous meta-analysis (MA)	49 (62.0%)
Median impact factor of the journal for which the MA was published (range)	5.18 (0.00–6.25)
Median number of review authors (range)	4.00 (2–8)
Location of corresponding author
North America	21 (26.6%)
Europe	27 (34.2%)
Australasia	21 (26.6%)
Other regions	10 (12.7%)
Type of treatment
Pharmacological	43 (54.4%)
Non-pharmacological	36 (45. 6%)
Total number of included primary studies	1,099
Total number of participants in included primary studies	801,294
Number of MA that reported harm of the intervention	46 (58.2%)
Funding location of the MA
Europe	23 (29.1%)
Australia or New Zealand	9 (11.4%)
Multiple region	9 (11.4%)
North America	8 (10.2%)
South America	1 (1.3%)
Not reported	29 (36. 7%)
Number of MA that declared conflicts of interest	15 (19.0%)
Number of MA that declared no conflicts of interest	26 (32.9%)
Number of MA that did not mention conflicts of interest	38 (48.1%)
Number of MA that searched international databases	78 (98. 7%)
Median number of international databases searched (range)	4.00 (0–12)
Number of MA searched non-English databases	8 (10.1%)
Reporting of coverage year of search:
Yes, reported both starting and ending years	26 (32.9%)
Partially, only reported starting years	48 (60.8%)
Not mentioned	5 (6.3%)
Search terms reported for one or more electronic databases
Topics/free text/keywords/MeSH	62 (78.5%)
Full Boolean	14 (17.7%)
Readers are referred elsewhere for full search strategy	3 (3.8%)
Eligibility of study design
RCT only	75 (94.9%)
RCT and observational studies	4 (5.1%)
Eligibility criteria based on language of publication
Included English publications only	11 (13.9%)
English and languages other than English	42 (53.2%)
Language criteria not reported	26 (32.9%)
Number of MA that included a PRISMA-Iike flow diagram	37 (46.8%)
Tools for assessing risk of bias of primary studies
Cochrane risk of bias tool	31 (39.2%)
Jadad scale	16 (20.3%)
Pedro scale	6 (7.6%)
Others	4 (5.1%)
Two or more of the above-mentioned tools	13 (16.5%)
Not reported	9 (11.4%)

Abbreviations: COPD, chronic obstructive pulmonary disease; MA, meta-analysis; RCT, randomised controlled trials.

Table 2

Methodological quality of 79 included meta-analyses on COPD treatments

Individual AMSTAR items	Yes (%)	No (%)	Cannot answer (%)
1. Was an 'a priori' design provided?	53 (67.1)	26 (32.9)	NA
2. Was there duplicate study selection and data extraction?	61 (77.2)	0 (0.0)	18 (22.8)
3. Was a comprehensive literature search performed?	73 (92.4)	3 (3.8)	3 (3.8)
4. Was the status of publication used as an inclusion criterion (i.e., grey literature would be included, if located)?	47 (59.5)	5 (6.3)	27 (34.2)
5. Was a list of studies (both included and excluded) provided?	48 (60.8)	31 (39.2)	NA
6. Were the characteristics of the included studies provided?	69 (87.3)	10 (12.7)	NA
7. Was the scientific quality of the included studies assessed and documented?	65 (82.3)	14 (17.7)	NA
8. Was the scientific quality of the included studies used appropriately in formulating conclusions?	14 (17.7)	65 (82.3)	NA
9. Were the methods used to combine the findings of studies appropriate?	39 (49.4)	40 (50.6)	NA
10. Was the likelihood of publication bias assessed?	28 (35.4)	51 (64.6)	NA
11. Were the sources of support for both the systematic review and the included primary studies reported?	2 (2.5)	77 (97.5)	NA

Abbreviations: AMSTAR, Assessing the Methodological Quality of Systematic Reviews; COPD, chronic obstructive pulmonary disease; NA, not applicable.

Association between bibliographical characteristics and methodological quality: multivariate analyses

Table 3 illustrates the results from logistic regression analyses (full results can be found in Supplementary Material 3). A higher journal impact factor was significantly associated with favourable methodological scoring on the following aspects: an ‘a priori’ design provided in the MA; duplicate study selection and data extraction; and provision of study lists. MAs published more recently were more likely to have conducted an assessment of the scientific quality of included studies, as well as an evaluation of publication bias. MAs on non-pharmacological treatments performed better in considering the scientific quality of included studies when formulating conclusions, compared with their pharmacological counterparts. Finally, MAs on non-pharmacological treatments also fared marginally better in using appropriate meta-analytic methods for combining findings. P values of Hosmer–Lemeshow tests for all logistic regression analyses were >0.05.

Table 3

Association between publication characteristics and methodological quality of MAs on COPD treatments: multivariate analyses

AMSTAR item (dependent variable)	Predictors	Adjusted odds ratio (99% CI)	P values
1. Was an 'a priori' design provided? (Yes versus No)	Higher impact factor	4.22 (1.50–11.86)	<0.0001
2. Was there duplicate study selection and data extraction? (Yes versus Cannot answer)	Higher impact factor	2.01 (1.19–3.38)	0.001
5. Was a list of studies provided? (Yes versus No)	Higher impact factor	6.85 (1.58–29.69)	0.001
7. Was the scientific quality of the included studies assessed and documented? (Yes versus No)	More recent publication years	1.40 (0.94–2.08)	0.030
8. Was the scientific quality of the included studies used appropriately in formulating conclusions? (Yes versus No)	Non-pharmacological treatment (pharmacological treatment as reference)	6.53 (0.92–46.36)	0.014
9. Were the methods used to combine the findings of studies appropriate? (Yes versus No)	Non-pharmacological treatment (pharmacological treatment as reference)	4.49 (0.98–20.72)	0.011
10. Was the likelihood of publication bias assessed? (Yes versus No)	More recent publication years	1.83 (1.13–2.97)	0.001

Abbreviations: AMSTAR, Assessing the Methodological Quality of Systematic Reviews; CI, confidence interval; COPD, chronic obstructive pulmonary disease; MA, meta-analysis.

Discussion

Main findings

This study assessed the methodological quality of 79 MAs on COPD treatment published between 2000 and 2013. The proportion of MAs that fulfilled each individual AMSTAR domain was similar to that of other medical fields like urology, orthodontics and nursing.[13-15] An overview of results from these studies is shown in Table 4. The majority of MAs were published after 2010, demonstrating an increasing popularity of systematic review methods among researchers in the field. The quality of more recent MAs in terms of assessing and documenting the scientific quality of the included studies was higher, reflecting progress in methodological sophistication. The frequency of updating was acceptable as >60% of the included MAs represented an update of a previous report. Higher impact factor is associated with clarification of an ‘a priori’ design, and this trend could be explained by the requirement of Cochrane MA in publishing a protocol. Other MA authors should consider registering their MA protocols on PROSPERO, which is an international online platform for prospective registration of systematic review protocols.[16]

Table 4

Methodological quality of meta-analyses in other health-care fields assessed by AMSTAR

Study	Fields	No.	Main results
MacDonald et al. [13]	Urology	57	49.1% SRs searched at least two databases; 31.6% SRs searched unpublished studies; 45.6% SRs provided a list of included and excluded studies; 63.2% SRs assessed and documented the methodological quality of included studies.
Papageorgiou et al. [14]	Orthodontics	110	20.0% clearly reported only the review question or only the inclusion criteria; 35.5% conducted in duplicate only study selection, but not data extraction; the grey literature was not scanned for relevant articles in 54 reviews (49.1%). 65.5% did not provided excluded studies; 8.2% did not provide included or excluded studies in a list or a table at all.
Seo and Kim[15]	Nursing	22	13.6% SRs were performed in duplicate study selection and data extraction; 18.2% SRs used publications status as an inclusion criterion; 63.6% SRs did not provide information on both included and excluded studies; 13.6% SRs assessed and documented the quality of the included studies and drew an appropriate conclusion reflecting the scientific quality of the included studies; 72.7% SRs appropriately combined the findings of studies using meta-analytic methods.

Abbreviations: AMSTAR, Assessing the Methodological Quality of Systematic Reviews; SR, systematic review.

Interpretation of findings in relation to previously published work

Despite these progresses, the overall methodological quality of MAs on COPD treatment can only be considered disappointing. Only 17.7% of included MAs explicitly considered the scientific quality of primary studies when formulating their conclusions. As trials with high risk of bias are more likely to report larger effect sizes,[17] readers of MAs on COPD treatments may consider reported conclusions more judiciously. Even when the trial’s risk of bias had been assessed, <40% of MAs had used the Cochrane risk of bias tool, a specialised instrument developed for appraising RCTs.[18] In the future, proper use of Cochrane risk of bias tools should be promoted and explicit consideration on the risk of bias of primary studies and its impact on the treatment results should be made.

These improvement measures are especially important for MAs on pharmacological treatments as it appears that they perform less well compared with non-pharmacological MAs. Less than satisfactory performance in this aspect could be related to the influence of funders, as previous studies have clearly shown the relationship between industry funding and positive results from MAs.[19-21] Another area of concern is the lack of reporting on conflict of interest among MA authors: in our results, only two MAs reported the funding sources for all included primary studies, as well as for the MA itself. More than 48% of the MA authors did not declare a conflict of interest. In addition, harmful effects of treatment were also under-reported in the sampled MAs. A total of 1,099 studies were summarised in these MAs, representing data from more than 800,000 patients. Despite the wealth of data reviewed, only 58% described the harmful effects of treatment. In the future, it is important for journal editors to encourage reporting in these aspects.

A substantial proportion of included MAs (64.6%) did not assess the likelihood of publication bias. As studies with positive results are published more often and more quickly compared with negative studies regardless of methodological quality,[22,23] an overestimation of COPD treatment effect could be observed among the MAs.[24] Treatment effect could be overestimated if publication bias existed even when the included individual trials have a low risk of bias.[25] To reduce the likelihood of overestimating results, in the future, MAs should explicitly assess publication bias using proper methods,[26] and appropriate adjustment on the meta-analytic results can be considered if publication bias is present.[27]

In addition to publication bias, language bias also increases the risk of overestimation as positive findings are more likely to be published in English language journals.[28] In our MA sample, only 10% of the MAs published had searched non-English databases. Building an international, multi-lingual team for conducting MAs could reduce English language bias by widening the coverage on non-English databases. Currently, more than half of the corresponding authors are from North America and Europe, and the review team size is rather small, with a median of four people. Adding authors from non-Western countries could be a solution for reducing language bias.

Strengths and limitations of this study

One of the strengths of this study was that we identified all MAs indexed in CDSR and DARE. The CDSR included all Cochrane Reviews prepared by Cochrane Review Groups within the Cochrane Collaboration, whereas DARE covers the majority of indexed non-Cochrane MAs.[29] One limitation of the present study is that the AMSTAR appraisal process was difficult to implement when the reporting quality was poor. This could be attributed to space restrictions in non-Cochrane journals, which limit detailed reporting.[30] Authors and journal editors are encouraged to adhere to the PRISMA requirement, which includes all required reporting components for authors to publish a systematic review.[31]

Conclusions

Our findings demonstrate that the methodological quality of MAs on COPD treatment is disappointing. Areas that require improvement in the future include the consideration of scientific quality when formulating conclusions, reporting of conflict of interest, assessment of publication bias, and prevention of language bias. Also, the harmful effects of treatment were often under-reported. Clinicians should be judicious when applying the conclusions of MA results to patients, taking into account the potential bias resulting from methodological limitations. Journal editors and peer reviewers will have a strong role in ensuring continual improvement of MA quality by upholding the methodological and reporting standards recommended by AMSTAR and PRISMA.[31]