Assessing the quality of meta-analyses in systematic reviews in pharmaceutical research in Iran by 2016: A systematic review.

Background: Meta-analyses, like all other studies, may be poorly designed and implemented. This study was designed to determine the quality of meta-analyses in systematic reviews in the field of pharmaceutical research in Iran.
Methods: Web of Science Core Collection, EMBASE, Ovid Medline, CINAHL, Scopus, and PubMed were systematically searched on June 4, 2017. The search was limited to the researches in the field of pharmaceutical studies. Based on inclusion criteria, 104 systematic reviews with meta-analysis (SRMA) were selected and assessed using quality assessment tools introduced by Higgins.
Results: Participants, experimental interventions, and outcomes were reported in all the articles. Comparator intervention and study design were correctly reported in 103 (99.04%) and 101 (97.12%) articles, respectively. The comprehensive search strategy was available only in 48 articles (46.16%), and there was no evidence of a comprehensive search in 56 articles (53.84%). Risk of bias was investigated in 78 articles (75%). Also, funnel plots were the most commonly used method for reporting the bias in 64 articles (46.42%).
Conclusion: In many of the meta-analyses, several items of the tool that represented a high-quality meta-analysis were absent. According to the findings, the comprehensive search and quality assessment were not at an appropriate level. Thus, the importance of reproducibility of information and quality assessment of included studies should be emphasized.

↑ What is “already known” in this topic:

The quality assessment of some types of articles is comprehensively implemented and published; however, the quality assessment of systematic reviews with meta-analyses (SRMA) articles has not yet been completely performed since few articles have been published in the pharmaceutical field.

→ What this article adds:

Researchers and readers interested in SRMA studies are aware of the weaknesses of existing articles and pay attention to the results of such articles to conduct their future studies.

Introduction

In the field of medicine, the SRMA articles (systematic reviews with meta-analyses) collate and analyze the results of several studies with similar objectives and results to provide a more valid and reliable estimate of the effect of the drug dose or other interventions (1).

The effect size of the pharmacological interventions is directly related to human health and life in most cases; therefore, the meta-analyzers provide more reliable results through aggregation of similar studies that include larger sample size and a shorter confidence interval and higher accuracy (2).

An appropriate and accurate evaluation of the quality of articles and the publication bias are inseparable parts that make the conclusion of this type of research valid (3). The recommendations and results that meta-analyses offer help researchers in the treatment of diseases based on the accuracy and precision with which the research has been done and written. In addition, adverse events were reported in clinical trials in certain circumstances. Writing adverse events in review articles makes researchers aware of the complications of interventions; therefore, the necessity of increasing the accuracy and comprehensiveness in this type of research is evident (4), to a degree that it is essential to mention the item related to the adverse events of drugs in the quality assessment tools (5). Meta-analyses with inappropriate statistics, poor resource, results with selection bias, and unfavorable access to the internet for sales and advertising increased the importance of paying attention to the quality assessment methods in this area (6, 7). Different biases have been observed in meta-analytical articles sponsored by a specific industry. Transparency and reporting the results in accordance with the aims of the organization are among the most prominent examples (7, 8).

Therefore, it is essential to assess the quality of SRMA studies. Several studies, including review articles and meta-analyses, have reported the use of various tools in quality assessment and measurement; PRISMA (9), AMSTRA (10), and quality assessment tool by Higgins et al are a few examples (11). In recent years, meta-analyses have played a significant role in the development of pharmaceutical science, and the specific analyses of the issues of efficacy and safety have been discussed more than other issues. In the meantime, the need for accuracy is more felt. This study was conducted to assess the quality of Iranian SRMA articles in the field of pharmaceutical research based on a quality assessment tool proposed by Higgins et al (11). Hence, the importance of the results of SRMA articles in medicine and pharmacology is a challenge.

Methods

Search strategy

One of the authors (F S) performed the online search for pharmacological meta-analysis articles in Web of Science Core Collection, EMBASE, Ovid Medline, CINAHL, Scopus, and PubMed (("Iran"[Mesh] OR Iran*[tiab] OR Persia*[tiab]) AND ("Meta-Analysis"[Publication Type] OR "Meta-Analysis as Topic"[Mesh] OR Meta-Analy*[tiab] OR Metaanalys*[tiab])) without date or language limitations on June 4, 2017. The studies were retrieved based on their title, abstract, and authors’ names in SRMA articles. For selective studies, the articles were independently screened by 2 of the authors (F S and A A) and the controversial issues were resolved in a debate.

Study selection and eligibility criteria

The inclusion criteria for the study were as follow:

1) Study design: systematic review and meta-analysis.

Data extraction

Two of the authors (A A and L M) extracted the data from the selected articles and a third author (F S) looked for contradictions. Articles selected for the study were reviewed based on the following criteria: 1. basic information; 2. data sources; 3. analysis of individual studies by the meta-analysis; 4. meta-analysis; and 5. reporting and interpretation.

The tool used in this study were consistent with the AMSTRA and Cochrane Handbook for Systematic Reviews of Interventions, and review articles and expert group were also used. The tool was used in pharmaceutical SRMA studies (11, 12). Unlike other quality assessment tools, this tool emphasizes on statistical and interpretative items. The main purpose of this tool is to evaluate the quality of meta-analyses for efficacy and safety in the pharmaceutical field.

Results

After screening the articles based on the inclusion criteria, 104 SRMA articles in the field of pharmaceutical research published since 2006 were investigated (Fig. 1). The growing trend of the articles over the years can be observed in Figure 2, and this growth was significant during 2015 and 2016 compared to the previous years.

Fig. 1

Fig. 2

PRISMA flow diagram

Distribution of the year of publication

Indicators related to the research question (eg, how to search and select information sources) are presented in Table 1. Regarding the inclusion criteria of the study, except for the study designs (97.1%) and the comparator intervention (99.0%), other elements were completely expressed in relation to the question. Published literature and bibliographic databases were searched in all articles; however, other elements were less explored. Of all articles, 93.2% of data disclosed by industry were not sought, although 43 (41.3%) SRMA articles considered limitations in terms of their characteristics, publication status, or language in literature search. The search strategy was presented in 26 articles (25.5%), not presented in 15 articles (14.5%), and was partially presented in 47 articles (45.1%). Search for evidence was reasonably and comprehensively expressed in 48 articles (46.1%), and 56 articles (53.8%) did not have any evidence for research comprehensiveness. The selection of studies was independently done by 2 or more people in 72 articles (69.23%), by one person in 3 articles (2.88%), and was unstated or unclear in 29 articles (27.89%).

Table 1

Resource search criteria in SRMA articles

Criteria	Frequency (%)
	Yes (%)	No (%)
Eligibility criteria were stated and suitably specific for
Participants	104 (100)	0 (0)
Experimental intervention(s)	104 (100)	0 (0)
Comparator intervention(s)	103 (99.04)	1 (0.99)
Out comes	104 (100)	0 (0)
Study designs	101 (97.12)	3 (2.88)
Further restriction on eligibility on studies or reports?Yes(n=43), No(n=45), unclear (16)
Specify restriction	33 (79.74)
Publication status restriction	6 (13.95)
Language restriction	3 (6.97)
Other restriction	1 (2.32)
Data for meta-analysis were sought from
Published literature	104 (0)	0 (0)
Online repositories	36 (34.62)	68 (65.38)
Correspondence with trialists	7 (6.73)	97 (93.27)
In-house IPD	1 (0.96)	103 (99.04)
Others IPD	0 (0)	104 (100)
The search for trials included
Bibliographic databases	104 (100)	0 (0)
Grey literature	3 (2.88)	101 (97.12)
The web	3 (2.88)	101 (97.12)
In-houses collections	2 (1.96)	102 (98.08)
Reference lists	54 (51.92)	50 (48.08)
Hand searching	11 (10.58)	93 (89.42)
Correspondence with industry	0 (0)	104 (100)
Other correspondence	1 (0.96)	103 (99.04)
Other sources	20 (19.23)	84 (80.77)
Which bibliographic databases are mentioned
PubMed/MEDLINE	103 (99.04)	1 (0.96)
EMBASE	38 (36.54)	66 (63.46)
CENTRAL/Cochrane Library	66 (63.46)	38 (36.54)
Science Citation Database	54 (51.92)	50 (48.08)
other(how many)	86 (86)	14 (14)

The quality assessment of methodology are presented in Table 2. The quality assessment of studies was performed in 78 (75%) SRMA articles, 22 (21.15%) of which were clear, and 4 (3.85%) were unclear in the quality assessment. Scale (eg, Jadad) was used to evaluate quality of methodology in 55 (70.5%) SRMA articles, the Cochrane risk of bias assessment tool was used in 20 (25.7%) articles, and checklist was used in 3 (3.8%) articles.

Table 2

Other clinical trial designs used in the SRMA articles

Study designs	Not found or not mention	Included appropriately	Included inappropriately	Explicitly excluded	Unclear
Cross-over trials	79(75.96)	19(18.27)	2(1.92)	2(1.92)	2(1.92)
Cluster randomized trials	95(91.35)	4(3.85)	0(0)	2(1.92)	3(2.88)
Other study designs	67(64.52)	14(13.46)	6(5.77)	10(9.62)	7(6.73)

In 6 articles (5.44%), a precise method was used to follow the outcome in missing data; in 12 articles (11.54%), no method was presented; and in 86 articles (82.7%), outcomes were unclear and data were lost. Other types of studies in which the authors have been included in their researches are shown in Table 2. Also, comparisons were reasonable in 95 SRMA articles (90.38%), the groups were not selected correctly in 5 articles (4.81%), and comparisons were unclear in 5 articles (4.81%). Moreover, the sensible outcome was reasonable and accurate in 97 articles (93.27%), it was not logical in 2 articles (1.92%), and was unclear in 5 articles (4.81%).

The double counting of individual studies was mentioned in 76 articles (73.08%), not mentioned in 4 articles (3.85%), and was unclear in 24 articles (23.08%). The methods used to assess heterogeneity in the included SRMA studies are shown in Figure 3. Homogeneity was investigated only in 1% of SRMA articles. Synthesis methods in SRMA articles are shown in Table 3. Funnel plot had the highest frequency in reporting the publication bias. Synthesis methods were classic-basic in 79 (76.7%) and classic-advance in 25 (23.3%) of the reviewed articles. Sensible strategy was seen in 53 articles (51%), was unclear in 32 articles (30.8%), and was clear in 19 articles (18.2%). Subgroup analyses were good in 24 articles (23.1%), unclear in 19 articles (18.3%), not applicable in 53 articles (51%), and none in 8 articles (7.7%). The appropriateness of interpretation of the subgroup analyses in SRMA articles was correct in 47 articles (45.2%), unclear in 15 articles (14.4%), not applicable in 31 articles (29.8%), and none in 11 articles (10.9%).

Fig. 3

Table 3

The methods used in risk of bias and reporting

Criteria	Frequency (%)
	Yes (%)	No (%)
Risk of bias (quality assessment) or eligibility criteria include
Generation of allocation sequence	58 (55.77)	46 (44.23)
Concealment of allocation sequence	58 (55.77)	46 (44.23)
Blinding	76 (73.08)	28 (26.92)
Attrition/drop out/ITT	73 (70.19)	31 (29.81)
Other	11 (10.58)	93 (89.42)
The synthesis methods used in the paper included
Pooling (no stratification by study)	101 (97.12)	3 (2.88)
Fixed-effect meta-analysis	57 (54.81)	47 (45.19)
Random-effect meta-analysis	82 (78.85)	22 (21.15)
Fixed-effect meta regression	4 (3.85)	100 (96.15)
Random-effect meta regression	21 (20.19)	83 (79.81)
Tools was used for assessed reporting bias
Funnel plots	64 (64.42)	37 (35.58)
Egger test	39 (37.50)	65 (62.50)
Begg-mazumdar rank correlation test	49 (74.12)	55 (52.88)
Other Funnel plots asymmetry test	1 (0.96)	103 (99.04)
Trim and Fill	3 (2.88)	101 (97.12)
Other	4 (3.85)	100 (96.15)

Methods used to assess heterogeneity

It is essential to focus on the studied variables in SRMA articles to judge the analysis. For example, when a studied variable is continuous, determining the cut-point of the studies is necessary. Certain important indicators considering the type of studied variables are listed in Table 4, and the status of each of these indicators was expressed in the studied SRMA articles.

Table 4

Testing and reporting issues based on data type

Criteria	Yes	Unclear	No	Not applicable
Issues for continuous data
Was the choice of effect size?	81 (77.88)	10 (9.62)	5 (4.81)	8 (7.69)
Was skew of data a potential problem not appropriately addressed?	11(10.58)	46 (44.23)	3 (2.88)	44 (42.31)
Issues for binary data
Were methods appropriate to rare events/sparse data?	4 (3.88)	36 (34.95)	6 (5.83)	57 (55.34)
Were cut-points to dichotomize continuous/ordinal out com justified?	19 (18.27)	29 (27.88)	2 (1.92)	54 (51.92)
Issues for time-to-event data
Were time-to-event data appropriately dealt with?	1 (0.96)	3 (2.88)	0 (0)	100 (96.15)
Issues for ordinal data
Were ordinal data appropriately dealt with?	3 (7.69)	3 (2.88)	0 (0)	93 (89.42)
Indirect comparisons
Were indirect comparisons performed appropriately?	6 (5.77)	9 (8.65)	10 (9.62)	79 (75.96)

Concerning the expression of risk of bias in SRMA articles, it was fully described in 41 articles (39.42%) in the table, fully described in the text of 28 articles (26.92%), but not mentioned or unclear in 35 (33.65%) of the remaining articles. Table 5 presents a general description of the interpretation and expression of the risk of bias in the text of the articles. In addition, the general interpretation of the questions in various sections of the checklist is expressed in Table 5. Only in 12 (11.5%) of the articles, the funding source and the study budget were specified in the text; they were not mentioned in 61 articles (58.7%), and they were unclear in 31 articles (29.8%).

Table 5

Summary judgment in parts of tool

Summary judgment	Yes	Probably yes	Unclear	Probably no	No	Not applicable
Were the review methods adequate such that biases in location and assessment of studies were minimized or able to be identified?	62(59.62)	14(13.46)	4(3.58)	11(10.58)	13(12.50)	0(0)
Were the individual studies analyzed appropriately and without avoidable bias?	32(30.77)	27(25.96)	19(18.27)	15(14.42)	11(10.58)	0(0)
Were the basic meta-analysis method appropriate?	8(7.69)	68(65.38)	7(6.73)	17(16.35)	1(0.96)	3(2.88)
Are the conclusions justified and the interpretation sound?	9(8.65)	67(64.42)	9(8.65)	17(16.35)	2(1.92)	0(0)

Discussion

The present study provided an overview of the trend in the publication of Iranian SRMA articles indexed in the Medline, EMBASE, CINAHL, and Web of Science Core Collection over the course of 11 years, and aimed to identify and improve the quality of reporting such studies. These types of studies use statistical methods to summarize the articles to provide the readers with a glimpse of overwhelming majority of the published articles with different results. Papers published on the quality assessment of the methodology of articles were studied in several ways to either review or evaluate the quality of methodology of meta-analyses in the articles published in journals and specific databases in a particular subject area or to compare them using different tools (13-15). The results of this study examined the methodology, design, and reporting of pharmaceutical meta-analyses with specially designed tools (11).

The results of this study revealed that online literature and online repositories were often used for the meta-analysis data searching, and search in bibliographic databases and reference lists was frequently done for the clinical trials. This information was presumably collected for the availability and familiarity of the researchers and the ease of searching. Insufficient familiarity with databases and searchable grey literature, inefficiency in providing the clinical trial results by the institutions, the publication of articles in non-English journals, nonpublication of articles with nonsignificant results and nonindexing of these types of journals at databases form the potential reasons for the use of full search strategy (16). The SRMA research does not include all available articles. However, the validity of these resources is subject to bias, as there is no peer-review and as the way grey literature is done by the researchers is an issue that influences the results of the study. All of these may lead to the selection of articles with significant results.

Among the bibliographic databases, PubMed/Medline, CENTAL Cochrane, and Web of Sciences were more common. Dahabreh et al (2013) reported that the use of the Cochrane and EMBASE databases increased over time, but this growth was not tangible in Medline. Based on the results of our study, Medline was searched as a resource in all the articles. The use of unpublished literature, conferences, and manual searches remained unchanged (13).

Adie et al (2015) made a comprehensive search in at least 2 databases for only 30% of the articles (17), while the search strategy was fully expressed in 25% of the articles, and partially in 45.19% of the articles. Among all examined articles, only 7% had linguistic limitations. Ho et al (2015), who studied the Cochrane meta-analysis on COPD disease, used only 10% of articles in non-English databases (18).

The risk of bias or quality of assessment was observed in 75% of the included articles in the SRMA, of which 70.5% used the Jadad quality tool. Zhu et al (2016) reported the quality of the articles’ methodology to be 82.3%, in which the risk of bias was assessed based on Cochrane checklist (39.2%) and Jadad scale (20.3%) (15). Adie et al (2015) assessed the risk of bias or the quality of the methodology in one-third of the surgical meta-analyses (17).

Double counting of the individual studies was clearly expressed in 73.08% of the articles. While evaluating the quality of meta-analysis in the field of depression, 5% of the articles were reviewed for double counting of the data (13). In this study, the amount of double counting was 77.2%, and the method of data deduplication check was unclear in 22.8% of them (18). The qualities of reporting articles, bias, and quality assessment have improved over time. Accordingly, the checklists or tools have also been updated. Yao et al (2016) highlighted the quality of meta-analyses. They mentioned that eliminating language limitation in literature search and evaluating the risk of bias would improve the quality of meta-analyses, although articles indexed in the Chinese databases were weaker than the ones indexed in the Cochrane database (19). Appropriate meta-analysis methods, avoiding language limitation, publication bias, and expressing conflict of interests can improve the quality of meta-analysis.

It is possible to determine the publication bias in the included studies using appropriate statistical methods, which may lead to the overestimation of results such as language bias. Articles with positive results were more likely to be published; moreover, the majority of these articles were in English and they probably were retrieved more when searched in databases or journals. The location and inclusion biases were also problematic in these studies (10, 18).

Considering the fact that these articles provide the highest level of evidence based on medicine and decision-making, the lack of bias and accuracy of performance in these articles are more crucial than other articles. Eventually, the results of this study may be affected by the design of the studies similar to the aforementioned studies on topics with conflicting results and challenges. In this study, only 2 articles (1.9%) lacked the application of pooling method, for which it was revealed that meta-analysis was not used. They had only conducted the systematic review, and meta-analysis was written mistakenly in the title or in the design of the study; however, according to a study by Zhu (2016), 4.1% of the articles did not refer to the pooling methods (15).Higher quality of the articles published in journals endorsed quality reporting statement compared to the articles in the journals that did not approve those statements (14, 15). However, preregistration and the published articles in Cochrane journals were also better than the non-Cochrane journals with regards to the quality of meta-analysis and systematic review (13, 17, 20).

The use of random effect and fixed effect in the synthesis methods was 78.85% and 54.81%, respectively, which was similar to the results of the academic financial sponsors of the pharmaceutical interventions compared to the industrial sponsors. Therefore, the synthesis methods were paid less attention in the studies with industrial sponsors. The results of this study on integrating the risk of bias, reporting bias, and multiplicity were slightly different from those of Lane et al study (13).

The mean scores of SRMA articles that only included RCT were more qualitative than the studies that included both observational scores and RCTs (13). Impressing the results of the articles by institutions that provide the funds, especially in pharmacological research, for which a lot of investment and profit is required, may affect the results of research by publishing, not publishing, or modifying the data or results (15, 21).

The attention of certain organizations to the quality of the report, the original articles and review articles has led to an increase in the precision and accuracy of the study design and reporting (22, 23). In this regard, the quality of reporting and performing meta-analyses in the field of medicine in the study of Philibert et al (24) was better than ecological areas. Increasing the quality of the methodology in clinical trials and the validity and reliability enhance the quality of the methodology in SRMA articles. When no comprehensive search was done in meta-analyses or the risk of bias and publication bias was not evaluated, the results were different from the clinical trials in some circumstances (13, 25).

Although the initial focus in pharmaceutical meta-analyses is on the efficacy of interventions, safety is also required in pharmaceutical strategies, which mostly depends on the reliability of the data rather than on the reporting (25). This is required to consider the importance of reporting in the articles compared to the safety of interventions along with their efficacy. Eventually, the sensitivity of the editors in peer-reviewed journals and their knowledge on the methodology of research presumably play a major role in the quality of articles (17, 19). Fleming et al found that the articles published in clinical journals had a high impact considering their received citations. For each value in journal’s IF (impact factor) score, the quality of the methodology of articles increases by 68% (26).

Conclusion

After the approval and publication of CONSORT and PRISMA statements by journals and editorial boards, the quality of the already performed and reported articles was more scientific and identical. These elements were improved in recent years and will even improve further in the upcoming years with further attention of scientific journals and communities. The quality of meta-analyses could improve by using appropriate methods, avoiding language limitation and publication bias, and expressing conflict of interests.

Contribution of methodologists (epidemiologists, biostatisticians, and medical librarians) is essential due to their familiarity with the statistical methods in SRMA articles and information resources for comprehensive search.

The articles were selected only by searching databases, so this was the limitation of the present study. Gray literature and hand searching were not done. In this study, it was assumed that SRMA articles are published in qualified journals indexed in credible databases and many databases were searched to decrease this limitation.

Acknowledgments

The researcher would like to thank Development Unit of Clinical Research, Hormozgan University of Medical Sciences, for their consultation.

Conflict of Interests

The authors declare that they have no competing interests.