Assessing the methodological and reporting quality of network meta-analyses in Chinese medicine.

BACKGROUND: An increasing number of network meta-analyses (NMAs) in traditional Chinese medicine (TCM) have been published recently, but the quality of them was lack of assessment. This study aims to evaluate the methodological and reporting quality of NMAs in TCM.
METHODS: Six electronic databases, including PubMed, the Cochrane Central Register of Controlled Trials (CENTRAL), Embase, China National Knowledge Infrastructure (CNKI), Wanfang and Chinese Biomedical Literature Database (CBM) from inception to January 2018, were searched. NMAs of TCM were included. A measurement tool to assess the methodological quality of systematic reviews (AMSTAR) and the PRISMA Extension Statement for Reporting of Systematic Reviews Incorporating Network Meta-analyses of Health Care Interventions (PRISMA-NMA) were used to assess the methodological and reporting quality of the included NMAs.
RESULTS: A total of 40 NMAs, including 2535 randomized controlled trials (RCTs), were included. They were published between December 2012 and November 2017. The median score and interquartile range of methodological and reporting quality was 7 (6-8) and 22 (19.1-27.1). Serious methodological flaws existed in the following aspects: the status of publication (22.5%), a list of studies provided (0%), assessment of publication bias (37.5%), and conflicts of interest (12.5%). Several items need to be improved in reporting, especially for Protocol and registration (2.5%), Data items (22.5%), Risk of bias across studies (Methods section) (37.5%), Results of individual studies (27.5%), Risk of bias across studies (Results section) (40%), Results of additional analyses (35%), and Funding (15%).
CONCLUSIONS: The methodological and reporting quality of NMAs in TCM is moderate. Identified shortcomings of published NMAs should be taken into consideration in further trainings of authors and editors of NMAs in TCM. Future researchers should be encouraged to apply PRISMA-NMA, and a recognized tool for the assessment of NMA methodology was wanted.

Introduction

Traditional Chinese Medicine (TCM) has a history of over 2000 years, and plays an important role in the healthcare system of China. Chinese medicine has several advantages over Western medicine, such as multitargets, multi-ingredients, and low cost. A group of network pharmacology methods appeared to predict the target profiles and pharmacological effects of Chinese medicine, to screen synergistic multicompounds from Chinese herbal formulae, and to illuminate the combinatorial rules and network regulation effects of Chinese medicine.[

A number of TCM treatments have been proved to be of significant efficacy,[ yet those studies are still lack of hard evidence. Well-conducted systematic reviews and meta-analyses of randomized controlled trials (RCTs) are considered the most valid research evidence to formulate policy and practice.[ However, meta-analyses can only compare the effect of head-to-head comparison interventions, and sometimes this high-quality evidence may not exist since direct evidence is often lacking.[ Network meta-analyses (NMAs), which were also called multiple treatment or mixed treatment comparison meta-analyses, can summarize a coherent and comprehensive set of comparisons based on all of the available evidence.[ NMAs could estimate the effectiveness of all the relevant interventions and rank them in order even though direct comparisons are lacking.[ NMAs are becoming increasingly popular as a new generalization of evidence synthesis toolkit which could make decisions or choices better than pairwise meta-analyses.[ NMAs are subject to similar methodological risks as traditional meta-analyses; however, it is recognized that NMAs may be affected by more risks due to its complexity of methodology.[ Several researches on the quality of NMAs have been conducted and showed that significant limitations exist in both the conduct and reporting of NMA, especially for statistical methodology and analytical process.[

About 30 tools have been used for the assessment of methodological quality of systematic reviews or meta-analyses recently.[ However, no recognized tool has been developed especially to assess the methodological quality of NMAs currently. AMSTAR (a measurement tool to assess the methodological quality of systematic reviews) may be the most commonly used tool for the methodology assessment of systematic reviews due to its good validity, reliability, and responsibility.[

Recently, there is an increasing number of NMAs in TCM published, but their quality was lack of evaluation. This study aimed to assess the methodological and reporting quality of NMAs in TCM.

Method

Ethics approval

Ethical approval and patient consent are not required since this study is an overview completely based on the published NMAs.

Literature search

Six electronic literature databases, including PubMed, the Cochrane Central Register of Controlled Trials, Embase, China National Knowledge Infrastructure (CNKI), Wanfang and Chinese Biomedical Literature Database (CBM), were searched from inception to January 2018. Searching terms were used as MeSH terms and free-text. The search strategy in PubMed was:

#1 ((((((((“Medicine, Chinese Traditional”[MeSH Terms]) OR “Chinese Medicine” [Title/Abstract]) OR “Traditional Chinese Medicine” [Title/Abstract]) OR “Chinese Traditional Medicine” [Title/Abstract]) OR herb∗[Title/Abstract]) OR zhongyi[Title/Abstract]) OR zhongyao[Title/Abstract])

#2 (((((((“Network Meta-Analysis” [MeSH Terms]) OR “Network Meta-Analys∗” [Title/Abstract]) OR “Network Meta Analys∗”[Title/Abstract]) OR “Mixed Treatment Meta-Analys∗” [Title/Abstract]) OR “Mixed Treatment Meta Analys∗” [Title/Abstract]) OR “Multiple Treatment Comparison Meta-Analys∗” [Title/Abstract]) OR “Multiple Treatment Comparison Meta Analys∗” [Title/Abstract])

#3 #1 AND #2

Eligible criteria

NMAs based on RCTs with the treatments of TCM, which included Chinese herbal medicine and patent medicine, were eligible in this review; other treatments like Western medicine could be included but there must be at least 1 TCM treatment in each NMA. Nonpharmaceutical treatments, like acupuncture, moxibustion, cupping, message, and others, were excluded. NMAs including observational studies or diagnostic test, studies on the theory of NMA, methodological articles, protocols, editorials, letters, commentaries, and conference paper were also excluded.

Study selection and data extraction

The titles and abstracts of each record retrieved were checked by 2 independent authors (HW and XJ) to determine whether they met the eligible criteria. The full texts of potentially relevant articles were retrieved for further assessment. Disagreements were resolved by discussion or the involvement of a third researcher. The information, including author, year of publication, disease, number of participants, number of interventions, description of interventions, number of original study, and outcome, was extracted from each study and entered into a preformulated spreadsheet.

Methodological and reporting quality assessment tools

Two independent reviewers (JZ and YC) assessed the methodological quality using AMSTAR checklist, of which 11 items were included. For each item, it was scored “1” if the answer was “Yes,” and “0” if the answer was “No,” “Can’t answer” or “not applicable.” [ The summary score for an NMA was calculated by counting the number of “Yes,” with a possible maximum of 11. Score of 9to 11 was identified as “high quality,” 5 to 8 as “moderate quality” and 4 or lower as “low quality.” “The PRISMA Extension Statement for Reporting of Systematic Reviews Incorporating Network Meta-analyses of Health Care Interventions: Checklist and Explanations” (PRISMA-NMA),[ which consists of 32 items, was used to evaluate the reporting quality by 2 independent authors (HW and XJ). Each of the items was scored “1” for full compliance, “0.5” for partial compliance, and “0” for noncompliance.[ The summary PRISMA-NMA score for a NMA was calculated by accumulating scores of each item, with a possible maximum of 32. Score of 26 to 32 was identified as “high quality,” 20 to 25.5 as “moderate quality” and 19.5 or lower as “low quality.” Any disagreement between reviewers was resolved by discussion or the involvement of a third reviewer (JZ).

Results

General information of included studies

The literature searches identified 219 records. After screening, a total of 40 were included,[ among which 32 were in Chinese (including 7 master's or doctor's degree theses) and the other 8 were in English. These NMAs were published in 17 Chinese journals and 6 English journals from December 2012 to November 2017. The reviews included 2 to 29 types of interventions and 5 to 371 RCTs, with a total of 2535 RCTs.

As many as 24 types of diseases were involved in the included NMAs, and 24 NMAs were covered with different types of cancers, others focusing on stroke, diabetic, atrial fibrillation, and so on. Characteristics of the included NMAs were shown in Table 1 .

Table 1

Characteristics of the included NMAs.

Methodological quality assessment

Compliance with the AMSTAR checklist, the median score and interquartile range (IQR) of included NMAs was 7 (6–8), detailed methodological quality assessment was shown in Tables 2 and 3.

Table 1 (Continued)

Characteristics of the included NMAs.

Table 2

Methodological quality assessment of the included NMAs.

The optimum item was “Comprehensive literature search”(100%). Items of “Provide a priori design” (97.5%), “duplicate study selection and data extraction” (97.5%), “quality of included studies assessment” (97.5%), “formulated conclusions concerning the quality of the included studies” (97.5%), “Provide characteristics of the included studies” (82.5%) and “methods used to combine the findings” (70%) were acceptable. However, severe flaws existed in 4 items: “Publication bias assessment” (37.5%), “status of publication used as an inclusion criterion” (22.5%), “interest conflict” (12.5%), and the worst compliant item “list of studies (included and excluded) provided” (0%).

Reporting quality assessment

After assessing the compliance of the NMAs using the 32-item PRISMA-NMA checklist, we got a median and IQR score of 22 (19.1–27.1), but none of the NMAs met all the 32 items, with the full details given in Tables 4  and 5. As many as 12 NMAs (30%) got the score of lower than 20, with the lowest of 14.

Table 3

Summary of methodological quality assessment.

Table 4

Reporting quality assessment of the included NMAs.

For 12 items, over 80% articles are in compliance with the criteria, but for the item of “structured summary,” “study selection,” “summary of evidence,” and “conclusions,” all articles have met the criteria. However, there were still 11 items whose compliance rates were below 50% (20/40), which were “Objectives,” “Protocol and registration,” “Search,” “Data items,” “Assessment of inconsistency,” “Risk of bias across studies,” “Results of individual studies,” “Exploration for inconsistency,” “Risk of bias across studies,” “Results of additional analyses” and “Funding.” Then, the quality of the remaining 9 items was moderate between 50% and 80% accordance with PRISMA-NMA checklist.

Throughout the reporting of Methods and Results sections, issues of inadequate or selective reporting also existed. Two NMAs[ (5%) reported to assess risk of bias within individual studies in Methods part (item 12) but not really did in Results (item 19). The same flaw showed in reporting of inconsistency assessment (items S2 and S5) in 4 NMAs[ (10%) and risk of bias across studies (items 15 and 22) in 1 NMA[ (2.5%).

Discussion

TCM is well known as a complementary and alternative therapy for its use of Chinese herbal combinations to treat the functional disorders. However, few studies directly revealed the relationship between multitargets and multi-ingredients in Chinese herbal formula by utilizing the network pharmacology methodologies due to the complexity of Chinese medicine in chemical composition and molecular mechanisms. The network pharmacology may contribute to generate the hypothesis, and further experimental validation was still needed.[

NMAs could provide useful evidence on relative effectiveness of different treatments for decision-making when head-to-head comparison trials are insufficient.[ To the best of our knowledge, this is the first study to comprehensively assess the methodological and reporting quality of NMAs in TCM, although several reviews that focused on the methodological or reporting problems of NMAs in other fields have been conducted.[

Methodological quality of NMAs is one of the key points for researchers and health care decision-makers. We assessed the methodological quality of NMAs in TCM based on AMSTAR checklist. Several methodological flaws were identified, especially regarding the status of publication (item 4), a list of studies provided (item 5), assessment of publication bias (item 10), and conflicts of interest (item 11). For each NMA, the highest score was 9 and the lowest was 4, with a median and IQR of 7 (6–8), which showed that the general methodological quality is moderate. Zarin et al[ did a research and analyzed 456 network meta-analyses, and it got the result that the overall median AMSTAR score and IQR was 6 (4–7), which was similar to this study. Reporting quality of NMAs is also vital, thus we evaluated the reporting quality of NMAs in TCM using PRISMA-NMA guideline. Several items need to be improved in reporting, especially for Protocol and registration (item 5), Data items (item 11), Risk of bias across studies (Methods section) (item 15), results of individual studies (item 20), risk of bias across studies (Results section) (item 22), Results of additional analyses (item 23), and Funding (item 27). From the angle of individual NMA, the highest score was 31 and the lowest was only 14, with a median and IQR of 22 (19.1–27.1), which indicated that the reporting quality of the included NMAs was also moderate.

Among the 40 NMAs, 10 were published between 2012 and 2015, with a median and PRISMA-NMA score and IQR of 18 (16.4–26.3). These studies may be conducted before the PRISMA-NMA published. While the other 30 NMAs published in 2016 and 2017 were 22.3 (20.4–27.5), which was higher than that of the former 10. To some degree, it showed that the PRISMA-NMA may have already helped improve the reporting quality of NMAs in TCM. Therefore, we suggest that the NMA authors of TCM follow the PRISMA-NMA checklist when reporting NMAs, further, Chinese journals should endorse PRISMA-NMA in manuscript requirement and it is necessary to check the manuscript submitted with this guideline. In addition, NMAs were reviews mainly based on clinical trials like RCTs, so the results of NMAs may be affected by the quality of included trials, so it is vital to improve the quality of clinical studies of TCM.

This study has several limitations. First, there has been no standard tool to assess the methodological robustness of NMAs recently, although AMSTAR was widely used in the quality assessment of systematic review and meta-analysis. Second, though the use of the summary AMSTAR and PRISMA score was validated in the previous studies,[ these checklist was not originally designed as a scored instrument.[ Third, even though thorough search strategy was employed, we cannot guarantee that all relevant articles were identified.

Conclusion

The methodological and reporting quality of NMAs in TCM is moderate. Some methodological and reporting flaws have been identified in the published NMAs, especially for the status of publication, a list of studies provided, assessment of publication bias, protocol and registration, conflicts of interest and funding. Identified shortcomings of published NMAs should be taken into consideration in further trainings of authors and editors of NMAs in TCM. Moreover, future researchers should be encouraged to apply PRISMA-NMA, and a recognized tool for the assessment of NMA methodology was wanted.

Author contributions

Authorship: FY and JZ conceived the study, developed the criteria, and wrote the paper. JT and LG searched the literature. HW and XJ exacted the data. JZ and YC assessed the methodological quality. HW and XJ assessed the reporting quality. XL and MSL revised the manuscript. All authors read and approved the final manuscript.

Conceptualization: Fengwen Yang, Junhua Zhang.

Data curation: Hucheng Wang.

Funding acquisition: Junhua Zhang.

Investigation: Jinhui Tian, Long Ge.

Methodology: Jiahan Zou, Xinyao Jin, Yawen Cao.

Writing – original draft: Fengwen Yang, Junhua Zhang.

Writing – review & editing: Xuemei Li, Myeong Soo Lee.

Table 4 (Continued)

Reporting quality assessment of the included NMAs.

Table 5

Summary of reporting quality assessment.