The Reporting Quality of Systematic Reviews in Japanese Physical Therapy Journals.

OBJECTIVES: The purpose of this study was to evaluate the quality of reporting of systematic reviews published in Japanese in the field of physical therapy.
METHODS: The study design was a bibliometric analysis of systematic reviews. Two Japanese physical therapy journals (Physical Therapy Japan and Rigakuryoho Kagaku) were analysed using J-STAGE. The inclusion criterion was that articles were systematic reviews. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist was used to score the reporting quality of eligible systematic reviews. The quality assessment was performed by two reviewers independently.
RESULTS: Of the 1578 articles identified, thirteen articles were included in this study. The median score of checklist items adequately adhered to across the included studies was 12 (range, 7-17). None of the studies adhered to the structured summary or additional analysis PRISMA items. The intention of bias assessment across studies was reported in only three studies (23%), and only two of these three reported the results.
CONCLUSIONS: The reporting quality of systematic reviews published in Japanese physical therapy journals was suboptimal. Therefore, readers should critically appraise the contents of systematic reviews. It is recommended that journals should strictly require their authors to adhere to reporting guidelines. ©2020 The Japanese Association of Rehabilitation Medicine.

INTRODUCTION

In clinical settings, research findings are often referred to when treatment decisions are made. Moreover, the synthesized information in systematic reviews is considered one of the most important sources of reliable evidence. The study design of a systematic review comprehensively evaluates and synthesizes all available evidence related to the topic. Therefore, it is recommended that clinicians seek out systematic reviews before searching for individual studies.[1]) Recently, the popularity of systematic reviews has increased. On MEDLINE, about 8000 systematic reviews were published in 2014, and the number published tripled throughout the decade.[1]) Up-to-date systematic reviews can be used not only in clinical settings but also in the political arena for implementing new rules or policies.[2]) The appropriate findings from systematic reviews support various clinical decisions.

Because some articles may provide misleading information due to flaws in the contents, the findings from such articles, including systematic reviews, should be assessed for validity before application in any setting. Even though systematic reviews are published after a rigorous peer-reviewed process, some may later be found to have serious methodological quality concerns.[3]) It is recommended that readers should always critically appraise articles for internal and external validity before using their findings. Therefore, the details of research should be presented in a clear and transparent way, something for which authors should take responsibility. Although an adequate amount of information is required for readers to judge the validity and applicability of an article’s content, the reporting quality of research articles is generally low.[4]) This makes it difficult for readers to interpret research appropriately.

To improve the reporting quality of articles, several guidelines have been published for different research designs. Consolidated Standards of Reporting Trials (CONSORT)[5]) and Strengthening the Reporting of Observational Studies in Epidemiology (STROBE)[6]) were designed for randomized controlled trials and observational studies, respectively. According to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement, systematic reviews should include all the important components of publication to increase the transparency of research.[7]) These guidelines are designed to guide authors to report appropriate information in their articles. Although the number of journals requiring authors to follow such instructions is increasing,[8]) many journals still do not require adherence to these instructions.[9])

Several systematic reviews have been published in the field of physical therapy research,[10]) but only a few studies have investigated the reporting quality of these articles. By using the PRISMA checklist to analyse physical therapy articles, a recent study investigated the reporting quality of systematic reviews published in Portuguese.[11]) The results of this study were that less than 30% of the checklist items were satisfied. These facts may provide both researchers and readers with important insights enabling them to avoid being misled or failing to understand research findings. Although the reporting quality may not be associated with specific countries,[12]) the impact of different languages used in articles is unclear. Therefore, it is worthwhile to investigate the reporting quality of articles in various languages other than English. Overall, scientific articles published in non-English languages comprise more than 30% of the total; however, this figure may depend on the research field.[13]) In the field of physical therapy, this proportion remains unknown. Ignoring important information from articles written in non-English languages can lead to a risk of bias.

The purpose of the current study was to assess the reporting quality of systematic reviews published in two Japanese physical therapy journals. Little evidence is available on the reporting quality of systematic reviews, and, as yet, no study has investigated this in Japanese physical therapy journals. With more than 83,000 members in 2018, the Japanese Physical Therapy Association is one of the largest associations of the World Confederation for Physical Therapy.[14]) Consequently, the results of the current study could have a large impact on the physical therapy community. Besides, the results of this study may motivate members from other countries to investigate the reporting quality of systematic reviews published in their own country.

METHODS

The current study was designed as a bibliometric analysis of systematic reviews in published articles. We defined Japanese physical therapy journals as journals associated with nationwide academic organizations currently publishing articles related to physical therapy in the Japanese language. To enhance the generalizability of our findings, we did not consider journals that were specific to subdisciplines of physical therapy, such as manipulative therapy. We selected two representative journals, namely Physical Therapy Japan and Rigakuryoho Kagaku. These two journals are published by Japanese academic societies in the field of physical therapy, including the Japanese Physical Therapy Association, and have the largest number of registered articles among physical therapy-related journals in J-STAGE (more than 3400 articles in Physical Therapy Japan and 2400 articles in Rigakuryoho Kagaku).[15]) Several previous studies[16],[17],[18],[19]) used these two journals to represent trends in Japanese physical therapy research.

Search Strategy

Eligible reviews on physical therapy were retrieved from Physical Therapy Japan (1984 to 24 December 2018) and Rigakuryoho Kagaku (1995 to 24 December 2018), via J-STAGE. J-STAGE is an electronic database managed by the Japan Science and Technology Agency that supports Japanese research societies and includes more than 2000 journals.[15]) In both these journals, “systematic review” and its synonyms were used separately as search terms to identify systematic reviews (Table 1). Because some articles contained English words, e.g., in keywords or abstracts, both Japanese and English language search terms were used. The electronic database search was conducted on 24 December 2018, and no limiting search filters were applied.

Table 1.

Search terms and the number of articles identified in Physical Therapy Japan and Rigakuryoho Kagaku

Search terms	n
Physical Therapy Japan
“システマティックレビュー” (“Systematic review” in Japanese) “レビュー” (“review” in Japanese) “メタアナリシス” (“Meta analysis” in Japanese) “Review” “Systematic review” “Meta analysis” “系統的総説” (“Systematic review” in Japanese) “総説” (“review” in Japanese)	55 93 67 316 132 87 8 43
Rigakuryoho Kagaku
“システマティックレビュー” (“Systematic review” in Japanese) “レビュー” (“review” in Japanese) “メタアナリシス” (“Meta analysis” in Japanese) “Review” “Systematic review” “Meta analysis” “系統的総説” (“Systematic review” in Japanese) “総説” (“review” in Japanese)	33 53 19 387 166 91 1 27

Study Selection

One author (AT) imported the titles and Digital Object Identifiers (DOIs) of all identified studies into Microsoft Excel, and then the selection of studies was conducted by both authors (AT and ID) independently. Studies that clearly mentioned “systematic review” or “meta-analysis” in the title, abstract, or text were included. We also checked the method for confirmation. Original studies, reviews, and proceedings were excluded. If any selection decisions conflicted, discussions were held to resolve the issue.

Reporting Quality Assessment

After the studies to be included were finalized, the reporting qualities of the systematic reviews were assessed by both authors independently using the PRISMA checklist.[20]) This checklist contains 27 items. Each item was rated by following a method used previously.[11]) We recorded the pages reporting the item if a study was found to satisfy the requirements of an item. The study was rated as “no” for items considered to be unsatisfied, incomplete, or having unclear reporting. According to the number of items satisfying the requirements (i.e., those with page numbers), each article was scored from zero (no item satisfied) to 27 (all the items satisfied).

Data Extraction

The following data were extracted from the included studies: the name of the journal, authors’ names, year of publication, protocol registration, type of systematic review (e.g., interventional, diagnostic, or prognostic), the number of studies included in the review, the target condition, types of intervention, the primary outcome, methods of data synthesis, and endorsement of the PRISMA Statement. One of the authors (AT) extracted the data using a form and the other author (ID) verified it.

Analysis

Adherence to the PRISMA checklist across all included studies was presented in a cross-tabulated form. The total number and the proportion of the items satisfied were calculated for each study, and the median and range were calculated across all studies. In addition, we present the median scores with ranges for articles that did and did not describe the PRISMA Statement in the article.

RESULTS

The search strategy retrieved 1578 studies from the two journals (Fig. 1). A total of 909 studies were included after the removal of duplicates. Finally, thirteen systematic review articles[21],[22],[23],[24],[25],[26],[27],[28],[29],[30],[31],[32],[33]) were included in the current study (Tables 2,3). Of these 13 studies, 4 endorsed the PRISMA Statement (Fig. 2). One study published its title and abstract in Japanese with the full text in English.[21]) We did not predict this possibility in the development phase of the current study proposal because the two journals basically publish articles in Japanese. The systematic review with meta-analysis by Hiraoka published in 1995[21]) was the only included article published before the PRISMA Statement was released. Since the small number of systematic reviews included in the current study, especially those with meta-analysis, the information contained in this study can be informative. Consequently, we considered this study to be worthy of inclusion in the current study. Five systematic studies used meta-analysis, and the remaining eight studies used qualitative analysis. The results showing adherence to the PRISMA checklist are presented in Table 4. Of the 27 PRISMA checklist items, “structured summary” (#2) and “additional analysis” (#23) were not adhered to by any of the included studies. One study[23]) satisfied “additional analysis” (#16) in the Methods section but did not report the data (#23). “Rationale” (#3) was the only item satisfied in all studies, and three items (“title” #1, “summary of evidence” #24, and “conclusions” #26) were satisfied by more than 90% of studies. The method of bias assessment across the studies (#15) was reported in only three studies (23%), and the results (#22) were presented in two of these three studies.

Fig. 1.

 Flowchart showing the selection process of articles included in the current study.

Table 2.

 Individual studies included in Physical Therapy Japan

Author	Year	Protocol registration	Type of SR	Included studies	Population	Intervention	Primary outcome	Type of analysis
Hiraoka[21])	1995	―	IE	n=6	Paediatrics	NDT	―	MA
Ozawa et al.[22])	2010	―	IE	Study 1: n=55 Study 2: n=11	Knee OA	Study 1:TE Study 2: Strengthening	―	Qualitative
Moriyama et al.[23])	2011	―	IE	n=25	Orthopaedics	Stretching	―	MA
Matsuda et al.[24])	2011	―	IE	n=8	Femoral proximal fracture	High intensity strengthening	―	MA
Tanaka et al.[25])	2013	―	IE	n=44	Knee OA	TE	Body structures & functions, activity, and QOL	MA
Umehara et al.[26])	2014	―	IE	n=13	THA for hip OA	TE	Body function and activity	MA
Mine et al.[29])	2016	PROSPERO	IE	n=6	Non-specific low back pain	Patient education	Pain, body functions, psychometric property	Qualitative
Ichikawa et al.[32])	2018	―	Methodology	n=8	Knee pain	n.a.	―	Qualitative

SR, systematic review; ―, not mentioned; IE, intervention effect; NDT, neurodevelopmental treatment; MA, meta-analysis; OA, osteoarthritis; TE, therapeutic exercise; QOL, quality of life; THA, total hip arthroplasty; n.a., not applicable.

Table 3.

 Individual studies included in Rigakuryoho Kagaku

Author	Year	Protocol registration	Type of SR	Included studies	Population	Intervention	Primary outcome	Type of analysis
Komukai et al.[27])	2016	―	Methodology	n=8	Healthcare students	n.a.	―	Qualitative
Sugita et al.[28])	2016	―	Risk factors	n=7	Caregivers of people with stroke	n.a.	Factors related to care burden	Qualitative
Takahashi et al.[30])	2017	―	IE	n=29	People with lung and colorectal cancer	Exercise	―	Qualitative
Komukai et al.[31])	2017	―	Outcome measurement	n=197	Rehabilitation specialties	n.a.	―	Qualitative
Ashizawa et al.[33])	2018	―	IE	n=10	Adults (age ≥20 years)	Reduction of sedentary behaviour	Risk factors of cerebrovascular disease	Qualitative

Fig. 2.

 The proportion of items in the PRISMA checklist satisfied by the 13 systematic reviews studied.

Table 4.

 Adherence to the 27 PRISMA checklist items by the 13 included studies

First author’s name	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	Score	%
Hiraoka[21])	○	n	○	n	n	○	n	n	n	n	n	n	○	n	n	n	n	○	n	n	n	n	n	○	○	n	n	7	26
Ozawa[22])	○	n	○	n	n	○	○	○	○	n	n	n	n	n	n	n	○	n	n	n	n	n	n	○	n	○	n	9	33
Moriyama[23])	○	n	○	n	n	○	n	○	○	n	n	○	n	○	○	○	○	○	○	n	○	n	n	○	○	○	n	16	59
Matsuda[24])	○	n	○	n	n	n	n	n	○	n	n	○	n	○	n	n	○	○	○	n	○	n	n	○	○	○	n	12	44
Tanaka[25])	○	n	○	n	n	○	n	n	n	n	n	○	○	○	○	n	n	n	n	n	○	○	n	○	n	○	n	11	41
Umehara[26])	○	n	○	○	n	n	n	n	○	n	n	○	○	○	○	n	○	○	○	○	○	○	n	○	○	○	n	17	63
Komukai[27])	○	n	○	n	n	○	○	○	n	○	○	n	n	○	n	n	○	○	n	n	n	n	n	○	○	○	n	13	48
Sugita[28])	○	n	○	n	n	○	○	○	○	○	○	○	n	n	n	n	○	○	○	n	n	n	n	○	○	○	n	15	56
Mine[29])	○	n	○	○	○	○	○	n	○	○	○	○	n	○	n	n	○	○	○	n	n	n	n	○	○	○	n	17	63
Takahashi[30])	○	n	○	○	n	○	○	n	n	n	○	○	n	n	n	n	n	○	○	n	n	n	n	○	○	○	n	12	44
Komukai[31])	○	n	○	n	n	○	○	○	○	○	○	n	n	n	n	n	○	○	n	n	n	n	n	○	○	○	n	13	48
Ichikawa[32])	n	n	○	n	n	○	○	n	○	n	○	n	n	n	n	n	○	n	n	n	n	n	n	n	○	○	○	9	33
Ashizawa[33])	○	n	○	○	n	n	○	n	n	n	○	○	n	n	n	n	n	○	○	n	n	n	n	○	○	○	n	11	41
Total	12	0	13	4	1	10	8	5	8	4	7	8	3	6	3	1	9	10	7	1	4	2	0	12	11	12	1
%	92	0	100	31	8	77	62	38	62	31	54	62	23	46	23	8	69	77	54	8	31	15	0	92	85	92	8

○, items satisfied; n, incomplete or unclear reporting.

The median score of the included studies was 12 items (range, 7–17). The median scores of the studies that did and did not describe the PRISMA Statement were 14 (range,13–17) and 11 (range, 7–17), respectively.

DISCUSSION

In the current study, systematic reviews in two Japanese physical therapy-related journals were analysed to investigate their reporting quality, based on the PRISMA checklist. Overall, less than half of the items were satisfied in the included studies. A previous study reported that adherence to the PRISMA checklist in physical therapy journals written in Portuguese was 29.8%.[11]) The adherence in the current study was higher than that in the previous study, but the adherence rate was still somewhat suboptimal. In comparison with the Portuguese study, our study included more recent articles, which might partly explain the higher adherence.[34])

One of the reasons for the low reporting quality identified in the current study could be that neither of the two journals states in their Guidelines for Authors that the PRISMA Statement should be adhered to. If journals set rules of reporting guidelines for authors, then papers without proof of the guideline endorsement would not be reviewed. A recent study compared the reporting quality between systematic reviews in the field of gastroenterology and hepatology published in journals with and without endorsements of the PRISMA Statement.[35]) The reporting quality was found to be higher in systematic reviews published in the endorsing journals. We could not perform a similar comparison because our study investigated journals that do not endorse PRISMA. Physical therapy-related journals published in other countries, such as USA,[36]) UK,[37]) and Canada,[38]) explicitly require their authors to follow the PRISMA Statement before publishing systematic reviews. The causal inference is still unclear, but to improve the quality of both journals analysed in the current study, the rules for reporting guidelines should be set.

Our finding of a low compliance rate with a structured abstract was consistent with previous studies. In oral implantology journals[39]) and general medical journals,[40]) the reporting quality of the abstracts in systematic reviews was suboptimal. For these reasons, it is recommended that the full text of articles should be read to avoid any misunderstanding.[40]) Presumably, requirements by each journal for authors to make their abstracts comply with the PRISMA Statement could simply address this issue. Clear instruction to provide structured abstracts, particularly in the methods section (e.g., descriptions of data sources, eligibility criteria, or evaluation of risk of bias) would help both authors and readers, in terms of transparency of reviews.

In the current study, the five systematic reviews that conducted a meta-analysis [21],[23],[24],[25],[26]) did not provide results of the additional analysis in their articles. Additional analysis includes meta-regression analysis that examines the impact of factors in individual studies that influence the effect size.[41]) This may give greater insight to the interpretation of results. However, even if an analysis is well-planned and conducted, there is a chance that authors choose to under-report their findings, which is well known as selective reporting bias. A majority of systematic reviews may undergo significant changes from the protocol to publication,[42]) and this could be indirect evidence of selective outcome reporting. Unfortunately, readers cannot assess this possibility because none of the five meta-analysis studies quoted the protocol registration number.

The level of risk of bias in the studies can inform readers about the possibility of publication bias in systematic reviews. Because less than half of conference abstracts are ultimately published as full texts,[43]) examining the risk of publication bias is imperative. Publication bias may change an overall inference because negative results remain under-reported.[43]) If authors intentionally or unintentionally exclude relevant evidence from their systematic reviews without information such as funnel plots[44]) or regression tests,[45]) readers cannot access the true results. Systematic reviews must be conducted in an appropriate way to access all available evidence. For systematic reviews handling qualitative data, searching unpublished data may be useful to minimize publication bias[46]) as long as the reviewer can appraise articles without the filter of the peer-review process.

Because evidence in this field is still scarce, our findings may contribute to the quantification of reporting quality in physical therapy-related systematic reviews. This can be helpful for many readers. For instance, researchers may conduct international comparisons based on the findings and reveal the factors associated with the reporting quality. Also, when clinical guidelines are being developed, this result can be informative for evaluating systematic reviews to be included. In addition, our findings may raise awareness among readers about the quality of reporting in Japanese physical therapy-related systematic reviews. Readers may need to thoroughly appraise the contents of systematic reviews to carefully interpret when they can be applied in clinical settings. The number of systematic reviews and meta-analyses is dramatically increasing; as a result, the number of articles that might mislead readers is also increasing.[47]) Authors should acknowledge our results and take responsibility for providing an adequate amount of information.

There are several limitations of the current research. One limitation is that we used only two journals, leading to inadequately broad searches. We acknowledged that physical therapy-related articles could be published in additional journals, including non-healthcare journals, for instance, in the field of engineering.[48]) However, the focus of this current study was to investigate systematic reviews in physical therapy journals. In addition, the two included journals are considered to be the primary Japanese physical therapy journals[16],[17],[18],[19]) in terms of the number of articles and their nationwide publication in Japan. Therefore, we believe that our findings are likely representative of systematic reviews published in Japanese physical therapy journals. Furthermore, we did not evaluate the risk of bias of included articles, so the relationship between reporting quality and risk of bias could not be considered. Articles that have a low quality of reporting may provide limited information for assessing risk of bias. Therefore, readers should critically assess the contents of such articles.

CONCLUSION

The current study investigated the reporting quality of physical therapy-related systematic reviews published in the major Japanese journals. The reporting quality of the systematic reviews examined in this study was suboptimal. Consequently, readers are encouraged to critically appraise their contents. We anticipate that our results will make readers seek clear and transparent information. Furthermore, researchers should consider following reporting guidelines to provide adequate levels of information even if it is not mandated by the journals. Future studies are needed to evaluate the relationship between the reporting quality and the risk of bias in this field. Furthermore, the impact of endorsement of reporting guidelines, such as the PRISMA Statement, on reporting quality should be examined.