A PRISMA assessment of the reporting quality of systematic reviews of nursing published in the Cochrane Library and paper-based journals.

OBJECTIVE: The Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) was released as a standard of reporting systematic reviewers (SRs). However, not all SRs adhere completely to this standard. This study aimed to evaluate the reporting quality of SRs published in the Cochrane Library and paper-based journals.
METHODS: The SRs which evaluate the effectiveness of nursing interventions in 2016 were identified via PubMed. The reporting quality of selected articles was evaluated using the PRISMA checklist. For comparison, we divided these articles into Cochrane review (CR) and non-Cochrane review (NCR). Based on the satisfaction of the applicable criteria, each article is assigned an accumulated score and a total percentage score.
RESULTS: Overall, 41.7% articles were concentrated in 19.0 to 22.5 points which represent the moderate quality, 22% articles were high quality. There were still 36.5% articles with low quality. The mean PRISMA score was 20.54 ± 2.367 for CRs, and 18.81 ± 2.536 for NCRs. Although no significant difference was exit between overall CR and NCR scores, there were differences between items 1, 5, 8, 16, 23. Analysis indicated that CR was significantly associated with the overall PRISMA score.
CONCLUSION: Compliance of CR and NCR with PRISMA checklist exhibited different strengths and weaknesses. Our study underscores that nursing researchers should pay more attention to comprehensive reporting of SRs in nursing to follow the PRISMA statement. IMPLICATIONS FOR NURSING AND/OR HEALTH POLICY: Nursing researchers who participate in SRs should follow the latest Cochrane Handbook to prepare such study. Meanwhile, the PRISMA statement should be followed strictly to report SRs, so as to improve the quality of SRs.

Introduction

Systematic review (SR) is an important method to summarize evidence accurately and reliably, which is considered the gold standard for evidence used to guide clinical practices.[ However, the value of SRs depends on how well authors have reported what they did, and what they found.[ The impact of SRs on practice and research makes the quality of their reporting particularly important.[ In 1996, a Quality of Reporting of Meta-analyses (QUOROM) statement with 18 items was developed to improve the reporting of meta-analyses.[ Ten years later, the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement was developed based on the QUOROM guidelines.[ The PRISMA was updated in 2010 and consisted of a checklist of 27 items which encompass 7 aspects of the report, along with a flow diagram.[ This guideline intended to guide authors on what need to be reported and how to report studies. Till 2016, more than 175 journals have endorsed the checklist.[ Study suggested that PRISMA statement adoption could help mitigate issues both in reporting and methodology.[ Reviews have been conducted for evaluation of the reporting quality of SRs of acupuncture,[ diagnostic tests,[ SRs published in Chinese Journal of Evidence-Based Pediatrics,[ in orthodontics,[ unfortunately, core information is inadequately reported in these studies. It is estimated that at least 50% of published research were poorly conducted making them difficult to interpret and use.[

With the increasing of published SRs in nursing area, researchers began to focus on the quality of nursing reviews. Studies investigated the adherence to recommended methodological and reporting guidelines for SRs published in Chinese nursing area,[ SRs of nursing on Traditional Chinese Medicine,[ SRs conducted by Korean researchers,[ SRs published in the top 10 nursing journals from 2009 to 2010,[ and SRs published in nursing journals which have endorsed the PRISMA statement.[ They found there were obvious deficiencies regarding the methodological and reporting issues in nursing SRs. To use the best available evidence in clinical practice, the author suggested that reviewers should conduct SRs using rigorous research methods to improve the quality of SRs in nursing area.

The PRISMA statement is recommended to the authors as a resource for reporting of SR. Cochrane Handbook for Systematic Review required all of the Cochrane review (CR) authors report the SRs strictly following PRISMA. SRs, both Cochrane and non-Cochrane, are of increasing importance in changing clinical practice and informing health policy. Study showed that comparing to non-Cochrane reports, compliance of reporting is superior in Cochrane.[ However, to our knowledge, there has been no study on comparing the characteristics and the reporting quality of nursing SRs both indexed in Cochrane and paper-based journals. It is unclear whether they have a similar reporting quality. For epidemiological characteristics, Cochrane SR requires a minimum of 3 authors.[ Meanwhile, the funding-supported papers have been fully demonstrated and evaluated by experts in the process of the project. Therefore, the quality of such kind of papers is comparatively higher. However, is the reporting quality really higher in funding supported nursing SRs? Thus, this study aimed to explore the epidemiological characteristics of included CR and paper-based non-Cochrane review (NCR); whether the reporting quality of nursing SRs is associated with epidemiological characteristics; and the deficiencies and to offer advice for achieving high reporting quality SRs.

Methods

Ethical review

Ethical approval was not necessary for this study as the study did not involve patient, and the included SRs can be traced from database.

Search strategy

Due to the volume of nursing literature, the search was limited to identify SRs of nursing intervention indexed between January 2016 and December 2016. We selected 2016, as the search was taken in July 2017 and it was the closest to when the protocol for this study was drafted. For retrieval of studies, we searched MEDLINE via PubMed. The search was conducted by a librarian and 1 investigator (TJH and ZJX) and the detailed strategy was shown in Appendix 1.

Study selection

For inclusion, studies must meet the following criteria: study design: SRs, meta-analysis (MA), or described as SRs and MA; assessing the effects of nursing interventions; reviews included only trials as primary studies; and full text available.

A “Cochrane review” is one that has been published in the Cochrane Database of Systematic Reviews; a NCR is one published in paper-based journals. Nursing interventions in our study are identified as nursing activities performed by registered nurses to improve patients’ health. There were no limitations in techniques or styles of interventions.

Titles and abstracts of initially retrieved studies were screened by 2 reviewers independently. The full texts of eligible reviews and uncertain reviews were obtained and examined to determine whether they met the inclusion criteria. Any disagreements were resolved by discussion. For study purpose, we divided the final reviews into CR and NCR.

Data extraction

Two authors (ZJX and WJC) independently extracted the epidemiology and reporting characteristic data of the included studies into Excel. The epidemiology characteristics include region of publication, number of authors, number of citations, number of included primary studies, affiliations of 1st authors, whether MA was undertaken, updates of previous reviews, and whether conflict of interest was stated. For reporting characteristics, the PRISMA checklist was chosen as a tool to assess reporting quality for this study. We assess the compliance of each report with 27 items in PRISMA checklist. If item was described in the report, we rated it as “Yes” with score 1, if described partly, we scored it as 0.5, then 0 for “No.” The total score for PRISMA was calculated by summing each score. To classify the quality of the included reviews, the following 3 categories were used: score >19.0 means low quality, 19.0 to 22.5 means moderate quality, and >22.5 was indicated as high quality.[

Statistical analysis

Descriptive statistics was performed to describe general characteristics. Pearson Chi-squared was performed to compare the percentage of CR and NCR under different characteristics. We performed on the distribution of scores per PRISMA item and summary statistics for the observed PRISMA scores for each SRs considered. We compared the quality of Cochrane and non-Cochrane SRs by the individual items of the PRISMA instrument. Data were summarized with descriptive statistical analyses. Pearson Chi-squared test was performed to analyze differences among 2 groups. We analyzed the possible associations of the reporting quality of SRs and independent predictors including authorship country of origin, number of authors, update of published reviews, CRs or not, and inclusion of a MA using simple and multiple linear regression analyses. P ≤ .05 was considered to indicate a statistically significant difference for all tests. Statistical analyses were performed with SPSS, version 19.0, software.

Results

Search results

Initially, 175 articles were obtained. After screening the titles and abstracts, 82 potentially eligible reviews were identified. Subsequently full text of each article was retrieved with a total of 68 SRs that were confirmed for further assessment. The search detail was given in PRISMA flow diagram.

Characteristics of included SR/MA

As showed in Table 1, there are number of researchers who play an important role in SRs production, as 86.8% of articles had researchers as 1st authors and 81.6% of 1st authors were from university. About 84.2% of studies were produced by authors more than 3, and most (78.9%) SRs were cited <5 times. MA was performed in 60.5% of SRs, and 36.8% of them were updates of a previous review. Only 21.1% of reviews stated conflict of interest in the study.

Table 1

General characteristics of the included SRs.

Our study showed that SRs published in paper-based journals tend to include more authors, and were cited less compare to CRs. However, more of the CRs were cited by other studies, perform MA, described the primary outcomes of interests, and had been updated.

PRISMA checklist assessment

The reporting quality results showed that CRs were superior to NCRs. Although none of the included 68 SRs were satisfied with items in PRISMA, more than 50% of CRs met the requirements of PRISMA domains (24 of 27). However, generally the compliance with PRISMA was poor in NCRs as only 17 items were complianced. Significant failings in the CRs were found in “identify the report as a SR/MA or both in title” (36.4%), and “the assessment of risk of bias (ROB) across studies” (36.4%). The main failings in NCRs were found in “protocol and registration” (12.5%), “search” (12.5%), “data collection process” (47.5%), “ROB across studies” (18.7%), and “additional analyses” (25%). The significant differences were observed within 2 groups in “protocol and registration,” “search,” and “additional analyses.” The least reported items in nursing SR/MA are “protocol and registration,” “search,” “additional analyses,” and “ROB across studies.” The overall PRISMA score for CRs was 20.54 ± 2.367 which indicated a moderate quality, and 18.81 ± 2.536 for non-Cochrane, respectively, which indicated a low quality. The difference in the 2 groups was statistically nonsignificant.

Supporting Information(Figure 1) shows according to summarized PRISMA scores, the SRs were divided into low, moderate, or high quality. The result indicated that CR was more likely to have a high PRISMA score compared with those from paper-based journals. The PRISMA scores ranged from 12.0 to 24. About 41.7% articles were concentrated in 19.0 to 22.5 points which means the quality of reports was moderate, 22% articles were high, and 36.5% articles were with low quality (Table 2).

Figure 1

The PRISMA score for Cochrane and Non-Cochrane systematic reviews.

Table 2

Summary table for PRISMA comparison.

Correlation between general characteristics and PRISMA scores

We found a positive relationship in reporting quality between CR and NCR. In addition, using of a MA was identified to be associate with better reporting quality (Table 3).

Table 3

Correlation between general characteristics and PRISMA scores.

Discussion

In a previous study, we assessed the reporting and methodological quality of SRs of nursing published in Chinese journals.[ This study differs in that it focuses primarily on reporting characteristics of SRs of nursing literature published in Cochrane and paper-based journals. This article is a systematic evaluation of PRISMA items used in reporting of nursing SRs, which does not represent the evaluation of SR evidence level. It is hoped that the present study will provide a benchmark against which the reporting of further nursing SRs can be compared.

Characteristics of included SRs/MAs

The SRs of nursing intervention come mainly from universities; meantime, most of these articles were conducted by nursing researchers. As SRs provide evidence for clinical practices, we suggest more clinical nurses should participate in the production of such research. SR is considered to be the highest level of evidence, but its findings will soon expire if they are not updated in time. Cochrane SR placed particular emphasis on updating the SR.[ Our study also revealed that CRs were updated more frequently than their paper-based counterparts. The low update rate among paper-based reviews suggested that editors of these journals are not sufficiently interested in publishing updated versions of SRs, or authors are not aware of such interest. Regardless of the causes, reviews should be updated timely.

Comparisons of the reporting quality of CR and NCR

The impact of SRs on practice and research makes their reporting quality especially important. Unfortunately, the reporting quality scores of nursing literature was not satisfied. Overall, CRs were more likely to have a high reporting quality compared with those from paper-based journals. Yao and colleagues showed the quality of SRs in Chinese was lower than that in CRs.[ Tian et al reported that quality of reviews from China and the United States is similar.[ It indicated the efforts of researchers from different countries in understanding and implementing the SRs reporting guideline are worthy of recognition, but there is still room for improvement.

Compared with reviews published in paper-based journals, CRs include elements which make them less prone to bias, such as the inclusion and exclusion criteria, and comprehensive search strategy.[ The reporting deficiencies in reviews published in paper-based nursing journals are similar to those reported by others. For example, registration information is seriously missing in paper-based reviews, which was also found in Tam's study that only 2 of 74 nursing SRs mentioned a lack of compulsory research protocols.[ Panic reported that only 4 of 90 SRs in gastroenterology and hepatology journals,[ 15% in radiology journals,[ 27% in orthodontic journals,[ and one-third in biomedical research[ mentioned research protocols. Reporting and publishing protocols are important steps in increasing the transparency of the research process and the reliability of published papers.[ It also can reduce the risk of the same topic.[ Prospective registration of researches is encouraged by initiatives including that of the International Committee of Medical Journal Editors.[ Cochrane SRs are required to be registered in advance.[ However, nursing journals do not have this requirement, which may lead to a large number of SR articles published repetitively in nursing journals. Present study suggested that nursing journals should have such requirements for authors, so that to avoid duplication of research projects with wasting of manpower and material resources.

Making a reasonable and detailed search strategy is the 1st prerequisite to improve the recall and precision of literature and ensure the quality of SRs.[ However, most reviews in paper-based nursing journals are relatively simple in the reporting of retrieval strategies, which is not conducive to readers from the perspective of credibility and methodology to evaluate the quality of SRs. This may be related to the layout and word limit of many nursing journals. It is suggested that the journal editor or peer reviewers should request relative data to be submitted online as supplementary material. It is best to describe the retrieval strategy of a representative database in detail, which can not only facilitate the evaluation of the quality of the research by the users of evidence, but also prompt the researchers to pay attention to the formulation of the retrieval strategy.[

Less than half of the nursing literatures reported items such as data extraction, research bias, and other analysis. An SR is often completed in the process of the need for researchers to adjust the original design, the lack of information in the methodological part, may be misleading to the completion of the SR, and affecting the methodological quality.[ Most of the nursing literatures did not report on inter-study bias and other analysis. One of the key steps in a SR is the assessment of internal validity (or ROB) of all studies included for evidence synthesis.[ Different types of bias may weaken the implementation and interpretation of SR, ignoring bias will reduce the credibility of SR/MA. SRs are urged to incorporate considerations of ROB into their results and undertake sensitivity analyses to evaluate the robustness of results.[

The CRs which accounted for 34% of the samples had more completed reports than their counterparts. CRs are significantly prior in the reporting of abstract, introduction, methods, results, and discussion sections. Multiple linear regression analysis showed that CRs are the most influential factors on PRISMA scores. This is possibly due to the use of strategies in the editorial process that promotes reporting guideline. CR editorial groups also use an internal checklist that reflects the content of PRISMA.[ We suggest the editors and reviewers of nursing journals should also strictly following PRISMA guideline in the peer review period to improve the reporting quality of evidence-based nursing research.

Strengths and limitations of this study

This is the 1st study to compare the reporting quality of SRs of nursing from Cochrane and paper-based journals. Because of the large amount of literature, we only chose those published in 2016, it may limit the generalization of the results to reviews. The results reflect a trend of reporting quality of SRs in nursing area.

Conclusion

The reporting quality of nursing SRs published in the Cochrane Library and paper-based journals was found to be deficient in certain areas, particularly with registration of protocol, detailed explanation of search strategy, additional analyses, and assessment of reports of ROB across studies. The result indicated that SRs published in the Cochrane Library were more likely to have a high PRISMA score compared with those from paper-based journals. The identified shortcomings in our study should be taken into consideration in further training of researchers who are about to participate in nursing SRs production. We propose following strategies for future research: since a protocol can prespecify the objectives and methods of the SR, it is important to register the article before further progress. Nursing authors should follow PRISMA to report SR especially in the area of search strategy, ROB across studies, and additional analyses. The researchers should update their knowledge timely regarding SRs methodology and reporting methods.

Implications for nursing and health policy

A strategic approach is crucial to improve reporting quality of SRs. This study calls nursing reviewers globally to action, to conducting SRs following the latest Cochrane Collaboration Handbook. Meanwhile, the PRISMA statement should be followed strictly to report SRs. Journals should have reporting requirements for submissions.

Acknowledgment

The authors thank Dr Jyh Boyette (RN, MSN, MBA, CIC, ACRN, Infection Prevention, Tampa General Hospital, Tampa, FL) for providing assistance with editing the final manuscript. They also thank Gansu Provincial Hospital for providing fund for this study.

Author contributions

Conceptualization: Juxia Zhang.

Data curation: Lin Han, Jiancheng Wang.

Funding acquisition: Juxia Zhang.

Methodology: Juxia Zhang, Lin Han.

Resources: Jinhui Tian.

Software: Jinhui Tian, Jiancheng Wang.

Supervision: Lin Han, Linda Shields, Jiancheng Wang.

Visualization: Jinhui Tian.

Writing – original draft: Juxia Zhang, Jiancheng Wang.

Writing – review & editing: Linda Shields.