Assessing abstracts of Iranian systematic reviews and metaanalysis indexed in WOS and Scopus using PRISMA.

Background: Systematic reviews and meta-analysis have significant advantages over conventional reviews in that all available data should be presented. This study aimed to evaluate Iranian systematic reviews and meta-analysis abstracts indexed in WOS and Scopus during 2003-2012 based on PRISMA checklist.
Methods: This is an analytical study. We evaluated 46 article abstracts indexed in WOS, 89 article abstracts indexed in Scopus and 158 article abstracts indexed in WOS and Scopus both (overlapped group). The quality of the abstracts was evaluated according to the PRISMA checklist for abstracts. Some indicators including distribution per year, total citation, average citations per year, average citations per documents and average citations per year in each article were determined through searching the WOS and Scopus Databases' analytical section. Then, the correlations between the abstract's PRISMA scores, average citations per year, and publication year were calculated.
Results: The abstract's quality is not desirable as far as the PRISMA criteria are concerned. In other words, none of the articles' abstracts is in line with the PRISMA items. The average of scores of the current study was 5.9 while the maximum score was 12. The PRISMA criteria showed the highest compliance with "Objectives" (98.6%), the second highest with "Synthesis of result" (85%) and "Title" (80.2%) and the lowest compliance with "Registration" (2%). There was a positive correlation between the compliance of PRISMA score and the average citations per year while there was a negative correlation between PRISMA score and the publication year.
Conclusion: It seems that the suggested criteria for reporting Iranian systematic reviews and meta-analysis are not considered adequately by the writers and even scientific journal editors.

↑ What is “already known” in this topic:

Systematic reviews and specifically meta-analyses are important in that they can bring a synthesis of knowledge in one article while the information originally appeared in several articles. Iranian researchers are increasingly developing such reviews.

→ What this article adds:

As the quantity of Iranian meta-analyses is increasing during the past 10 years, their quality is decreasing each year. This was apparent from the received number of citations.

Introduction

The past few years have witnessed a large increase in the use of systematic reviews in both medicine and the social sciences. A systematic review attempts to compare all empirical eligible evidence to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimize the bias. Key characteristics of a systematic review are:

• A clearly stated set of objectives with pre-defined eligibility criteria for academic studies;

• An explicit, reproducible methodology;

• A systematic search that attempts to identify all studies that would meet the eligibility criteria;

• An assessment of the validity of the findings of the included studies, for example, through the assessment of the risk of bias; and a systematic presentation, and synthesis of the characteristics and findings of the included studies (1).

Thus, a systematic review attempts to collate all relevant evidence that fit pre-specified eligibility criteria to answer a specific research question. It uses explicit, systematic methods to minimize bias in the identification, selection, synthesis, and summary of studies.

When done well, this provides reliable findings from which conclusions can be drawn and decisions made (2-4). The method is intended to identify the studies related to a common subject that aims to get unbiased knowledge in a comprehensive, systematic and replicable review of the scientific literature (5). Indeed, a systematic review is the reference standard for synthesizing evidence in health care and attempts to collate all relevant evidence that fit pre-specified eligibility criteria to answer a specific research question (4).

One of the subsets of a systematic review is a meta-analysis. As aptly defined by many authors, meta-analysis is a quantitative approach for systematically assessing the result of previous research in order to arrive at conclusions about the body of research (6-9) or it is the use of statistical techniques to combine and summarize the results of multiple studies; they may present a systematic review. By combining data from several studies, meta-analyses can provide more precise estimates of the effects of health care than those derived from the individual studies (4).

PRISMA is an evidence-based minimum set of items for reporting systematic reviews and meta-analyses. PRISMA not only focuses on the reporting of reviews evaluating randomized trials but also can be used as a basis for reporting systematic reviews of other types of research, particularly evaluations of interventions. This guideline prepares 27 preferred reporting items for systematic reviews and meta-analyses items (4). One item in this checklist describes the main criteria for evaluating abstract as one of the most important parts of each article. A well-written abstract provides a description of a clinical problem or research question; the methods used to address it and the significant results and implications (10). In other words, the main function of an abstract of a systematic review should be to signal its systematic methodology. For most readers, the findings described in the abstract will also be of special significance, either as the sole part of an article that will be read and will determine whether to read the full text. Abstracts of systematic reviews are very important, as some readers cannot access the full paper, then abstracts may be the only option for gleaning research results (11).

Nowadays, we see a significant growth in the systematic reviews and meta-analysis studies published by researchers. If systematic reviews and meta-analysis articles do not structurally comply with internationally accepted protocols, they cannot effectively meet the objectives of these studies. Thus, in this study first we intended to describe the Iranian systematic reviews and meta-analysis productions indexed in WOS and Scopus and to evaluate their abstracts, based on the PRISMA checklist. Then, some scientometric indicators including distribution per year, total citation, average citations per year, average citations per documents and average citations per year in one article will be identified by searching in the WOS and Scopus Databases. In the end, the correlation between the abstract's PRISMA Scores, average citations per year, and publication year was calculated.

Methods

This is an analytical study. We did some “advanced search” in WOS and “document search” in Scopus during 2003-2012. The search strategies in WOS were as follows:

Title=("meta-analysis") OR Title=("meta analysis") OR Title=("meta_analysis") OR Title=("metaanalysis") OR Topic=("meta-analysis") OR Topic=("meta-analysis") OR Topic=("meta_analysis") OR Topic=("metaanalysis") OR Title=("systematic review") OR Title=("systematic reviews") OR Topic=("systematic review") OR Topic =("systematic reviews") Refined by: Countries /Territories=(Iran) Timespan=2003-2012. Databases =SCI-EXPANDED.

The search strategies in Scopus was:

(TITLE-ABS-KEY(systematic review) OR TITLE-ABS-KEY (systematic reviews) OR TITLE-ABS-KEY(meta-analysis) OR TITLE-ABS-KEY(meta-anlysis) OR TITLE-ABS-KEY(meta_anlysis) AND AFFILCOUNTRY(Iran)) AND SUBJAREA (mult OR agri OR bioc OR immu OR neur OR phar OR mult OR medi OR nurs OR vete OR dent OR heal) AND PUBYEAR >2002 AND PUBYEAR <2013 AND (LIMIT-TO (EXACTKEYWORD, "Systematic review") OR LIMIT-TO (EXACT KEYWORD, "meta-analysis")) AND PUBYEAR >2002 AND PUBYEAR <2013

Total number of retrieved records were 677 (Scopus (261) and WOS (416)) of which 158 were overlapped; So, with the overlaps dropped, 519 remained. Then, the 158 overlapped articles, which had been indexed in both the WOS & Scopus were also studied and labeled as “overlapped group”. Furthermore, there were some irrelevant articles indexed in WOS and Scopus. They included different issues like letters, editorials, congresses, incorrect affiliations, inaccessible abstracts, wrong research methodologies, and wrong titles. As such, after eliminating the irrelevant records 293 remained.

We evaluated 46 article abstracts indexed in WOS, 89 indexed in Scopus and 158 indexed in WOS and Scopus both (overlapped group). Quality of the abstracts were evaluated according to the PRISMA checklist for abstracts as its bibliographic information comes in the 6th references.

According to Figure 1 , we assigned one score for each item defined in Figure 2. Thus, the maximum score for article abstracts evaluated by PRISMA checklist is 12 and the minimum is zero.

Fig. 1

Fig. 2

The PRISMA for Abstracts Checklist.

Comparison of the rate of articles published during 2003 to 2012.

The average of received citations per year, the number of authors, the publication year and the organization affiliation were determined through searching in WOS and Scopus Databases analytical section. These indicators in “overlapped group” were obtained from Scopus database because this database covers more articles than ISI.

ANOVA and Duncan tests were used to compare the databases in point of the difference between PRISMA score, and article's average of received citation and the Pearson correlation coefficient was used to calculate the correlation between the abstract's PRISMA score with the average of received citations per year, the number of authors and the publication year.

Results

The results indicated that the number of Iranian systematic review and meta-analysis articles had an increasing rate during 2003 to 2012 (Table 1).

Table 1

Distribution of Iranian systematic review and meta-analysis articles publication year indexed in WOS, Scopus and Overlapped group

Publication year	Database
	WOS		Scopus		Overlapped group		Total
	n	%	n	%	n	%	n	%
2003	0	0	0	0	1	0.6	1	0.34
2004	0	0	1	1.1	1	0.6	2	0.68
2005	0	0	1	1.1	0	0	1	0.34
2006	3	6.5	4	4.5	4	2.5	11	3.75
2007	3	6.5	4	4.5	8	5.1	15	5.12
2008	1	2.2	3	3.4	18	11.4	22	7.51
2009	7	15.2	9	10.1	21	13.3	37	12.63
2010	10	21.7	5	5.6	29	18.4	44	15.02
2011	6	13	26	29.2	23	14.6	55	18.77
2012	16	34.8	36	40.4	53	33.5	105	35.84
Total	46	100	89	100	158	100	293	100

According to Figure 2, there was 1 article published in 2003 while this rate was 105 in 2012.

Figure 3 shows that the most of compliance with PRISMA criteria is observed in “Objectives” (98.6%) then “Synthesis of result” (85%) and “Title” (80.2%). The less compliance with PRISMA is observed in “Registration” (2%), ”Strengths & limitation of evidence” (3%), “Eligibility criteria” (3.4%) and “Risk of bias” (4.4%), respectively.

Fig. 3

Compliance with PRISMA in Iranian Systematic Reviews and Meta-analysis abstracts indexed in WOS and Scopus

Table 2 shows that the average score of Iranian systematic review and meta-analysis articles indexed in WOS and Scopus is 5.9, while the highest score could be 12. The highest compliance with PRISMA criteria is observed in 2003 while the lowest was in 2004. None of the abstracts in this study obtained the complete score which is 12.

Table 2

Average score of PRISMA for the article abstracts indexed in WOS, Scopus and overlapped group during 2003-2012

Year	WOS	Scopus	Overlapped group	Average score
2003	0	0	6.63	6.63
2004	0	4.59	5.5	5.05
2005	0	6.09	0	6.09
2006	6.39	4.88	6.17	5.81
2007	6.65	6.20	6.38	6.41
2008	4.37	5	7.03	5.47
2009	6.97	5.14	6.7	6.27
2010	5.96	5.77	6.63	6.12
2011	6.26	5	6.02	5.76
2012	6.11	4.73	6.36	5.73
Ave.	6.24	5	6.47	5.9
std	1.68	1.22	1.41	1.55

P value <0.001

Table 3 represents that the average rate of total citations in Iranian systematic review and meta-analysis articles publication year indexed in WOS during 2003 to 2012 was 16 (STD=24.67). The average rate of citation/year was 2 in 2008 in the minimum rate while it was 31.7 in 2007 in the maximum rate. The average rate of citation /documents was 3.4 in 2012 in the minimum rate while it was 46.7 in 2006 in the maximum rate. The average citation/year in one article was 3.4 (STD=3.39). Since there was no publication during 2003 - 2005, we were unable to determine other indicators.

Table 3

Average citation per year and per document of Iranian systematic reviews and meta-analysis articles in WOS

Publication year	The articles indexed in WOS
	Article numbers	Total citation	Average citations/ year	Average citations /documents	Average citation / year in one article
2003	0	0	0	0	0
2004	0	0	0	0	0
2005	0	0	0	0	0
2006	3	140	17.5	46.7	5.8
2007	3	222	31.7	74	10.6
2008	1	12	2	12	2
2009	7	136	27.2	19.4	3.9
2010	10	98	24.5	9.8	2.5
2011	6	75	25	12.5	4.2
2012	16	54	27	3.4	1.7
Total	46	737	154.9	177.8	30.6
Ave.	--	16	--	--	3.4
STD	--	24.67	--	--	3.39

Table 4 represents that the average rate of total citation in Iranian systematic review and meta-analysis articles publication year indexed in Scopus during 2003 to 2012 was 4.6 (STD=10.93). The average rate of citation/year was 0.1 in 2004 in the minimum rate while it was 40.5 in 2012 at the maximum rate. The average rate of Citation /documents was 1 in 2004 in the minimum rate while it was 17.7 in 2008 at the maximum rate. The average citation/ year in one article was 1.2 (STD=2.29). Since there was no publication during 2003 - 2005, we were unable to determine other indicators.

Table 4

Average citation per year and per document of Iranian systematic reviews and meta-analysis articles in Scopus

Publication year	The articles indexed in Scopus
	Article numbers	Total citation	Average citations/ year	Average citations /documents	Average citation / year in one article
2003	0	0	0	0	0
2004	1	1	0.1	1	0.1
2005	1	22	2.4	22	2.4
2006	4	6	0.8	1.5	0.2
2007	4	40	5.7	10	1.4
2008	3	53	8.8	17.7	2.9
2009	9	142	28.4	15.8	3.2
2010	5	21	5.3	4.2	1.1
2011	26	45	15	1.7	0.6
2012	36	81	40.5	2.3	1.1
Total	89	411	107	76.1	13
Ave.	--	4.6	--	--	1.2
Std.	--	10.93	--	--	2.29

Table 5 represents that the average rate of total citation in Iranian systematic review and meta-analysis articles publication year indexed in overlapped group during 2003 to 2012 was 19.9 (25.78 STD). The average rate of citation/year was 1.5 in 2003 in the minimum rate while it was 157.8 in 2009 at the maximum rate. The average rate of Citation /documents was 5 in 2012 in the minimum rate while it was 116 in 2004 at the maximum rate. The average citation/year in one article was 4.3 (STD=4.17). Since there was no publication during 2003 - 2005, we were unable to determine other indicators.

Table 5

The Status of Iranian systematic review and meta-analysis articles publication year indexed in Overlapped group*

Publication year	The articles indexed in Overlapped group
	Distribution	Total citation	Average citations/ year	Average citations /documents	Average citation / year in one article
2003	1	16	1.5	16	1.5
2004	1	116	11.6	116	11.6
2005	0	0	0	0	0
2006	4	149	18.6	37.3	4.7
2007	8	521	74.4	65.1	9.3
2008	18	726	121	40.3	6.7
2009	21	789	157.8	37.6	7.5
2010	29	392	98	13.5	3.4
2011	23	170	56.7	7.4	2.5
2012	53	264	132	5	2.5
Total	158	3143	671.6	338.2	49.6
Ave.	---	19.9	---	---	4.3
Std.	---	25.78	---	---	4.17

*As we mentioned above, overlapped articles are the ones that indexed in WOS & Scopus both. The indicators in “overlapped group” were obtained from Scopus database because this database covers more articles than ISI.

Table 6 represents that the average rate of total citation in Iranian systematic review and meta-analysis articles publication year indexed in all databases during 2003 to 2012 was 14.6 (STD=23.04). The average rate of citation /year was 1.5 in 2003 in the minimum rate while it was 213.4 in 2009 at the maximum rate. The average rate of citation /documents was 3.8 in 2012 in the minimum rate while it was 58.5 in 2004 at the maximum rate. The average citation / year in one article was 3.2 (STD=3.90).

Table 6

Average citation per year and per document of Iranian systematic reviews and meta-analysis articles in WOS, Scopus & Overlapped group.

Publication year	The articles indexed in WOS, Scopus and Overlapped group
	Distribution	Total citation	Average citations/ year	Average citations /documents	Average citation / year in one article
2003	1	16	1.5	16	1.5
2004	2	117	11.7	58.5	5.9
2005	1	22	2.4	22	2.4
2006	11	295	36.9	26.8	3.4
2007	15	783	111.9	52.2	7.5
2008	22	791	131.8	36	6
2009	37	1067	213.4	28.8	5.8
2010	44	511	127.8	11.6	2.9
2011	55	290	96.7	5.3	1.8
2012	105	399	199.5	3.8	1.9
Total	293	4291	933.5	261	38.9
Ave.	---	14.6	---	---	3.2
Std.	---	23.04	---	---	3.90

Table 7 shows that if we consider the results of Tables 3-7, there is a positive correlation between PRISMA score and average citation/year (p<0.001, r=0.309). It means that the more PRISMA score rate, the more citation/ year.

Table 7

Pearson correlation coefficient test result for determining the relation between PRISMA score and average citation / year

Database	r	P
WOS	0.305	0.039
Scopus	0.210	0.048
Overlapped group	0.159	0.046
Total	0.309	0.001

Table 8 shows that if we consider the results of Table 3, (As we mentioned above, the average score of Iranian systematic review and meta-analysis articles indexed in WOS and Scopus is 5.9, while the highest score could be 12. The highest compliance with PRISMA criteria is observed in 2003 while the less of them is in 2004. None of the abstracts in this study could obtain the complete score which is 12), there is a negative correlation between PRISMA score and publication year (P=0.021, r=-0.135).

Table 8

Pearson correlation coefficient test result for determining the relation between PRISMA score and publication year

Database	r	P
WOS	-0.078	0.605
Scopus	-0.175	0.100
Overlapped group	-0.086	0.285
Total	-0.135	0.021

It means that from 2003 to 2012, the quality of abstracts according to PRISMA checklist is decreased.

Discussion

Nowadays, evidence-based studies have become an important concept in public health and community health promotion. In most evidence paradigms, systematic reviews, along with meta-analyses, are most highly valued (12). As the importance of systematic review conclusions relies upon the scientific rigor of methods and the currency of evidence, we aimed to evaluate Iranian systematic reviews and meta-analysis abstracts and clarify the real situation of articles including their strengths and weaknesses in an attempt to offer some recommendation to enhance the quality of the articles.

According to the results, PRISMA checklist complied differently with various sections of the abstracts as follows: “Registration”(2%), “Strengths & limitation of evidence”(3%), “Eligibility criteria” (3.4%), “Risk of bias” (4.4%), “Included study” (5.8%), “Funding” (25.3%) , “Information sources” (40.6%), “Description of the effect” (44%); on the other hand, the PRISMA showed the highest compliance with “Objectives” (98.6%) , “Synthesis of result” (85%) , “Title” (80.2%) and “Interpretation” (70%), respectively. Ideal systematic review and meta-analysis should be conducted comprehensively, by maximizing the precision level, minimizing the bias and narrating the concepts clearly. In other words, out of 12 main items of the PRISMA checklist considered for systematic reviews & meta-analysis abstracts evaluation; only three items (25%) were reported higher than 50%. These results are consistent with the findings of Le et. Al (13), Xiao et al. (14) and Willis et al. (15) results who reported low quality for a set of articles published in China, according to PRISMA checklist. They reported that “compliance with the PRISMA statement was generally poor: none of the reviews completely adhered to all checklist items”. This may be the result of lack of awareness of authors of forms and standard checklists. Similar problems have been indicated in other related studies as well (e.g. Hróbjartsson, 2009). According to Hróbjartsson (2009) research authorities are required to consider to the development of international guidelines for the reporting of trial protocols and public access to the protocols (16). Added to this is the important role that can be accomplished by the journal editors. They can consider a specific framework, based on accepted criteria (e.g PRISMA), for the reviewing process. This framework may then lead to the publication of scientifically more qualified articles. As Soori (2002) said, most scientific journals in Iran and many other countries worldwide do not have proper specific instruction for writing and reviewing epidemiologic articles (17).

This study shows that the quantity of Iranian meta-analysis and systematic reviews’ articles were increased during 2003-2012 while their citations have decreased. Greater attention should be paid to the quality of the articles rather than just the number of publications. According to Valayi et al. there is a sharp increase in the number of peer-reviewed articles in Iran in recent years. The quality of the articles, however, was not satisfactory (18).

Our results showed that there is a positive correlation between PRISMA score and average citation per year, which means greater PRISMA score, can result in the more citations/years. So, the low quality of Iranian systematic review and meta-analysis articles could have a direct impact on general aspect of rating based on citation levels. This result, as a predictable outcome, is approved by Panic et al. who found that the endorsement of PRISMA can improve both the methodology of the study and the way it is reported (19-21). Therefore, it is better to pay attention to “reporting guidelines” which have been developed to improve the quality of the manuscript as well.

Conclusion

Compliance with the PRISMA statement in abstracts of Iranian systematic reviews and meta-analysis was poor. None of the reviews completely observed all the PRISMA checklist items. It can be concluded that the more compliance with the criteria, the more citations to systematic reviews and meta-analysis studies. Thus, by paying more attention to the accepted publication criteria, the authors can obtain more citation impact.

Acknowledgements

This study is based on the a thesis, entitled " Assessment of Iranian systematic reviews and meta-analysis indexed in Web of Sciences & Scopus with PRISMA checklist and determine the correlation between the obtained results with scientometric indicators" which was presented to the Department of Medical Library and Information Science, Paramedical School, Shahid Beheshti University of Medical Sciences.

Conflict of Interests: None declared.