Comparative Analysis of Web of Science and Pubmed Indexed Medical Journals Published in Former Yugoslav Countries.

INTRODUCTION: The source of scientific information, methods for their evaluation, and methodology of their use are critical for serious scientific research and publishing of the scientific research results. Certain methodological principles should be inexcusably followed when designing clinical or observational research to avoid bias and presentation of results that do not reflect the truth about the phenomenon that is the object of the study. AIM: The aim of this study was to compare the methodological quality of clinical trials and observational studies published in medical journals from ex-Yugoslav countries indexed in Web of Science (WoS) and Pubmed/MEDLINE.
METHODS: Clinical studies published in medical journals of ex-Yugoslav countries were retrieved from the WoS and Pubmed database, and the sample for analysis was randomly chosen from the retrieved publications. The rate of the most common errors in the design of clinical/observational studies was established by a careful reading of the sampled publications and their checking against predefined criteria.
RESULTS: Number and percent of the evaluated studies that failed to meet each of the methodological criteria tested, number of the evaluated criteria not satisfied per database and number of studies that satisfied more than 4 criteria were analyzed per database. When explanatory potential of journal impact factor, number of citations, time elapsed from publication and a database where a journal is referred were tested by linear regression in regard to the number of methodological criteria satisfied per study, the linear regression model was obtained by backward deletion method and achieved R2 adjusted of 0.166 (F=13.827, df1 = 2, df2 = 127, p=0.000). The methodological quality of studies was directly related to impact factor of the journals (B = 0.976, 95% confidence interval 0.539 - 1.413, p=0.000) and inversely with the database where a journal is referred (B =-0.444, 95% confidence interval-0.824 - -0.064, p = 0.022). Each additional unit of impact factor increased number of satisfied methodological criteria for about 1, while referring a journal only in WoS decreased number of satisfied criteria for 0.45 points in comparison with journals referred in both WoS and Pubmed/MEDLINE, and for 0.9 points in comparison to journals referred only in MEDLINE.
CONCLUSION: Methodological and scientometric quality of clinical studies published in medical journals from ex-Yugoslav region varies significantly, and the variations are higher in journals referenced only in WoS than in journals referenced in Pubmed/MEDLINE only, or in both Pubmed and Web of Science databases.

INTRODUCTION

A scientific research is a process that has several distinct components.(1) These are: To identify the key research questions, choice of scientific approach for the study and data collection, data analysis, and reporting on results (2) Science and technology play a key role in the development of modern society and scientific research, and if they stand on the ethical principles, they can certainly provide answers to others to remix, tweak, and build upon the previous work non-commercially, as long as the author is credited and the new creations are licensed under the identical terms (2). The production and exchange of knowledge are important issues of human existence that are reflected in relevant scientific communication, which is established and implemented by scientific publications (1). The reliability and soundness of the scientific knowledge of each scientist and researcher should be important for him/her and for the professional community to which it belongs. In this way, the doors would potentially turn open for their successful scientific and possible academic career.

The source of scientific information, methods for their evaluation, and methodology of their use are critical for serious scientific research and publishing of the scientific research results (1). The society normally imposes to science some rules of conduct and rules of the game. But scientific knowledge still largely depends on the actions that are at least in the initial phase and basically individual. It generally depends, to a large extent, on the creativity, skills, and talents of individuals. Creativity and critical thinking are just some of the essential characteristics of the scientific research process (2). From the standpoint of content, in order to distinguish papers with scientific ambitions from those which are called professional, we must recognize some of their basic characteristics.

Scientometrics is part of Scientology (Science of sciences) that analyzes scientific publications and their citation in the selected sample in scientific journals (1-3). Scientometrics is the science of measuring and analyzing science using qualitative, quantitative and computational approaches (2).

An important way of systematically reviewing studies is through meta-analysis (1). Meta-analysis is a statistical and analytical method that combines and synthesizes various mutually independent studies and integrates their results into a single, common result. If well designed and properly implemented, it can be a very powerful tool for proving hypotheses. It is based on strictly established mathematical and statistical principles for critical analysis of medical data. If the results are obtained by proper meta-analysis supervised by experts, they are considered valid and there is no need for further testing (2).

Meta-analysis became indispensable in understanding a large collection of raw data or literature that is sometimes contradictory, inconsistent, and unclear on a topic, and in understanding the true importance of statistical results when addressing a scientific topic, such as efficiency.

Scientometric quality of articles published in medical journals is a function of the quality of the journal itself, but also of the major database where the journal is referenced. It is well known that quality of the articles also vary over time (1), underlining its multifactorial nature. Although Web of Science (WoS) covers the most cited articles published in the journals with the highest impact factor (2), is also hosts a multitude of low impact factor journals which publish articles with very low scientometric rank. The latter is especially true for journals coming from certain geographical regions with moderate research potentials. Another major database of medical journals, Pubmed/MEDLINE, also covers journals of various scientometric quality (3), but has somewhat different inclusion and exclusion criteria from WoS. In practice, it often happens in certain developing regions that some medical journals are covered only by WoS, some only by Pubmed/MEDLINE and other by both databases. Such situation creates different opportunities for the journals, since depending on which database is preferred by national scientific authorities (e.g. Ministry of Science) and included in scores for ranking national journals, regional authors of scientific papers will choose some of the journals, and ignore the others. Region that covers former Yugoslav countries is just one of the kind, and regional medical journals are differently valued by local authors depending on their reference status in WoS and Pubmed/MEDLINE. However, whether the methodological validity of the articles (i.e. their true quality) follows regional popularity or prestige of WoS or MEDLINE remains to be established.

AIM

The aim of our study was to compare methodological and scientometric characteristics of clinical trials or observational clinical studies published in journals from former Yugoslav countries and referenced in either WoS, Pubmed/MEDLINE or both databases.

METHODS

Our study was of the cross/sectional type. The studies were retrieved for analysis from the journals referred in PubMed/MEDLINE (4), Web of Science database - Master journal list (5), or in the both databases. The published articles were included into the study based on the following criteria: journal article, published in a journal issued in Slovenia, Croatia, Bosnia and Herzegovina, Serbia, Montenegro or North Macedonia, original clinical trial, original observational clinical study, and full text availability. The exclusion criteria were: articles published in a language other than English or Bosnian/Croatian/Serbian, review articles, animal studies, in vitro studies, modeling studies and case reports or case series. In order to create and employ search strategies, at first the following journals published in Ex-Yugoslav countries were identified: journals referred in Web of Science (WoS) and not in Pubmed/MEDLINE database (Journal of Medical Biochemistry, Acta Facultatis Medicae Naissensis, Acta Stomatologica Croatica, Alcoholism and Psychiatry Research, Signa Vitae, Onkologija, Balkan Journal of Medical Genetics, Montenegrin Journal of Sports Science and Medicine, Vojnosanitetski Pregled, and Srpski Arhiv za Celokupno Lekarstvo), journals referred only in Pubmed/MEDLINE database, and not in WoS (Acta medico-historica adriatica, Acta Informatica Medica, Materia Socio-Medica, Medical Archives, Acta Medica Academica and Medicinski Glasnik), and journals referred in both databases (Acta Dermatovenerologica Alpina, Pannonica, et Adriatica, Radiology and Oncology, Zdravstveno Varstvo, Acta Dermatovenerologica Croatica, Biochemia Medica, Acta Pharmaceutica (Zagreb, Croatia), Croatian Medical Journal, Acta Clinica Croatica, Psychiatria Danubina, Arhiv za Higijenu rada i Toksikologiju, Bosnian Journal of Basic Medical Sciences and Prilozi). The following search strategy was used for each country separately to implement inclusion and exclusion criteria and select clinical/observational studies for further analysis: ("title of the journal 1" [All Fields] OR „title of the journal 2“ [All Fields] OR ...) AND (("observational study"[Publication Type] OR "observational studies as topic"[MeSH Terms] OR "observational study"[All Fields]) OR ("clinical trial"[Publication Type] OR "clinical trials as topic"[MeSH Terms] OR "clinical trial"[All Fields])) Filters: Free full text. The data for journals referred in both WoS and PubMed were used from our previous study (6).

Size of the study sample in regard to the number of studies per database (n=43) was calculated on the basis of the following assumptions: rate of inappropriate research design 0.5 (7) and width of the 95% confidence interval ± 0.15. The formula n = (1.96)2 x 4*p*(1-p)/d2 was used for the calculation, where „n“ is the sample size, „p“ probability of inappropriate research design and „d“ width of the confidence interval (8). Since the studies retrieved were numbered orderly in the respective database, the study sample was extracted by simple randomization, using random number generator in Excel. The sample of studies published in journals referenced in both WoS and PubMed was used as analyzed in our previous study (6).

The extracted clinical/observational studies were analyzed for common errors in design and statistics, as earlier described in the literature (searching for data distribution, randomization techniques, inclusion and exclusion criteria, intention-to-treat analysis, etc.) (9). For the purpose of analysis of the extracted studies, the checklist with 9 questions was prepared (Table 2). Results of the analysis of the extracted studies are shown in the Tables 1 and 2. Number of citations for each study was taken from Google Scholar after performing search with the study title. The results were described by rates and percentages when categorical, and by means, standard deviations, medians and interquartile ranges, if continuous. Normality of the data distribution was checked by Kolmogorov-Smirnov test, and if not achieved, Kruskal-Wallis nonparametric analysis of variance was used for comparisons among the databases. Values of categorical variables were compared among the databases by Chi-square test or by Fisher’s exact test, if assumptions for the Chi-square test were not met. Association between number of satisfied methodological criteria and journal impact factor, number of citations or time elapsed from the publication date was tested by multiple linear regression. Maximum acceptable probability of null hypothesis was set at 0.05. All calculations were performed by SPSS statistical program, version 18.

Table 2.

Results of the methodological evaluation. * probability of null hypothesis calculated by Chi-square test or Fisher’s exact test (if frequencies less than five) for categorical data, and by Kruskal-Wallis non-parametric analysis of variance for continuous data

	Indexed only in WoS	Indexed only in Pubmed/MEDLINE	Indexed both in WoS and Pubmed/MEDLINE	p*
Failure to specify the inclusion and exclusion criteria?	23 (54%)	7 (16%)	4 (9%)	0.000
Failure to determine and report the error of yourmeasurement methods?	19 (44%)	10 (23%)	19 (44%)	0.068
Failure to specify the exact statistical assumptionsmade in the analysis?	31 (72%)	31 (72%)	22 (51%)	0.063
Failure to perform sample size analysis before thestudy begins?	39 (91%)	39 (91%)	37 (86%)	0.725
Failure to implement adequate bias control measures?	31 (72%)	23 (54%)	21 (49%)	0.069
Failure to vigorously recruit and retain subjects?	0 (0%)	1 (2%)	1 (2%)	0.601
Failure to examine for normality of the data?	25 (58%)	29 (67%)	21 (49%)	0.217
Failure to report missing data, dropped subjectsand use of an intention to treat analysis?	38 (88%)	25 (58%)	11 (26%)	0.000
Failure to point out the weaknesses of your own study?	26 (60%)	24 (56%)	12 (28%)	0.005
Number of satisfied criteria per study: mean, standard deviation, median, interquartile range	3.6 ± 1.8,3.0, [3.0]	4.7 ± 2.0,5.0, [3.0]	5.5 ± 1.5,6.0, [2.0]	0.000
Number of studies that satisfied more than 4 criteria	12 (28%)	23 (52%)	32 (74%)	0.000

Table 1.

Characteristics of the study samples per country. * standard deviation, ** interquartile range, *** probability of null hypothesis calculated by Kruskal-Wallis non-parametric analysis of variance

	Indexed only in WoS	Indexed only in Pubmed/MEDLINE	Indexed both in WoS and Pubmed/MEDLINE	p***
Number of medical journals publishing clinical studies	10	6	12	-
Number of retrieved studies	523	91	285	-
Size of the evaluated random sample	43	43	43	-
Years passed from a study publication (mean, SD*, median, IQR**)	3.2 ± 2.6,2.0 [4.0]	4.3 ± 2.5,4.0 [4.0]	10.1 ± 6.7,7.0 [11.0]	0.000
Number of citations (mean, SD*, median, IQR**)	2.9 ± 4.0,1.0 [5.0]	8.5 ± 13.6,4.0 [10.5]	11.2 ± 18.2,6.0 [7.0]	0.008
Impact factor (mean, SD*, median, IQR**)	0.3 ± 0.4,0.2 [0.2]	0.6 ± 0.3,0.6 [0.5]	1.3 ± 0.9,1.0 [1.0]	0.000
Percent of foreign studies in the sample	34%	49%	35%	-

RESULTS

There are ten biomedical journals from the six ex-Yugoslav countries referred only in WoS and not in Pubmed/MEDLINE database: Journal of Medical Biochemistry, Acta Facultatis Medicae Naissensis, Acta Stomatologica Croatica, Alcoholism and Psychiatry Research, Signa Vitae, Onkologija, Balkan Journal of Medical Genetics, Montenegrin Journal of Sports Science and Medicine, Vojnosanitetski pregled, and Srpski Arhiv za Celokupno Lekarstvo. Another six medical journals are referred only in Pubmed/MEDLINE database, and not in WoS (Acta Medico-historica Adriatica, Acta Informatica Medica, Materia Socio-Medica, Medical Archives, Acta Medica Academica and Medicinski Glasnik). Finally, both in WoS and MEDLINE are referred the following medical journals published in Ex-Yugoslav countries: Acta Dermatovenerologica Alpina, Pannonica, et Adriatica, Radiology and Oncology, Zdravstveno Varstvo, Acta Dermatovenerologica Croatica, Biochemia Medica, Acta Pharmaceutica (Zagreb, Croatia), Croatian Medical Journal, Acta Clinica Croatica, Psychiatria Danubina, Arhiv za Higijenu Rada i Toksikologiju, Bosnian Journal of Basic Medical Sciences and Prilozi.

Number of medical journals indexed in WoS, Pubmed/MEDLINE, or both databases, number of retrieved studies, size of random samples, number of citations, years elapsed from the publication, impact factors of the journals in the year of publication and percent of the samples with authors from foreign countries are shown in the Table 1.

Number and percent of the evaluated studies that failed to meet each of the methodological criteria tested, number of the evaluated criteria not satisfied per database and number of studies that satisfied more than 4 criteria per database are presented in the Table 2. References of the analyzed studies referred in both WoS and MEDLINE could be found in our previous publication (6), while the references of the studies covered only by WoS or only by Pubmed/MEDLINE (10-96) medical journals are given at the end of this article (Table 3 and 4). When explanatory potential of journal impact factor, number of citations, time elapsed from publication and a database where a journal is referred were tested by linear regression in regard to the number of methodological criteria satisfied per study, the linear regression model was obtained by backward deletion method and achieved R2 adjusted of 0.166 (F=13.827, df1 = 2, df2 = 127, p=0.000). The methodological quality of studies was directly related to impact factor of the journals (B = 0.976, 95% confidence interval 0.539 – 1.413, p=0.000) and inversely with the database where a journal is referred (B =–0.444, 95% confidence interval–0.824 – -0.064, p = 0.022). Each additional unit of impact factor increased number of satisfied methodological criteria for about 1, while referring a journal only in WoS decreased number of satisfied criteria for 0.45 points in comparison with journals referred in both WoS and MEDLINE, and for 0.9 points in comparison to journals referred only in Pubmed/MEDLINE.

Table 3.

The list of articles published in the Journals referred in WoS only (first author and number in the cited list of references)

Author’s of the Study	Failure to specify the inclusion and exclusion criteria	Failure to determine and report the error of yourmeasurement methods	Failure to specify the exact statistical assumptionsmade in the analysis	Failure to perform sample size analysis before thestudy begins	Failure to implement adequate bias control measures	Failure to vigorously recruit and retain subjects	Failure to examine for normality of the data	Failure to report missing data, dropped subjectsand use of an intention to treat analysis	Failure to point out the weaknesses of your ownstudy	Citation	IF of a journal
Vilibić M, et al. (10)	No	Yes	No	No	Yes	No	No	No	No	1	0.390
Blažinović I, et al. (11)	No	NA	Yes	Yes	Yes	No	Yes	No	Yes	0	0.390
Jiang B, (12)	No	Yes	Yes	Yes	Yes	No	Yes	Yes	No	3	1.593
Sivaci R, et al. (14)	No	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	0	0.622
Zogović D , et al. (13)	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	No	7	0.919
de Almeida O, et al. (15)	Yes	Yes	Yes	Yes	Yes	No	No	Yes	Yes	1	0.403
Stefanović A, et al. (17)	No	No	Yes	Yes	Yes	No	No	Yes	No	3	1.592
Al-Zeidaneen AS, et al. (16)	No	Yes	Yes	Yes	Yes	No	Yes	Yes	No	0	0.240
Đorđević N, et al. (18)	Yes	NA	Yes	Yes	Yes	No	Yes	Yes	Yes	0	0.210
Živković VD, et al. (19)	Yes	NA	Yes	Yes	Yes	No	Yes	Yes	Yes	0	0.200
Kozomara S, et al. (20)	No	NA	Yes	Yes	Yes	No	Yes	Yes	Yes	1	0.200
Nešić I, et al. (21)	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	0	0.240
Gašpar M, et al. (22)	Yes	NA	Yes	Yes	Yes	No	Yes	Yes	Yes	1	0.220
Haliti F, et al. (23)	Yes	NA	Yes	Yes	Yes	No	Yes	Yes	Yes	8	0.320
Alajbeg IŽ, et al. (24)	No	No	Yes	Yes	No	No	No	No	No	10	0.640
Čelebić A, et al. (25)	No	NA	Yes	Yes	Yes	No	No	No	No	1	0.640
Muhić E, et al. (26)	No	No	Yes	Yes	No	No	No	Yes	Yes	2	0.340
Erceg T, et al. (27)	Yes	NA	Yes	Yes	Yes	No	Yes	Yes	No	1	0.390
Grizelj M, et al. (28)	Yes	Yes	No	Yes	Yes	No	No	Yes	Yes	1	0.390
Lotar Rihtarić M, et al. (29)	No	Yes	Yes	Yes	Yes	No	Yes	Yes	No	5	0.390
Sidlauskiene A, et al. (30)	No	No	No	Yes	No	No	No	Yes	No	3	0.152
Grbović V, et al. (31)	No	NA	No	Yes	No	No	No	Yes	Yes	0	0.152
Stefanović Z, et al. (32)	No	NA	Yes	Yes	Yes	No	No	Yes	Yes	0	0.152
Ivanović S, et al. (33)	No	No	No	No	Yes	No	No	Yes	Yes	4	0.267
Gajić M, et al. (34)	Yes	No	Yes	Yes	No	No	Yes	Yes	Yes	5	0.267
Kim JY, et al. (35)	Yes	NA	No	Yes	Yes	No	Yes	Yes	No	0	0.170
Karlović Z, et al. (36)	No	NA	No	Yes	No	No	No	Yes	No	0	0.173
Prkačin I, et al. (37)	Yes	NA	Yes	Yes	Yes	No	Yes	Yes	Yes	4	0.126
Kosmač N, et al. (38)	Yes	Yes	No	No	No	No	No	No	Yes	0	0.136
Marić N, et al. (39)	Yes	NA	Yes	Yes	NA	NA	Yes	Yes	Yes	2	0.154
Marciuš VŽ. et al. (40)	Yes	Yes	No	Yes	Yes	No	No	Yes	Yes	0	0
Kukovica MG, et al. (41)	No	NA	Yes	Yes	Yes	No	Yes	Yes	No	0	0
Škof E, et al. (42)	Yes	NA	Yes	Yes	Yes	No	Yes	Yes	Yes	0	0
Demir S, et al. (43)	Yes	Yes	Yes	Yes	Yes	NA	Yes	Yes	Yes	0	0.692
Terzic M, et al. (44)	No	Yes	Yes	Yes	Yes	No	yes	yes	No	0	0.692
Vachev TI, et al. (45)	No	Yes	No	Yes	No	No	No	Yes	Yes	5	0.553
Khabour OF, et al. (46)	No	Yes	No	No	No	No	No	Yes	Yes	6	0.200
Sobti RC, et al. (47)	Yes	Yes	No	Yes	Yes	No	No	Yes	Yes	5	0.375
Wertheimer V, et al. (48)	Yes	Yes	Yes	Yes	No	NA	Yes	Yes	No	7	0
Arazi H, et al. (49)	Yes	Yes	Yes	Yes	No	No	No	Yes	Yes	6	0
Muratović A, et al. (50)	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	No	18	0
Zhang Y, et al. (51)	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	15	0
Brás R, et al. (52)	Yes	No	Yes	Yes	Yes	No	Yes	Yes	No	1	0

Table 4.

The list of articles published in journals referred in Pubmed only (first author and number in the cited list of references)

Author’s of the Study	Failure to specify the inclusion and exclusion criteria	Failure to determine and report the error of yourmeasurement methods	Failure to specify the exact statistical assumptionsmade in the analysis	Failure to perform sample size analysis before thestudy begins	Failure to implement adequate bias control measures	Failure to vigorously recruit and retain subjects	Failure to examine for normality of the data	Failure to report missing data, dropped subjectsand use of an intention to treat analysis	Failure to point out the weaknesses of your ownstudy	Citation	IF of a journal
Duc NM, et al. (53)	Yes	NA	Yes	NA	NA	No	Yes	NA	No	0	1.03
Askari-Majdabadi H, et al. (54)	Yes	NA	Yes	Yes	Yes	No	Yes	No	No	0	1.03
Selmanovic S, et al. (55)	No	Yes	Yes	Yes	No	No	Yes	Yes	Yes	11	0.76
Gholipour Baradari A, et al. (56)	No	No	No	Yes	No	No	No	No	No	6	0.76
Lajqi N, et al. (57)	Yes	No	Yes	Yes	Yes	No	Yes	Yes	Yes	14	0.90
Berisha M, et al. (58)	No	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	0	1.17
Gojkov-Vukelic M, et al. (59)	No	No	Yes	Yes	Yes	No	Yes	Yes	Yes	32	0.99
Forootan M, et al. (60)	No	NA	Yes	Yes	No	No	Yes	No	No	0	0.93
Latsou D, et al. (61)	No	No	No	No	No	NA	No	No	No	3	0.69
Moghadam MP, et al. (62)	No	No	Yes	Yes	No	No	Yes	Yes	Yes	4	0.78
Jafarzadeh L, et al. (63)	No	NA	Yes	Yes	Yes	Yes	Yes	No	Yes	13	0.86
Jafari F, et al. (64)	No	No	Yes	Yes	Yes	No	Yes	Yes	Yes	14	0.63
Petrovic M, et al. (65)	No	Yes	Yes	Yes	Yes	Not clear	Yes	Yes	Yes	10	0.25
Garousi S, et al. (66)	Yes	No	Yes	Yes	No	Not clear	Yes	Yes	No	5	0.25
Rohani F, et al. (67)	No	NA	Yes	Yes	Yes	No	Yes	Yes	Yes	0	0.76
Hoxha A, et al. (68)	No	NA	Yes	Yes	No	No	Yes	Yes	No	3	0.76
Torkaman A, et al. (69)	No	NA	Yes	Yes	Yes	No	Yes	No	Yes	4	0.61
Vanis-Vatrenjak S, et al. (70)	No	NA	Yes	Yes	Yes	No	Yes	No	Yes	2	0.61
Haxhibeqiri-Karabdic I. (71)	No	NA	No	Yes	NA	No	No	No	Yes	32	0.65
Djedovic M, et al. (72)	No	Yes	Yes	Yes	Yes	No	Yes	No	Yes	4	0.66
Rahmanovic E, et al. (73)	No	NA	No	Yes	No	No	Yes	Yes	Yes	3	0.55
Zonić-Imamović M, et al. (74)	No	NA	Yes	Yes	No	No	No	Yes	Yes	0	0.17
Bajrić B, et al. (75)	No	No	No	Yes	Yes	No	No	No	No	2	0.17
Račić M, et al. (76)	No	No	Yes	Yes	NA	No	Yes	Yes	No	1	0.27
Godinjak A, et al. (77)	No	No	Yes	Yes	NA	No	Yes	No	No	47	0.14
Meneses Calderón J, et al. (78)	No	NA	Yes	Yes	No	No	Yes	Yes	No	2	0.14
Krstović-Spremo V, et al. (79)	No	No	Yes	No	No	No	Yes	No	No	11	0.27
Loghmani A, et al. (80)	No	No	No	No	No	No	No	No	No	66	0.27
Iglica A, et al. (81)	No	NA	No	Yes	Yes	No	Yes	No	Yes	0	0.38
Rakanović-Todić M, et al. (82)	No	NA	Yes	Yes	Yes	No	Yes	Yes	Yes	0	0.38
Vegar-Zubović S, et al. (83)	No	NA	No	Yes	No	No	No	Yes	Yes	5	0.29
Okanović A, et al. (84)	No	Yes	No	Yes	Yes	No	No	Yes	No	33	0.28
Mujarić E, et al. (85)	No	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	6	0.66
Mehmedović A, et al. (86)	No	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	6	0.53
Gholipour Baradari A, et al. (87)	No	Yes	Yes	Yes	No	No	Yes	No	No	6	0.28
Stevanović S, et al. (88)	No	No	Yes	Yes	Yes	No	Yes	NA	No	13	0.12
Nouraei SM, et al. (89)	No	No	No	No	No	No	No	No	No	4	0.61
Ðug H, et al. (90)	Yes	Yes	Yes	Yes	NA	No	No	Yes	Yes	2	0.29
Asheghan M, et al. (91)	No	NA	Yes	Yes	Yes	No	Yes	Yes	Yes	2	0.78
Stanetić K, et al. (92)	Yes	No	No	Yes	Yes	No	No	Yes	No	0	0.17
Alimian M, et al. (93)	No	Yes	No	Yes	Yes	No	No	Yes	Yes	5	0.65
Alkhaldi HM, et al. (94)	Yes	NA	Yes	Yes	Yes	No	NA	Yes	Yes	2	0.61
Esparza M, et al. (95)	No	NA	Yes	Yes	Yes	No	NA	Yes	No	0	0.17
Secik Arkin F, et al. (96)	No	No	No	Yes	No	No	No	NA	No	0	1.03

Interesting facts are shown in Table 3 and 4 (in the last two columns) - that papers with numbers 50 (Muratovic et al.) and 51 (Zhang et al.) have the highest number of citations but papers are published in WoS indexed journals with IF 0. Or, the fact is that the most cited papers published in Pubmed journals (Loghman et al. - 66 citations, Haxhibekiri-Karabdic et al. - 32 citations), were published in journals with relatively low IF (0,27 and 0,65), but Duc et al. has not citations, and he published paper in the journal with higher IF than those of 95% of journals where the analyzed studies in our sample were published. Also, citation of the paper from Pubmed list, published by author Gojkov-Vekelic et al. in journal indexed in Pubmed has higher number of citations, and journal in which paper is published has higher IF, but paper belongs to Dental medicine scientific area, not to areas of medicine, which are more important or attractive, or have more influence in medical sciences - within the scientific medical subdisciplines, like Internal medicine or Surgery, etc. In this study we did not compare authors end their achieved, e.g. number of citations, or IF scores of the journals, for how long time journals existed, in which country journals are printed, or how long time journals are deposited in WoS or Pubmed/MEDLINE databases, or to which scientific fields journals belong, or with score of Google Scholar indexes of authors which are included in our meta-analysis. It will be, probably, scope of our next analysis in the future. Generally, we proved that number of citations and IF scores in our analysis was not strongly connected with belonging of the journals to WoS or Pubmed/MEDLINE databases.

DISCUSSION

Scientometrics with its various indices is a reliable method for evaluation of scientific development (1, 2, 97). Name bibliometrics in the seventies was introduced to denote a quantitative study of the communication process using mathematical and statistical methods to books and other media of communication (6). More specifically, in 1969 was introduced the name scientometrics relating to scientific field that deals with the study of science as an information process by applying quantitative (statistical) method, and later Tibor Braun (founder of the International Journal of Scientometrics), introduced the name Scientometrics (97). Modern Scientometrics is based largely on the work of Derek J. de Solla Price and Eugene Garfield, founder of Institute for Scientific Information (ISI), named as the father of Scientometrics and methods of evaluation of scientific publications (1, 97, 102). Garfield developed several factors that allow the assessment value and importance of scientific publications, including the most important impact factor (IF) and the H-index. IF is the number of citations of articles published in the journal during the previous two years divided by the total number of articles published in the journal during the same period (97, 102).

In the August 2005, Jorge Hirsch introduced a new indicator for quantifying the research output of scientists (1, 97). Hirsch’s so-called H index was proposed as an alternative to other bibliometric indicators - such as the number of publications, the average number of citations and the sum of all citations (97) - and is defined as follows: “A scientist has index h if h of his or her Np papers have at least h citations each and the other (Np - h) papers have ≤ h citations each” (102). All papers by a scientist that have at least h citations are called the “Hirsch core” (13). An H-index of 5 means that a scientist has published five papers that each have at least five citations (97). An H-index of 0 does not inevitably indicate that a scientist has been completely inactive: he or she might have already published a number of papers, but if none of the papers was cited at least once, the H-index is 0 (97). H-index is an index that attempts to measure the productivity and impact of published work of scientists (the index is based on the basis of the most cited papers and the number of citations that papers received in other publications). One criticism is that the excess citations, i.e., all citations which exceed h for a given publication do not have an effect. Therefore Egghe (97, 102) proposed the g-index which is given by the largest number g of papers which have received at least g citations on average (It was argued recently that the g-index is a measure of a researcher’s specific impact (i.e., impact per paper) as much as it is a measure of overall impact) (97). The main difference between the g-index and the H-index is that the former penalizes consistency of impact whereas the latter rewards such consistency (it is concluded that the H-index is a better bibliometric tool than is the g-index) (1, 2).

For all authors is very important question for thinking before making decisions about his/her written paper: How to choose an academic journal for their research works? (97, 98). "Find out information about the relative quality of a journal either by using the programs Publish or Perish and Scopus to obtain data on the impact factor and citation rate of journals in your area of interest; by accessing Web of Knowledge or Science through your institution to obtain rankings of the journals in a particular area; or by visiting the journal’s website to find out information about the journal in question" (98, 99).

Our study showed that studies published in journals from former Yugoslav countries were of the highest methodological quality if referenced in both WoS and Pubmed/MEDLINE, and of the lowest if referenced only in WoS. Especially constructive self-critique, better definition of inclusion and exclusion criteria and missing data were more prevalent in studies published at journals covered by Pubmed/MEDLINE. Methodological quality was correlated with scientometric quality, since impact factor of the journals and number of citations were also higher for articles covered by Pubmed/MEDLINE or both WoS and MEDLINE in comparison to those referenced only in WoS.

This article represents a continuation of consideration on a very important and methodologically significant problem within the scientific field - Scientometry, especially in the field of biomedical sciences, about which the experiences in written form are very scarce. Few authors in the field of medicine in the former Yugoslavia have dealt with this aspect (several articles in this field have been published by the authors: Lackovic, Rumbolt, Srica, Marusic, mostly descriptively without meta-analysis), and written experiences are insufficient even globally in medical sciences. Prior to its disintegration in 1991, the former Yugoslavia had over 22 million inhabitants in 6 independent states and over 15 medical and dental faculties, and more than 15,000 teachers elected in academic titles in over 40 medical scientific disciplines. Also, more than 100 indexed biomedical journals have been continuously published.

These indicators are presented in more detail in several articles by both authors of this article and readers can find more about this topic from them (99-102). This study, with its results of applied meta-analyzes and a unique and specific approach, provides several important conclusions, but also messages and recommendations to readers, not only informative, but also educational, for authors who write articles, for reviewers who evaluate articles before publication, and especially for journal evaluators in their indexing application in index and citation databases.

The idea for this study arose after the First and the Second SWEPs (Mediterannean Seminar on Science Writing, Editing and Publishing), held in Sarajevo in 2016 and 2018, and after we in a previous study (103-105), were analyzing errors in statistical analysis of data in articles indexed in WoS journals, with IF ranging from 2.5 to 4.5, and found a large number of methodological errors. We came to the conclusion that the editors of journals and their associates, as well as mentors of master’s and doctoral theses (because a large number of published papers were taken from the content of these defended theses) had insufficient knowledge in the field of Medical Statistics. A special case are expert evaluators of potential journals for admission to index databases such as WoS, Medline, Scopus, etc. The experience we have with applications in these three databases indicate that expert evaluators take as leading criteria: scope of the journal, regional character of the journal, quality of the language used (they prefer US rather than UK English), technical features, quality of the journal web page, content of the Instructions for authors with COPE Guidelines and based on the ICMJE criteria (http://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html), "Sarajevo Declaration on Integrity and Visibility of Scholarly Journals" (104). other rules, etc.

In this article, we cannot analyze in detail the facts that are the arguments for our allegations, but we refer readers to the last two columns of Tables 3 and 4, which list the citations of articles and Impact Factors of journals in which articles are published and the correlation between them, and which indicate to and argument the above allegations. Let’s say for example: what is the correlation between WoS and PubMed indexing of the journal and Impact factors, the correlation between the length of existence (age) of the journal and Impact factors, the correlation between the importance of the topic of the article and Impact factors, etc. A special case for analysis is the role and significance of the effects of the List of Predatory Journals by American librarian Jeffrey Beall, whose list of Potential Predatory journals is cited by many authors, based on his own criteria that none of the world’s scientific and academic institutions have analyzed or accredited, but which could be officially used. His list has provoked a storm of outrage among thousands of publishers and editors of journals, who have been put in a position to be scientifically belittled by his criteria and list, and many authors to avoid them as potential journals in which to publish their articles. The authors which are quoting Beall and his list did not use a scientific method similar to ours at all - a method of meta-analyzing articles from his list in which they could confirm Beall’s assessments and the content and quality of articles from the list of predatory journals. An illustrative article by Refat Aljumili on a serious and critical approach has revealed what we have stated in this article: "The story of „Beall’s List“ started probably in early or late 2010 when a guy called Jeffrey Beall - a librarian at Auraria Library, University of Colorado, in Denver, Colorado - came up with a blog „Scholarly Open Access“, as well as a list of questionable journals and publishers, or as Jeffrey Beall likes to call it „Potential, possible or probable predatory scholarly open-access publishers“, and gave himself the right to ward academic scholarly academic publication" (98).

Shortly after this list established, Jeffrey Beall added many open-access publishers to it and continued to update it regularly - by adding to the list and removing from it - and introduced many authors and researchers to the assumption that Open Access Journals (OAJ) are essentially “Predatory publishers and low-quality journals”. Sadly, few authors, academics, or researchers who read or heard about “Beall’s List” without much previous knowledge of the academic publication situation in general and without any prior knowledge about Jeffrey Beall’s background and intentions in particular came away believing that “Beall’s list” is a recognized authority in evaluating scholarly journals (98). "Well, the truth is “Beall’s list” has no affiliation to any governing body or organization accredited to scholarly publishing, and has no legal or academic value. If you follow some of Beall’s work on his blog, and it makes no sense whatsoever! His official web blog exposed the truth about Jeffrey Beall, particularly Walt Crawford’s 2014 article “The Sad Case of Jeffrey Beall - Case and Insights”, which provides a very detailed history about OAJs and directly addresses some of the broader issues with “Beall’s list”., (available on: http://citesandinsights.info/civ14i4.pdf). (98, 99).

Additionally, two examples of arguments for statements in this article are Scopus indexed journals: at the https://blog.cabells.com/category/predatory-publishing/ we found interesting fact that 81 Vietnamese authors who published their articles in the 2019 IJEAT ("International Journal of Engineering and Advanced Technology") and were rewarded, were also indicated as predatory, but journal was indexed in Scopus in 2019 (https://www.scopus.com/sourceid/21100899502) (106). According to Scopus, in 2018, there were only 280 documents from International Journal of Engineering and Advanced Technology indexed in Scopus content coverage. Surprisingly, in 2019, number of documents, indexed in Scopus, raised up to 4455. It is remarkable that no external citations from these indexed documents were noticed. Eventually, coverage discontinued in Scopus in 2019 (Figure 1). We believe that the Content Selection and Advisory Board of Scopus should consider their workflow again when evaluating new applicants and indexing massively documents from new members to Scopus database like aforementioned example.

Figure 1.

Scopus discontinued indexing "International Journal of Engineering and Advanced Technology" in their content coverage in 2019.

Second case is journal "Folia Medica Facultatis Medicinae Universitatis Saraeviensis" (http://www.foliamedica.mf.unsa.ba/index.php/FM), (in journal has commonly been published papers of Medical faculty in Sarajevo teachers), which was also accepted to Scopus several years ago without serious evaluation (re-established after more than 20 years of brake). It is stated that its last issue is printed in March 2019, and journal is signed as Croatian journal which belongs to University of Zagreb, Croatia. But the journal is published in Sarajevo (Bosnia and Herzegovina), and its h-index is 1 (107).

On the other side, the journal Materia Socio-Medica, one of the oldest journals on the Public Health in Europe (established in 1978) was rejected by Scopus several times, while in this study it has papers with high IF (papers 60 to 64, with Ifs from 0.63 to 0.93), better than 50% of IFs of WoS journals in our study.

The most important facts is that Editors and reviewers need to approach the evaluation of manuscripts submitted to journals more responsibly. Editors and reviewers should not reject articles without arguments, nor accept articles without checking that the submitted articles are written in accordance with the appropriate elements of the methodology that guarantee impartiality and proper application of statistics, all in order to reach the scientific truth in medicine (97, 102, 108, 109).

This is reason why studies like this one need to be made much more frequently in the praxis, especially for evaluations of the journals quality.

Limitation of the study: Our study had several limitations: a) only two investigators from two of the six former Yugoslav countries rated all articles, which created an opportunity for rating bias; b) the criteria for methodological quality were based on just one reference, therefore some of important methodological qualities could have been missed; and c) gold standard for overall methodological quality of clinical trials and observational clinical studies is still lacking, therefore our results should be taken with a reserve.

CONCLUSION

Methodological and scientometric quality of clinical studies published in medical journals from ex-Yugoslav region varies significantly, and the variations are higher in journals referenced only in WoS than in journals referenced in Pubmed/MEDLINE only, or in both Pubmed and Web of Science databases. Editors and reviewers of the submitted papers need to keep respect of the journal and check all methodological elements in assessment of the quality of the submitted papers, especially of the used correct statistical analysis and presentation of the results, trying to keep the scientific truth in medicine.