Impacts of the Journal Evaluation Program of the Korean Association of Medical Journal Editors (KAMJE) on the Quality of the Member Journals.

BACKGROUND: In 1997 the Korean Association of Medical Journal Editors (KAMJE) instituted a program to evaluate member journals. Journals that passed the initial evaluation were indexed in the KoreaMed. Here, we report changes in measures of quality of the KAMJE member journals during the last 20 years.
METHODS: Quality measures used in the study comprised 3 assessment categories; self-assessment by journal editors, assessment of the journals by KAMJE reviewers, and by Korean health science librarians. Each used detailed criteria to score the journals on a scale of 0 to 5 or 6 in multiple dimensions. We compared scores at baseline evaluation and those after 7 years for 129 journals and compared improvements in journals indexed vs. not-indexed by the Web of Science (Science Citation Index Expanded; SCIE).
RESULTS: Among 251 KAMJE member journals at the end of 2015, 227 passed evaluation criteria and 129 (56%) had both baseline and 7-year follow-up assessment data. The journals showed improvement overall (increase in median [interquartile range; IQR] score from baseline, 0.47 [0.64]; 95% confidence interval [CI], 0.44-0.61; P < 0.001) and within each category (median [IQR] increase by editor's assessment, 0.17 [0.83]; 95% CI, 0.04-0.26; P = 0.007; by reviewer's, 0.45 [1.00]; 95% CI, 0.29-0.57; P < 0.001; by librarian's, 1.75 [1.08]; 95% CI, 1.77-2.18, P < 0.001). Before the foundation of KAMJE in 1996, there were only 5 Korean medical journals indexed in the MEDLINE and none in SCIE, but 24 journals in the MEDLINE and 34 journals in SCIE were indexed by 2016.
CONCLUSION: The KAMJE journal evaluation program successfully contributes improving the quality of the member journals.

INTRODUCTION

There are several evaluation systems for scholarly journals. Most of them intend to select journals for indexing in databases such as MEDLINE, PubMed Central (PMC), Web of Science (Science Citation Index Expanded; SCIE), Scopus, KoreaMed, and Korea Citation Index. One exceptional type of evaluation is selecting journals to support financial boost by the Korean Federation of Science and Technology in Korea.

The Korean Association of Medical Journal Editors (KAMJE) was founded in 1996 and instituted a program to evaluate Korean medical journals.1 Korean medical journals were evaluated by peer review and those passed the evaluation criteria were indexed in the KoreaMed, the Korean version of PubMed.2 The items of evaluation were comprised of key points for high quality, such as timeliness, quality of editorial works, flawlessness of bibliography, and citation numbers.3 The journals indexed in the KoreaMed underwent re-evaluation after 7-year interval. KAMJE also organizes regular annual meetings and educational programs for authors, reviewers, editors, and manuscript editors of member journals. Here we report changes in measures of quality associated with the KAMJE program.

METHODS

Quality measures used in the study comprised three categories; self-assessment by journal editors (Category I), and assessment by peer reviewers (Category II), and by Korean health science librarians (Category III).3 Each used detailed criteria to score the journals on a scale of 0 to 5 or 6 in multiple dimensions (Table 1). The KoreaMed-indexed journals were reevaluated after 7 years with slightly modified criteria. We compared scores at baseline evaluation and after 7 years for all journals, and also assessed scores among KAMJE member journals indexed vs. non-indexed in SCIE.

Table 1

Criteria of journal evaluation by categories for KAMJE journalsa

Category	Items	Score range
I. Self-assessed	1-1. Number of change of Editor-in-Chief during the last 15 years	1–5, +1 for full-time editor
	1-2. Editorial board	1–5
	1-3. Editorial system	1–5
	1-4. Peer review system	0–5
	1-5. Duration of peer review	0–5
	1-6. Proportion of revised manuscripts	1–5
	1-7. Rejection rate including withdrawal	0–5
	1-8. Other issues in peer review system	1–5
	1-9. Copyright protection	0–5
	1-10. Distribution of hard copies	1–5
II. Reviewer assessed	2-1. Cover, design and error	0–5
	2-2. Errors in editorial portion	0–5
	2-3. Editorial Guidelines (I), general introduction	1–5
	2-4. Editorial Guidelines (II), preparation of manuscripts 1	1–5
	2-5. Editorial Guidelines (III), preparation of manuscripts 2	1–5
	2-6. Errors in title, author name, affiliation	0–5
	2-7. Errors in abstract	0–5
	2-8. Errors in text	0–5
	2-9. Errors in references	0–5
	2-10. Editing tables	1–5, −3 for misconduct
	2-11. Editing figures	1–5, −3 for misconduct
	2-12. Quality of line drawing	1–5, −3 for misconduct
	2-13. Copyright record and journal website introduction	1–5
III. Librarian assessed	3-1. Publication delay	0–5
	3-2. Citation of Korean literature	0–5
	3-3. Indexed in global databases	0–5
	3-4. Journal impact factor by Web of Science	0–5
	3-5. Errors in references	0–5
	3-6. Online visibility	1–5
	3-X. Bonus by CrossRef h-index or DOI resolution number	1–5

aTranslated of Korean document available at https://www.kamje.or.kr/data/evaluate/evalcriteria7.pdf.

Statistical analysis was done using SPSS ver. 20 (IBM, Armonk, NY, USA). Student's t-test, independent sample and paired sample, was used and significance was verified by level of significance was 0.05.

RESULTS

Among 251 KAMJE member journals at the end of 2015, 227 (90.4%) passed evaluation criteria and 129 (56%) had both baseline and 7-year follow-up assessment data. The journals showed improvement overall and within each category of evaluation with statistical significance (Table 2). Before the foundation of KAMJE in 1996, there were only 5 Korean medical journals indexed in the MEDLINE and none in SCIE. By 2016, moreover, there are 24 MEDLINE-indexed and 34 SCIE-indexed member journals (Fig. 1). There was no statistically significant difference in scores on initial assessments between 21 SCIE-indexed and non-indexed journals but the scores on follow-up assessments were significantly higher in SCIE-indexed journals than those of SCIE-non-indexed journals (Table 3).

Table 2

Change of scores of journal evaluations between initial and follow-up assessments (n = 129)

Categoriesa	Scoresb by median ± IQR			95% CI	P valuec
	Initial	Follow-up	Difference
I	3.71 ± 0.56	3.83 ± 0.70	0.17 ± 0.83	0.04–0.26	0.007
II	3.23 ± 0.73	3.69 ± 1.08	0.45 ± 1.00	0.29–0.57	< 0.001
III	0.50 ± 0.50	2.50 ± 0.88	1.75 ± 1.08	1.77–2.18	< 0.001
Total	2.91 ± 0.35	3.40 ± 0.68	0.47 ± 0.64	0.44–0.61	< 0.001

IQR = interquartile range, CI = confidence interval.

aCategory I, II, and III assessed by editors, reviewers and librarians respectively. bScores for each item are rated based on detailed description (KAMJE's criteria for journal evaluation); range, 0.09–5.09. cPaired sample t-test.

Fig. 1

Increasing numbers of Korean medical journals indexed in global databases. The number is steadily increasing and has risen more rapidly in recent years.

PMC = PubMed Central, SCIE = Science Citation Index Expanded (core collection of Web of Science).

Table 3

Difference in scores between SCIE-indexed and non-indexed journals

Time	Category	Scoresa of journals		95% CI	P valueb
		SCIE-indexed	SCIE-non-indexed
Initial evaluation	I	3.50 ± 0.51	3.71 ± 0.58	−0.14–0.35	0.39
	II	3.64 ± 0.67	3.22 ± 0.71	−0.62–0.13	0.18
	III	0.50 ± 0.86	0.50 ± 0.47	−0.85–0.01	0.06
	Total	2.96 ± 0.63	2.90 ± 0.34	−0.41–0.10	0.22
Re-evaluation	I	4.27 ± 0.63	3.80 ± 0.69	0.26–0.61	< 0.001
	II	4.31 ± 0.77	3.62 ± 0.96	0.35–0.82	< 0.001
	III	3.00 ± 1.19	2.50 ± 1.00	0.28–1.09	< 0.001
	Total	4.07 ± 0.47	3.35 ± 0.66	0.42–0.79	< 0.001

SCIE = Science Citation Index Expanded (core collection of Web of Science), CI = confidence interval.

aScore for each item is rated based on detailed description (KAMJE's criteria for journal evaluation), median ± interquartile range. bIndependent sample t-test.

DISCUSSION

Improvement of the quality of medical journals is of prime concern for all medical journal editors. The journal quality mainly depends on quality of its articles but also on editing and publishing practice, both of which are assessed by global databases. The MEDLINE and PubMed is a supreme global database of quality medical journals managed by the US National Library of Medicine. When a journal is indexed in the MEDLINE or PMC, abstracts of articles in the journal are delivered to global researchers via the PubMed which can call global citations.4 The inclusion criteria of the MEDLINE or PMC are global standards of journal evaluation.5

In order to achieve its purpose, KAMJE developed a system of journal peer review and applied the system to evaluate member journals.1,3 In the 129 journals which were compared of their scores during the 7 years after KoreaMed enlisting, total score was increased 0.47, from 2.91 to 3.40 (95% confidence interval, 0.44–0.61; P < 0.001). All of the scores by 3 categories were raised significantly and this raise meant general upgrade of journal editing and publishing. Especially, the score in the Category III, which was assessed by librarians, improved most as 1.75, from 0.50 to 2.50 (Table 2). The items in the Category III include database indexing status, citation numbers, correctness of references, and online visibility. The significant increase in these items demonstrates that the journals make less mistakes and receive more citations. Enlisting of journals on the KoreaMed provides more chances of both domestic and global citations, and therefore the significant increase of the Category III score of the 129 journals is evidence of journal quality upgrade by the KAMJE system.

Similar difference was also noticed between SCIE-indexed and non-indexed journals. All scores of the 3 categories and total scores were significantly higher in SCIE-indexed journals than those in non-indexed journals (Table 3). Especially the scores in Category I and II were 4.27 ± 0.63 and 4.31 ± 0.77 respectively. These high scores indicate overall editing and management of the SCIE-indexed journals meet global standard. The KAMJE evaluation system is well-correlated with global criteria for scholarly journals.

In addition to the journal evaluation, KAMJE made continuous teaching efforts to achieve its goals in various fields of publishing including improvement of editing and publishing system, ethics in research and publishing, and continuous education of editors, reviewers, and authors. KAMJE developed the KoreaMed which freely spreads title, author information, and abstract with tables or figures of articles of indexed KAMJE member journals to global researchers.2 As a result of the global spreading of abstracts, the KoreaMed has played a powerful role of watching duplicate publications in Korean medical journals. This watching system through the KoreaMed as well as KAMJE education programs achieved significant reduction of duplicate publications by Korean medical researchers from 5.9% in 2005 to 1.2% in 2009.6 The various KAMJE activities continuously upgrade member journals' quality until now.

‘Peer review of journals’ performed by KAMJE member journal editors, not by government or public institution, encourages fellow journals for better quality not for criticism. That is another unique characteristic of the KAMJE program. The 20-year history of KAMJE is dotted with a number of milestones of improvement of Korean medical journals; launching of the KoreaMed, KoreaMed Synapse, and Korean Medical Citation Index (KoMCI).7 During the past 20 years of KAMJE activity, significant numbers of KAJME member journals have been indexed in global databases such as PubMed, SCIE, and Scopus (Fig. 1). That is another evidence of quality upgrade of member journals.

KAMJE may be a role-model for other countries or societies, especially for developing countries, who want to improve the quality of their medical journals. KAMJE will guide the member journals continuously for updated or newly developed global editing and publishing as Gasparyan et al.8 introduced. Close synergistic cooperation between member journals within KAMJE and its successful achievements recall a famous African proverb, ‘If you want to go fast, you should go alone, but if you want to go far, you should go together.’

In conclusion, the present findings suggest a significant correlation between quality improvement of member journals and the peer review evaluation program of KAJME. Our study confirmed that KAMJE has accomplished its goal and mission successfully by both journal evaluation and education programs.