A bibliometric analysis of research productivity of emergency medicine researchers in South Korea.

OBJECTIVE: During the past 20 years, over 1,400 doctors have been certified as emergency physicians in Korea. The number of scientific publications in the field of emergency medicine has also increased. This study aims to evaluate the research productivity of academic emergency physicians in South Korea.
METHODS: Articles published from 1996 to 2015 by authors affiliated with Korean emergency departments were retrieved using Pubmed, Embase, and Web of Science. Research productivity was analyzed quantitatively to ascertain the number of articles for publication type and year. The performance of these articles was also analyzed qualitatively using impact factor, citation number, and Hirsch index. Bibliometric analysis was performed by researching Web of Science, Scopus, and Google Scholar.
RESULTS: A total of 858 articles with 293 Korean authors as the first or corresponding authors were published across 191 journals. The number of publications increased continuously. The most common publication type was original article (n=618), the most commonly studied research topic was resuscitation medicine (n=110), and the average impact factor of the original articles was 2.158. The highest h-index was 17 and, using Web of Science, the maximum number of citations was found to be 85.
CONCLUSION: This study suggests that the research productivity of Korean authors in the emergency medicine field has progressed steadily during the last 10 years. However, qualitative indexes, such as the number of citations and h-index value, remain low.

INTRODUCTION

Twenty years have passed since a board-certified system for emergency physicians was introduced in South Korea. By 2015, the number of board-certified emergency physicians had increased to 1,418. According to the results of the 2015 emergency physician survey, there are 1,096 emergency physicians working at emergency departments in 294 hospitals in Korea. In conjunction with such an increasing number of emergency physicians, academic advances in the field of emergency medicine are also being made. Initially, emergency medicine-related papers were mainly published in domestic journals. For example, founded in 1990, the Journal of the Korean Society of Emergency Medicine (JKSEM) has been published bimonthly since June 2003, and published approximately 2,000 articles by February 2016. However, the journal impact factor (IF) of JKSEM is just a maximum of 0.288 [1].

The first article published in an international journal by a researcher affiliated with an emergency medicine department in South Korea was a letter published in 1996 in the Annals of Emergency Medicine [2]. Since then, many articles by Korean researchers have been published in international journals. Many Korean researchers, including emergency physicians, have tended to publish their research output in international journals. The number of first authors from South Korea who have published in PubMedlisted journals since 2010 is estimated to be more than twice the number of those published in KoreaMed-listed journals. Although there have been studies analyzing articles published in JKSEM, no study has analyzed the research productivity of Korean researchers affiliated with emergency medicine departments [1].

The goal of this study is to analyze the research productivity of Korean authors affiliated with emergency medicine departments using both quantitative and qualitative bibliometric analysis.

METHODS

The present study is a retrospective observational study that did not involve human subjects and, thus, approval from the institutional review board was not required or obtained.

Search process

In order to retrieve the studies of Korean researchers affiliated with emergency medicine departments, major article sites were searched. Articles were searched by using the keywords “Korea” [ad] AND “emergency” [ad] in PubMed; “Korea:ca” AND “emergency medicine” [ad] in Embase; and “Korea” AND “emergency medicine” for the names and addresses of research institutions in Web of Science (WoS). Additionally, the search period was limited to between 1996 and 2015. The articles that were retrieved included 970 articles from PubMed, 1,459 from Embase, and 1,705 from WoS. The search results were saved to a file before being exported to Microsoft Excel in order to allow sorting by topic. Duplicate articles were deleted, and articles that had only been published as e-publications by 2015 were excluded from the analysis. Non-original articles such as letters and editorials were included, but replies to letters were excluded. Analysis was only conducted on articles for which the first or corresponding author was affiliated with an emergency medicine department, which was determined by analyzing author information for each article. We excluded articles for which only a co-author was affiliated with an emergency medicine department, as well as excluding articles by authors who were affiliated with other related departments, including emergency medical services. Authors who overlapped by being affiliated with both other clinical departments and emergency medicine departments were included only if they actually worked in an emergency medicine department. The search process was conducted collaboratively by two authors.

Classification of results

For the included articles, the abstracts or, if necessary, the full texts, were reviewed. The publication type was classified into original article, case report, editorial, or letter in order to examine the total number of articles by year and the number of articles by publication type. Articles that were in the form of a letter or image, but were contextually case reports, were classified as case reports. In addition, articles were classified as basic experimental studies, simulation studies, or clinical studies depending on the topic. Subsequently, clinical studies were subdivided into categories such as prospective studies, retrospective studies, observational studies, or randomized controlled trials, depending on the study design. In the case of original articles, the topic of each article was classified by area in order to determine the research areas that have been frequently studied.

For qualitative analysis, we used Journal Citation Reports or Researchgate to investigate whether the journals in which each article was published were listed on the Science Citation Index (SCI), the Science Citation Index Expanded (SCIE), or Medline, and obtained the IF for the publication year. To determine the number of citations per article, WoS, Scopus, and Google Scholar were searched. For WoS, the terms “(emergency medicine) AND (Korea)” were searched for in the names and addresses of research institutions while Scopus was searched using “AFFILCOUNTRY (Korea) AND AFFILORG (emergency medicine)”. The retrieved articles were sorted by the number of citations, and the most-cited articles were confirmed accordingly. h-index values were confirmed by searching Scopus for individual authors.

RESULTS

Quantitative analysis of total articles

Between 1996 and 2015, a total of 858 articles were published in international-level journals by 293 Korean authors, who were affiliated with emergency medicine departments and acting as the first or corresponding author. By publication type, there were 618 original articles, 179 case reports, 48 letters, 11 review articles, and three others. The number of published articles by year is shown in Table 1. The growing trend in the number of published articles by year in comparison with the increasing number of certified emergency physicians is shown in Fig. 1. The total number of journals in which Korean authors published their articles was 191. The journals in which articles were regularly published by Korean authors are shown in Table 2. There were 463 articles published in SCI-indexed journals, 341 articles in SCIE, and 54 articles not included in the SCI(E) category.

Table 1.

The number of articles published between 1996 and 2015 by Korean authors from the emergency medicine sector, represented in regard to publication type

Year	Total	Original	Case	Letter	Review	Editorial
1996	1	–	–	1	–	–
1997	1	–	1	–	–	–
1998	–	–	–	–	–	–
1999	–	–	–	–	–	–
2000	1	1	–	–	–	–
2001	3	3	–	–	–	–
2002	6	5	1	–	–	–
2003	9	6	2	–	1	–
2004	6	3	2	1	–	–
2005	3	1	1	1	–	–
2006	10	6	4	–	–	–
2007	27	11	13	2	1	–
2008	45	23	14	5	2	1
2009	53	32	16	5	–	–
2010	62	39	19	4	–	–
2011	98	71	18	8	1	–
2012	123	99	21	2	1	–
2013	135	99	21	12	3	–
2014	108	78	23	5	2	–
2015	167	141	23	2	1	–
Total	858	618	179	48	12	1

Fig. 1.

The cumulative number of published articles and board-certified emergency physicians (EPs) from 1996 to 2015. The cumulative number of papers per 10 EPs is rapidly increasing. EM, emergency medicine.

Table 2.

The eleven journals most commonly published in by Korean authors from the emergency medicine sector

Ranking	Journal title	2015 IF	2015 EM ranking	Papers	%
1	Am J Emerg Med	1.504	11	139	16.2
2	Resuscitation	5.414	1	106	12.4
3	Emerg Med J	1.836	9	77	9.0
4	J Korean Med Sci	1.256	NA	52	6.0
5	J Emerg Med	1.048	15	42	4.9
6	Clin Toxicol	2.886	NA	41	4.8
7	Emerg Med Australas	1.223	12	17	2.0
8	Acad Emerg Med	2.537	4	13	1.5
9	Yonsei Med J	1.154	NA	11	1.3
10[a)]	PLoS One	3.057	NA	10	1.2
10[a)]	Eur J Emerg Med	2.026	7	10	1.2

IF, impact factor; EM, emergency medicine; NA, not applicable.

Tie.

Quantitative analysis of the original articles

After 618 original articles had been analyzed, there were 450 (73%) clinical trial-related articles, 107 (17.3%) basic research-related articles, and 61 (9.7%) simulation study-related articles. In the case of articles pertaining to clinical trials, there were 18 randomized controlled trials, one systematic review, two case control studies, 278 retrospective studies, and 106 prospective observational studies. There were also five before-and-after comparative studies, 18 surveys, 15 experimental studies, and seven case series. By research topic, resuscitation was the most commonly studied with 110 articles (24.4%), followed by 42 articles on clinical toxicology, 33 articles on intensive care medicine, 32 articles on traumatology, and 28 articles on emergency radiology. Clinical studies were published in 113 journals, of which 74 articles (16.4%) were published in the American Journal of Emergency Medicine, 53 in Resuscitation, 42 in the Journal of Korean Medical Science, 36 in the Emergency Medicine Journal, and 13 articles in Clinical Toxicology. By basic experimental research material, rats were the most commonly used as they were involved in 44 articles (41%), followed by 23 articles involving gerbils, 14 involving pigs, 13 involving cell lines, five involving dogs, four involving bacteria, three involving mice, and one article involving rabbits. Basic experimental research was the most commonly published, with 14 articles (13%) in Resuscitation, nine articles in the Journal of Neurological Sciences, and seven articles in American Journal of Emergency Medicine. As for simulation studies, there were 55 articles (90%) involving mannequins, three involving cadavers, two involving animations, and one article involving a computer. The number of published articles in this regard was 16 (26%) in Emergency Medicine Journal, 13 articles in Resuscitation, and 10 articles in American Journal of Emergency Medicine.

Qualitative analysis of the articles

After analysis of 826 articles for which IF was identifiable, the mean IF was found to be 2.124. The highest-ranked articles in relation to journal IF are shown in Table 3. Upon comparing the IFs of clinical studies in relation to study design, randomized controlled trial and systematic review were found to have the highest values (Table 4). The mean IFs of clinical studies and basic studies were the same at 2.16. In terms of the IFs of 618 original articles, there were 99 articles with an IF of less than 1, 298 articles (48.2%) with an IF of 1 to less than 2, 108 articles (17.5%) with an IF of 2 to less than 3, 59 articles with an IF of 3 to less than 4, 34 articles with an IF of 4 to less than 5, and 29 articles with an IF of more than 5. Among original articles, the most highly cited article was an article published in Critical Care Medicine in 2011, with 85 citations in WoS, 97 citations in Scopus, and 145 citations in Google Scholar (Table 5). The highest h-index value for an author of an article with a high number of citations was 17 points.

Table 3.

High-ranking articles by Korean authors from the emergency medicine sector, arranged by journal IF

Ranking	IF	Journal title	Year	Design	PMID
1	50.955	N Engl J Med	2012	RCT	22533576
2	28.409	Lancet	2008	Editorial	18603292
3	17.047	Circulation	2015	Retrospective	26269576
4	8.736	Clin Infect Dis	2015	Retrospective	25694652
5[a)]	7.584	Arch Neurol	2011	Retrospective	21825248
6[b)]	7.422	Crit Care Med	2015	Retrospective	26468697
6[b)]	7.422	Crit Care Med	2015	Retrospective	26284621
8	7.078	Int J Cardiol	2011	Retrospective	20471704
9	6.499	Stroke	2008	Letter	18162615
10[b)]	6.330	Crit Care Med	2011	Rat experiment	20975550
10[b)]	6.330	Crit Care Med	2011	Prospective	21705883
10[b)]	6.330	Crit Care Med	2011	Retrospective	21057309
13	6.312	Crit Care Med	2014	Pig experiment	24145844
14	6.147	Crit Care Med	2013	Rat experiment	23648567

IF, impact factor; PMID, PubMed identifier; RCT, randomized controlled trial.

Corresponding author,

Tie.

Table 4.

The number of articles and average IFs in relation to study design in clinical studies

Study design	Total no.	Average IF according to index category			Average IF
		Not SCI(E)	SCIE indexed	SCI indexed
RCT	18	–	1.29	5.93	4.38
SR	1	–	–	4.10	4.10
Case control	2	–	3.23	1.29	2.26
Retrospective	278	0.51	1.71	2.76	2.20
Prospective	106	0.79	1.52	2.31	1.94
Before after	5	–	1.22	2.28	1.86
Survey	18	0.04	1.72	2.66	1.43
Experimental	15	0.21	0.54	2.14	1.52
Case series	7	–	1.41	1.36	1.39
Total	450	0.39	1.58	2.76	2.34

IF, impact factor; SCIE, science citation index expanded; SCI, science citation index; RCT, randomized controlled trial; SR, systematic review.

Table 5.

Top ten journal articles ranked by the number of citations, as calculated by Web of Science, Scopus, and Google Scholar

Ranking	Web of Science	Scopus	Google Scholar	Title (journal, year)
1	85	97	145	Extracorporeal cardiopulmonary resuscitation in patients with inhospital cardiac arrest: a comparison with conventional cardiopulmonary resuscitation (Crit Care Med, 2011)
2	70	80	125	Low-dose abdominal CT for evaluating suspected appendicitis (N Engl J Med, 2012)
3	59	63	100	Does therapeutic hypothermia benefit adult cardiac arrest patients presenting with non-shockable initial rhythms? A systematic review and meta-analysis of randomized and non-randomized studies (Resuscitation, 2012)
4	51	62	98	Diaphragm dysfunction assessed by ultrasonography: Influence on weaning from mechanical ventilation (Crit Care Med, 2011)
5	42	46	61	The density ratio of grey to white matter on computed tomography as an early predictor of vegetative state or death after cardiac arrest (Emerg Med J, 2008)
6	40	45	57	Diffusion-weighted magnetic resonance imaging for predicting the clinical outcome of comatose survivors after cardiac arrest: A cohort study (Crit Care, 2010)
7	38	37	66	Epidemiology and outcomes from non-traumatic out-of-hospital cardiac arrest in Korea: A nationwide observational study (Resuscitation, 2010)
8	35	34	44	Effects of audio tone guidance on performance of CPR in simulated cardiac arrest with an advanced airway (Resuscitation, 2008)
9	34	38	50	Is the inter-nipple line the correct hand position for effective chest compression in adult cardiopulmonary resuscitation? (Resuscitation, 2007)
10[a)]	32	39	65	Comparison of the GlideScope video laryngoscope and Macintosh laryngoscope in simulated tracheal intubation scenarios (Emerg Med J, 2008)
10[a)]	32	32	60	Red cell distribution width is a prognostic factor in severe sepsis and septic shock (Am J Emerg Med, 2008)

Tie.

DISCUSSION

The research productivity of Korean emergency medicine researchers is gradually increasing, and is showing a similar trend to the increase in the number of certified emergency physicians that has occurred throughout the 2010’s. 72% of articles published by emergency medicine researchers were types of original articles. Of these, 73% were clinical trials and 62% of the clinical trials were retrospectively designed studies. By research topic, resuscitation was the most commonly studied with the highest number of articles. By basic experimental research, studies involving rats were the most common.

Only papers that were published in international-level journals and, thus, are searchable in international databases such as PubMed were studied in the present study. An increasing number of Korean researchers prefer to publish their research output in international journals. The reason for this is thought to be that articles in domestic journals may have a low chance of being cited. In the results of a study that reviewed the number of citations of articles published in JKSEM between 1992 and 2013, 25 articles were reported to have only 30 citations. Consequently, excellent research output tends to be submitted to international journals, so domestic article searches on sites such as KoreaMed were not included in the present study. For reference, when KoreaMed was searched for articles from between 1996 and 2015 that feature the keyword “emergency medicine” in the author affiliation, a total of 2,650 articles were retrieved.

In a study by Wilson and Itagaki [3] that analyzed approximately 14,000 articles published between 1996 and 2005 originating from authors affiliated with emergency medicine departments, the results showed that the US published 58.5% of articles, followed by the United Kingdom with 8.4%, and then a series of other countries with more than 1%, such as Japan, Australia, Italy, Canada, Taiwan, China, Turkey, and Austria. In a study by Li et al. [4] that analyzed articles on emergency medicine published in 13 journals between 2006 and 2010, it was reported that contributions from authors originating from the US were highest with 46.3%, while South Korea ranked joint 10th with 1.3%. It was also reported that the proportion from the US had decreased from 69% in 1960, whereas the proportion from Europe had increased from 22% to 37% over the same period [5]. Further, 37,934 articles originating from authors affiliated with emergency medicine departments were indexed in PubMed between 1996 (when a board-certified emergency physician system was introduced in South Korea) and 2015 and, of these, 970 articles (2.6%) indicated South Korea in the institutional affiliation of the authors. When WoS was searched in the same way, 55,543 articles were found to have originated from authors in the field of emergency medicine, of which the contribution by South Korea was 3.1% at 1,705 published articles. Therefore, the quantitative contribution from South Korean authors is considered to be approximately 3%. Wilson and Itagaki [3] stated that 41.8% of articles by authors affiliated with emergency medicine departments were published in journals in the field of emergency medicine. In the present study, 52% (447/ 858) of the total articles were found to be published in emergency medicine-category journals.

The number of citations is considered a quality indicator of articles. In the present study, the most cited article received 85 citations in WoS. In another study examining the top 100 cited articles published in six emergency medicine journals in 2015, it was reported that the most cited article received 954 citations and the hundredth-most cited article received 181 citations. Of the 100 articles, 65 were published in the Annals of Emergency Medicine [6]. Meanwhile, in Scopus, the median citation number for articles on ultrasonography performed in emergency departments was 115 (range, 75 to 681) [7]. An audit of the top 100 cited articles published in Pediatric Emergency Care found that the number of citations ranged from 132 to 42 [8]. In comparing the number of citations of articles in different fields, this is considered low. It was reported that the most highly cited article in the field of critical care medicine was cited 4,909 times [9]. For information, the most frequently cited article in the field of emergency medicine is an article related to injury severity scores, which was published in 1974 in the Journal of Trauma and has 4,020 citations [10]. The citation counts differ because the number of aggregated articles varies depending on search engines. Generally, Scopus and Google Scholar retrieve more citations per article than WoS [11].

IF was used in the present study to perform a qualitative analysis of research productivity. The term IF first appeared in 1963 [12]. If the Journal Citation Reports IF of a journal is two, it means that articles published in that journal during the two preceding years were cited twice each on average in SCI-indexed journals during the corresponding year. Therefore, IF was originally developed as a tool to evaluate journals, but not to evaluate individual articles or authors. The mean IF of articles included in this study was found to be 2,011. In this study, the journals carrying articles with the highest IF scores were the New England Journal of Medicine, Lancet, and Circulation, respectively. Among original articles, there was no difference in the mean IF between clinical studies and basic experimental studies.

Recently, other qualitative indicators in addition to IF have been suggested. The h-index is an alternative index that was suggested in 2005 by Hirsch [13]. If a researcher has published papers, of which h papers (where h is some number) have been cited h times in other papers and the remaining papers have been cited less than h times, the researcher’s h-index is h [14]. When the authors with high rankings in terms of number of citations were examined in this study, the results showed that the highest h-index was 17 points. According to a paper analyzing the research productivity of academic emergency medicine professors in the US, it was reported that the highest h-index was 51 points based on WoS and 63 points based on Google Scholar [15]. It is also argued that the h-index is advantageous for researchers with a long academic career because of an increasing, accumulative number of citations. Consequently, another index, m-quotient, which divides the h-index by the number of years since the first published paper, might be more ideal [16].

Recently, altmetrics have been used to evaluate the impact of articles by analyzing citation counts and the online environment. Altmetrics are a generalization of the impact of an article and analyze aspects such as its citations, page views, downloads, and mentions in social media and news media, after which the articles are indexed as viewed, discussed, saved, cited, or recommended. According to a recent study analyzing the altmetric scores of articles in the field of emergency medicine, the altmetric score for an article related to early goal-directed therapy published in New England Journal of Medicine in 2001 had the highest score with 176 [10]. When altmetric scores for highly cited articles in the present study were calculated using the altmetric bookmarklet available on Google Chrome, it was found that altmetric scores for the first, second, and third most cited articles were 14, 38, and 11 points, respectively.

In summary, research productivity in the field of emergency medicine in South Korea has rapidly increased over the past 10 years. The most common type of study concerns retrospectively designed clinical studies in the area of resuscitation. However, qualitative indexes such as the number of citations and h-index have remained low.

In regard to shortcomings, the limitations of this study are as follows: first, analyses by authors and affiliated institutions were not conducted. It was difficult to conduct such an analysis by author because the English name of a Korean author can be expressed in various different ways and there may be individuals with the same acronymic name in English. The same issue arises for analysis by institution. It was not easy to conduct analysis by institution because in some cases authors may not have clearly indicated their affiliation to a university or a hospital. Second, analyses were conducted with a focus on the first author. In order to avoid duplicate counting of articles, we put priority on the first author, and corresponding authors were taken into account only if the first author was not affiliated with emergency medicine while the corresponding author was affiliated with emergency medicine. In addition, this is also related to the practical issue that it is much more difficult to identify the corresponding author. Third, the classification of research area may not be accurate. For example, although an article on pediatric resuscitation concerns both resuscitation medicine and pediatric emergency medicine, it was only classified as resuscitation medicine in this study. Since a clear classification of disciplines has not yet been established and the author’s discretion might be reflected, caution is required in interpreting the findings for research area in this study. Fourth, because databases including PubMed were analyzed in the present study, there is a possibility that articles that do not specify institutional affiliation for authors might be excluded. If missing articles were identified, they were manually added as much as possible.