What can we learn from top-cited articles in inflammatory bowel disease? A bibliometric analysis and assessment of the level of evidence.

BACKGROUND AND OBJECTIVES: Despite increasing number of publications in inflammatory bowel disease (IBD), no bibliometric analysis has been conducted to evaluate the significance of highly cited articles. Our objectives were to identify the top-cited articles in IBD, assessing their characteristics and determining the quality of evidence provided by these articles. DESIGN AND OUTCOME MEASURES: IBD and related terms were used in searching the Web of Science to identify English language articles. The 50 top-cited articles were analysed by year, journal impact factor (JIF), authorship, females in authorship, institute, country and grants received. The level of evidence was determined using the Oxford Centre for Evidence-Based Medicine guidelines.
RESULTS: The number of citations varied from 871 to 3555 with a total of 74 638, and a median 1339.50 (IQR=587). No correlations were found between the number of citations and number of years since publication (r=0.042, p=0.771), JIF (r=0.186, p=0.196), number of authors (r=0.061, p=0.674), females in authorship (r=0.064, p=0.661), number of institutes (r=0.076, p=0.602), number of countries (r=0.101, p=0.483) or number of grants (r=-0.015, p=0.915). The first authors were from the USA (n=24), the UK (n=6), Germany (n=5), France (n=5), Belgium (n=3) and Canada (n=3). The levels of evidence were 12 articles at level 1b, 9 articles at level 3a and 15 articles at level 3b and fewer were at other levels.
CONCLUSIONS: Research papers represented 66% of articles. The majority of items have reasonably high levels of evidence, which may have contributed to the higher number of citations. The study also shows a gender gap in authorship in this area. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

More than one method was used in searching the Web of Science to maximise the yield of the search.

The articles represent a historic perspective on key discoveries in the area of inflammatory bowel disease.

A number of characteristics including journal impact factor, number of authors, number of females in authorship, number of institutes, number of countries involved and grants received have been studied against citations received.

The level of evidence for each article has been evaluated.

The search was limited to articles in the English language, raising the possibility of failing to include top-cited articles in other languages.

Introduction

The number of citations received by an article has been used for several years by universities and grant funding bodies in assessing the quality of research produced by researchers, ranking research performance and in making decisions regarding professional promotion and grant applications.1–4 Thomson Reuters gather the names of these authors in the Highly Cited Researchers database, where they are acknowledged for the quality of their work and the size of their research input in a particular area.5 Although the number of citations cannot explain why researchers cited a particular paper nor reflect the quality of the research and the outcomes identified in a manuscript, the number of citations and the reputation of contribution to research in a particular field cannot be ignored. This is particularly important when there is a pattern of consistency and progressive input into a discipline over years (a life experience), and a demonstration from publication records and citation history of collaboration with other researchers from other institutes at local and international levels. Hence, the credits given to an author or a group of authors and the impact of the work in a particular area can be proportionally related to the citation records.6 The greater the citation history over years, the more influential they are in their specialty.7 In this study, we hypothesised that examining the most cited in IBD may provide more insight into the significance of these articles and the level of evidence they present.8 Considering the fact that there are over 91 000 articles on inflammatory bowel disease (IBD) in the literature as per our preliminary search of Web of Science, it is important to consider the characteristics of highly cited articles.

IBD comprises ulcerative colitis (UC) and Crohn’s disease (CD); both pursue a relapsing and remitting course over years. While the two diseases share several similarities, they have a number of differences with regard to structures involved, pathogenesis, clinical presentation and management approaches. UC was first reported briefly in mid-1800,9 while CD was reported in the Journal of the American Medical Association (JAMA) later by Crohn et al in 193210; describing it as a chronic IBD of the ileum. Later, it was discovered that CD can involve any part of the gastrointestinal tract from the lips to the anal margin but the ileocolonic disease represents the common presentation. Since then both diseases have been extensively researched in their different aspects.

We assumed that these articles may be a cornerstone in IBD, and may enable researchers in understanding a range of aspects related to IBD. Therefore, the bibliometric analysis of these articles including journal impact factor (JIF), authorship, females in authorship, institute, country and grants received may explain key features of a successful or influential article in IBD. The aims of the present research were (1) identify the top-cited articles in IBD, and analyse their characteristics and (2) assess the quality of evidence provided by articles.

Methods

Study design

The Web of Science database was searched for the identification of the top-cited articles and tracking the citation records of each publication. Although Scopus and Google Scholar also provide citation records, it was decided to limit the search to Web of Science database. Compared with other databases, the Web of Science is regularly updated and the 2015 Journal Citation Reports (JCRs) included >6500 journals across 150 disciplines. Although Google Scholar database is freely available, it was not used because it is difficult to search, and it cites textbooks, monographs, conference proceedings, as well as non-peer-reviewed publications.11 It is also not possible to track the yearly records of citations attracted by each article since publication. Scopus database was not used because it is not extensive in its coverage and its records only go back to 1996.12 Furthermore, several other researchers have used Web of Science to identify top-cited articles.13–15

To achieve the aims of this study, we planned to identify the highly cited articles in IBD and assess if there were any correlation between the number of citations and any of the parameters characterising these highly cited articles. We also aim to grade each article against the level of evidence hierarchy as per the Oxford Centre for Evidence-Based Medicine (OCEBM-2011 levels of evidence and the accompanying table of evidence glossary).16 17

Searching the Web of Science database

On 15 and 16 April 2018, the authors searched the Web of Science database to retrieve top-cited articles in IBD. The search words used were the following: ‘Inflammatory bowel disease’, ‘Ulcerative colitis’, ‘Crohn’s disease’, ‘IBD’, ‘Experimental colitis’, ‘Animal models for Colitis’, ‘Animal models for inflammatory bowel disease’, ‘Pathology IBD’, ‘Pathology UC’, Pathology CD’, ‘Pathogenesis IBD’, ‘Pathogenesis UC’, ‘Pathogenesis CD’, ‘Treatment IBD’, ‘Treatment UC’, ‘Treatment CD’, ‘Investigation IBD’ and ‘Regional ileitis’. To increase the yield of the search, we used the full terms— inflammatory bowel disease, ulcerative colitis and Crohn’s disease for the abbreviations IBD, UC and CD, respectively. These search words were identified from the terminology used in gastroenterology journals and the proceedings of major conferences on IBD and gastroenterology such as the British Society of Gastroenterology, the American Gastroenterological Association, American College of Gastroenterology, Canadian Association of Gastroenterology, Scottish Society of Gastroenterology and Gastroenterological Society of Australia. For each search word, the results were arranged using a link on the Web of Science system, ‘sort-by’—‘Time Cited—highest to lowest’. The results showed the articles organised in a descending order with the articles most frequently cited at the top. A copy of the results was printed out for further analysis. The findings from each search word were then arranged on one Excel sheet (Microsoft Excel 2010, Microsoft, Redmond, Washington, USA) in descending order based on the number of citations. Duplicate articles and articles not in the English language were excluded. In addition to the absolute number of citations, we calculated the average citations per year for each article. The average number of citations per year is the ratio calculated from the number of citations obtained by an article divided by the number of years since publication.18

Using the above-mentioned search words, we conducted another search of major gastroenterology journals and the 2016 JCR under the category ‘Gastroenterology and Hepatology’. This category comprises 76 journals at the time of conducting the search, of these 7 journals were in languages other than English and were not searched. Gastroenterology journals publishing articles not in the English language were excluded because neither the author nor the assistant researchers are competent in the Spanish, Italian or German languages. Since the language recommended by the journal publishing this work is English and its readers are most likely interested in research publications in the English language, we decided not to search these journals.

Interestingly, after identifying the list of top-cited articles in IBD, and again checking these seven non-English journals, none had a paper with a citation higher than the paper ranked number 50 on the list.

These keywords were also used in searching the websites of major general medicine, surgery and research journals including the New England Journal of Medicine, Lancet, the British Medical Journal, the Journal of the American Medical Association, Annals of Internal Medicine, Archives of Medicine, PLOS Medicine, Annals of Surgery, Archives of Surgery, British Journal of Surgery, American Journal of Surgical Pathology, Nature, Science, Nature Reviews Cancer, Nature Genetics, Nature Medicine, Cell, Nature Reviews Microbiology, Immunity, Nature Reviews Immunology, Nature Reviews Molecular Cell Biology and Journal of Immunology.

A list identifying the 50 top-cited articles was reviewed again and checked regarding authorship, year of publication, title of the article, journal publishing the work, the JIF at the time of the search, the number of citations and the institution of the first author (see online supplementary appendix 1).

Inclusion and exclusion criteria

The inclusion criteria were: papers focusing on IBD (UC or CD) written in the English language. The exclusion criteria were: (i) articles on IBD in languages other than English, (ii) articles that focused on other diseases and IBD was not the main focus and (iii) studies that focus on other types of colitis and not IBD.

Assessing articles

The full text of the identified 50 top-cited articles was obtained and a copy was given to each researcher. The following information was collected for each article: (i) the authors’ names, the number of authors, their affiliations and the number of females contributing to authorship, (ii) the number of institutes involved in the publication, (iii) the city and country of the origin of the publication, (iv) the total number of citations obtained up to the day of searching the database, and the number of yearly citations since publication, (v) the year of publication and the calculated number of years since publication and (vi) the number of funds/grants stated in the publication and the Web of Science.

We have not used the classification provided by the Web of Science regarding study type because we noted that the Web of Science groups publications as original research, articles, practical guides and reviews and identifies them as articles or reviews. For consistency and the purpose of this study, the top-cited articles were grouped into four types—article, review paper/meta-analysis, report and research. A definition of each type is given in the glossary (see online supplementary appendix 2). Two researchers independently allocated each of the top-cited articles under its type as per the definition given. Any differences between the researchers were discussed in a meeting until a decision was reached.

The topics covered by the top-cited articles were identified by each researcher independently and were discussed in a meeting to harmonise the grouping into a logical, simple and practical way. Articles that covered more than one topic were classified on the basis of the aim of the study and the main outcomes. For other evaluations of an article including the number of authors, the number of females represented in authorship, the number of institutes and countries contributing to the work and the number of grants/funds received, we checked the original article for such details. Institute was defined as the university where an author belonged. If an author belonged to two universities, this was considered to be two different institutes. Regarding the identification of females in the authorship, we noted that several journals use abbreviations of the first and second name rather than the full name. In order to identify the females in these articles, we tried to search the Google database to find the university, personal website of the author, their LinkedIn or ResearchGate accounts. We also tried to identify them by searching the Google Scholar database and identify their account, where we can find other publications under their name and the full first name. In two papers, despite our efforts, we failed to identify the gender of five authors and we contacted the corresponding authors. We received a response from one correspondence, making us unable to identify the gender of three authors in the second paper.

Evaluating the journals

The publishing journals of the top 50 articles in IBD were identified and evaluated in regard to the following: (i) the 2016 JIF of each journal and (ii) the ranking order of each article in comparison to other articles published in that journal. This was based on the number of citations obtained in comparison to the citation numbers received by other articles published in the journal. For example, an article ranked number one, in its publishing journal, means that the article received the highest number of citations in comparison to all other articles published in that journal. This evaluation aimed at assessing the position order of articles identified among the 50 top-cited articles in IBD in regard to their ranking among other articles published in the journal. Such assessment highlights the significance of the IBD articles among other topics published in gastroenterological journals as well as general medicine journals such as The New England Journal of Medicine, The Lancet, the British Medical Journal, Medicine and top research journals such as Nature, Nature Genetics, Science, Cell and top journals in immunology such as Nature Reviews Immunology (see online supplementary appendix 1).

Assessing level of evidence

Two researchers independently used the OCEBM-2011 levels of evidence and the accompanying table of evidence glossary16 17 to rank each article regarding level of evidence. In 1998, this hierarchical of evidence was first produced to make the process of finding relevant evidence feasible. Since then the levels have been reviewed and amended, and the version used in this research is the currently available version. This evaluation aimed at identifying the level of evidence of each article and assessing whether the highly cited articles have received higher scores in regard to level of evidence as per the Oxford hierarchy. The assessment required extensive review of each article since quality descriptors exist for different types of studies and level of evidence vary depending on therapeutic, prevention, prognostic, diagnostic or prevalence design.13 14

Statistical analysis

Pearson’s correlation coefficient (r) was calculated to determine if the high citation numbers obtained was related to the age of the article. Other correlations were between the number of citations and the number of authors, the percentage of females in authorship, the number of institutes, the number of countries involved, the number of grants received and the JIF of the journals in which articles were published. Because of the assumption that researchers usually cite recently published articles, it was decided to compare the mean yearly citations received of articles published before the year 2000 and compare them with those published after the year 2000. All analyses were conducted using SPSS Software (IBM SPSS Statistics Premium V.22.0 for Mac OS-SPSS, Chicago, Illinois, USA) and the results were reported at total, mean, median, IQR and percentage. The inter-rater agreement between evaluators was calculated using the Fleiss kappa scale.19

Patient and public involvement

This study did not involve patients or the public.

Results

Top-cited papers identified

The 50 top-cited articles in IBD identified by searching the Web of Science20–69 have been summarised in online supplementary appendix 1. The articles are listed in an order from 1 to 50 with the highest absolute citation number is ranked 1 and the article with the lowest citation ranked 50 as per the day of the search.

Table 1 summarises the year of publication and article type. The articles were published over 57 years (from 1955 to 2012). During the period from 1955 to 1976, only three articles (6%) were published. However, the number increased significantly from 1977 to 1994 making a total of 12 (24%) articles. The number of publications increased significantly to 35 (70%) during the years from 1995 to 2012. No correlation was found between the number of citations of these papers and the number of years since published (Pearson’s correlation (r)=0.042, p=0.771). To assess if there were differences between old articles (published before the year 2000) and those published after the year 2000, it was decided to study the mean number of citations received by top-cited articles in each year after their publications. The year 2000 was taken as a mid-point for comparison because the majority of articles identified were published in the period after the year 1986, and hence the year 2000 could represent such point. As shown from figure 1A and B, the mean numbers of citations were higher for articles published after the year 2000 compared with those published before the year 2000. Approximately one-fourth of the top-cited papers were reviews/meta-analysis (n=14, 28%), two-thirds were research papers (n=33, 66%) and the remaining were an article (n=1, 2%) and two reports (n=2, 4%).

Table 1

Top-cited papers in inflammatory bowel disease identified by searching the Web of Science, summarised by year of publication and category

Article type	Year of publication: no. of articles (references)
	1950–1958	1959–1967	1968–1976	1977–1985	1986–1994	1995–2003	2004–2012	Total (%)
Article				139				1 (2%)
Review/meta-analysis			161			525 36 41 55 60	827 33–35 38 45 57 63	14 (28%)
Report							243 59	2 (4%)
Research	128		123	337 58 67	8 22 40 44 48–50 56 66	10 20 21 24 26 31 46 62 65 68 69	1029 30 32 42 47 51–54 64	33 (66%)
Total (%)	1 (2%)	0 (0%)	2 (4%)	4 (8%)	8 (16%)	15 (30%)	20 (40%)	50 (100%)

Figure 1

(A) Number of citations of papers published before the year 2000 (mean±SD). (B) Number of citations of papers published after the year 2000 (mean±SD).

Table 2 summarises the distribution of IBD topics covered in the top-cited articles. These can be summarised as follows: epidemiology and prevalence (n=2, 4%), CD genetic susceptibility and NOD2 mutation (n=6, 12%), animal models (n=5, 10%), pathogenesis of IBD (n=15, 30%), classification and index of disease activity (n=5, 10%), risk of developing colorectal cancer (n=2, 4%), extraintestinal complications (n=1, 2%), infliximab in CD (n=6, 12%), corticosteroids in UC (n=1, 2%), drug treatment and ciclosporin in UC (n=2, 4%), adalimumab in CD (n=1, 2%), 5-aminosalicylic acid in UC (n=1, 2%), 6-mercaptopurine in CD (n=1, 2%) and monoclonal antibiotics and antitumour necrosis factor in CD (n=2, 4%). As shown in online supplementary appendix 1, 14 (28%) were basic research and 19 (38%) were clinical research. The level of evidence is discussed later in the results.

Table 2

Top-cited papers in inflammatory bowel disease (IBD) identified by searching the Web of Science, summarised by category and topic

Topics on IBD	Category: no. of articles (references)
	Article	Review/meta-analysis	Report	Research	Total (%)
Epidemiology, prevalence		235 38			2 (4%)
CD genetic susceptibility and NOD2 mutation		234 45		420 21 51 54	6 (12%)
Animal models for IBD		160		422 40 44 48	5 (10%)
Pathogenesis of IBD		725 27 33 36 55 57 63		829 32 42 62 64–66 68	15 (30%)
Classification, index of disease activity	139		243 59	223 37	5 (10%)
Risk of developing colorectal cancer		141		149	2 (4%)
Extraintestinal complications		161			1 (2%)
Infliximab in CD				624 30 31 46 47 52	6 (12%)
Corticosteroids in UC				128	1 (2%)
Drug treatment and ciclosporin in UC				256 58	2 (4%)
Adalimumab in CD				153	1 (2%)
5-Aminosalicylic acid in UC				150	1 (2%)
6-Mercaptopurine in CD				167	1 (2%)
Monoclonal antibodies, antitumour necrosis factor in CD				226 69	2 (4%)
Total (%)	1 (2%)	14 (28%)	2 (4%)	33 (66%)	50 (100%)

CD, Crohn’s disease; UC, ulcerative colitis.

The articles were published in the following journals: New England Journal of Medicine (n=12, 24%), Gastroenterology (n=12, 24%), Nature (n=4, 8%), Nature Genetics (n=4, 8%), the Lancet (n=2, 4%), Proceedings of the National Academic of Sciences of the United States of America (n=2, 4%), Cell (n=2, 4%) and Gut (n=4, 8%). See table 3 for more details about the journals publishing these articles.

Table 3

The journals that published the top-cited inflammatory bowel disease articles included in the study, the journal impact factor (JIF) and the number of papers published and reference number

Journal*	2016 JIF	Number of papers published (references)
New England Journal of Medicine	72.406	1225 26 30 31 46 47 49 50 52 56 57 67
Gastroenterology	18.392	1223 35 36 38 40 53 58 60 63 66 68 69
Nature	40.137	420 21 27 33
Nature Genetics	27.959	434 45 51 54
The Lancet	47.831	224 37
Proceedings of the National Academy of Sciences of the United States of America	9.661	232 42
Cell	30.410	222 44
Gut	16.658	441 59 62 64
British Medical Journal	20.785	128
Science	37.205	129
Human Pathology	3.014	139
Laboratory Investigation	4.857	148
Canadian Journal of Gastroenterology and Hepatology	2.147	143
Nature Reviews Immunology	39.932	155
Medicine (Baltimore)	1.804	161
Journal of Immunology	4.856	165

*Gastroenterology-related journals (Gastroenterology and Gut) only published 16 (32%) articles out of the top highly cited 50 articles.

Looking at the ranking of the top-cited articles in IBD in the journals they were published in could provide a better picture about the influence of these articles. This influence would be clearly demonstrated when the journal has a relatively higher JIF. The ranking is based on the number of citations received by an article compared with the number of citations received by other articles that were published in that journal. The article by Best et al23 is ranked number 4 in the list, and also ranked number 2 among all articles published in Gastroenterology (2016 JIF=18.392). The article by Eaden et al41 is ranked number 22 in the list and also ranked number 1 among all articles published in Gut (2016 JIF=16.658). The ranking of other articles in list and their rankings in the journals in which they were published are shown in online supplementary appendix 1.

The first authors of the top-cited articles were from the USA (n=24, 48%), the UK (n=6, 12%), Germany (n=5, 10%), France (n=5, 10%), Belgium (n=3, 6%), Canada (n=3, 6%), Japan (n=2, 4%), Sweden (n=1, 2%) and the Netherlands (n=1, 2%).

Table 4 summarises the 70 authors who have published two papers or more in the top-cited IBD. Of these, 18 authors were the first author and coauthors of more than two papers in total, all the remaining 52 were coauthors of more than two papers. Top authors were Rutgeers, P (n=10 papers), Targan, SR (n=8 papers), Schreiber, S (n=7 papers), Cho, JH (n=7 papers), Colombel, JF (n=7 papers), Hanauer, SB (n=7 papers), Silverberg, MS (n=6 papers) and each of the following authors have five papers: Podolsky, DK, Rioux, JD, Daly, MJ, Steinhart, AH, Rotter, JI, Schumm, LP, Taylor, KD, Vermeire, S, Duerr, RH and Regueiro, M.

Table 4

Authors and coauthors of two or more articles of the top-cited articles in inflammatory bowel disease identified by searching the Web of Science

Author’s name*	Number (references)		Author’s name*	Number (references)
	First author	Coauthor		First author	Coauthor
Hugot, JP	120	234 45	Daly, MJ	–	529 33 34 45 51
Podolsky, DK	125	327 28 31	Steinhart, AH	–	529 34 43 45 51
Targan, SR	126	729 31 33 34 43 45 51	Griffiths, A	–	429 34 45 51
Hanauer, SB	124	621 26 30 31 53 56	Dassopoulos, T	–	229 34
Xavier, RJ	127	333 34 51	Bitton, A	–	329 33 34
Duerr, RH	129	421 33 34 45	Datta, LW	–	329 34 51
Present, DH	231 67	326 30 56	Kistner, EO	–	229 34
Rutgeerts, P	130	924 26 31 34 45–47 52 53	Rotter, JI	–	529 33 34 45 51
Fiocchi, C	136	165	Schumm, LP	–	529 34 45 51 52
Barrett, JC	134	233 45	Lee, J	–	233 45
Riddell, RH	139	243 60	Lees, CW	–	233 45
Loftus, EV Jr	135	143	Sandborn, WJ	–	330 47 53
Franke, A	145	233 54	Barmada, MM	–	329 34 51
Silverberg, MS	143	529 33 34 45 51	Nicolae, DL	–	421 29 34 51
Rioux, JD	151	429 33 34 45	Sands, BE	–	330 31 52
Lichtiger, S	156	147	Belaiche, J	–	220 34
Colombel, JF	247 53	520 24 30 43 45	Laukens, D	–	333 34 45
Abraham, C	157	229 33	Lawrance, I	-–	233 45
Becktel, JM	–	223 58	Louis, E	–	333 34 45
Singleton, JW	–	223 58	Vos, M	–	333 34 45
Kern, F Jr	–	223 58	Vermeire, S	–	533 34 43 45 46
Van Deventer, SJ	–	426 31 52 69	Satsangi, J	–	433 34 43 45
Mayer, L	–	324 26 31	Bernstein, CN	–	243 52
Braakman, T	–	226 31	Tremelling, M	–	233 34
DeWoody, KL	–	226 31	Mansfield, J	–	333 34 45
Schaible, TF	–	226 31	Jewell, D	–	234 43
Feagan, BG	–	324 30 52	Mathew, CG	–	433 34 45 54
Lichtenstein, GR	–	224 30	Parkes, M	–	333 34 45
Schreiber, S	–	724 33 43 45 53 54 68	Georges, M	–	333 34 45
Rachmilewitz, D	–	424 30 47 52	Karban, A	–	243 45
Wolf, DC	–	224 52	Gossum, A	–	234 45
Olson, A	–	224 30	Franchimont, D	–	333 34 45
Taylor, KD	–	529 33 34 45 51	Newman, W	–	233 45
Bayless, TM	–	221 45	Regueiro, M	–	529 33 34 45 51
Cho, JH	–	721 29 33 34 45 51 57	Kornbluth, A	–	247 56

*Author’s name=family name, abbreviations of first or first and second names.

The leadership of universities and institutes that have contributed to the creation of these publications were Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts, the USA, Mount Sinai School of Medicine, New York, the USA, Mayo Clinic, Rochester, Minnesota, the USA, Cedars-Sinai Medical Center, Los Angeles, California, the USA, University of Chicago Medical Center, Chicago, Illinois, the USA, University of Pittsburgh, Pittsburgh, Pennsylvania, the USA, University Hospital of Cleveland Case Western Reserve University School of Medicine, Ohio, the USA, Wellcome Trust Centre for Human Genetics, University of Oxford, Roosevelt Drive, Oxford, the UK, Wellcome Trust Sanger Institute, Wellcome Trust Genome Campus, Cambridge, the UK and Institute of Virology and Immunobiology, University of Würzburg, Germany (see online supplementary appendix 1 for more detail).

Characteristics of the top-cited articles

These articles were created by 667 authors, median 7.0, minimum 1, maximum 106, IQR 9.5; the females in authorship were 111, median 1.0, minimum 0, maximum 20, IQR 2. It is worth mentioning here that it was difficult to identify the gender of some authors in two papers because the full first and middle names were not shown and it was difficult to find more information or clues to make a decision. We contacted the corresponding authors of these two articles; we received information for one article but we had no response from the corresponding author of the second article. Thus, the gender of three authors could not be identified. The number of institutes involved were 436, median 3.0, minimum 1, maximum 88, IQR 9; the countries involved were 141, median 1.0, minimum 1, maximum 16, IQR 2.75 and the number of grants/funds were 328, median 1.0, minimum 0, maximum 94, IQR 2.75. No correlations were found between the number of citations and the JIF (Pearson’s correlation (r)=0.186; p=0.196), the number of authors (r=0.061; p=0.674), number of females in authorship (r=0.064; p=0.661), the number of institutes involved (r=0.076; p=0.602), the number of countries involved (r=0.101; p=0.483) and the number of grants received (r=−0.015; p=0.915).

Level of evidence

Table 5 summarises the grading of articles according to the OCEBM. The table shows that most articles were graded at levels 1b and 3a and 3b evidence (12 papers had level 1b evidence, 9 papers at level 3a and 15 papers had level 3b evidence). Five articles had a level of evidence of 4 and no article at level 5. The remaining articles were at levels 1a, 2a, 2b and 2c. The overall agreement between the evaluators was acceptable; Fleiss kappa=0.8252; 95% CI 0.79 to 0.85.

Table 5

Grading the top-cited articles in inflammatory bowel disease according to the Oxford Centre for Evidence-Based Medicine levels of evidence

Level	Domain	Characteristics and description	Articles number (references)
1a	Therapeutic/prevention, aetiology/harm	Systematic reviews of RCTs (with consistent results from individual studies)	133
	Prognosis	Systematic reviews with homogeneity of inception cohort studies
	Diagnosis	Systematic reviews with homogeneity of level 1 diagnostic studies
	Differential diagnosis/symptom prevalence study	Systematic reviews with homogeneity of prospective cohort studies
1b	Therapeutic/prevention, aetiology/harm	Individual RCT (with narrow CIs)	1224 26 30 47 50 52 53 56 58 59 66 67
	Prognosis	Individual inception cohort study with >80% follow-up
	Diagnosis	Validating cohort study with good reference standards
	Differential diagnosis/symptom prevalence study	Prospective cohort study with good follow-up
2a	Therapeutic/prevention, aetiology/harm	Systematic review of cohort studies (with consistent results from individual studies)	234 63
	Prognosis	Systematic review with homogeneity of either retrospective cohort studies or untreated control groups in RCT
	Diagnosis	Systematic review with homogeneity of level 2 diagnostic studies
	Differential diagnosis/symptom prevalence study	Systematic review with homogeneity of 2b and better studies
2b	Therapeutic/prevention, aetiology/harm	Individual cohort study (including low-quality RCT, eg, <80% follow-up)	331 46 49
	Prognosis	Retrospective cohort study or follow-up untreated control patients in an RCT
	Diagnosis	Exploratory cohort study with good reference standards
	Differential diagnosis/symptom prevalence study	Retrospective cohort study or poor follow-up
2c	Therapeutic/prevention, aetiology/harm	Outcome studies (analysis of large registries)	338 60 61
	Prognosis	Outcomes research
	Diagnosis
	Differential diagnosis/symptom prevalence study	Ecological studies
3a	Therapeutic/prevention, aetiology/harm	Systemic reviews of case-control studies (with consistent results from individual studies)	925 27 35 36 41 43 45 55 57
	Prognosis
	Diagnosis	Systematic reviews with homogeneity of 3b and better studies
	Differential diagnosis/symptom prevalence study	Systematic reviews with homogeneity of 3b and better studies
3b	Therapeutic/prevention, aetiology/harm	Individual case-control study	1520–22 28 29 32 40 42 44 51 54 62 64 65 68
	Prognosis
	Diagnosis	Non-consecutive study, or without consistently applied reference standards
	Differential diagnosis/symptom prevalence study	Non-consecutive cohort study or very limited population
4	Therapeutic/prevention, aetiology/harm	Case series (and poor quality cohort and case-control studies)	523 37 39 48 69
	Prognosis	Case-series (and poor quality prognostic cohort studies)
	Diagnosis	Case-control study, poor or non-independent reference standard
	Differential diagnosis/symptom prevalence study	Case-series or superseded reference standards
5	Therapeutic/prevention, aetiology/harm	Expert opinion without explicit critical appraisal or based on physiology, or bench research	0 (0)
	Prognosis	Expert opinion without explicit critical appraisal or based on physiology, or bench research
	Diagnosis	Expert opinion without explicit critical appraisal or based on physiology, or bench research
	Differential diagnosis/symptom prevalence study	Expert opinion without explicit critical appraisal or based on physiology, or bench research

RCT, randomised controlled trial.

Discussion

This study aimed at identifying the characteristics of the top 50 most frequently cited papers in IBD and assessing the quality of evidence provided. The papers covered a number of key topics related to IBD including: epidemiology and prevalence, pathogenesis and genetic susceptibility, animal models, clinical classification and indices of disease activity, risks of developing colorectal cancer, extraintestinal complications and use of infliximab, adalimumab, monoclonal antibodies and antitumour necrosis factor and 6- mercaptopurine in CD, and use of 5-aminosalicylic acid, corticosteroids and ciclosporin in treating UC. While these topics cover key issues related to IBD, topics related to molecular biology, surgical management, patient education, nutritional aspects, radiological and other investigations were not represented in the top-cited articles list.70 71

The assessment of the characteristics of the top 50 highly cited articles identified in this study reveals the following:

First, the lack of correlations between the number of citations and the number of years since publication may indicate that the high citations is not caused by ageing of these articles. In fact, only 15 articles (30%) were published in the period from 1955 to 1994 (39 years), while the majority, 35 (70%) were published in the last 17 years. No papers were published after 2012. This article also shows that the mean number of citations of articles published before the year 2000 were much lower than those of articles after the year 2000. This may be related to the tendency of researchers to cite recently published research and new findings72 and papers that they have read.18 Also the number of citations and the papers’ age might be attributed to the changes in citation behaviour over time. For example, in the 50s and 60s, the authors used to cite fewer papers compared with what was practised over the last 20 years.73 The higher citations have been found to be the strongest predictor of current online availability after a long time since publication. Therefore, the higher citation could be a protective mechanism for continued availability of a publication despite ageing and hence continuing citation.74

Second, the lack of a correlation between the number of citations and the JIF. The JIF has been widely used in ranking and evaluating journals. It stands as a proxy for the relative importance of a journal with its field.75 Although the top-cited articles identified were published in journals with high impact factors, the impact factors of some journals were not necessarily the highest in their fields. For example, Laboratory Investigation, Canadian Journal of Gastroenterology and Hepatology, Medicine (Baltimore) and Journal of Immunology had JIFs in the range of 1.804–4.857. With this information in mind one may postulate that the high JIF is not necessarily related to the higher citation numbers received. Two recent works showed that the JIF is not an accurate indicator of citations an average article receives, articles published in low impact factor journals can still be highly cited and vice versa.76 77

Third, the study showed negative correlation between the number of citations and the number of authors, the number of female authors or the number of institutes. The number of authors and females in authorship varied from 1 to 106 and from 0 to 20, respectively. Also the number of institutes involved varied from 1 to 88. The question that can be raised in this regard is: are we expecting an increase in number of citations as the number of authors or the number institutes involved increased? The work of García-Aroca et al78 shows that collaboration between authors increases their impact and increases citation rates. However, they showed that publishing in English in certain journals and collaborating with certain authors and institutes increase the visibility of the manuscripts published on the subject. Hence, it is the quality of collaboration rather than the absolute number of these parameters. Recently, Tanner-Smith and Polanin showed that studies conducted by more established authors (have higher h-indices) and reported in more prestigious journal outlets are more likely to be cited by other scholars, even after controlling for various proxies of study quality.79

Although the proportion of women in authorship of original research in the USA in general has significantly increased in the last four decades, women still compose a minority of the authors of original research.80 In the field of gastroenterology, the percentage of the US female physician authors of original research in the field has relatively increased over time, yet the senior author position remains lower than expected.81 82

Fourth, the study showed no correlation between the number of citations and the number of funds/grants received. This finding is not surprising. Recently, it was shown that too many of the US authors of most influential papers in science do not receive NH funding.83 Another group of researchers found no association between grant percentile ranking and grant outcome as assessed by number of top 10% articles per dollar million spent.84

The USA, the UK, Germany, France, Belgium and Canada contributed to the majority of these articles. The leadership of universities from these countries in gastrointestinal research particularly IBD is no surprise, top universities identified from this study were Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts, the USA, Mount Sinai School of Medicine, New York, the USA, Mayo Clinic, Rochester, Minnesota, the USA, Cedars-Sinai Medical Center, Los Angeles, California, the USA, University of Chicago Medical Center, Chicago, Illinois, the USA. Other institutes and research centres that had led these studies are shown in online supplementary appendix 1.

All articles were published in the English language. The most productive journals were the New England Journal of Medicine and Gastroenterology with a total of 24 articles. Others were Nature, Nature Genetics, the Lancet and Proceedings of the National Academy of Sciences of the United States of America, Cell, Gut, British Medical Journal, Science and Nature Reviews Immunology making a total of 20 articles. While these journals have a relatively high JIF, other journals published one article each and had a relatively low JIF compared with journals included in their categories. For example, Human Pathology listed number 22 under the category of Pathology, and Canadian Journal of Gastroenterology and Hepatology, listed number 74 under the category Gastroenterology and Hepatology. The high level of evidence as outlined in the top-cited articles could be an important contributing factor to the higher number of citations received by these articles.85

After the submission of our study we came across the study by Connelly et al86 on top 100 articles on IBD. While they indicated that their findings covered the period between 1955 and 2012, which is consistent with our findings, our study has examined a range of parameters including correlation of citation numbers with JIF, number of years since publication, number of authors, females in authorship, institutes and number of countries and grants, which were not addressed by Connelly et al. Also we have examined the level of evidence of highly cited articles.

Conclusions

Our list of top-cited articles in IBD highlights key contributions that based the foundation of research and examination of different aspects of the disease over 57 years. This scholarly contribution came from universities and research centres in the USA, the UK, Germany, France, Belgium and Canada. The findings may be consistent with the concept that it is not the absolute number of collaborators that makes an impact on the citation number or the influence of a publication but rather the quality of such collaboration with regard to the researchers involved, their institutes and the ongoing contribution to the advances of research. The relatively small number of females in the authorship reflects the gender gap and the fact that women still compose a minority of the authors of original research and reviews in gastroenterology. The higher level of evidence demonstrated in most top-cited articles may have contributed to the higher number of citations received by these articles.