Institution Origin and Medical School Rank Impact the Citation Frequency and Publication Rate in Orthopaedic Sports Medicine Journals.

Purpose: To examine the trends between various categories of institutions with their respective published orthopaedic sports medicine content and to determine the publication output and citation rate from the 25 highest-ranked medical schools compared with lower-ranked institutions.
Methods: Publications between 2015 and 2019 from the American Journal of Sports Medicine, Journal of Bone and Joint Surgery, Journal of Shoulder and Elbow Surgery, Clinical Orthopaedics and Related Research, and Arthroscopy were categorized into university/university affiliated hospitals, non-university affiliated teaching hospitals, public/semi-government research institutes, nonprofit research institutes, private sector institutions, government institutions, and other institutions. Citation rates were collected from PubMed for the first and corresponding author. Similarly, corresponding authors were stratified by U.S. News and World Report 2021 medical school research rankings.
Results: Of the 12,152 publications identified, 5,044 publications met the inclusion criteria. Nonprofit research institutions garnered the greatest number of citations on average (6.44 based on first author, SD 8.83, n = 214; 6.62 based on corresponding author, SD 9.65, n = 208; P < .001), while university/university-affiliated hospitals produced the majority of published articles (77.0% based on first author, 76.8% based on corresponding author), but had lower average citation rates (4.48 based on first author, SD 6.67, n = 3,886; 4.44 based on corresponding author, SD 6.55, n = 3,873; P < .001). Furthermore, of 1953 medical school publications, the top 25 accounted for 53.1% of publications; however, there was no statistical difference between their citation rates and those of lower rankings (P = 0.47). Conclusions: Publications are cited at different rates, depending on their institution of origin. In addition, high-ranking medical schools produce a disproportionately greater output of publications than lower-ranking schools, but there is no statistically significant difference in citation rates on an individual publication basis. Clinical Relevance: Knowing how an institution's ranking influences publication and citation rates can help us understand bias in the scientific literature.

The reduction of bias is important for producing high-quality research. However, as rankings and prestige of academic institutions are in part attributed to their publication output, publication bias may arise. Academic institutions are pressured to produce more publications, and while this finding likely applies to basic science research, the same could be true for clinical research, as millions of people have coauthored at least one biomedical paper from various institutions. Ioannidis has explored the various biases that exist in clinical research, stating that universities and other institutions demand a high output of publications by their researchers, potentially at the expense of clinical practicality. As a result, understanding the impact of research according to specific institution types in which they originate is important, which could influence where physicians interested in research would consider employment to publish high-impactful research.

With an increased push toward evidence-based medicine, the impact factor of a journal becomes increasingly important as this characteristic is a reflection of the readership’s trust in the quality of their information. Impact factor is a metric that is used to measure the importance of a journal within its field and has been used to determine guidelines for evidence-based medical practices. It is determined by the number of citations a publication receives and has been calculated for many orthopaedic journals. Previous studies have sought to determine which countries are publishing the most orthopaedic-related research to determine the trends of publications as well as the quality and quantity of publications from several countries and have found that the United States is consistently publishing significantly more than any other country.5, 6, 7, 8 However, the publication output and number of citations received based on the type of institution or specific institutions themselves has yet to be elucidated in the field of orthopaedic sports medicine.

The purposes of this study are to examine the trends between various categories of institutions with their respective published orthopaedic sports medicine content and to determine the publication output and citation rate from the 25 highest-ranked medical schools compared with lower-ranked institutions. Our primary null hypothesis was that there would be no difference between institution category and citation rate for orthopaedic research, while additionally, our secondary null hypothesis was that the top 25 medical schools would not produce a significantly greater amount of orthopaedic sports medicine related research, with an overall greater citation rate as well.

Methods

Study Selection

The data gathered for this study were obtained from 5 major orthopaedic journals from the years 2015 to 2019. These include the American Journal of Sports Medicine (AJSM), Journal of Bone and Joint Surgery (JBJS), Journal of Shoulder and Elbow Surgery (JSES), Clinical Orthopaedics and Related Research (CORR), and Arthroscopy. These journals were chosen as they represent the 1st (AJSM), 2nd (JBJS), 4th (Arthroscopy), 5th (CORR), and 12th (JSES) highest ranked orthopaedic journals based on impact factor. The 3rd and 6th-11th journals were not selected, as they did not include a sufficient number of sports medicine publications. First, extraction of all shoulder-, elbow-, and sports medicine–relevant publications was performed. Publications pertaining to orthopaedic hand, non-sports foot and ankle, spine, oncology, trauma, prosthetics, and total joint reconstruction (besides total shoulder arthroplasty), congenital diseases, and autoimmune diseases were excluded. Editorials, society newsletters, specialty updates, errata, abstracts, and correspondences were all excluded as well.

Data Collection

While we gathered the articles, both the first and corresponding author affiliations were acquired in case they have different affiliations. In the instances where an author would have multiple personal affiliations, the first one listed would be used. Citation rate was defined as the number of times an article was cited according to PubMed. The number of citations listed on PubMed for each article were recorded, and each article was classified twice, first based on the first author’s institution, and second from the corresponding author’s institution. Using a modified classification system derived from Hottenrott and Lawson, each article was divided into one of the following categories:

Higher education sector

University/university-affiliated hospitals

Non–university-affiliated teaching hospitals

Public or semi-government research institutes

Nonprofit research institutes

Private sector institutions

Government Institutions

Other institutions

Hottenrott and Lawson combined “1a” and “1b” into one classification of “Higher Education Sector,” but we divided this into 2 categories for increased differentiation. The websites of each unique institution were used to place them into one of the classifications (Fig 1).

Fig 1

Flow chart describing the process of classifying sports medicine articles from 5 high-impact orthopaedic journals into 7 categories between the years 2015 and 2019. Inclusion and exclusion criteria are listed, as well as the total number of articles in each step of the process.

Statistical Analysis

One-way analyses of variance were run to determine whether a statistically significant difference existed in number of citations among different institution classifications. This was repeated for the first and corresponding authors of each publication. Another analysis of variance was run to determine whether medical school rank had any effect on the number of citations. Tukey post hoc analysis was conducted to reveal any significant difference within groups. The data were analyzed in SPSS (IBM Corp. Released 2017. IBM SPSS Statistics for Windows, Version 25.0; IBM Corp., Armonk, NY), and a P value of <.05 was considered statistically significant.

Results

Of the 12,152 publications identified between the years 2015 and 2019 when considering all 5 journals, 6902 were omitted based on the inclusion criteria, and 206 (3.9%) publications were omitted because of the inability to classify them into 1 of the 6 categories used in this study. This left 5,044 articles to be analyzed that fit the inclusion criteria (Fig 1). In total, 1,798 articles with their respective number of citations were extracted from AJSM between February 2 and February 14; 340 from JBJS between February 15 and February 21; 1,328 from JSES between February 22 and February 29; 201 from CORR between March 1 and March 18; and 1,377 from Arthroscopy between April 5 and April 29. There is a statistically significant difference in the number of citations recorded by each classification for both the first and corresponding author, as publications originating from nonprofit research institutions have the highest number of citations on average (6.44 based on first author, standard deviation [SD] 8.83, n = 214; 6.62 based on corresponding author, SD 9.65, n = 208; P < .001) whereas non–university-affiliated teaching hospitals have the lowest number of citations on average (3.98 based on first author, SD 6.32, n = 155; 3.96 based on corresponding author, SD 6.35, n = 153; P < .001).Universities and university-affiliated hospitals produced majority of the articles published (77.0% based on first author, 76.8% based on corresponding author) but have a lower average citation rate (4.48 based on first author, SD 6.67, n = 3,886; 4.44 based on corresponding author, SD 6.55, n = 3,873; P < .001). Additional information, such as mean citation rate for the first and corresponding authors as well as the percentage of publication distribution based on author affiliation, can be found in Table 1, Table 2, Table 3.

Table 1

Citation Rate Stratified by First Author

Classification (First Author)	Sample Size	Mean Number of Citations	Standard Deviation	95% Confidence Interval	P Value
University/university–affiliated hospitals	3,886	4.48	6.67	4.27-4.69	–
Non–university-affiliated teaching hospitals	155	3.98	6.32	2.98-4.98	–
Public or semi-government research institutes	11	5.09	5.15	1.63-8.55	–
Nonprofit research institutes	214	6.44	8.83	5.25-7.63	–
Private sector institutions	482	5.34	8.26	4.60-6.08	–
Government institutions	287	4.15	5.56	3.50-4.80	–
Other institutions	9	4.33	4.36	0.98-7.68	–
Total	5,044	4.61	6.88	4.42-4.80	<.001

Table 2

Citation Rate Stratified by Corresponding Author

Classification (Corresponding Author)	Sample Size	Mean Number of Citations	Standard Deviation	95% Confidence Interval	P Value
University/university-affiliated hospitals	3,873	4.44	6.55	4.23-4.65	–
Non–university-affiliated teaching hospitals	153	3.96	6.35	2.95-4.97	–
Public or semi-government research institutes	13	4.62	4.91	1.65-7.58	–
Nonprofit research institutes	208	6.62	9.65	5.30-7.93	–
Private sector institutions	519	5.41	8.35	4.69-6.13	–
Government Institutions	270	4.43	5.85	3.73-5.13	–
Other institutions	8	4.13	5.11	–0.15-8.40	–
Total	5,044	4.61	6.88	4.42-4.80	<.001

Table 3

Publication Distribution by Institution and Author Affiliation

Percentage of Publications by Author Affiliation (First and Corresponding)
Author	University/University-Affiliated Hospitals	Non–university-Affiliated Teaching Hospitals	Public or Semi-Government Research Institutes	Nonprofit Research Institutes	Private Sector Institutions	Government Institutions	Other Institutions
First author	77.0%	3.07%	0.22%	4.24%	9.56%	5.69%	0.18%
Corresponding author	76.8%	3.03%	0.26%	4.12%	10.3%	5.35%	0.16%

Of the 3,873 university/university-affiliated hospitals when sorting by corresponding author, 2,048 publications were extracted, as they were United States medical schools or affiliates. Medical schools that were “unranked” from U.S. News and World Report 2021 research edition were omitted from analysis (n = 95, 4.64% of data set). Excluding these unranked schools, 1,953 publications were analyzed. In the few instances in which a hospital had more than 1 medical school affiliate, the higher-ranked school was used for classification. Comparisons were made between their ranks, showcasing the top 10 accounting for 27.1% (n = 529) of all sports medicine publications, 11-25 at 26.0% (n = 508), 26-50 at 24.0% (n = 469), 51-75 at 19.9% (n = 389), and 76+ at 3.0% (n = 58) (Table 4). No statistical difference between the rank of a medical school and the average number of citations they garner was present (P = .47).

Table 4

Citation Distribution by Medical School Rank

Percentage of Citations by Medical School Rank (Corresponding Author)
Medical School Rank	Percentage of Total Citations	Mean Number of Citations	Standard Deviation	95% Confidence Interval
1-10	27.1% n = 529	4.36	5.83	3.86-4.86
11-25	26.0% n = 508	5.06	7.24	4.43-5.69
26-50	24.0% n = 469	5.04	8.25	4.30-5.79
51-75	19.9% n = 389	4.57	6.84	3.89-5.25
76+	3.0% n = 58	4.81	8.65	2.54-7.08

NOTE. n = sample size.

Discussion

In this study of 5 orthopaedic medical journals, we observed that with regard to stratifying orthopaedic sports medicine publications produced by U.S. medical schools, the top 25 medical schools produce greater than one-half of the publications. Also, there was a significant difference between the number of citations a publication received and the origin of the publication when accounting for both the first author and corresponding author. In both instances, publications originating from nonprofit research institutions garnered the most citations on average (P < .001), while papers originating from non–university-affiliated teaching hospitals garnered the least amount of citations on average. In addition, our study found that orthopaedic surgeons who work in the private sector publish significantly less than orthopaedic surgeons who work at academic institutions, and that university-affiliated hospitals publish substantially more than non–university-affiliated hospitals.

The importance of publishing research in the field of academic medicine has been displayed in previous studies and have shown an association between the amount of funding an institution receives and their research output., A previous study by Hottenrott and Lawson found that authors of bioscience publications affiliated to the greater education sector composed the bulk of published articles, similar to those our study. A study by Movassagi et al. found that the number of authors or number of institutions contributed per publication did not increase the citation rate in orthopaedic sports medicine publications, but they did not break down the specific citation rates per institution type. As a result, there is no conflict between our studies.

Aside from the pressure to publish in academic settings, the reason for a great number of publications originating in academic centers could stem as far back as medical school and residency. In order for a medical school applicant to be competitive in obtaining an orthopaedic surgery residency spot, they must complete research. Schrock et al. found that when comparing matched U.S. seniors with unmatched U.S. seniors, matched seniors had more research products (abstracts, presentations, posters, and publications) (4.6) compared with unmated seniors (3.0), a finding that was statistically significant. Once in residency, those who pursue an academic career postresidency publish more during residency than those who pursue a nonacademic career., Namdari et al. found that during orthopaedic residency, those continuing to an academic career had 4.8 publications, compared with only 2.4 for those who did not pursue an academic career, although this finding was not significant. However, when stratifying the publications, they found that there was a statistically significant difference in the number of basic science (0.7 vs 0.4) and review articles (0.9 vs 0.4). This phenomenon was not only seen in orthopaedics, as a systematic review by Straus et al. found that publishing during residency is also associated with a career in academic radiology. The importance of publishing high-impact literature can also largely be attributed to perceived success amongst colleagues, as well as being the basis for promotions.,16, 17, 18 Furthermore, it has been stated that top universities from high research output cities produce more research at a relatively lower cost, which can be a contributing factor as to why academic publications exceed private practice and non–university affiliated hospital research output. Similarly, it was found that increased research output was associated with dedicated research time and a strong mentorship program. These 2 factors are more likely to be associated with a top university than in private practice or a non–university-affiliated hospital. Nevertheless, the publication rate of biomedical sciences originating from private practice institutions has been increasing, but is still low.

Nonprofit research institutions were determined to have the highest citation rate on average, with a higher average citation rate than university and university affiliated publications (P < .001 based on first author and corresponding author). This could be due to many underlying factors in which more research is needed, such as funding, types of research being performed, and reputation of specific institutions themselves. The sample size of this group is also much smaller than the university and university affiliated group, which could affect results.

U.S. News and World Report medical school rankings are determined in large part by National Institutes of Health research funding. This, in turn, can support more intangibles of conducting research, such as manuscript preparation, biostatistical support, or other ancillary support, resulting in the increased publication output from greater ranked schools. However, when comparing the average number of citations per publication, there was no significant difference, implying that when lower ranked schools do publish, the quantity of viewership of their publications are similar to their higher ranked counterparts.

Different research and citation trends of published articles in the field of orthopaedic surgery have been studied. For example, Movassagi et al. identified 2 independent predictive factors that result in greater citation rates of published articles: the first being articles published in AJSM were more likely to have a greater citation rate and the second being articles focusing on the hip were more likely to have greater citation rates. Furthermore, in a study conducted by Kortlever et al., there was no difference in the likelihood of an article being cited based on whether the article was published in a subscription-based journal versus an open-access journal. Though the results from these studies analyze different factors than those analyzed in this study, it provides additional support to the citation trends seen in published articles within the field of orthopaedic surgery.

Limitations

This study was not without limitations. Since data on the number of citations were gathered in the first quadrimester of 2020, most articles published in 2019 had either 0 or 1 citations present at the time of gathering according to PubMed. This has the potential to decrease the average number of citations. In addition, there are many nonprofit or academic institutions with orthopaedic surgery residency programs that are not affiliated directly with a medical school that were omitted from this analysis. Also, a factor of the exclusion criteria was based on the inability to understand the language of certain non-English international programs, and as a result, were omitted from analysis. Furthermore, only the first and corresponding authors’ institutions were extracted and analyzed in this study. The affiliated institution of the remaining authors was not included in our analysis.

Conclusions

Publications are cited at different rates, depending on their institution of origin. In addition, high ranking medical schools produce a disproportionately greater output of publications than lower ranking schools, but there is no statistically significant difference in citation rates on an individual publication basis.