An Evaluation of Self-Reported Publications in Orthopaedic Sports Medicine Fellowship Applications.

BACKGROUND: Orthopaedic sports medicine fellowship positions are increasing in popularity, as evidenced by the increasing number of applicants to these programs. As positions have become more competitive, greater emphasis has been placed on an applicant's research experience. However, there has been a lack of research evaluating the accuracy of self-reported publications from fellowship applications.
PURPOSE: To evaluate the accuracy of self-reported research publications and the outcomes of studies submitted for publication by applicants to an Accreditation Council for Graduate Medical Education (ACGME)-accredited sports medicine fellowship in the United States (US). STUDY
DESIGN: Cross-sectional study.
METHODS: Demographic and research publication data were retrospectively collected from 435 applications to an ACGME-accredited orthopaedic sports medicine fellowship program at a single high-volume academic institution from 2013 to 2017. All self-reported manuscript publications and studies in progress were analyzed with a minimum 2-year follow-up. "Submitted" publications were reviewed by searching the originally submitted journal and all publicly available sources. Publications were verified on PubMed, MEDLINE, and other open access journals. Journal impact factors were collected through use of InCites Journal Citation Reports.
RESULTS: Only 5.7% (85/1504) of papers reported as "completed" were inaccurately self-reported, with 44 (51.8%) remaining unverified and 41 (48.2%) reporting discordant authorship, in which the published study listed a different author order than reported on the application. Further, 28.3% (197/696) of papers self-reported as "submitted" remained unpublished, 21.8% (152/696) were published in a different journal than originally reported, and 7.6% (53/696) were published with a different authorship order than reported. Among 95 applicants whose papers were published in different journals than originally reported, the mean impact factor of the final accepting journal was significantly lower than that of the journal of original submission (0.97 ± 0.13 vs 3.91 ± 0.79, respectively; 95% CI of the difference, 1.34-4.54; P < .01). Univariate analysis showed no significant relationships between variables of interest (age, sex, US Medical Licensing Examination Step 1 score, American Orthopaedic Association membership, medical school ranking, and advanced degree) and the presence of an inaccuracy.
CONCLUSION: There is a low rate of inaccurate self-reporting of "completed" publications on applications for orthopaedic sports medicine fellowships. The majority of papers listed as "submitted" on these applications were not published in the journals to which they were originally submitted.

Since the establishment of subspecialty orthopaedic programs in the 1970s, an increasing number of orthopaedic residents have elected to pursue subspecialty fellowship training after residency graduation.[3,6] From 2003 to 2013, the percentage of fellowship-trained orthopaedic surgeons in the United States (US) increased from 76% to 90%.[6] Sports medicine fellowships that are accredited by the US Accreditation Council for Graduate Medical Education (ACGME) are historically the most sought-after orthopaedic fellowship programs. According to a 2013 organized match, 256 of 1207 participating orthopaedic residency graduates (21.2%) applied to sports medicine fellowship programs, with 42% of applicants matching into their first choice.[3] Additionally, sports medicine fellowships offer the most positions in which to match. As recently as 2019, sports medicine fellowships had the most positions (n = 228) offered to fellowship applicants and the second highest number of participating programs (n = 89) in the San Francisco Residency and Fellowship Match Services (SF Match).[11]

Given their popularity, sports medicine fellowship positions are becoming increasingly competitive, which has placed a greater emphasis on research experience during residency as a factor for programs to grant interviews and establish a rank list.[1] Reasons for valuing research experience include assessing an applicant’s proficiency in the scientific method, commitment to ongoing projects, and potential to continue research during fellowship. In the SF Match, applicants report their published research and any works currently being submitted. For fellowship directors and attendings evaluating these applications, it is especially difficult to evaluate self-reported works that are currently submitted to a journal or are in preparation for submission to a journal. To our knowledge, there is a paucity of research on the outcomes of works listed by applicants as being submitted to a journal. The percentage of submitted studies that ultimately reach publication is unknown to application reviewers, and it may be hard to gauge an applicant’s aptitude for research without knowing this information.

The purpose of this study was to evaluate the accuracy of self-reported research publications by applicants to a US ACGME-accredited sports medicine fellowship and to analyze the publication outcomes of research listed by applicants as being submitted to a journal. We hypothesized that there will be a low rate of inaccurate reporting of publications and that a significant percentage of studies submitted for publication will not be published in the journal to which they were reported as being submitted. We also investigated the relationship between applicants who self-reported published papers and the following demographic variables: (1) age, (2) sex, (3) US Medical Licensing Examination (USMLE) Step 1 score, (4) American Orthopaedic Association (AOA) membership, (5) medical school type (US doctor of medicine [MD] program, US doctor of osteopathy [DO] program, or international MD program), (6) attendance at a National Institutes of Health (NIH) top-40 medical schools for research,[13] and (7) attainment of an advanced degree (master’s or doctorate). We expect these results to provide improved data and an overall better understanding of applicants’ self-reported research experience to evaluate future applicants. We also intend for this study to provide a framework for future research of “submitted” publications within residency and fellowship applications.

Methods

Data were retrospectively collected from applications to an ACGME-accredited orthopaedic sports medicine fellowship program at a single high-volume academic institution from 2013 to 2017. Demographic data included the 7 areas listed earlier, and publication data included listings for which we could conduct a minimum 2-year audit. These research papers were classified as either “completed” or “submitted.” Completed papers included those that were self-reported as either published in a print journal or electronically published ahead of print. Submitted papers included those that were self-reported as being “accepted,” “in press,” “under review,” “in progress,” “in preparation,” or “submitted.” Reported abstracts, book chapters, magazine entries, and web-based articles were excluded from our analysis.

Completed papers and their associated author lists were verified on PubMed, MEDLINE, and other listed open access journals. Papers that could not be confirmed as published were recorded as “unverified.” Papers in which the applicant’s name was listed in a different order than what appeared on the verified publication were recorded as having “discordant authorship.” The sum of papers recorded as unverified or having discordant authorship were classified as “inaccurate completed” papers. A univariate analysis was conducted using a chi-square test for categorical variables and a Student t test for continuous variables to assess for variables that influenced inaccurate reports. Significance was set at P < .05.

Submitted papers and their associated author lists were also investigated on PubMed and other listed open access journals. Papers unable to be confirmed as published were recorded as “unpublished.” Papers in which the applicant’s name was listed in a different order than what appeared on the verified publication were recorded as having “incorrect authorship.” We also recorded papers that were published in a different journal than the one listed on the application. Journal impact factors were derived from InCites Journal Citation Reports (Thomson Reuters); unlisted journals were assigned an impact factor of 0. Among eligible applicants, the mean difference in average impact factor between the originally listed journal and the journal in which papers were ultimately published was calculated by use of a paired-samples t test. Significance was set at P < .05.

Results

Overall, data from 435 applicants between 2013 and 2017 were analyzed. The cohort consisted of 77.7% US MD medical graduates, 8.0% US DO applicants, and 14.3% international MD graduates. Additional demographics are displayed in Table 1.

Table 1

Demographics

Total applicants, N	435
Mean age, y (n = 305)	31.25
Median age, y (n = 305)	31
SD age, y	3.032
Male, n (%)	389 (89.4)
Female, n (%)	46 (10.6)
AOA, n (%)	32 (21.4)
US MD medical graduates, n (%)	338 (77.7)
US DO medical graduates, n (%)	35 (8.0)
International medical graduates, n (%)	62 (14.3)
Applicants who attended NIH top-40 medical school, n (%)	138 (31.7)
Advanced degree, %	17.7
USMLE Step 1 mean, median (n = 399)	235.8, 238
USMLE Step 2 mean, median (n = 390)	242.0, 245
USMLE Step 3 mean, median (n = 360)	218.5, 220

AOA, American Orthopaedic Association; DO, doctor of osteopathy; MD, doctor of medicine; NIH, National Institutes of Health; USMLE, US Medical Licensing Examination.

The applicants reported a total of 2200 auditable research studies. Of these, 1504 (68.4%) were reported as “completed” papers, and 696 (31.6%) were reported as “submitted” papers. We further found that 74.3% of applicants reported at least 1 paper as completed, and 51.7% of total applicants reported at least 1 paper as submitted. Stratified results are reported in Table 2.

Table 2

Publications Reported as “Completed” and “Submitted”

	Total	US MD Graduates	US DO Graduates	International MD Graduates
Papers reported as “completed,” n	1504	1216	28	260
Papers reported as “completed” per applicant, mean (median)	3.5 (2.0)	3.6 (2.0)	0.8 (0.0)	4.2 (1.5)
Applicants with ≥1 paper reported as “completed,” n/N (%)	323/435 (74.3)	266/338 (78.7)	15/35 (42.9)	42/62 (67.7)
Papers reported as “submitted,” n	696	610	11	75
Papers reported as “submitted” per applicant, mean (median)	1.6 (1.0)	1.8 (1.0)	0.3 (0.0)	1.21 (0.0)
Applicants with ≥1 paper reported as “submitted,” n/N (%)	225/435 (51.7)	190/338 (56.2)	9/35 (25.7)	26/62 (41.9)

DO, doctor of osteopathy; MD, doctor of medicine; US, United States.

Among the 1504 completed papers, 85 (5.7%) were classified as “inaccurate” (Table 3). These inaccurate publications consisted of 44 (51.8%) unverified publications and 41 (48.2%) discordant authorships. Papers reported among US DO graduates exhibited the highest inaccuracy percentage (17.9%), with the least number of total papers (n = 28). Papers reported among international MD graduates exhibited the lowest inaccuracy percentage (3.9%). Among the 435 applicants, 85 (19.8%) had at least 1 inaccuracy in their self-reported published studies.

Table 3

Inaccuracy in Reporting of Publications as “Completed”

	Total	US MD Graduates	US DO Graduates	International MD Graduates
Inaccurate completed papers	85/1504 (5.7)	70/1216 (5.8)	5/28 (17.9)	10/260 (3.9)
Unverified completed papersb	44/1504 (2.9)	37/1216 (3.0)	3/28 (10.7)	4/260 (1.5)
Completed papers with discordant authorship	41/1504 (2.7)	33/1216 (2.7)	2/28 (7.1)	6/260 (2.3)
Applicants with ≥1 inaccuracyc	64/323 (19.8)	52/266 (19.5)	5/15 (33.3)	7/42 (16.7)

Values are expressed as n/N (%). DO, doctor of osteopathy; MD, doctor of medicine; US, United States.

Journals verified through PubMed and open access journals.

In applicants with ≥1 paper reported as “completed” on fellowship application.

Univariate analysis of fellowship applicants with at least 1 completed paper showed no significant relationships between the variables of interest (age, sex, USMLE Step 1 score, AOA membership, medical school type, NIH top-40 medical school for research, and attainment of an advanced degree) and the presence of an inaccuracy (Table 4).

Table 4

Univariate Analysis of Fellowship Applicants With at Least 1 Publication Reported as “Completed”

Variable	≥1 Inaccuracy (n = 64)	No Inaccuracies (n = 259)	P Value
Age, y, mean ± SD	31.0 ± 2.5	31.5 ± 2.5	.28
Sex			.21
Male	60 (93.8)	229 (88.4)
Female	4 (6.3)	30 (11.6)
USMLE Step 1 score, mean ± SD	236.4 ± 17.2	236.4 ± 15.9	.97
AOA	16 (25.0)	57 (22.0)	.63
US MD graduates	52 (81.3)	214 (82.6)	.80
US DO graduates	5 (7.8)	10 (3.9)	.18
International MD graduates	7 (10.9)	35 (13.5)	.58
Attendance at an NIH top-40 medical school	30 (46.9)	95 (36.7)	.13
Advanced degree	8 (12.5)	51 (19.7)	.18

Values are expressed as n (%) unless otherwise noted. AOA, American Orthopaedic Association; DO, doctor of osteopathy; MD, doctor of medicine; NIH, National Institutes of Health; USMLE, United States Medical Licensing Examination.

Among the 696 submitted papers, 294 (42.3%) were published in their originally reported journal (Table 5). A further 28.3% of papers remained unpublished after 2 to 6 years. The percentage of unpublished papers per year remained stable between 2013 and 2017 (22.8%, 34.0%, 22.7%, 30.5%, and 28.0%, respectively; P = .23). US MD graduates accounted for 87.3% of the unpublished papers (Figure 1). Further, 21.8% of papers were published in a different journal than originally reported, and 7.6% (53/696) were published with a different authorship order than originally reported. Among applicants (n = 95) whose papers were published in different journals than originally reported, the mean impact factor of the final accepting journal was significantly lower than that of the journal of original submission (0.97 ± 0.13 vs 3.91 ± 0.79, respectively; P < .01; 95% CI of the difference, 1.34-4.54) (Figure 2).

Table 5

Outcomes of Papers Reported as “Submitted”

	Total	US MD Graduates	US DO Graduates	International MD Graduates
Submitted papers published in original journal	294/696 (42.3)	259/610 (42.5)	3/11 (27.3)	32/75 (42.7)
Unpublished submitted papersb	197/696 (28.3)	172/610 (28.2)	4/11 (36.4)	21/75 (28.0)
Submitted papers with incorrect authorship	53/696 (7.6)	44/610 (7.2)	1/11 (9.1)	8/75 (10.7)
Papers published in a different journal than originally reported	152/696 (21.8)	135/610 (22.1)	3/11 (27.3)	14/75 (18.7)
Difference in average impact factor, mean ± SD (n = 95)c	2.94 ± 0.81 (P < .01)	2.99 ± 0.89 (P < .01)	2.83 ± 0.89 (P = .20)	2.53 ± 2.45 (P = .33)
Applicants with ≥1 paper published in a different journal than originally reportedd	95/225 (42.4)	82/190 (43.2)	3/9 (33.3)	10/26 (38.5)

Values are expressed as n/N (%) unless otherwise noted. DO, doctor of osteopathy; MD, doctor of medicine; US, United States.

Journals verified through PubMed and open access journals.

Mean difference in average impact factor between journals in which papers were reportedly submitted and journals in which papers were ultimately published per applicant.

In applicants with ≥1 paper reported as “submitted” on fellowship application.

Figure 1.

Percentage of papers reported as submitted for publication that remained unpublished. Percentages stratified by graduating medical school type: US doctor of medicine (MD) program, US doctor of osteopathy (DO) program, and international MD program. N signifies the total number of papers per category.

Figure 2.

Average impact factor of publication ultimately accepted to a different journal. Journal impact factors were verified through InCites Journal Citation Reports. The average impact factor of the original journals of submission (3.91 ± .079) was significantly higher than the average impact factor of the accepting journals (0.97 ± 0.13). ***P < .01.

Discussion

This study sought to evaluate the accuracy of self-reported research publications by applicants to a single ACGME-accredited orthopaedics sports medicine fellowship and to analyze the outcomes of research classified as “submitted” on fellowship applications. Upon retrospective review with minimum 2-year follow-up, we found that 5.7% of papers reported as “completed” were either unpublished or contained a different author order than reported and that 19.8% of applicants had at least 1 inaccuracy in self-reported “completed” publications. We also found that 28.3% of papers reported as “submitted” remained unpublished and 21.8% of submitted papers were published to a different journal than originally reported.

There have been many proposed reasons for the inaccuracy of self-reported publications. These include honest mistakes by applicants when reporting citations, the pressure applicants feel to overstate their accomplishments to secure competitive positions, and the perception that programs do not verify publications.[2,7,14] Of the papers reported as “completed,” our finding of 85 of 1504 (5.7%) inaccurately reported papers was relatively low compared with previous studies of inaccuracies within orthopaedic residency applicants. In 1999 and 2007, the Journal of Bone and Joint Surgery published inaccuracy results of 18.0% and 20.6%, respectively, in applications for orthopaedic residency programs.[5,8] These rates influenced the implementation of a PubMed Identifier (PMID) number for applicants’ reported works in the peer-reviewed publication section by the Electronic Residency Application Service (ERAS) in 2014. A follow-up study reported only 13 of 1100 (1.18%) papers as inaccurate during the 2016-2017 application cycle, which is much closer to the rate of inaccuracy found in our study.[10] The slight increase in our inaccuracy rate could be because the SF Match does not require a PMID field for each citation. Fellowship applicants may also demonstrate more diligence when reporting their research experience. Overall, the inaccuracy of self-reported research publications by applicants to a single ACGME-accredited sports medicine fellowship was low. This suggests that the number of self-reported “completed” papers could serve as an appropriate and reliable measure of an applicant’s research experience.

We also sought to investigate the relationship between papers reported as published in a journal and the following demographic factors: (1) age, (2) sex, (3) US USMLE Step 1 score, (4) AOA membership, (5) medical school type, (6) NIH top-40 medical school for research, and (7) attainment of an advanced degree. Previous studies that examined publications within the residency applications for other specialties detected relationships between having an inaccurate publication and demographic factors such as age, USMLE score, graduation from an international medical school, and enrollment within a top-10 medical school for research.[9,15] Our univariate analysis detected no significant relationships, which suggests that objective demographic data are not predictive of a fellowship applicant having an inaccurate publication. Although our findings showed that papers reported as completed exhibited a higher inaccuracy percentage among US DO graduates (17.9%) compared with US MD applicants and international MD graduates (5.8% and 3.9%, respectively), univariate analysis showed no statistical significance of these findings, as the proportion of DO applicants studies was very low (8.0%).

To our knowledge, no previous research has followed the outcomes of works listed by fellowship applicants as submitted for publication. One study[4] examining the publication history of neurological surgery residency applicants excluded manuscripts listed as “in preparation” or “submitted for publication,” as the authors found it difficult to evaluate applicants using these reports. Our study found that a substantial number of papers categorized as submitted remained unpublished after a minimum of 2 years (28.3%). A 2017 study examining the duration of time from first submission to publication for 337 publications to peer-reviewed journals showed a median of 30 weeks for online publication and 36 weeks for final publication.[12] One potential reason for a submitted paper remaining unpublished beyond these median times is rejection by the original journal of submission and an inability to gain acceptance to an alternative journal. Other potential reasons include continuous requests for revision by the original journal of submission, prolonged data collection to increase statistical power, and rigorous work hours during residency that delay the submission or revision of a paper. Last, as there was not a mechanism to verify whether the study listed as submitted was actually reviewed by the journal listed, we could not verify whether these studies completed the submission and review process.

Our findings also showed a moderate number of papers that were published in different journals than originally reported (21.8%). The average impact factor of the original journals of submission was significantly higher than the average impact factor of the accepting journals. These findings suggest that unforeseen challenges and circumstances prevent applicants from publishing in their originally intended journal. Given the substantial number of unpublished papers and papers published in journals with lower impact factors than originally reported, it may be in the reviewer’s best interest to place more emphasis on “completed” papers when assessing a fellowship applicant’s research experience.

Our study had several limitations. A retrospective analysis of applications exclusively from 2013 to 2017 limits the conclusions we could draw from our results, particularly when sample sizes decrease upon stratification of the data among US MD graduates, US DO graduates, and international MD graduates. Conducting the study at a single institution also limits the total applicant pool and the generalizability of our results. However, the institution from which the study data were collected is a high-volume and highly ranked program that attracts a large, diverse applicant pool. Further, because the US Orthopaedic Sports Medicine fellowship match uses the SF Match registration platform, applicants will submit the same application and publication listings to all programs in the SF Match to which they applied. Therefore, all other programs in the country will receive the same information as our institution, which makes the analysis from our institution generalizable to others. The minimum 2-year audit period may have been too brief for submitted papers, which would lead to a higher number of unpublished papers. In addition, it is difficult to compare the publication rate of submitted papers listed on fellowship applications with the overall publication rate of submissions for each journal. Last, some submitted papers may have been published in journals that were not accessible through PubMed, MEDLINE, and open access methods, in which case we were not able to verify their publication status.

Conclusion

We noted a low rate of inaccurate self-reporting of completed publications in applications for orthopaedic sports medicine fellowships. The majority of papers listed on applications as submitted for publication were not published in the journal to which they were originally submitted.