Analyzing the Role of Research in the Radiation Oncology Match.

PURPOSE: Program directors consider scholarly output to be integral in matching applicants with radiation oncology residencies. However, applicants' research productivity can be quantified in several ways, and the results can be misleading for both applicants and program directors. We conducted a bibliometric analysis to quantify the research productivity of applicants who had successfully matched to radiation oncology residencies and to test for associations between research productivity and residency program rankings. METHODS AND MATERIALS: We identified U.S. radiation oncology residency programs from the Accreditation Council for Graduate Medical Education website and sorted the findings into 4 tiers based on the programs' reputation and research output per Doximity's Residency Navigator. First-year (post-graduate year-2) radiation oncology residents starting in 2020 were identified on residency program websites. Residents' research productivity was estimated by identifying peer-reviewed research articles (published before the residency applications began) via PubMed, Scopus, and Google Scholar for each resident. Bibliometric variables were analyzed for potential association with matching to higher-tier residencies.
RESULTS: We identified 187 first-year residents in 83 U.S. residency programs. The mean number of National Resident Matching Program publications (which was self-reported and included presentations and abstracts) was 18.3 per applicant; the mean (± standard deviation) peer-reviewed publications was 2.47 (±2.88) per resident. Multivariate analysis showed that number of first-author publications was associated with matching to a higher-tier program, based on the program's reputation (P = .019) and research output (P = .010); numbers of radiation oncology-specific publications (P = .039) and h-index (P = .024) correlated with matching to a higher-tier residency based on the program's research output.
CONCLUSIONS: The number of first-author publications was significantly associated with matriculating into a higher-tier residency ranked by both reputation and research output. Significant correlations were also found for number of radiation oncology-specific publications and h-index. Applicants may find publishing meaningful radiation oncology articles, especially as first author, to be more valuable than meeting National Resident Matching Program numbers.

Introduction

The radiation oncology match was extremely competitive in the past, with the ratio of applicants to residency spots having almost doubled between 1993 to 2003. Although the number of radiation oncology residency spots has grown and the number of applicants for those spots has declined in recent years, the National Resident Matching Program (NRMP) indicates that the quality of successful matriculants into radiation oncology (defined by routinely reported NRMP metrics) has improved.2, 3, 4, 5 Aspects considered by residency programs when inviting applicants for interviews include the United States Medical Licensing Examination (USMLE) Step 1 score, letters of recommendation, medical school grades, and involvement in research, with 73% of radiation oncology program directors citing research as an important factor. According to a 2020 report from the NRMP titled Charting Outcomes in the Match, senior year U.S. medical school candidates who matched successfully into radiation oncology reported an average number of 18.3 publications on their Electronic Residency Application Service (ERAS) submission, which is much higher than the 6.9 reported for all matched U.S. medical school seniors., This rather large number may be misleading, however, as ERAS data on research productivity are self-reported and include not only peer-reviewed scientific articles but also abstracts, posters, and book chapters, among others. Thus, identifying which types of scholarly activities are valued by radiation oncology residency programs based only on NRMP data is challenging.

Previous studies of other residency programs in neurological surgery, otolaryngology, and plastic surgery have used bibliometric analyses to clarify applicants’ research productivity.8, 9, 10, 11 Although some bibliometric analyses have been used in radiation oncology, those analyses focused on either departmental research productivity or residents’ research productivity during and after their residency. Here, we sought to specifically assess the importance of peer-reviewed research published before the residency application process in matching to radiation oncology residencies in the United States. We used bibliometric analysis to quantify the research output of radiation oncology residency matriculants prior to residency and sought correlations between that research output and matching into higher-tier residency programs.

Methods and Materials

A list of all radiation oncology residency programs in the United States was compiled from the website of the Accreditation Council for Graduate Medical Education (www.acgme.org). First-year (post-graduate year-2) radiation oncology residents for the 2020 to 2021 year were identified by examining each institution's radiation oncology website. Program websites lacking resident information and residency programs with no first-year residents were excluded from the analysis. International medical graduates were excluded from the analysis, as were residents who transferred to radiation oncology residencies after spending more than 1 year in a different specialty, although residents who had transferred from an internship or transitional year were allowed. These exclusions were implemented to ensure that only post-graduate year-2 residents were considered in the analyses.

For each resident included in the final cohort, we searched the Scopus database for all peer-reviewed research articles published on or before September 1, 2018, with the intent of ensuring that all identified publications had also been included in the ERAS database. PubMed and Google Scholar were cross-referenced for further accuracy. Resident identities were confirmed by searches of ResearchGate, LinkedIn, and Doximity, and cross-checked by undergraduate institution, medical school, graduating year, gap year, and middle names. Bibliographic metrics compiled for each resident were the total number of publications, number of radiation oncology–specific publications, number of first-author publications, and h-index. The h-index, defined as the number of articles, h, that have been cited at least h times, is used as both an objective measure of research productivity and an indicator of the meaningfulness of the research produced. Only peer-reviewed original research papers, systematic reviews, and case reports were included. Conference papers, presentations, posters, abstracts, book chapters, errata, and letters to the editor were excluded from the analysis.

Next, all Accreditation Council for Graduate Medical Education–accredited U.S. radiation oncology residency programs were sorted via Doximity's rankings for “reputation” and “research output” and divided into four tiers: tier 1 (1-20), tier 2 (21-40), tier 3 (41-60), and tier 4 (the remaining residency programs). Doximity's “reputation” ranking represents the pooled results of nomination surveys over the previous 3 years that reflect the opinion of Doximity members who were board-certified in radiation oncology and asked which residency programs provide the best clinical training. The research output ranking is established based on the total h-index of alumni graduating in the past 15 years from a specific program, as well as the number of research grants awarded to each department. Bibliometric measures were analyzed for potential association with matching to higher-tier residencies based on reputation and research output, as has been done by other groups.7, 8, 9, 10 Institutional review board approval was not required for this retrospective database review.

A logistic regression was used for univariate analysis of all bibliometric measures. Variables found to be significant in univariate analysis were applied as inputs into a multivariate ordinal regression model. Least square means with a Bonferroni correction was used for significant variables in the multivariate ordinal regression model to pinpoint differences between matriculated program tiers. All statistical analyses were completed in RStudio version 1.4.1106. Statistical significance was set as P < .05.

Results

Of the 91 U.S. radiation oncology residency programs identified, 6 programs were excluded because of missing resident information on their respective websites, and another 2 were excluded for having no first-year residents in the 2020 to 2021 academic year. In the remaining 83 residency programs, 198 first-year residents were identified, 9 of whom were excluded for being international medical graduates and 2 for transferring to radiation oncology residencies after spending more than 1 year in a different specialty, leaving a final cohort of 187 first-year residents in 83 residency programs. Bibliometric variables for those residents, all reflecting papers published before they applied to radiation oncology residency, were as follows. The mean (± standard deviation) number of peer-reviewed articles published was 2.47 ± 2.88 per resident; number of radiation oncology–specific articles was 0.70 ± 1.40; and number of first-author publications was 0.78 ± 1.24. The mean (± standard deviation) h-index for successfully matched applicants was 1.84 ± 2.09. These variables were also sorted by program tier and by Doximity rankings for program reputation and research output (Table 1).

Table 1

Means (with standard deviation) of bibliometric measures between matriculated program tiers based on reputation and research output

	Publications	Radiation oncology publications	First-author publications	H-index
All programs	2.47 (2.88)	0.70 (1.40)	0.78 (1.24)	1.84 (2.09)
Reputation
Tier 1	3.64 (3.24)	1.13 (1.77)	1.29 (1.53)	2.68 (2.32)
Tier 2	2.05 (2.38)	0.44 (0.98)	0.58 (1.03)	1.58 (1.71)
Tier 3	1.58 (2.09)	0.63 (1.36)	0.53 (0.95)	1.13 (1.51)
Tier 4	1.68 (1.89)	0.24 (1.42)	0.30 (1.64)	1.32 (2.11)
Research
Tier 1	3.51 (3.31)	1.10 (1.77)	1.26 (1.54)	2.59 (2.38)
Tier 2	1.60 (1.82)	0.55 (1.04)	0.40 (0.77)	1.33 (1.44)
Tier 3	2.51 (2.80)	0.65 (1.44)	0.89 (1.26)	1.78 (1.92)
Tier 4	1.55 (1.74)	0.20 (1.29)	0.23 (1.50)	1.18 (1.96)

All bibliometric variables were non-normally distributed and right-skewed; Figure 1A shows the distribution of number of publications per resident and Figure 1B the distribution of first-author publications per resident. Univariate analysis revealed that the number of publications, number of radiation oncology–specific publications, number of first-author publications, and h-index all correlated significantly (P < .05) with the matriculated program tier for the program's reputation and research rankings. Multivariate analysis showed that the number of first-author publications was significantly associated with matriculating to a higher-tier program, for both program reputation (P = .019) and program research output (P = .010), whereas the number of radiation oncology publications (P = .039) and h-index (P = .024) correlated with matriculation to a higher-tier residency only for the research ranking (Table 2). Least square means with a Bonferroni correction showed that for program tiers based on reputation, the number of first-author publications for each tier 1 matriculant was statistically different from those for matriculants in tiers 2, 3, and 4 residencies, with adjusted P < .05 (Fig 2). For program tiers based on program research output, students matriculating into tier 1 programs had higher numbers of first-author publications (P = .004 for tier 1 vs tier 2, P < .001 for tier 1 vs tier 4) and higher h-index (P = .015 for tier 1 vs tier 2, P < .001 for tier 1 vs tier 4) than did those matching to other tiers. Furthermore, students who matriculated in a tier 3 residency ranked according to research output had a higher h-index than matriculants to tier 4 programs (P = .045). Also, for program tiers based on program research output, the numbers of radiation oncology–specific publications were significantly different for matriculants to tier 1 compared to those to tier 4 (P = .016).

Fig. 1

Non-normal, right-skewed distribution of (A) number of publications per resident and (B) first-author publications per resident. Number of residents represented by each bar shown.

Table 2

Univariate and multivariate effects on matriculated program tiers based on reputation and research output

Tier		Value*	P value (univariate)	P value (multivariate)
Reputation
	Publications
	1	3 (1-6)	<.001†	.209
	2	2 (0-3)
	3	1 (0-2)
	4	1 (0-2)
	Radiation oncology publications		.002†	.104
	1	0 (0-2)
	2	0 (0-0)
	3	0 (0-1)
	4	0 (0-0)
	First-author publications		<.001†	.019†
	1	1 (0-2)
	2	0 (0-1)
	3	0 (0-1)
	4	0 (0-0)
	H-index		<.001†	.058
	1	2 (1-4)
	2	1 (0-2)
	3	1 (0-2)
	4	1 (0-2)
Research
	Publications		<.001†	.055
	1	2 (1-5)
	2	1 (0-2)
	3	2 (0-4)
	4	1 (0-2)
	Radiation oncology publications		.002†	.039†
	1	0 (0-2)
	2	0 (0-1)
	3	0 (0-0)
	4	0 (0-0)
	First-author publications		<.001†	.010†
	1	1 (0-2)
	2	0 (0-1)
	3	0 (0-1)
	4	0 (0-0)
	H-index		<.001†	.024†
	1	2 (1-4)
	2	1 (0-2)
	3	1 (0-3)
	4	1 (0-1)

Median (interquartile range).

Statistically significant, P < .05.

Fig. 2

Boxplots presenting the number of first-author publications in comparison to matriculated program tier based on reputation. Least square means with a Bonferroni correction confirmed that the number of first-author publications of each tier 1 matriculant was statistically significant compared to those of matriculants in tier 2, 3, and 4 residencies with adjusted P < .05, as shown by (*). Outliers are designated by (x).

Discussion

We identified significant correlations between the number of first-author publications and matriculating into a higher-tier residency based on Doximity rankings of program reputation and program research output. We also found significant associations between the number of radiation oncology publications and h-index and matching at a higher-tier program based on Doximity rankings of program research. Notably, differences in bibliometric variables between individual tiers were not all statistically significant, but the tier 1 program matriculants seemed to produce more meaningful (ie, higher h-index) and more first-author publications, particularly radiation oncology–specific papers, than did those who matched to other program tiers. We also found that the number of peer-reviewed publications identified through searches of Scopus, PubMed, and Google Scholar (2.47 per resident) produced by successful radiation oncology matriculants before their residency application was much lower than the number of publications (which also includes presentations and abstracts) in the NRMP database (18.3 per resident).

To the best of our knowledge, this is the first study in which bibliometric data were used to analyze research productivity among radiation oncology matriculants before they submitted their residency applications for comparison between different program tiers. Other studies used the data reported by the NRMP Charting Outcomes in the Match to draw conclusions about research output among medical students before they matriculated to radiation oncology residency programs., By manually reviewing various database records, we were able to distinguish peer-reviewed publications from all other research activities in the NRMP data, enabling us to collect more detailed information and draw more robust conclusions about the role of research in the radiation oncology match.

Other groups have utilized bibliometric analyses to study the importance of research in matching to other competitive and similarly research-heavy specialties, such as neurological surgery, otolaryngology, and plastic surgery.8, 9, 10, 11 Prior studies have differed slightly in methodology and findings, with 1 reporting only h-index to correlate significantly with matriculating to higher-tier residencies, while another found both h-index and total number of publications to be significant. In general, these studies also found that increased research productivity during medical school correlated with matching at higher-tier residency programs.8, 9, 10, 11 The protocols used for data identification, collection, and statistical analysis in these studies were modified and refined to implement the best approach to assess research productivity in radiation oncology residents. For example, 2 studies of residents in neurological surgery and otolaryngology used univariate and multivariate statistical analysis, but data were collected only from Scopus, and the respective residency programs were ranked based only on research output., Likewise, a similar study of residents in plastic surgery used several databases to compile data and stratified their residency programs into tiers based on both Doximity reputation and research output; however, the investigators reported findings from only univariate statistical analysis. In another study of otolaryngology residencies, multiple resources were used to collect data and verify residents’ identities, but only the top 25 residencies were sorted by Doximity reputation, and no statistical analyses were reported. We considered and applied all of these aspects in our own methods to ensure the rigor, reliability, validity, and utility of our results, a notable strength of this study.

The cause of the discrepancy between the NRMP data (mean 18.3 publications, presentations, and abstracts) for radiation oncology matriculants and our own findings (mean 2.47 peer-reviewed research publications) undoubtedly reflects several factors, as noted in reports of studies in other specialties.8, 9, 10, 11 One study found that 22% of radiation oncology applicants to a particular institution had falsified the number of publications listed on their residency applications in a single year. Although these errors may not have been deliberate given the ambiguous instructions for listing research on the ERAS submission, this finding does underscore the need for clarity in the application process, for the benefit of both students and program directors. Concern that the radiation oncology job market may be becoming increasingly saturated also underscores the importance of enabling potential applicants to better understand the “soft requirements” needed to match into higher-tier programs. Data from the NRMP archives indicate that the mean number of publications per resident in 2016 was 12.7, and that number rose to 15.6 in 2018., It is interesting to observe that the data reflect this trend despite the clear downtrend in the number of U.S. medical school applicants into the field of radiation oncology. While the amount of research published before matching into radiation oncology has been shown to predict both resident productivity and the probability of choosing an academic career,, it will be interesting to understand trends in research productivity in the upcoming years, as more applicants may become deterred from pursuing a career in radiation oncology.

The USMLE Step 1 score, one of the most important factors considered by program directors for applicant interviews, is shifting to a pass/fail format on January 26, 2022,, thereby increasing the impetus for medical students to shift their focus to increasing their extracurricular activities, with research being one of these potential areas. However, an overly intense focus on research productivity may translate into neglecting other skills that are crucial to medical education, such as additional learning opportunities in medicine and developing leadership skills. The field of radiation oncology is currently at a critical transition point in the context of educating and training future physicians. Our results suggest that higher-tier residency programs seem to value meaningful, first-author publications in radiation oncology more than total number of publications. We propose that the NRMP Charting Outcomes in the Match should consider reporting number of publications, presentations, and abstracts in separate categories, beginning with the 2022 edition. This straightforward solution may provide transparency to the residency application process and would encourage medical students to contribute to more meaningful participation in research as opposed to chasing a single number that can be easily inflated and manipulated. Not only will this lead to more well-informed medical students who are interested in the field, it may also aid program directors in ranking their interviewees for the match.

Nevertheless, a few limitations to this study must be noted. First, only peer-reviewed articles published on or before September 1, 2018, were analyzed to ensure that all identified publications had been included in the submissions to the ERAS. Thus, we did not consider research published after this date but before the match results had been finalized, a period during which applicants could have provided updates of their new work. Also, the h-index is a powerful tool for determining research impact, but it does not account for position in the list of authors; however, we addressed this limitation by including first-author publications as a separate bibliometric variable. Second, the Doximity rankings of reputation of radiation oncology residency programs is based on the opinions of survey-eligible physicians who responded to a questionnaire. Moreover, the method by which Doximity ranks radiation oncology residencies based on research output has not been objectively validated. However, these rankings are easily accessible and provide a valuable resource for medical students interested in a particular specialty, and many past studies have used this method to stratify residency programs.8, 9, 10, 11 In addition, residency tier is hardly the only component for an applicant to consider when asked to rank programs; higher percentages of U.S. potential radiation oncology matriculants cited overall goodness-of-fit, interview day experience, and desired geographic location over the reputation of the program. Third, the role of research in the radiation oncology match was analyzed independently of all other factors that program directors considered important when inviting applicants for interviews, such as scores on Step 1 of the USMLE, letters of recommendation, and medical school grades. Fourth, while the focus of our bibliometric analysis was centered on peer-reviewed research, other forms of research that were excluded, such as presentations and abstracts, may play a role in program decisions regarding research productivity. Finally, although Scopus is an extremely useful tool, it too has limitations with regard to incompleteness, as not all peer-reviewed publications are listed and Scopus profiles are generated from public author information, which complicates confirmation of which publications are attributable to a given resident. For example, it may be difficult to collect all applicant publications if the author changed their surname. Despite our meticulous attempts to ensure accurate data collection, quite possibly other peer-reviewed publications were missed or not accounted for in this analysis.

We fully encourage that similar studies be undertaken in different fields to assess the importance of research in matching to other specialty residency programs through bibliometric analysis. It would be interesting to see if the substantial difference between the NRMP average number of publications, presentations, and abstracts and the actual number of peer-reviewed publications holds true across all specialties.

Conclusion

With this bibliometric analysis, we hope to better inform potential radiation oncology applicants about the realities of the research needed to match into their desired residency programs. Notably, our results indicate that higher-tier residencies value first-author publications and articles with higher impact (ie, higher h-indices), in addition to publications focusing specifically on radiation oncology. We hope to encourage change in how the NRMP reports research productivity numbers to more accurately reflect the types of research produced by residency program matriculants, for the benefit of both students and program directors.