Orthopaedic Surgery Residency Application, and Selection Criteria Adaptations, in Times of COVID-19: A Survey Study.

Amid the COVID-19 pandemic, medical education and residency application have faced unprecedented changes. This has forced residency directors to alter their selection criteria in the absence of away rotations and the implementation of nationwide virtual interviews. Purpose: The purpose of this study was to assess how residency directors have adapted their selection criteria in light of this unique application cycle and to look at the effect, and future, of the different changes.
Methods: A 16-question online survey was disseminated to 31 residency programs gathering data about new opportunities offered this cycle, changes to selection criteria, match outcomes, as well as the number of applicants to their program.
Results: Twenty-nine respondents completed the survey (94% response rate). There was a significant rise in the number of applications received by programs this cycle (p < 0.05). Programs have unanimously altered their selection processes. The biggest changes in selection criteria were putting more weight into communication from mentors, emails from the applicants, home applicant status, and virtual information session attendance. Some programs used additional application requirements beyond Electronic Residency Application Service, which were often uncompleted, and cut the number of eligible applications by up to 46%. Among the new opportunities offered this cycle, virtual information sessions and social media platforms seem to be the most commonly offered and are anticipated to grow. Discussion and
Conclusion: Orthopaedic surgery residency continues to become more competitive with a significant rise in the number of applications during the COVID-19 pandemic. Amid this increasingly demanding virtual application cycle, a holistic application review was more challenging. More weight was put that cycle into communication from faculty mentors, emails from the applicants, home applicant status, and virtual information session attendance. Supplementary applications and virtual informative opportunities are likely to last and change the future of the orthopaedic surgery residency application process.

Introduction

Amid the coronavirus disease 2019 (COVID-19) pandemic, medical education and the residency application process have faced unprecedented changes. In response to the pandemic, multiple specialty associations had to develop new policies to ensure student safety while striving for a fair and equitable residency application process. Some of the unprecedented changes seen that cycle were the new virtual platforms for engagement with residency programs, the termination of away rotations, and nationwide virtual interviews. Orthopaedic surgery residency has been greatly affected by these changes because of the competitiveness involved in matching this specialty. This increasing competitiveness is a result of a larger pool of applications for each program, which has continued to grow every year[1]. Although the number of applications has risen across all specialties, the growth was disproportionate, with almost triple the growth in orthopaedic surgery residency applications compared with the other specialties[1].

With the rising volume of applications, many residency programs institute screening tools to decrease the time reviewing applications. One particular criterion that has been frequently cited is setting a minimum US Medical Licensing Examination (USMLE) step 1 score to screen applications[2,3]. When Schrock et al. surveyed programs in 2016, 89 programs (83% of respondents) reported having used a USMLE step 1 score to reduce the volume of applications for further consideration. Although this cutoff has varied among programs, the average step 1 score for seniors that match in orthopaedic surgery residency has continuously increased, from 234 points in 2007 and rising to 248 points in 2020[4,5]. In addition, other metrics have been frequently cited to correlate with higher match success rates, such as USMLE step 2 Clinical Knowledge, American Orthopaedic Association (AOA) membership status, third-year clerkship grades, and research productivity[6]. Akin to USMLE step 1, the averages for some of these other metrics have increased in recent years for the successfully matched orthopaedic surgery applicants[6].

Performance on away rotations (also called visiting rotations, externships, and subinternships) is another important aspect of the application process[7-10]. Programs use this time to learn about an individual on a personal basis, enabling applicants to leave a more significant impression compared with a standard interview[10]. In this extended period, a program can evaluate an individual beyond a few conversations and can better evaluate an applicant's fit within a given residency program[8,9]. To some, an applicant's performance on an away rotation is arguably the biggest determinant of an applicant's strength[11]. In response to the pandemic, the Association of American Medical Colleges (AAMC) has dissuaded away rotation programs in a press release on May 11, 2020[12]. Furthermore, in an AAMC article published on July 17, 2020, 98% of medical schools responded to a previous survey stating that they have decided to curtail away rotations[13]. This was clearly a drastic change when considering that orthopaedic applicants went on an average of 2.4 away rotations before the COVID-19 pandemic[14]. Bernstein et al. report that program directors ranked students' performance on an audition rotation at their institution to be their highest of all selection criteria[15]. How this has changed program selection criteria is still unclear.

Objective

The primary objective of this project is to study how orthopaedic surgery residency programs have adapted their application selection criteria in light of the unique changes to this cycle. The secondary objectives of the study were to evaluate how these changes and virtual opportunities affected the number of applications submitted, number of interviews offered, match outcomes, and the impact it had on the admissions committees.

In light of the lack of travel time and costs with virtual interviews, we predicted substantial growth in the number of applications this cycle. We also hypothesized programs would go further down their rank list assuming an overlap between top-ranked applicants among programs across the nation.

Materials and Methods

Survey Construction

A 16-question survey was built in Qualtrics online platform (Qualtrics) (Appendix 1, http://links.lww.com/JBJSOA/A385, https://qfreeaccountssjc1.az1.qualtrics.com/jfe/form/SV_czRJjGJgO5FoWuW). The survey was reviewed by the study team and presented to the Collaborative Orthopaedic Educational Research Group (COERG) of the AOA. In an effort to increase the response rate, program identification was optional.

Survey Dissemination

The COERG is a national collaboration of orthopaedic residency program directors and interested faculty whose mission is to assist the orthopaedic training community and improve the quality of resident education and faculty development research.

The project was submitted to the COERG governing board for endorsement, and on approval, the proposal was presented at the COERG monthly meeting. The survey was then disseminated to the 31 residency programs that had expressed interest in participation at the monthly COERG meeting or on receiving the monthly newsletter about the ongoing projects (11 south, 10 northeast, 5 west, and 5 midwest). A follow-up email was sent 3 weeks later.

Data Collection and Statistical Analysis

Survey responses were captured by Qualtrics and exported into Microsoft Excel. Descriptive analysis and bar graph representation were used to report categorical responses about changes in applications in times of COVID-19. The survey gathered historical data for the years 2019 and 2020 for intraprogram control. Statistical analyses were conducted in R (R Core Team, 2020). The paired-samples t test was used to test for significant changes in baseline annual rise in the number of applications.

Results

Twenty-nine respondents completed the survey (94% response rate). The majority of respondents were program directors (21), and they have served 8 years on average in their current role (Table I). This year the average number of applications reported per program has grown in comparison with previous years (682 applications in 2020 vs. 620 in 2018), with a statistically significant rise in the annual growth in the number of applications (42 ± 47 vs. 13 ± 30 applications per year, p = 0.007) (Tables II and III). A subgroup of programs (n = 4) added new secondary application requirements (surveys and/or videos) to their application process beyond the Electronic Residency Application Service. For this group of programs with the additional requirements, on average, only 60% of their applications were completed to their entirety (range 54%-70%).

Despite the overall growth in the number of applications per program, all programs still spent an equal (14 programs) or greater (15 programs) time reviewing each application. In addition, despite the growth in the number of applications, only 1 program had an increase in the number of faculty on their application review committee. The majority of the programs (23 programs) still had the same number of faculty, or some even had less faculty on their application review committee (4 programs). In addition, although some programs were able to expand the number of interviews that cycle (10 programs), the majority of the programs (16 programs) had no changes in the number of interviews, despite the growing number of applications. The majority of the programs matched applicants at about the same ranking compared with previous cycles (14 programs) or even better (11 programs). Only 3 programs had to go further on their rank list in comparison with previous cycles (went about 5-10 beyond their usual).

All selection criteria have unanimously experienced some degree of change (Table IV). The biggest changes in review criteria were putting more weight into communication from faculty mentors (59%), emails from the applicants (45%), home applicant status (41%), and virtual information session attendance (55%), which was a new consideration that year. In addition to those virtual information sessions, 93% of the programs have offered a multitude of opportunities for students in light of the cancellation of away rotations (Table V). Virtual information sessions and residency-specific social media accounts were the most common new opportunities offered that cycle (23 and 18 programs, respectively), and more programs anticipate to offer them in the future (25 and 20 programs, respectively). Mentorship pairing models were the least commonly offered that cycle (5 programs). However, all programs that offered this opportunity anticipate continuing it in the future. Structured virtual curricula, as well as individualized meetings with program leadership, were offered by some of the programs (8 and 5 programs, respectively); however, close to half of those programs anticipate to discontinue those opportunities in the future. In addition, none of the programs offered interviews to all their virtual rotators and participants. At the most, programs offered interviews to approximately 75% of their participants, but some offered to as little as 10 to 25% of their virtual rotators.

Discussion

The orthopaedic surgery residency match process has become increasingly competitive over the years. Along with the growing number of applicants each cycle and the limited number of positions, medical students also submit more applications each cycle[1]. This trend has continued for several years, which has expanded the financial burden on the applicants and resulted in a larger task placed on the residency programs' review committee[16-19]. Surveying programs, it was noticeable that the number of applications received that cycle has significantly increased in comparison with past years. Although this rise can be partially attributed to the ongoing growth for several years, the growth during the 2019 to 2020 application cycle was significantly higher than that reported between previous cycles (p < 0.05). In our opinion, this significant rise in the annual growth in the number of applications can be attributed to the virtual nature of interviews that cycle that has slashed a huge financial burden[20]. In addition, we theorize the ease of scheduling and stacking virtual interviews from coast to coast has also contributed to this growth. Yet, despite the increase in the number of applications, most programs were not able to reciprocate that growth in the number of interviews offered nor the number of faculty on the application review committee. With more applications to review, and a lack of a reciprocate growth in the review faculty, this inevitably increases the challenge for a holistic application review. Especially, when none of the programs spent any less time reviewing each application received.

Although it has been believed that many programs would start transitioning away from the use of USMLE step 1 scores in the screening process (in light of the upcoming transition of scoring into a pass/fail scoring)[21], only 2 programs reported to have put less weight into step 1 during that cycle. This mismatch was likely due to increasing evaluation demands with the significant rise in the number of applications that cycle.

Although the rising number of applications was the prevailing trend witnessed in our survey, a subgroup analysis of our data demonstrated a unique subgroup. Programs with supplemental application requirements that cycle experienced no significant rise in the annual applications received; in addition to gathering data for additional applicant evaluation, this additional requirement has helped to limit the significant rise in the number of applications. In addition, a substantial number of applicants did not complete the additional requirements, rendering as high as 460 applications at 1 program to be incomplete and cutting the eligible application number to almost half (54%). The statistical comparison for the overall number of applications has a low power, given the small sample size of this subgroup. Nonetheless, there was a substantially reduced workload for admissions committees, with 176 to 460 fewer applications to review at those programs.

In regards to the match results, our initial hypothesis was that programs would go further on their rank list matching lower ranked applicants than in previous years. This hypothesis was initially based on the maldistribution of residency interview invitations communicated through the AAMC's letter in December of 2020[22]. We hypothesized that with the virtual nature of interviews, top-tier applicants would complete a higher number of interviews than in previous cycles. Consequently, programs would go further on their rank list, given a redundancy in the top-ranked applicants among programs. However, our data demonstrate otherwise. Despite 89% of programs matching similar or better-ranked applicants than usual, only 46% of the programs had anticipated to continue offering virtual interviews when taking the survey in March 2021.

Finally, Undergraduate Medical Education to Graduate Medical Education Review Committee released their recommendations in April 2021, suggesting that interviewing should be virtual for the 2021 to 2022 residency recruitment season[23]. This will work to control applicant expenses and enable students to dedicate more time to their clinical education[23]. Our study, demonstrating that most programs still matched similar or even better-ranked applicants than usual, suggests that theoretically, programs only serve to benefit from welcoming virtual interviews for numerous future cycles.

Current study limitations are the limited number of respondents who had used supplemental application requirements. The study also lacks data regarding when and how those requirements were communicated with the applicants. In addition, for programs that have identified themselves to use applicants' attendance at virtual information session during their application review process, the study does not identify whether programs make distinctions beyond attendance. Finally, there may have been a component of selection bias based on responses coming only from the subset of COERG programs that chose to participate.

Conclusion

The number of applications to orthopaedic surgery during the COVID-19 pandemic has continued to rise and surpass the annual growth in what was already a highly competitive specialty. Selection considerations were reassessed to accommodate the changes in that application cycle. The biggest changes in review criteria were putting more weight into communication from faculty mentors, emails from the applicants, home applicant status, and virtual information session attendance. Most programs still continue to use the USMLE step 1 score in the application review process, despite approaching changes to pass/fail score reporting this upcoming January. Virtual information sessions and residency-specific social media accounts were the most common new opportunities that cycle and are anticipated to grow in the future. Supplemental application questions, although theoretically more work for the applicant and faculty reviewer(s), may have dramatically cut down the number of complete applications to review.

Appendix

Supporting material provided by the authors is posted with the online version of this article as a data supplement at jbjs.org (http://links.lww.com/JBJSOA/A385).

TABLE I

Survey Participants' Demographics

Role	No. of respondents
Program director	21
Associate program director	7
Chairman	1
Time in current role	Time in years
Minimum-maximum	1-19
Mean ± SD	8 ± 5.4
No. of residents	No. per year
Minimum-maximum	3-12
Mean ± SD	5 ± 2

TABLE II

Number of Applicants to Respondent's Program

Year	Minimum	Maximum	Mean
2018	210	900	620
2019	220	960	647
2020	340	1055	682

TABLE III

Annual Growth in the Number of Applications at Each Program*

Annual Growth of Applications	Mean ± SD
2018-2019	13 ± 30
2019-2020	42 ± 47

The number of applications in 2020 increased significantly beyond the growth that was anticipated based on the annual growth that was witnessed between the years 2018 to 2019. There was a statistically significant rise in the annual growth in the number of applications (p = 0.007).

TABLE IV

Changes in Selection Criteria This Cycle in Comparison with Previous Years*†

Criteria	N/A: Did Not Use		Put Less Weight This Cycle		No Change from Previous Years		Put More Weight This Cycle or New Consideration
Additional mentor communication (phone call, email, etc.)	3.5%	1	0.0%	0	34.5%	10	62.1%	18
Applicant attended virtual information sessions	24.1%	7	0.0%	0	20.7%	6	55.2%	16
Applicant had applied for an away before cancellation	13.8%	4	6.9%	2	27.6%	8	51.7%	15
Email communication from applicant	3.5%	1	0.0%	0	44.8%	13	51.7%	15
Home applicant	10.3%	3	0.0%	0	48.3%	14	41.4%	12
Geographic ties	3.5%	1	13.8%	4	44.8%	13	37.9%	11
Applicant reached out through social media	27.6%	8	3.5%	1	31.0%	9	37.9%	11
Applicant completed a virtual curriculum	51.7%	15	0.0%	0	13.8%	4	34.5%	10
Letters of recommendation	0.0%	0	3.5%	1	65.5%	19	31.0%	9
Personal statement specific to your program	27.6%	8	0.0%	0	44.8%	13	27.6%	8
Standardized letters of recommendation	0.0%	0	0.0%	0	75.9%	22	24.1%	7
Supplemental application requirements (additional essays and videos)	58.6%	17	0.0%	0	17.2%	5	24.1%	7
Home subinternship performance	3.5%	1	3.5%	1	75.9%	22	17.2%	5
Other life experience and full-time work	0.0%	0	0.0%	0	82.8%	24	17.2%	5
Essay requirements about applicant's interest in your program specifically	62.1%	18	0.0%	0	24.1%	7	13.8%	4
Former varsity or Olympic athlete status	6.9%	2	3.5%	1	75.9%	22	13.8%	4
Presence/absence of a home residency program	20.7%	6	0.0%	0	69.0%	20	10.3%	3
Surgery and internal medicine clerkship grades	3.5%	1	0.0%	0	86.2%	25	10.3%	3
Other third-year clerkship grades	3.5%	1	0.0%	0	86.2%	25	10.3%	3
Research involvement experience	0.0%	0	0.0%	0	89.7%	26	10.3%	3
Research productivity (posters/podiums/published or unpublished manuscripts volume)	0.0%	0	0.0%	0	89.7%	26	10.3%	3
Personal statement	3.5%	1	0.0%	0	89.7%	26	6.9%	2
USMLE step 2 CK	6.9%	2	0.0%	0	86.2%	25	6.9%	2
Volunteer/leadership/extracurricular experiences	0.0%	0	0.0%	0	93.1%	27	6.9%	2
Alpha Omega Alpha membership	6.9%	2	3.5%	1	82.8%	24	6.9%	2
Other awards/special honors	0.0%	0	0.0%	0	93.1%	27	6.9%	2
Medical school reputation	3.5%	1	0.0%	0	93.1%	27	3.5%	1
Applicant type (MD/DO/IMG)	0.0%	0	0.0%	0	96.6%	28	3.5%	1
USMLE step 1	0.0%	0	6.9%	2	89.7%	26	3.5%	1
Reapplicant status	6.9%	2	6.9%	2	86.2%	25	0.0%	0
Aways performance if completed	31.0%	9	0.0%	0	69.0%	20	0.0%	0

CK = Clinical Knowledge, DO = Doctors of Osteopathic Medicine, IMG = international medical graduate, MD = Doctors of Medicine, and USMLE = US Medical Licensing Examination.

Data presented in descending order based on new considerations, and criteria weighed more heavily this cycle than previous years.

TABLE V

Opportunities that Programs Have Offered This Cycle in Light of Virtual Interviews and Canceling Away Rotations*†

Opportunity	Offered This Cycle	Anticipate to Offer in the Future
Virtual interviews	29	13
Structured virtual curriculum	8	5
Individualized 1 on 1 meetings with PD, APDs, and chairman	5	2
Mentorship pairing model	5	5
Virtual information sessions	23	25
Residency-specific social media account	18	20
Other‡	5	2

APD = associate program director, and PD = program director.

Although all programs offered virtual interviews this cycle, only 13 programs anticipate to continue offering those in the future. Virtual information sessions and social media accounts were very common and are most likely to grow in future cycles.

Other opportunities were direct calls from residents to all applicants confirmed to interview, calls and email communication with applicants who requested communication with leadership, matching applicants who applied for acting internship with residents, offering virtual acting internships, and offering no additional opportunities.