Recommendations on the use of virtual interviews in the orthopaedic trauma fellowship match: A survey of applicant and fellowship director perspectives.

OBJECTIVE: The aim of this study was to assess applicant and fellowship director (FD) perspectives on virtual interviewing based on the 2019 to 2020 orthopaedic trauma fellowship interview experience and to develop recommendations for future application cycles.
METHODS: Web-based surveys were distributed to all matched applicants and orthopaedic trauma FDs after the 2019 and 2020 orthopaedic trauma fellowship match. Thirty-one applicants and 23 FDs completed the survey-response rates of 34% and 38%, respectively.
RESULTS: Virtual interviews were completed by 68% of applicants and 17% of FDs. Twenty-nine percent of applicants felt they were able to familiarize themselves with the culture of programs, and 38% of applicants were satisfied with their ability to present themselves. Most (62%) were comfortable ranking programs based on the virtual interview, but 38% reported the format influenced them to rank a program lower. Among all applicants, 77% preferred the in-person interview. Most FDs (75%) reported virtual interviews limited their ability to familiarize themselves with an applicant, and only 50% were comfortable ranking an applicant afterward. Still, 78% of FDs believe there is a role for virtual interviews in the fellowship match. Choosing a virtual interview may negatively affect applicants as 97% of applicants worry the choice conveys less interest to programs, while 43% of FDs would interpret it as less interest.
CONCLUSIONS: Virtual interviews have multiple shortcomings but are technically feasible and provide reasonable information to applicants and FDs to complete the match process. Our recommendations, based on the perspectives of applicants and FDs, can guide their implementation.

Introduction

Orthopaedic trauma fellowships were forced to adapt to the COVID-19 pandemic during the latter part of the 2019 and 2020 interview season when many hospitals enacted travel restrictions. In March 2020, several programs responded by transitioning interviews to a remote format, using telephone or videoconferencing applications. Subsequently, the Coalition for Physician Accountability recommended all residency programs commit to online and virtual interviews for the 2020 and 2021 interview season.[ This position was reinforced by the American Orthopaedic Association and Council of Orthopaedic Residency Directors .[ The Orthopaedic Trauma Association (OTA) has also followed suit, disallowing traditional, on-site interviews.

Virtual interviews utilizing videoconferencing applications have been described in various other specialties during the residency interview process.[ However, there has been relatively limited study on the use of the virtual interview format in the orthopaedic surgery match process. Healy and Bedair[ described videoconferencing as a valuable tool in the interview process for the Newton-Wellesley Hospital Adult Reconstruction fellowship, though they found applicants had mixed opinions when surveyed. In particular, 30% of respondents to their survey felt videoconferencing was not a good format, and 34% ranked the Newton-Wellesley program lower due to the interview format.[ Despite their tempered enthusiasm, they were still able to present several useful general recommendations on the use of videoconferencing.[

As virtual interviews will play a greater role in both residency and fellowship interviews in 2020 and 2021, understanding the perspectives of key stakeholders is critical to the efficient and successful use of the virtual interview format. The aim of this study was to assess the perspectives of applicants and fellowship directors (FDs) on virtual interviewing based on the 2019 and 2020 orthopaedic trauma fellowship interview experience and to use their systematic feedback in the development of recommendations that might improve the virtual interview process in future interview cycles.

Material and methods

Study design

Institutional review board approval was obtained prior to the initiation of this study. To understand the point of view of orthopaedic trauma fellowship applicants and orthopaedic trauma fellowship directors, we distributed web-based surveys to each group after the fellowship match. The study population included all applicants who matched to an orthopaedic trauma fellowship during the 2019 and 2020 cycle (n = 91) and all orthopaedic trauma FDs (n = 61). We designed separate versions of the survey, one for applicants and another for FDs, on a secure, web-based survey platform (Qualtrics, Provo, Utah). Email distribution of the survey was facilitated by the OTA Fellowship Committee. All responses were kept anonymous, and participation was voluntary. To improve the response rate among matched applicants, two $50 Amazon gift cards were used as incentives and were given to randomly selected applicants who completed the survey. We assumed FDs would not require further incentivization as improving the fellowship match process for subsequent years was considered to be of more direct benefit to them.

Survey design

The applicant survey consisted of a brief series of questions on demographics followed by general information about their experience in the fellowship match, including—number of applications, number of interviews, details about travel and expenses, and their match result. The remainder of the survey was organized into a series of domains focusing on more detailed information about: OTA fellowship information sessions, virtual interview experience (only if they completed any), perspectives on virtual interviews (all respondents), and a free response section to express any remaining thoughts or concerns not captured by the structured portion of the survey (Appendix 1).

The structure of the FD survey was similar; however, demographics were limited to information about their fellowship program including regional location and program size. General questions on the 2019 and 2020 application cycle included: the number of applicants to their program, number of interviews completed, and estimates of interview day expenses. The remainder of the FD survey included 3 domains: experience with virtual interviews (only if completed), perspectives on the use of virtual interviews (all FDs), and a free response section as in the applicant survey (Appendix 2).

Both surveys were piloted at the primary institution. The applicant survey was piloted among 4th year orthopaedic surgery residents, since they had recently completed the fellowship match process. Attending surgeons trialed the FD version of the survey. In both cases, focus groups were conducted, and feedback was used to modify the content of our questions, the structure of the surveys, and our response sets prior to distribution.

Statistical analysis

Descriptive statistics were used to summarize data derived from survey responses, using the Qualtrics survey platform. Key themes and trends identified within the free text response sections of the survey were summarized qualitatively.

Results

The applicant perspective

The response rate among applicants was 34% (31 of 91 matched applicants). The majority of respondents were males (26; 84%) from residency programs in the South (12; 39%) and Northeast (10; 32%) with a mean age of 33.2 years (SD 4.2). On average, respondents applied to 23.3 programs (SD 7.5) and attended 13.1 interviews (SD 3.0), of which 11.5 (SD 2.6) required significant travel. Forty-two percent matched at their top ranked program (Table 1).

Table 1

Information on applicant respondents (N = 31).

Age (years)	33.2 (4.2)
Gender
Male	84% (26)
Female	13% (4)
Program location
Northeast	32% (10)
Midwest	19% (6)
South	39% (12)
West	10% (3)
Number of programs applied to	23.3 (7.5)
Number of interviews completed	13.1 (3.0)
Number of interviews requiring travel	11.5 (2.6)
Position matched on rank list
1	42% (13)
2	10% (3)
3	13% (4)
4	3% (1)
5+	32% (10)
Completed at least 1 virtual interview	68% (21)

Ninety percent of the applicants who responded attended information sessions at the 2019 OTA Annual Meeting in Denver. Nearly all (97%) believed that a virtual alternative for the OTA information sessions would be beneficial to future applicants, yet the majority endorsed a preference to attend in-person information sessions (58%).

Virtual interviews were completed by 68% of respondents (Table 1). Of those completing virtual interviews, 24% completed more than 1, and 10% matched at a program where they performed a virtual interview. During virtual interviews, more respondents were able to familiarize themselves with the structure of a program (rotation schedule, call schedule, composition of the team, etc) than the culture of the program, 52% compared with 29%, respectively. Further, only 38% of respondents were able to present themselves to their satisfaction through the virtual format. These issues may have contributed to the fact that only 62% were comfortable ranking programs where they completed a virtual interview, while the format itself negatively impacted the way applicants ranked a program in 38% of cases (Fig. 1). Despite the issues, technical and scheduling difficulties were infrequently reported (29% and 19%, respectively). In general, respondents felt that the process could have been improved if programs supplied information about the fellowship in advance of the interview, afforded more consistent access to the current fellow(s), and were more organized overall.

Figure 1

Perspectives of applicants (N = 21) and fellowship directors (N = 4) on their experiences with virtual interviewing in the 2019 and 2020 orthopaedic trauma fellowship match. ∗Indicates the percentage of respondents responding that the overall ability to familiarize yourself with an applicant is worse than an in-person interview.

Among all respondents (applicants that did and did not complete virtual interviews), 97% of respondents expressed concerns that choosing a virtual interview over an in-person interview would convey less interest to the FD of that program (Fig. 2). This may be why 77% preferred an in-person interview relative to videoconferencing and telephone, and 58% would interview at more programs if there was access to a virtual option. When asked to rate the important components of virtual interviews (on a 1 to 5 Likert scale), respondents gave the opportunity to speak to the current fellow(s) the highest rating 4.7 (SD 0.6), followed by opportunity to speak to all faculty (4.4, SD 0.8).

Figure 2

Pie charts demonstrating the responses of applicants (N = 31) and fellowship directors (FDs; N = 23) regarding the perception of an applicant choosing a virtual interview over an in-person option. The majority of applicants worry that FDs interpret this choice as less interest in their program. A significant portion of FDs admit that it does, in fact, represent less interest on behalf of the applicant.

Commonly, applicants perceived cost savings and reduced time away from their residency programs to be the primary benefits of the virtual interview format. In contrast, commonly identified disadvantages included difficulty getting a sense of the personalities of the faculty, how the applicant might fit in, and the facilities and location. A final important point highlighted among the free responses was the concern that the discussion and networking that occurs between applicants at the in-person interviews would be lost through a switch to entirely virtual interviews.

The fellowship director perspective

The response rate among FDs was 38% (23 of 61 FDs). The majority of FDs who completed our survey were from fellowship programs in the Midwest (43%) with 1 (61%) or 2 (26%) fellows. On average, they received 35.8 applications (SD 14.5) and completed 19.5 interviews (SD 8.2) (Table 2).

Table 2

Fellowship director respondents (N = 23).

Program location
Northeast	9% (2)
Midwest	43% (10)
South	22% (5)
West	26% (6)
Number of fellows
1	61% (14)
2	26% (6)
3	13% (3)
Average number of applications to program	35.8 (14.5)
Average number of interviews	19.5 (8.2)
Offer a virtual interview (pre-COVID19)	13% (3)
Performed virtual interviews	17% (4)
Believe there is a role for virtual interviews	78% (18)

Only 13% of FDs reported that their fellowship program offered a virtual interview option (pre-COVID19), while 17% completed virtual interviews due to COVID-19-related travel restrictions during the 2019 and 2020 interview season. All FDs reported using a videoconference application, and 25% of FDs offered telephone interviews as well. Technical issues were encountered infrequently (25% of the time); however, 75% of FDs reported scheduling difficulties. Half of FDs reported being able to adequately familiarize themselves with an applicant's personality and qualifications through the virtual interview format. Ultimately, most still felt it was inferior compared with an in-person interview (75%), and only 50% of FDs felt comfortable ranking applicants based on the virtual interview experience (Fig. 1). Among all FD respondents, only 2 (9%) reported matching an applicant who completed a virtual interview.

Despite many of the shortcomings referenced above, 78% of FDs believed there is a role for virtual interviews in the orthopaedic trauma fellowship match, even in the absence of COVID-19. The vast majority of FDs still endorsed a preference for in-person interviews (91%) over both videoconference and telephone interviews. A significant proportion of FDs (43%) would interpret an applicant choosing a virtual interview over an in-person interview as an expression of less interest in their program (Fig. 2).

FDs largely agreed with applicants regarding cost and time being the primary benefits of the virtual interview format. They reported similar concerns about the impersonality of virutal interviews, with difficulty getting a sense of the personality and mannerisms of applicants. Additional drawbacks highlighted by FDs were an inability to assess applicant dexterity through an assigned task and that applicants would not have an opportunity to interact with residents at their programs. In general, attitudes expressed by FDs were heavily in favor of in-person interviews.

Cost considerations

A cost savings of $844 (SD $1028) across the entire interview process would lead applicants to prefer the virtual interview format. FDs responded inconsistently when considering a cost savings at which they would prefer a virtual interview format, with 57% of FDs responding with $0 and 2 others responding with values in excess of $50,000. This likely highlights the strong feelings FDs have about virtual interviews relative to in-person interviews.

Discussion

The disrupted 2019 and 2020 fellowship interview season created an opportunity to better understand the experiences and perspectives of uniquely qualified key stakeholders, who, due to the COVID-19 pandemic, had first-hand experience with both in-person and virtual interviews within the same interview season. The findings of the present study have immediate implications for the coming orthopaedic residency and fellowship interview seasons. Based on this information, we have formulated a list of recommendations for the use of virtual interviews (Table 3).

Table 3

Key recommendations for the use of virtual interviews.

Continue exclusive use of virtual interviews during the pandemic

Develop a virtual OTA information session or webinar for interested applicants

Utilize a videoconferencing application rather than telephone for interviews

Include ample access to the current fellows

Provide information on the program to applicants in advance of the virtual interview (virtual OTA information session/webinar, PowerPoint presentation, information packet or pamphlet, video introduction)

Postpandemic, consider using a virtual interview for all applicants by the fellowship director to allow applicants to evaluate programs

Once in-person interviews resume, consider an initial virtual interview to screen programs/applicants

Select fewer applicants for on-site, in-person interviews

Overall, there appears to be a reasonable level of acceptance, with a majority of FDs who responded to our survey envisioning a role for virtual interviews in the fellowship match. It is important to consider several shortcomings associated with virtual interviews identified through our survey. Namely, applicants expressed substantial concerns about their ability to familiarize themselves with programs and to adequately present themselves to interviewers. It is also notable that a considerable proportion endorsed ranking a program lower due to the virtual interview format. From the FD perspective, many felt the virtual format was inferior to in-person interviews, with only half expressing comfort ranking applicants based on the virtual interview. Both applicants and FDs demonstrated strong preferences for in-person interviews over remote interviews whether by videoconference or telephone. We also identified what we believe to be a critical obstacle to widespread implementation of virtual interviews: an applicant choosing a virtual interview over an in-person interview results in a negative assessment of the applicant by the program. Applicants feared this, and it appeared to be true. Lastly, cost-savings seem to be a somewhat minor benefit to virtual interviews. In particular, 1 FD relayed, “cost and convenience is a flawed and cheap surrogate for a quality fellow.” This response seems to convey the underlying sentiment of both parties—that saving money and avoiding time away are much lower in priority than the ultimate goal of a successful match. Although the trauma fellowship may only be 1 year, it is a critical year in training, during which many lasting relationships are forged. As such, a “good fit” is essential. This information can guide the academic orthopaedic trauma community in the adoption of virtual fellowship interviews in a way that fosters an effective experience for both applicants and programs.

Many studies assessing the use of virtual interviews have revealed advantages in terms of cost and time.[ Most of these studies focused on other specialties. Healy and Bedair[ surveyed a population most comparable to our own, and our findings stand in contrast to some of the successes they highlighted. Eighty-five percent of arthroplasty fellowship candidates responding to their survey felt videoconference interviews allowed them to present themselves to their satisfaction, and 81% of candidates were comfortable ranking their program based on the videoconference interview. They also endorsed high levels of satisfaction among faculty participating in the videoconference interviews. Ultimately, they were forced to temper some of their enthusiasm due to similar concerns identified in our study, namely that 34% of candidates reported the videoconference interview negatively impacted their ranking of the Newton-Wellesley arthroplasty fellowship. The fact that they provided applicants a fellowship manual and a video tour prior to the interviews and were able to refine their process over the 2 years preceding their study likely explains some of the discordance with respect to our findings. In comparison, many of the orthopaedic trauma fellowships were unexpectedly forced to transition to virtual interviews, leaving them unable to prepare such an organized approach.

More recently, Vining et al[ presented their single-institution experience with videoconference interviews in the surgical oncology match during the COVID-19 pandemic. They utilized a well-structured virtual interview day with personalized interview schedules, premade videoconference links, a preparatory session with an overview of the fellowship from the FD, and additional provisions. Eighty-one percent of applicants in their study reported being able to convey themselves well or very-well, and all endorsed a good or very good understanding of their program based on the virtual interview day experience. Further, all faculty interviewers in their study felt applicants were able to convey themselves well or very well. However, 69% of applicants and 50% of interviewers still preferred in-person interviews. These experiences suggest a carefully organized approach to the virtual interview format may compensate for difficulties applicants face in presenting themselves to interviewers and getting a better sense of programs. However, the widespread preference for in-person interviews by applicants and interviewers across specialties is strongly ingrained.

Our survey data also revealed applicants are appropriately worried that if they choose a virtual interview over an in-person interview, FDs will likely interpret that as a lack of serious interest in their program. For this reason, providing a virtual interview option alongside an in-person interview is likely not a viable solution. Rather, the virtual interview may be better suited as a supplement to in-person interviews and may function best as a screening tool to generate a more targeted pool of applicants for the in-person interview day.[ This can still achieve the benefits of saving cost to applicants and programs secondary to travel and time away by reducing the number of interviews applicants complete in-person. As long as potential COVID-19 travel restrictions continue to vary by region, it will remain necessary to foster a more equal experience for applicants by recommending all programs perform virtual interviews instead of in-person interviews.

Though somewhat tangential to the orthopaedic trauma fellowship application process, OTA information sessions represent a primary avenue through which applicants learn about programs. Our data suggest that offering a virtual option for OTA information sessions would be highly beneficial to applicants. By offering a virtual option and providing early information to a larger proportion of applicants, FDs may be able to generate a more focused applicant pool for their program. This may be even more beneficial as many applicants admitted they would likely interview at more programs if virtual interviews were widely adopted.

There are several limitations to our study. First, the response rates for both survey populations (34% of applicants and 38% of FDs) may limit generalizability and the strength of our conclusions. Though our response rates are comparable to several other survey studies about applicant populations,[ this represents a significant limitation. In particular, the limited number of FD respondents who completed virtual interviews (n = 4) represents only 7% of all orthopaedic trauma FDs, and our conclusions about this subpopulation must be considered with this in mind. Further, we were only able to distribute our survey to applicants who matched; we were unable to assess the perspectives of unmatched applicants. As a result, we fail to capture their potentially valuable insights. Almost half of the applicants who responded matched at their top program. This may represent a source of bias as these individuals may be more satisfied with whichever interview format led to their successful match result. Second, since our survey was self-administered and anonymous, questions and response sets are subject to interpretation of survey respondents, and our results are dependent on the accuracy of self-reported data. For instance, inconsistency in FD responses to survey questions on cost limits our ability to interpret this data in a meaningful way. Additionally, many of our response sets were kept dichotomous rather than categorical to induce a clear choice, though this may sacrifice granularity. Third, the study population includes only orthopaedic trauma fellowship match applicants and fellowship directors and limits the ability to extrapolate our findings to the residency match or to other fellowship matches. Fourth, our study is cross-sectional by design and limited to a single point in time. Thus, we are unable to make judgments about how the perceptions of virtual interviews may evolve over time. As a society, the pandemic has forced many to become more familiar with videoconferencing, and as a result, aversion to the format may change. In particular, the completely virtual 2020 and 2021 interview season may generate different impressions of virtual interviews as there will be no option for in-person interviews. Sixth, we did not ask respondents to distinguish between videoconference and telephone interviews, which may impact our results as there are different advantages and disadvantages to each of these formats. Despite these issues, there are many strengths of our study. We utilized a rigorous approach to survey design that involved refinement of questions and response sets after pilot distribution. The inclusion of multiple free response items allowed us to capture concepts that may have been missed by the structured domains of our survey. By surveying both applicants and FDs, we were able to describe the perspective of the primary stakeholders on both sides of the virtual interview process. Additionally, our study captures the opinions of applicants and interviewers at multiple institutions, while many other studies have surveyed primarily within individual fellowship programs.[ Our investigation adds to the limited body of evidence on the use of virtual interviews for resident and fellow selection in orthopaedic surgery, providing additional context to a pressing issue facing the orthopaedic community. The widespread use of virtual interviews in 2020 and 2021 may provide more insight into their role in the selection process, but many questions remain. Future research on virtual interviews might focus on how their importance compares to other components of a candidate's application such as recommendations and personal connections. Additionally, there should be continued focus on optimizing the use of virtual interviews in the fellowship match process.

Conclusion

There were multiple shortcomings associated with the use of virtual interviews identified by this cohort. Most importantly, both applicants and programs remained concerned about their ability to get a sense of each other's culture/personality, which substantially limited their comfort level in ranking each other based on the virtual interview. Some of the other issues may stem from inadequate preparation in the face of suddenly implemented travel restrictions secondary to the COVID-19 pandemic. A well-organized and carefully planned virtual interview process can potentially compensate for some of the issues applicants and programs face in familiarizing themselves with each other. It is more difficult to overcome the perceptions generated by applicants opting for a virtual interview over an in-person interview; many of the entrenched preferences for in-person interviews are valid. There are many intangible benefits to in-person interviews when selecting an applicant who will not only be entering a critical year of training but will also be joining the legacy of the fellowship program. This is where the impersonality of the virtual interview is most limiting. As a result, we believe there is a role for virtual interviews, but they remain limited in comparison with in-person interviews. Under normal circumstances, virtual interviews may be most useful as screening tools or adjuncts to the in-person interview. On the other hand, while the COVID-19 pandemic is ongoing, it is likely more equitable to have all programs offer virtual interviews instead of in-person interviews. Ultimately, the most effective role for the virtual interview in the fellowship match process remains to be determined.