What Are the Lessons Learned From the Virtual Interview Process for Oral and Maxillofacial Surgery Residency Programs Affected by COVID-19?

PURPOSE: With the outbreak of COVID-19, residency programs for the 2020 to 2021 OMS interview cycle were forced to quickly adapt to a new method of assessing candidates-virtual interviewing-for the first time. The purpose of this study is to describe applicants' perspectives regarding the advantages and disadvantages of the virtual interview process through an online descriptive survey.
METHODS: This cross-sectional study utilizes a descriptive survey sent to the 2020 to 2021 interview cycle applicants of the University of Illinois at Chicago Oral and Maxillofacial Surgery (UIC OMS) residency program. The survey consisted of questions employing both scaled and open-ended designs. The questions were fabricated to extrapolate applicants' perceived advantages and disadvantages of virtual interviewing compared to their expectations of in-person interviewing by focusing on interactions, number of interviews received/attended, and overall applicant satisfaction. Descriptive statistics were computed for each variable for data analysis.
RESULTS: In the 2020 to 2021 UIC OMS residency cycle, there were 288 applications. A total of 102 surveys were collected (response rate = 35.4%). Respondents attended more interviews during the virtual format due to accessibility (64.7%), and cost savings (63.7%). 42.2% of applicants felt they did not present themselves as well during the virtual interview as they would have in person and 46.1% felt they did not have a good understanding of the culture of the program after the interview. 49.0% of all participants do not feel that virtual interviews should continue in the future.
CONCLUSIONS: Virtual formats would allow access to a greater number of interviews; however, they lack the ability to promote effective interaction between applicants, residents, and faculty members. The results of the study show that although applicants agreed that interviews were more accessible this year, they felt that the "personal" interaction was lacking and they could not present themselves effectively with the virtual format. Ultimately, almost half (49%) of the respondents do not wish for virtual interviews to continue in the future.

With the outbreak of COVID-19 starting in early 2020, residency programs across the country decided to conduct virtual interviews in order to comply with CDC guidelines for social distancing. Although no official statement was put forth by the American Association of Oral and Maxillofacial Surgeons (AAOMS), the Coalition for Physician Accountability stated that “all programs commit to online interviews and virtual visits for all applicants.” Such guidelines were adhered to by most interviewing oral-maxillofacial surgery (OMS) residencies for the 2020-21 application cycle (also described as the application cycle for the 2021-2022 residency positions).

In a short period of time, OMS programs needed to create and conduct the residency interview process in an unfamiliar format. The alteration of the traditional in-person interview process created some confusion for both the interviewees and interviewers alike. In addition to the concerns regarding the organization and accessibility of interviews, the loss of in-person interaction, facilities tours, and social events were distinct differences from interviews in years past. In order to maintain the ability of an applicant to assess a program adequately, and vice-versa, the virtual interviews needed to abide by some basic principles. Programs had to develop an organized structure for the interview day, be mindful of personal issues that may arise (eg, technical difficulties), ensure individualized applicant interview time, and replace/eliminate the interview dinner and social events usually held in the past.

The focus of residency programs was to conduct a virtual interview process that would closely resemble an in-person interview. The interview still had to accomplish 2 main goals: 1) an assessment of each applicant by the program, and 2) presentation of various aspects of the program to the applicants. The ability of applicants and programs to evaluate each other and determine “the perfect match” is difficult enough in years when in-person interviews were in place. The lack of in-person interaction undoubtedly alters the ability of the “perfect match” to occur. Applicants may have difficulty assessing the culture of a program through a virtual forum, and programs may become more reliant on objective metrics (eg, dental school class rank, individual course grades, CBSE score, letters of recommendation, etc) in their evaluation of applicants in the virtual interview process.

Although several medical residency programs and fellowship programs have evaluated their virtual application and interview process, no thorough review has been conducted yet in the field of OMS. There are many similarities between the OMS residency application process that reflect other medical specialties, as well as many unique aspects that may be investigated. In addition, an assessment of the virtual interview process may have many positive aspects that may/should be maintained in future application cycles (eg, a hybrid part-live, part-virtual format).

For these reasons, it is critical to evaluate the interviewees perspective of this first virtual OMS interview cycle, which may be instrumental in making adjustments in forthcoming interview cycles. The purpose of this study is to describe applicants’ perspectives regarding the advantages and disadvantages of the virtual interview process through an online descriptive survey. The surveys were distributed to all oral and maxillofacial surgery applicants to the University of Illinois at Chicago Oral and Maxillofacial Surgery (UIC OMS) program in the 2020 to 2021 interview cycle. The aims of this cross-sectional study were to design and implement a reliable survey, distribute the survey online to all eligible participants, and use descriptive statistics to summarize, report, and describe potential relationships of variables included in the survey.

Materials and Methods

Study Design

To address the research purpose, we designed and implemented a cross-sectional study utilizing an online descriptive survey as data collection sent to the 2020 to 2021 interview cycle applicants of the UIC OMS residency program. The survey consisted of a total of 16 questions employing Likert scales (Strongly Agree to Strongly Disagree) numeric scales (0 to 20+) and an open-ended question. The questions were fabricated to extrapolate applicants’ perceived advantages and disadvantages of virtual interviewing compared to their expectations of in-person interviewing by focusing on interaction/communication, the number of interviews attended/received, and overall applicant satisfaction assessed post-Match.

Study Sample

The study population was composed of 2020 to 2021 interview cycle applicants to the UIC OMS residency program. To be included in the study, participants must have completed their Postdoctoral Application Support Service (PASS) application to UIC and have an email on file at the America Dental Education Association (ADEA). There were no exclusion criteria, therefore all 288 applicants to the UIC OMS residency program were included in the study sample. Applicants were sent the survey via the email address present in their ADEA PASS application. All fifteen survey questions were required to have an answer for submission of the survey, with an optional comments section at the end of the questionnaire. All completed surveys were included in the analysis. This project received Institutional Review Board exemption from the University of Illinois at Chicago (UIC IRB Protocol #2021-0399).

Study Variables

Study design and variables attempted to reflect those proposed by Majumder, et. al., 2021, who utilized primary assessment categories of feasibility, appropriateness, and acceptability of virtual interviews. The survey consisted of a total of 16 questions including the following: thirteen questions utilizing a Likert scale with 5 outcome options ranking from Strongly Agree to Strongly Disagree, 2 questions utilizing a numeric scale ranging from 0 to 20+, to result in quantitative response variables. The last question was an optional Comments Section in free-text format. The individual survey questions are listed in Table 1 .

Table 1

Survey Questions

1. I attended more interviews due to the accessibility of virtual interviews. 2. I attended more interviews due to the cost-savings of not traveling for virtual interviews. 3. I accepted virtual interview offers this year that I may not have accepted in years when it would have required me to travel. 4. How many virtual interview invitations did you receive from residency programs?* 5. How many virtual interview invitations did you accept and attend?* 6. I felt I presented myself as well during the virtual interview as I would have in person. 7. It was easy for me to interact with the program director during the interview. 8. It was easy for me to interact with the interviewing faculty. 9. It was easy for me to interact with the current residents. 10. I found the pre-interview program overview to be helpful in understanding the program. 11. I found that the post-interview social gave me a good perspective of the “vibe” of the program. 12. My experience of virtual interviews did not vary from one residency program to the other. 13. I felt confident after the interview that I had a good understanding of the culture of the program. 14. I felt my CBSE score and class rank were a more significant factor with the virtual interviews. 15. I think that virtual interviewing should continue in the future (post-COVID). 16. What comments or suggestions do you have regarding the virtual interview format, or any suggestions regarding changes for the future?†

Answer choices 1 to 20+.

Open-ended, free-text.

Data Collection, Management, Analysis

Data for this study was collected through an online survey. An email with the survey link was distributed to each eligible applicant using Google Forms. Three reminder emails were sent to each applicant in order to increase the response rate. The survey remained open from April 18th (initial email sent) to May 10, allowing a total of 3 weeks for entry of responses. All completed surveys were converted into Google Sheets for data analysis using descriptive statistics, where appropriate.

Results

In the 2020 to 2021 UIC OMS residency cycle, there were 288 applications. A total of 102 surveys were collected (response rate = 35.4%) Demographic information about respondents versus non-respondents was not collected in this study. Responses to survey questions are shown in Figure 1 .

Figure 1

Survey responses.

Respondents agreed that they accepted interview invitations, and attended more interviews during the virtual format due to: accessibility (64.7% Agree or Strongly Agree), followed by cost savings (63.7% Agree or Strongly Agree), and finally, fewer interview limitations due to travel (44.1% Agree or Strongly Agree). Overall, of the 102 total respondents, 88 respondents accepted every interview they were offered (86.3%) and the acceptance rate of an interview offer from this population was 96.8% (819 total interview offers, 793 accepted).

Respondents felt that their CBSE score and class rank were more heavily weighted with the virtual format (69.7% Agree or Strongly Agree) and with a majority response of 64.7% Disagree or Strongly Disagree, participants felt that their overall interview experience did vary significantly between programs.

Respondents were split in their opinions concerning interactions during the interview. 42.2% (Strongly Disagree and Disagree) of applicants felt they did not present themselves well during the interview and 38% (Strongly Disagree and Disagree) felt they were not able to easily interact with the program director. Similarly, 46.1% (Strongly Disagree and Disagree) felt they did not have a good understanding of the culture of the program after the interview and 43.1% were neutral in their opinion on understanding the “vibe” of the program as a whole.

Overall, about half of all participants did not think that the virtual interview process should continue in the future (49.0% Disagree or Strongly Disagree), compared to only 24.5% who believed that the virtual format should continue (24.5% Agree or Strongly Agree).

Discussion

The purpose of this study is to describe applicants’ perspectives regarding the advantages and disadvantages of the virtual interview process through an online descriptive survey. Using 1 OMS program as an example, the results demonstrated an acceptance rate of interview offers by applicants (96.8%; 793 of 819 total interview offers) and although most applicants cited the ease of virtual accessibility as well as the cost-savings as major factors for attending more virtual interviews (64.7 and 63.7%, respectively), there is a clear divide in respondents in regards to continuing virtual interviewing in the future with 49.0% of applicants not wanting virtual interviews to continue post-COVID. This does correlate, however with the result of 42.2% of participants who felt they did not have the opportunity to present themselves effectively during the virtual interview compared to how they felt they would have done in an in-person interview setting along with 46.1% (Strongly Disagree and Disagree) who felt they did not have a good understanding of the culture of the program after the interview, and 43.1% who were neutral in their opinion on understanding the vibe of the program as a whole. There is a consensus that applicants are experiencing a difficulty communicating with faculty and residents with virtual interviewing. In fact, many free-text comments from applicants described the “…inability to grasp a program's culture” in the virtual environment, and how this virtual format “…made a commitment to a program for 4-6 years ‘difficult’ and/or ‘uncomfortable’.”

The free-text comments responses also suggested that virtual interviewing be used as a “pre-interview screening process” to better assess applicants before offering a traditional in-person interview. This would allow both the OMS program and the applicant to better assess each other before making a full commitment to traveling on the part of the applicant, and to making time for interviewing on the part of the programs. However, compared to fully virtual interviews, this suggestion does require additional time commitment and increased expenses by both parties, including travel for the applicant and dinner/social events for the program.

A comparison of the National Matching Services (NMS) statistics regarding oral and maxillofacial surgery programs from 2019, 2020, and 2021, shows that there were 416, 438, and 493 applicants, respectively, each year. The 2020 to 2021 virtual cycle showed the largest recorded increase in applicants to OMS residency programs since the availability of statistics began in 2012. This increase in OMS applicants was also associated with a 12.4% increase in overall dental specialty residency applicants; also the largest increase on file thus far. However, the number of available OMS residency positions remained relatively constant, with a minor increase from 235 to 236 resident positions.

This increased number of applicants, combined with the virtual interview format, does not appear to have affected the rankings of programs by matched residents. In 2021, 118 applicants matched at their number 1 ranked program, followed by 34 at their number 2 ranked program, and 19 at their number 3 ranked program. This is comparable to 2020, with 117, 49, and 21 matching at their first, second, and third ranked programs.

In addition, programs do not seem to have suffered based upon the National Matching Services (NMS) data. Comparing the ratio of “least preferred matched rank to number of positions filled,” oral and maxillofacial surgery programs averaged 2.9 in 2020, with a minor increase to 3.0 in 2021. This indicates that oral and maxillofacial surgery programs matched with applicants ranked in relatively the same position compared to the prior year (ie, not further down on their rank order list (ROL)). Furthermore, there was only 1 unfilled position in 2021, compared to 5 unfilled positions in the previous application cycle.

The NMS also provides more in-depth statistics for the 2017 cycle. In 2017, the average number of interview offers to applicants was 7.9. In our survey, respondents reported an average of 8.0 interview offers. Compared to previous data of residents who matched to an OMS program, the average number of interviews attended was 9. This data would suggest that applicants were not offered more, or less, interviews due to the virtual application process.

The data available above would indicate that neither the OMS applicant nor the OMS program suffered from the virtual interview process by not matching at a high level on their individual or programmatic ROL. However, given a larger application pool with the same number of available positions, there was, of course, a larger number of unmatched applicants compared to years past.

It seems that although the average data has not changed with regards to the NMS Match, the pool of applicants who were offered virtual interviews may have become more unevenly distributed. For instance, our survey data shows that applicants accepted interview offers at what appears to be exceedingly high rates (96.8%). In our survey population alone, 19 applicants reported attending 15 or more virtual interviews; this is a number which would seem unfathomable in previous years that required even minimal travel time (and time off from dental school obligations) and/or travel costs/expenses. This group may represent the small cohort of applicants who filled a disproportionate number of interview positions, previously expected with the announcement of virtual interviews.

The results of this study do show trends in OMS applicant sentiments towards virtual interviewing not previously evaluated in current literature. However, we do acknowledge the limitations to this study. The response rate of 35.4% is well below the 80% value suggested to prevent non-response bias, however, it is still higher than recent resident surveys in the oral and maxillofacial surgery field which have ranged from 5.9 to 26.1%. As this study was conducted using an anonymous online survey that lacked collection of demographic information, it is impossible to assess variations between respondents and non-respondents and confirm statistically significant equal baseline groups. Similarly, there can be no analysis of the responses gathered against study sample differences. For example, responses between applicants who Matched into a program versus applicants who did not Match this cycle could not be evaluated within the limitations of this study.

In addition, the survey was distributed in April 2021, in an effort to be mindful of the emotional post-match process for many (∼55% of applicants do not match). However, an increase in the length of time between the applicants’ last interview, and receiving the survey, may increase the risk of recall bias among the respondents.

Finally, the limited answer choices of the individual survey questions do limit the analysis that can be performed in this study. With a “Neutral” option answer choice, results were not fully dichotomized leading to difficulty in assessing responses of applicants quantitatively. Similarly, the free-text Comments Section question did have responses stating some of the questions “did not apply to them”—this forces the respondent to answer “Neutral” which again, skews the results. Finally, the forced cap of “20+” for interviews numbers offered/accepted removes higher end outliers from the data.

In conclusion, there have been several lessons learned regarding the first virtual OMS interview cycle. Virtual format would allow access to a greater number of interviews however, would lack the ability to promote effective interaction between applicants, residents, and faculty members which is a vital component to assess for determining a “perfect match” of an applicant to an OMS program. The results of the study do in fact show that although applicants agreed that interviews were more accessible this year, they felt that the “personal” interaction was lacking and that they could not present themselves effectively with the virtual format (42.2% of participants felt they did not have the opportunity to present themselves effectively during the virtual interview). Ultimately, almost half (49%) of the respondents do not wish for virtual interviews to continue in the future. In order to utilize the advantages of the virtual format (eg, decreased costs involved), and minimize the disadvantages of the virtual format (eg, lack of personal/social interaction) perhaps future OMS residency program interviews may adopt a hybrid format by incorporating an initial virtual screening component, followed by an in-person live interview. This would then allow the virtual interview to become another component of the assessment of the applicant by the program (ie, in addition to CBSE, class rank, etc), as well as the assessment of the program by the applicant (ie initial meeting with the program director and resident(s)). This would require an additional time commitment for the applicant and the program without an increased cost to either party. Although since this present study evaluated the UIC OMS program, we plan to pilot this hybrid format for the 2021 to 2022 residency program application cycle to use the advantages and minimize the disadvantages of each format based upon the results of this study.

Additional research is necessary to assess definitively the feasibility of virtual interviewing in the future. These studies should evaluate the differences in responses between Match and non-Match applicants as well as the differences in relation to general participant demographics.