Finding the Next Resident Physicians in the COVID-19 Global Pandemic: An Applicant Survey on the 2020 Virtual Urology Residency Match.

OBJECTIVE: To assess interviewing applicant perceptions of a virtual urology residency interview in the setting of changes mandated by COVID-19 and to determine applicant preference for virtual or in person interviews. Applicant perceptions of multiple interview components were queried to identify program specific and interview modality specific strengths or weaknesses in the 2020 to 2021 Urology Match.
METHODS: A 12 question multiple choice and free text survey was emailed to 66 virtually interviewed applicants for open residency positions at a metropolitan training program after conclusion of interviews. Items of interest included interview type preference, overall interview impression, and recommendations for improvement.
RESULTS: A total of 50 of 66 (76%) applicants completed the survey corresponding to approximately 11% of the 2020 national urology applicant pool. A total of 49 of 50 (96%) respondents assessed faculty interaction and the virtual platform positively. A total of 38 of 50 (76%) was satisfied with their resident interaction and 32 of 50 (64%) applicants stated they were able to satisfactorily evaluate the site and program. Ultimately, 39 of 50 (78%) respondents would have preferred an in person interview to our virtual interview. Respondents cited challenges in assessing program culture and program physical site virtually.
CONCLUSION: The majority of survey respondents indicated a preference for in person interviews. A smaller proportion of applicants preferred virtual interviews citing their convenience and lower cost. Efforts to improve the virtual interview experience may focus on improving applicant-resident interaction and remote site assessment.

The Covid-19 pandemic created an immediate national need to prioritize safety and disease containment. All global elements of educational and social life were emergently adjusted. One uniquely impacted aspect of the national health care system was the residency interview process for medical students. Instead of traditional in person interviews requiring travel and lodging most, if not all, of the 256 US urology residency programs conducted virtual interviews. We wished to gauge the applicant experience of virtual interviews through a customized survey in hopes of improving applicant interviews for future training cycles.

Prior research on virtual interviews suggests applicant preferences for in person or remote interaction are difficult to define. , A recent study by Female Pelvic Medicine and Reconstructive Surgery (FPMRS) program directors found that in person interviews are preferable to applicants but virtual interviews are acceptable especially in select cases. Similar preferences for in person interviews are reported across surgical specialties including cardiothoracic surgery and pediatric surgery. , Despite the limited research consensus thus far it was imperative to limit widespread travel and attendant risks posed by the traditional match process and virtual interview processes were adopted overnight. New studies exploring applicant perceptions of virtual interviews would expand the literature and help decide whether virtual interviews should be the “new normal” for postgraduate urology, and other medical specialty, program interviews. Herein, we conducted an anonymous survey study of interviewing applicants at our program with a focus of describing their perceptions of virtual interviews with a goal of improving this process for the future.

MATERIALS AND METHODS

Our Virtual Interview

We received 312 applications for the 2020 to 2021 urology match cycle and 66 applicants completed a remote interview with our program. The initial interview season began with several virtual open houses for the applicants to familiarize themselves with our program. Interviews were offered on the basis of internal objective and subjective criteria including personal statements, letters of recommendation, research endeavors and several other common components. Interviewing applicants were offered 5 interview days and divided into cohorts with 12 to 13 applicants per day. For our virtual interview process, we used the Zoom platform (zoom.us; San Jose, CA, USA).

Interview days were approximately 5 hours in duration and faculty members attended from different locations. The interview process began with a 30 minute introduction to the entire group by the program director (JP). This introduction to the program detailed curriculum structure, faculty, residents, and sites followed by a Q&A session. Afterwards, interviewees were separated into 2 groups. Group 1 proceeded to interview with 1 to 2 faculty members at a time in “breakout rooms” for a total of 8 mini-interviews that lasted 15 minutes each. Each mini-interview focused on various themes personal statement, education, leadership, research, extracurricular activities, letters of recommendation and similar topics. Interview rooms were given a “fair warning” at 5 and 2 minutes prior to interview conclusion. Meanwhile, Group 2 applicants joined a resident room which began with a 45 minute virtual site tour followed by 75 minutes of informal interaction with junior and senior residents. After faculty interviews concluded there was a brief intermission and interview groups were crossed over (ie, Group 1 entered the resident room and Group 2 began mini-interviews with faculty). At the end of the interview day, applicants were thanked for their participation and the Zoom conference was concluded.

Survey Study

We conducted a voluntary and anonymous post-interview electronic survey using Survey Monkey (www.surveymonkey.com; San Mateo, USA) to evaluate the medical student perspective and perception of our virtual interview process. This survey was designed during a consensus meeting of matched residents and faculty. It was based on prior internal non-validated questionnaires used to assess previous interview quality. The survey (appendix 1) consisted of 12 questions meant to capture demographic data (age, gender, medical school region and prior application), as well as details regarding their perception of the online interview. Interview specific items included perception of platform (ie, Zoom), meeting faculty, meeting residents, learning about sites, and overall interview experience. These questions were scored on a Likert scale (very poor, poor, satisfactory, good, and very good). We also queried whether the applicant would have preferred an in person interview or a virtual interview. Finally, applicants were given the opportunity to share free text responses on questions of particular interest including preference for in person or virtual interviews, overall interview quality, and areas for future improvement.

The invitation for this online survey was distributed electronically to all 66 interviewees by our Program Coordinator after conclusion of our virtual interview period. Anonymized data was extracted and analyzed using SPSS v.21.0 software (IBM Corp., Armonk, NY, USA).

RESULTS

Of 66 total interviewees, 50 (76%) completed the survey. Extrapolating a 450 applicant class in accordance with the previous application cycle, this represented roughly 11% of the 2020 to 2021 urology applicant class. Applicants took an average of 2 minutes to complete the survey. Survey responder demographics are outlined in Table 1 . Based on these demographics, interview responders are representative of our typical applicant. Median age was 27, 30% were female, and 80% were attending medical schools in the Northeast region of the US.

Table 1

Survey respondent demographics

Total no. of Respondents (%)	50 (75.8)
Median age (IQR) – year	27 (26, 28)
Gender – no.(%)
Male	34 (68)
Female	15 (30)
Unknown	1 (2)
Region – no. (%)
Northeast	40 (80)
Midwest	7 (14)
Southwest	3 (6)
Prior applicant – no. (%)	1 (2)

Applicants viewed the virtual interview process favorably (Fig. 1 ). Notably 96% of respondents rated the online platform and faculty interaction as “good” or “very good”. A total of 50/62 (76%) respondents listed resident interaction as “good” or “very good”. Interestingly despite the virtual nature of interviews and lack of site visit, 64% of applicants rated their ability to get to know the program and training sites as “good” or “very good”. In accordance with the favorable impressions of individual interview components, 94% of responders stated their overall experience was “good” or “very good”. Most respondents (86%) felt interview duration was appropriate (10% felt it was “too long”; 4% felt they needed “more time”). All 62 applicants (100%) felt able to express themselves as “individuals” during the virtual interview process.

Figure 1

Respondent perception of virtual interview components and overall interview.

Despite favorable individual component ratings and a favorable aggregate impression for the virtual interview, the majority of responders (n = 39, 78%) would have preferred an in person interview. This preference was explored further using a semantic approach to thematic analysis of applicants’ responses. This was done by 2 resident authors and 1 faculty author during a consensus meeting using a deductive framework. This analysis is provided in Table 2 alongside selected representative uninterpreted responses. The most identified factor favoring in person interviews among survey responders preferring in person interviews was improved resident interaction (n = 22, 56%) followed by getting to know the site and program better (n = 16, 41%) and better evaluating program culture (n = 12, 31%). One representative responder stated “[It was] hard to pick up on the intangibles (seeing how residents interact with each other, more ability for one-on-one conversations)”. Another applicant stated a desire to “get to know [the] site and residents better in their “natural habitat””. These comments relayed the importance of intangible aspects of the interview process which may be difficult to replicate virtually. Fewer respondents prefer virtual interviews (n = 11, 22%). These applicants overwhelmingly noted cost benefits (n = 11, 100%) virtual interview efficiency (n = 9, 82%) and simplicity of the virtual interview process (n = 5, 45%). Some applicants favoring in person interviews nonetheless identified advantages for virtual interviewing including cost and travel.

Table 2

Thematic analysis of applicant responses and key quotes

Interview Preference	In-person	Virtual
No. (%)	39 (78)	11 (22)
Summary of comments	• Residents (22)• Site (16)• Culture (12)• Faculty (9)• Know me as an applicant (7)• Location (6)• Meet other applicants (3)	• Lower cost (11)• Time saving (9)• No travel (5)• Simple (5)
Key comments	“Getting to know the program better. Allowing the program get to know me better. Seeing facilities first-hand rather than via video (i.e. resident workroom, average OR setup, outpatient clinic area, etc). Could this potentially be mitigated by a combination of virtual interview and second-look?”“While well done, the virtual medium made it difficult to get a good feel for the residents and faculty of the program. I worry that it might have been difficult to assess me as a candidate as well.”“Cost of travel. The virtual tour and resident question and answer sessions were helpful.”“There are definitely pros to interviewing virtually that I can't deny, i.e. convenience, cost, reducing implicit biases associated with arbitrary things like height. However, nothing replaces in-person interactions, and while more costly, interviewing costs are small compared to ending up at a program where you could potentially be unhappy for 5-6 years. Being able to interview with more programs starts to yield diminishing returns at a certain point, particularly if the quality of the interactions are lower than with in-person interviews. Obviously these responses are going to vary based on applicants' values.”	“Decreased cost and time of travel. Medical students are already in so much debt. We are one of the only groups who has to pay thousands of dollars just to apply for jobs. the more we can reduce this burden, the better.”“Urology being a competitive specialty I think most candidates just want to match anyway. To be honest, I think the whole thing about "getting a feel for the program by being in person" is overrated.”“I thought I got a good sense of the program remotely. It was also much easier to attend due to traveling and cost!”“An in-person interview would have given me the opportunity to see the physical site and program. This would also have allowed the faculty/residents to get to know me better as an applicant. With that being said, the Zoom interview did a fine job of introducing us to the department.”

Applicants were also queried on areas for improvement in our interview process. Some identified a lack of inter-applicant interaction compared to previous years which customarily included a pre or post-interview social for applicants and residents stating, “It would be nice to let applicants be amongst themselves for a bit… …to foster camaraderie.” Others identified problems with resident interaction which could be a source of improvement, suggesting, “[Use] breakout rooms whenever interacting with residents [to make interactions more personal].” Several applicants stated interview quality was excellent and did not provide clear areas for improvement.

DISCUSSION

In the setting of substantial changes to postgraduate medical training candidate selection mandated by COVID-19 we conducted a survey of all interviewing applicants at a metropolitan urology residency program in the Northeast. Extrapolating a 450 applicant class in accordance with the previous application cycle, this represented roughly 11% of the 2020 to 2021 urology applicant class. While applicants have overall positive perceptions of virtual interviews, they prefer in person interviews by a significant margin. This finding agrees with prior work, like a study in 2012 comparing randomized virtual to in person interviews, which also suggested an adjunct role for virtual interviewing.

This preference for in person interviews persists despite favorable impressions of all measured virtual interview components. Based on thematic analyses and free responses, it appears intangible effects like improved resident interactions, specifically resident applicant interactions, and a perceived ability to better assess program culture in the “natural habitat” may also contribute to this preference. Another possible component is that one-to-one applicant-resident and inter-applicant interactions were not achieved during this virtual interview. One previous survey of 156 urology applicants from the in person 2019 and 2020 interview cycle by Kenigsburg et al demonstrated that among responders 34% preferred virtual interviews, 41% preferred in person interviews and 25% preferred regional/centralized interviews. Further work reiterates this point that applicants and program directors view virtual interviews as useful adjuncts for generally preferable in person interviews. , , Additionally, some applicants report faculty interviews can be replicated virtually, but that this is more difficult for resident interactions and city visits. In our study, 10% of respondents rated the quality of their resident interactions as “poor”. This possibly corresponds with minimal one on one interactions between applicants and residents or dissatisfaction with the 75 minute informal group discussions between residents and applicants.

Advantages for virtual interviews seemingly centered on well-defined limitations of in person interviews. Cost was the predominant criticism of in person interviewing with one representative applicant noting, “Medical students are already in so much debt. We are one of the only groups who has to pay thousands of dollars just to apply for jobs. The more we can reduce this burden, the better.” There are clear costs associated with in person interview processes and previous reports estimate these at $7000 per urology applicant for a net spend of $3.1 million. For applicants located outside metropolitan hubs, particularly outside the NYC program cluster, these expenses are likely even higher. This is especially important as 20% of applicants report limiting interview attendance due to financial constraints. Conversely other applicants noted the substantial costs associated with interviews were nonetheless small compared to committing to at least 5 years at a program where they may be disappointed and unproductive. Other responders noted possible advantages in reducing implicit biases stating, “There are definitely pros to interviewing virtually that I can't deny, i.e., convenience, cost, [and] reducing implicit biases associated with arbitrary things like height.”

In the context of limited in person interaction during this cycle some urology programs adopted virtual educational activities open to applicants who might otherwise be conducting away rotations. One such example is the role of virtual grand rounds. Xu et al report virtual grand rounds at one center were viewed positively by applicants and that program faculty reported they were useful for determining whether to offer an interview. This topic was further explored by Jian et al in an online survey of 70 urology applicants connected through the UroResidency platform. They report applicants were generally engaged by virtual open houses and felt direct interaction with residents was important to assessing program culture. Resident involvement in the virtual interview process is critical, particularly without allowing site visits for in person assessment of a residency program's culture. Furthermore, in that same survey, 64% of responders felt resident interaction was the most important interview day component. A further 81% stated this could not be replicated virtually.

From a faculty and program director perspective, the virtual interview process was seamless and well organized. In accordance with a survey of Female Pelvic medicine and Reconstruction program directors, our faculty were generally satisfied with the virtual process and found this style of interview effective for assessing applicants. Some of our faculty also found it helpful to “visit the homes” of applicants and see purposeful or inadvertent background objects. For example, particular talking points to learn more about the candidates were prompted by musical instruments, artwork, books or hobbies noted in the background that were in some cases even brought to the screen for demonstration.

Due to the ongoing COVID 19 pandemic, the 2021 application cycle will also proceed virtually. When constructing this project, we recognized interviewing applicants are an incredible wealth of information and are one of the few parties able to directly compare interview processes between residency programs. As such they are the de-facto experts on urology residency interviews and we sought their expertise to identify areas for improvement. Applicants were overwhelmingly positive regarding the operation of the chosen online platform, Zoom. This may be attributable to widespread adoption of this platform in the wake of COVID-19 mediated changes to all facets of medical education. Notably costs under this platform are borne by the program. Future work could focus on whether other platforms offer improved interpersonal or resident interaction as these areas were identified for improvement.

Responders were overwhelmingly positive in their assessment of virtual interview suitability for meeting faculty but rated it poorer for meeting residents and assessing the site. We are actively considering several improvements to our virtual interviews. To improve resident interaction with applicants, we would plan on arranging a resident only virtual applicant happy hour prior to the interview. We would potentially send out meal vouchers for online restaurant delivery meal service platforms such as GrubHub or Seamless. This would in some ways replicate the “Interview Dinner” from in person interviews. More importantly, this would allow residents and applicants to interact more informally to get a better sense of each other outside of the work environment. Furthermore, to improve the applicants’ ability to “visit” the sites and city, we could make more dedicated videos of our various training sites and neighborhoods and post these on our website for applicants to view at their leisure. Perhaps GoPro (gopro.com; San Mateo, USA) video capturing equipment could be utilized to allow medical students to virtually visit areas of the hospital including operating rooms, clinics and patient wards. Finally, we will consider medical student and applicant only breakout rooms so students can freely converse with each other to foster comradery.

Limitations of our work include that the anonymous nature of the survey prevented further efforts to reach non-responders. Conversely it is also possible that despite assurances of anonymity, some responders may not have believed the survey was completely anonymous and reported erroneously positive perceptions. Furthermore, as in any survey, we suffered participation bias. By choosing to only survey interviewing applicants at our program our sample size was decreased and participants may have felt more pressure to respond positively. This effect is likely also strengthened by the high stakes of match interviews. More broadly New York Medical College is in a major metropolitan center with many transportation options and applicants from the northeast, who were the majority of interviewing applicants, may not have viewed travel to our center as highly burdensome. It is also probable that our interview population underrepresented applicants outside the northeast. Thus, our study may underestimate benefits of virtual interviews for less metropolitan centers or applicants. We also made the survey anonymous and short to increase response rate. Thus, we were unable to dive deeper into the responses of the applicants. Furthermore, we only surveyed urology applicants and thus, the results may not be applicable to other specialties and/or interview processes. This may be particularly true for larger programs such as family medicine or internal medicine. (Fig.2 )

Figure 2

Key takeaway points.

Despite these limitations, virtual interviews are likely far more economical and more efficient by eliminating travel burdens. Amidst these cost benefit analyses it is hard to quantify on the ground impressions and in person interactions which are nonetheless highly valued by applicants and programs alike. What remains to be seen is whether the innovation embraced during this crisis can improve future Urology Matches, and potentially matches across specialties, by creating a more equitable system for choosing the next generation of urologists and physicians at large.

CONCLUSION

Despite favorable impressions of all measured virtual interview components, the majority of interviewing applicants still prefer in person interviews despite increased burdens of travel and expense. This preference appears to be explained by difficult to measure variables like “program culture” and “face-to-face interaction.” Further work could focus on improving intangible facets of virtual interviews or alleviating burdens imposed by in person interviews. More robust study of applicant preferences regarding virtual interviews could be accomplished with multi-centric or match-wide applicant surveys. Nonetheless virtual interviews are viewed favorably by applicants and will likely be a significant component of the 2021-22 urology interview cycle.