Understanding General Surgery Applicant Expectations and Perceptions in the Virtual Interview Process.

OBJECTIVE: The COVID-19 pandemic has played a lasting role on residency recruitment through the virtual interview process. The objective of this study was to 1) examine general surgery applicants' priorities and perceptions following pre-interview virtual open houses and 2) to assess applicant expectations and efficacy of the virtual interview day process. DESIGN/SETTING/PARTICIPANTS: This study utilized two voluntary and anonymous cross-sectional surveys administered via email to evaluate the virtual interview process of a general surgery residency program. The first was administered to registrants following completion of three open houses of various topics. The second was administered following each interview day. The post-open house survey had 78 respondents, two excluded for no open house attendance. The post-interview survey was completed by 44 applicants (62.9% response rate).
RESULTS: Majority of respondents reported that attending virtual open houses made them want to apply to (90.9%) and improved their perception of the program (94.7%). Applicants who felt a sense of obligation to attend open houses (68.4%) were significantly more likely to feel that they contributed to the stress and time commitment of applications (81.8% vs 18.2%, p=0.028). Interview expectations were identified in recurrent themes: 1. Clear organization with breaks, 2. Interactive resident sessions, 3. Meetings with program leadership, 4. Additional information unavailable on other resources. The pre-interview social and interview day improved 90.2% of the applicants' perceptions of the program. The interview significantly improved applicants' ability to assess nearly all aspects of the program, notably resident camaraderie and culture (30.8% vs 97.4%, p=0.01) and strengths and weaknesses (30.8% vs 92.3%, p=0.04).
CONCLUSIONS: While virtual open houses can improve applicants' perceptions and desire to apply to a program, the associated stress and obligation should be considered. Virtual interviews should provide information unavailable using other resources and provide avenues for conveying the resident culture and camaraderie.

INTRODUCTION

The coronavirus disease 2019 (COVID-19) pandemic has brought significant changes to the residency interview process. In March 2020, in response to the pandemic, the Association of American Medical Colleges (AAMC) published guidance urging residency programs to conduct virtual interviews to minimize the spread of the virus. Given the ongoing concerns regarding COVID-19, this recommendation was extended to the 2021-2022 residency interview season. As the virtual interviews have continued for a second cycle, residency programs have adapted through hosting virtual clerkships,2, 3, 4 virtual open houses,5, 6, 7 improving program websites, and expanding social media usage. ,

Studies have examined many individual aspects of the interview process that optimally showcase the program, and provide information to the applicants in a virtual format. Virtual open houses or information sessions prior to the interview have become new tools for recruitment, with a national survey of general surgery program directors finding a significant increase in their perceived importance during the pandemic. Applicant surveys evaluating virtual open houses have demonstrated that up to 100% of applicants have felt satisfied with their experiences during the virtual open houses. ,

Traditional interview days usually encompassed a pre-interview social event with a half or full day of program related information, facility tours, question and answer sessions with the residents, and interviews with residents/faculty members. In the transition to virtual interviews, programs have worked to incorporate these activities through a virtual platform. Previous studies have suggested that while up to 81% of applicants prefer in-person interviews, , there can be high overall satisfaction with the virtual interview process, how it enhances the understanding of the residency program, and provides access to the residents.

The virtual interview concept has been rapidly adopted and adapted by programs and applicants over the past two years. Given the current status of the pandemic, successful implementation of virtual interviews and other benefits (costs, equity, reduced stress of travel), , it is likely that the residency interview process may remain virtual in the coming years. As the virtual interview process progresses, it is important to continue assessing the value and efficacy of this new format. While we are growing in our understanding of the virtual interview process, we have yet to examine applicant expectations and their ability to assess the training programs through this virtual experience. Furthermore, few articles examine the general surgery interview process, which presents likely different applicant perspectives and expectations than those of other fields. The objective of this study was to 1) examine applicants’ priorities and perceptions of a general surgery program following the completion of virtual open house programming and 2) to assess applicant expectations and efficacy of the virtual interview day process.

MATERIALS AND METHODS

This study encompassed two different cross-sectional surveys of virtual open house participants and applicants to a single general surgery residency program's virtual interview process in the 2021-2022 application cycle. This study was deemed exempt by the Northwestern University Institutional Review Board.

Virtual Programming

Three 75-minute virtual open house events were held with topics: (1) clinical training/education, (2) research opportunities, and (3) resident culture/diversity, equity, and inclusion. The open houses were held from September 2021 to October 2021. The sessions were advertised on the general surgery program website and through social media posts (Twitter and Instagram) and required a registration for attendance. The sessions were held on the Zoom platform. Each session had an approximately 45-minute presentation by faculty and residents followed by a moderated question and answer session. The questions were submitted by applicants prior to the open house and questions were also answered from the text chat. Following the completion of all webinars, the recordings were posted on the program website and remained available on-demand.

The virtual interview programming was structured with a pre-interview social event the night before the interview, followed by the formal interview day. The pre-interview socials were 75-minute casual resident-only sessions in which applicants rotated to four breakout rooms. Each break out room was approximately 18 minutes with five to six applicants and two to three residents per room. The virtual interview day lasted 5.5 hours and began with information sessions and welcomes by the department chair and residency program directors. The applicants rotated through four 20-minute interviews, three with faculty and one with a senior resident. Additionally, all the applicants had brief one-on-one meetings with the department chair and the residency program director. There was a 45-minute session in a resident room that consisted of a presentation by current residents followed by question and answer. Three 20-minute, off camera breaks were included in addition to a 40-minute lunch break. Each applicant was provided with a personalized schedule with links to each breakout room throughout the day.

Survey Design

A single post-virtual open house survey was emailed to all registrants following completion of the third virtual open house event. This survey was administered through Qualtrics with no personalization of links or tracking of participation. The survey was clearly noted as anonymous and voluntary for the participants. While respondents who registered but did not attend may have received the link, those who did not participate in any live virtual open houses were excluded from the analysis. The second survey was distributed through email following the virtual interview day to all applicants who attended the virtual interview. This survey was also anonymous and voluntary with data stored on the Qualtrics server. No demographic information was collected in the post-interview survey given the small number of survey respondents to protect applicant's anonymity. The data for both surveys were compiled and analyzed by two authors (JH, RJ) who played no role in the interview invitation and ranking process. The clarity and coherence of the complete survey was assessed using cognitive interviews with a sample of general surgery residents and recruitment team, including program coordinators, program directors, and faculty. The survey then underwent multiple rounds of iterative revisions with extensive pilot-testing and feedback.

A Likert scale was used to assess respondents’ perceptions and impressions. Responses were dichotomized with 4=agree and 5=strongly agree or 4=important and 5=most important versus 3 or below. Free-text answers were included for applicant expectations and additional comments. Chi-square, Fisher's Exact Test, and McNemar's test were used to evaluate differences in gender preferences for topics and pre/post interview ability to assess information, respectively. The descriptive statistics and data analysis were completed using Qualtrics and R software version 4.1.1 (R Foundation, Wien, Austria). A p-value of 0.05 was used to determine significance.

RESULTS

Virtual Open Houses

A total of 78 responses were received. Two respondents were excluded because they did not attend an open house session per their survey response. Respondent characteristics are displayed in Table 1 . The attendees were 65.8% female, with 43.2% white and 83.6% non-Hispanic. Overall, 76.3% were graduating in 2022, the current match year, and 73.3% were between 25-29 years of age. Most respondents (82.0%) found the open houses through the program website or social media (Twitter and Instagram), whereas few found them through word of mouth or advisors. Of the respondents, 31.0% attended the program introduction, clinical training, and education session, 17.7% attended the research opportunities session, and 51.3% attended the diversity, equity, and inclusion and resident culture session (Fig. 1 ).

TABLE 1

Post-Virtual Open House Survey Respondent Characteristics

Gender	Female	50 (65.8%)
	Male	26 (34.2%)
Race	White	35 (43.2%)
	Black	13 (16.0%)
	Asian	26 (32.1%)
	Other	5 (6.2%)
	Declined	2 (2.5%)
Ethnicity	Not Hispanic or Latino	61 (83.6%)
	Hispanic or Latino	9 (12.3%)
	Declined	3 (4.1%)
Age	Below 24	7 (9.3%)
	25-29	55 (73.3%)
	30-34	8 (10.6%)
	35-39	4 (5.3%)
	40+	1 (1.3%)
Graduation Year	2021 or prior	15 (19.7%)
	2022	58 (76.3%)
	After 2022	3 (3.9%)
How was the open house found?	Word of mouth	3 (3.9%)
	Program Website	33 (43.4%)
	Twitter	26 (34.2%)
	Instagram	8 (10.5%)
	Medical School advisors etc	2 (2.6%)
	Other	4 (5.2%)
How many open houses have you attended?	1-5	39 (51.3%)
	6-10	19 (25%)
	11-20	12 (15.8%)
	21+	6 (7.9%)

FIGURE 1

Virtual Open House attendance amongst respondents.

A total of 90.8% (n= 69/76) of respondents reported that participation in the process increased their desire to apply to the program, and 94.7% (n=72/76) had an improved perception following the open houses. There was no significant difference in gender among those who had an improved perception of the program (male vs female, 92.3% vs 96.0%, p=0.494). Additionally, there was no significant difference in in those who attended one of the program open houses compared to more than one (95.9% vs 92.6%, p=0.534). There were 7.9% (n=6) of respondents who were not planning to apply to the general surgery program prior to the open houses. Of those respondents, 100% had an improved perception of the program and felt that participation in the events made them want to apply.

Respondents noted operative experience (73.3%), strengths and weaknesses of program (72.0%), and mentorship (62.7%) as the most important topics for discussion during the virtual open houses (Table 2 ). Furthermore, 100% of respondents rated mentorship and strengths and weaknesses of the program as important. Notable topics rated as less important include the history of the program and chair introduction with 59.5% and 39.2% of respondents rating it neutral to least important, respectively. There were no significant differences in topics valued between the genders.

TABLE 2

Percentage of Applicants Who Perceive Topic to be of Importance (4=important, 5=most important) for Discussion in Pre-Interview Open Houses. N Includes Those who Rated Topic as a 4 or 5

Topics	N (%)
Program Director and/or Associate Program Director introduction	63 (85.1%)
Chair Introduction	45 (60.8%)
History of Program	30 (40.5%)
Clinical Rotations and training sites	72 (97.2%)
Education curriculum	72 (97.2%)
Operative experience and autonomy	71 (95.9%)
Department/Faculty Research	61 (82.4%)
Resident research opportunities	66 (89.2%)
Mentorship	74 (100%)
Strengths and Weaknesses of Program	74 (100%)
Diversity, Equity, and Inclusion Principles	69 (94.5%)
Wellness Initiatives	64 (86.5%)
Community service & outreach opportunities	59 (79.7%)
Life outside of residency	68 (91.9%)
Q&A	64 (87.7%)

In total, 68.4% of respondents reported feeling a sense of obligation to attend the open houses. These applicants were significantly more likely to feel that open houses contributed to the stress and time commitment for the applications (81.8% vs 18.2%, p=0.028). Including other surgery programs’, at the time of the survey, 51.3% of respondents had attended a total of 1-5 open houses, 25.0% attended 6-10, and 23.6% attended over 11 events. There was a trend toward a significantly increased rate of respondents reporting an obligation to attend events (88.9%, p=0.091) in those who attended 11 or greater open houses. There was no significant difference in feeling an obligation to attend (65.3% vs 74.1%, p=0.607) or open houses contributing to stress of applications (49.0% vs 33.3%, p=0.231) in those who attended 1 versus 2 or more of our program's open houses.

Positive qualitative comments noted good organization and well-timed open houses (October). Some suggestions for improvement include more interaction with other participants, breakout rooms, and doing fewer open houses.

Virtual Interview Day: Perceptions and Expectations

Forty-four of 70 (62.9% response rate) applicants responded. No demographic or applicant information was included to assure anonymity. When asked about their perceived importance of program aspects when selecting a program, 100% of applicants rated operative/clinical experience and fellowship match of the graduates as important. In contrast, size of the city/things to do in city and location were rated least important with 27.9% and 18.2% rating it neutral to least important, respectively. There were no significant differences in perceived importance of topics by gender. These topics are reflected as themes in applicant responses describing their ideal program. The majority of responses included excellence in clinical and operative training, a culture of camaraderie and collegiality, and the presence of research opportunities.

Applicants were asked about general expectations for programs in the virtual interview process through an open-ended question. The most notable themes to highlight are displayed in Table 3 . The majority of applicants expected a well-organized agenda with clear communication. Other prominent themes included providing adequate time for resident interaction and to better appreciate the culture and camaraderie of the program.

TABLE 3

Emergent Themes From General Applicant Expectations of Programs During The Virtual Interview Process

1. The virtual interview process should be well organized with breaks throughout the day

“Punctual, organized”

“Clear schedule and organization”

“Interview schedule ahead of time”

“Good working knowledge of Zoom, maintaining the schedule and staying on time”

“Break times are either structured or applicants are allowed to be off camera”

2. Pre-interview socials and interview day should allow for interaction and Q&A with residents

“Get a feel for resident camaraderie”

“Opportunities for chats with small groups of residents”

“Pre-interview social with residents to have a chance to ask questions and assess fit”

3. Allocated time to meet with the chair and program directors

“Time with PD and chair”

“Smaller groups in order for adequate time to ask questions and to have adequate time w/PD/Chair”

“Engagement from both residents and faculty”

4. Gain new information on aspects of the program unable to be found using other resources

“Attempts to showcase what you miss not being able to visit in person”

“Get a feel for culture”

“1. To be able to assess program culture through interactions among residents 2. To get a general idea about the dynamics between surgical autonomy and supervision”

“An overview of the program including things you can't find online”

“Leave with an understanding of program's education focus or mission”

Overall, participation in virtual pre-interview socials and the interview day improved 90.2% of applicant's perceptions of the program. 90.2%, and 92.6% of applicants gained new knowledge about the program that they found useful following the pre-interview social and interview day respectively. Of note, 97.5% of applicants felt that the resident camaraderie was well-reflected in the pre-interview social, and 82.9% of applicants felt that the faculty and department leadership's commitment to the residency program were well reflected during the interview day.

Additional positive qualitative comments noted good organization and an appreciation of talking to residents and faculty from his or her subspecialty of interest. Some areas for improvement were designing a pre-interview social that allows for a more natural flow of conversation and including more information on operative volume, community outreach opportunities, and family dynamics.

Applicant Assessments of General Surgery Programs

The most valued resources for information prior to the interview day was the program website (84.1%) and communication with residents (59.1%). In total, 54.5% of applicants reported the virtual open houses both live and recorded as valuable resources, and 29.5% of applicants watched one or more recorded sessions.

Applicants’ ability to assess the program before and after attending the virtual pre-interview social and interview day were examined (Table 4 ). Prior to the virtual interview, 76.9% reported a good or excellent ability to assess resident research opportunities and the department/faculty research. Only 30.8% of applicants were able to assess the strengths and weaknesses of the program and resident camaraderie and culture. In general, fewer than 50% of applicants of applicants reported being able to assess most aspects of the program prior to the virtual interview process. After the interview day, applicants had significantly improved ability to assess nearly all aspects of the program. Of note, 97.4% were able to assess the resident camaraderie and culture following the interview, significantly increased from prior to the interview.

TABLE 4

Percentage of Applicants Able to Assess Topics (4=good or 5=excellent) Prior to Attending Pre-Interview Social and Interview Day and Impressions Following Attendance

Topics	Before, N (%)	After, N (%)	p-value
Clinical rotations and training sites	23 (59.0%)	38 (97.4%)	<0.01
Education curriculum	28 (71.8%)	36 (92.3%)	<0.01
Operative experience and autonomy	14 (35.9%)	35 (89.7%)	0.03
Department/faculty research	30 (76.9%)	37 (94.9%)	<0.01
Research opportunities	30 (76.9%)	37 (94.9%)	<0.01
Mentorship	14 (35.9%)	38 (97.4%)	0.002
Diversity, Equity, Inclusion Priorities	17 (43.6%)	33 (84.6%)	0.04
Strengths & Weaknesses of Program	12 (30.8%)	36 (92.3%)	0.04
Wellness Initiatives	18 (46.2%)	38 (97.4%)	<0.01
Community service & Outreach opportunities	14 (35.9%)	29 (74.4%)	0.54
Life outside of residency	18 (46.2%)	38 (97.4%)	<0.01
Resident camaraderie and culture	12 (30.8%)	38 (97.4%)	0.01
Program facilities	19 (48.7%)	35 (89.7%)	0.004
Living in the city	22 (56.4%)	37 (94.9%)	<0.01

DISCUSSION

As the COVID-19 pandemic continues to affect residency recruitment and we strive to improve our virtual interview process, it is important to assess the design, implementation, and alignment with applicant expectations of programmatic recruitment efforts. This study finds that applicants can have an improved perception of programs following virtual open houses and a remote interview process. Furthermore, the interview day significantly improved applicants’ ability to assess nearly all aspects of the program, most notably resident camaraderie and culture and strengths and weaknesses of the program.

Virtual open houses are pre-interview information sessions hosted by programs to provide information to prospective applicants in a live or recorded format. Programs across the country in various specialties have adopted the idea of using virtual open houses with overall reported positive responses from applicants. , , , Similarly, our results also indicate that most participants had a positive perception of the program following the virtual open house. More importantly, 100% of those who were not planning to apply to the program prior to the open house were planning to apply following the event.

While applicants found value in the information presented in open houses, we should remain cognizant of the potential for overburdening applicants during the interview cycle. One prior study examining pathology applicants has similarly showed that 30% of applicants felt obligated to attend events and 17% felt overwhelmed. We found that 68.4% of participants felt an obligation to attend open houses and felt that the events contribute to stress and increase the time commitment for the applications. Additionally, there was no significant difference in feeling an improved perception of the program between those who attended one compared to two or more open houses. Thus, hosting only one open house may likely have produced similar results. Also, nearly half of the participants had attended 6 or more events at the time of survey. Programs appear to host varying numbers of open houses with up to six reported for one program. , These findings should help inform programs on the number and length of open houses, favoring fewer sessions that focus on providing concise and valuable information i.e. difficult to distribute using other resources. Programs should also consider avenues to notify applicants that open house attendance is not considered in the evaluation of applications.

Applicants reported that the most important topics to be discussed in virtual open houses are mentorship, strengths and weaknesses of the program, clinical rotations and training sites, and educational curriculum. These topics of importance are similar to those reported in other articles examining open house events. , When evaluating how interviewed applicants assess programs, we found that their most important considerations were operative/clinical experience and fellowship match of the graduates followed by resident camaraderie. These findings show that as applicants progress through the interview cycle, their prioritized topics shift to focus more on evaluating program fit. In contrast to other studies, our study did not identify gender differences in applicant prioritization of factors. , This is likely due to a result of a different survey design.

Throughout the transition to virtual processes, programs have significantly adjusted communication strategies with applicants. , , Our program advertised the open house events on our website and social media including Twitter and Instagram. Applicants most often learned about open houses on the program website (43.3%) and on Twitter (34.2%). Some other studies have found that most applicants found their events through Twitter with minimal to none finding it on the formal website. , This may reflect a change in the usage of resources over the second virtual application cycle, as program websites have become prioritized and improved compared to the first cycle. , Applicants may also differ in their social media usage for application information by specialty, as there are few general surgery articles on this topic. Our post-interview survey also found that the most valuable resource for applicants was the program website, consistent with the post-open house survey. Additionally, our program maintained recordings of the virtual open houses on the website for on-demand viewing. While it was only listed as the most important resource for 12.7% of applicants, the availability of on-demand open houses affords significant flexibility for applicants and may decrease the added stress of attending live open houses. We also found that 59.1% of applicants listed communication with residents to be the most important resource. Our residents reach out to applicants prior to interview day through email and remain available during and after the interview days for questions. This finding highlights the critical importance of resident involvement throughout the interview process.

Another important aspect in creating recruitment content and shaping the process is understanding the applicant expectations of the programs. As applicants gain a better grasp of the virtual application process, expectations of programs will continue to evolve. While one prior study has evaluated applicant applications in their perceived importance of each application component, there are no existing studies examining applicant expectations for virtual programming. We found that applicants expect the process to be well organized, have allocated time with chairs and program directors, interaction time with residents, and to gain new information unavailable using other resources. These findings are in line with topics that applicants find most important from interview day, as mentioned above.

Overall, we found that participation in virtual pre-interview social and interview day improved 90.2% of applicants’ perceptions of the program. Most applicants felt that the resident camaraderie was well reflected throughout the process. Notably, the applicants’ ability to assess the program significantly improved in nearly all areas of interest. This demonstrates that a virtual interview may offer a significant amount of information despite the limitations of a virtual interview. While the resources available prior to the interview seem to showcase the research opportunities and educational curriculum well, it is difficult to fully reflect the resident camaraderie and culture without the live interview day interaction. Understanding the resident culture is critical for applicants to assess their fit with the program. , , A previous study assessing applicants’ perception of fit in to residency programs found that regardless of race or gender, the three most important factors for assessing applicant fit were how much the program seems to care about its residents, how satisfied the residents are with their program, and how well residents get along with each other. These findings are consistent with our findings that applicants assess programs by resident camaraderie and culture and expect adequate time with residents during the virtual interview events.

There are several limitations to this study. As those who attended the open houses were not recorded, we were unable to obtain a response rate for the first survey. While the surveys were clearly denoted as anonymous, applicants may feel an obligation to complete the survey and provide positive responses in an effort to gain favor with the program. Additionally, these surveys were voluntary and contain an inherent response bias self-selecting for those who may have had a favorable opinion of the program. This study setting is a single urban academic center and may not represent the responses of general surgery applicants at other programs. The priorities of applicants may be different for those applying to rural programs or community-based programs. While the survey was administered immediately following the interview, there may be recall bias in responses regarding assessment of topics prior to the interview. Due to the small sample size, we were unable to fully assess demographics and recognize that this represents a specific subset of the general surgery applicant population. These limitations may limit the generalizability of these results and should be interpreted within the context of each program.

CONCLUSION

As the COVID-19 pandemic led to the transition to virtual interviews and continues to present uncertainty regarding residency interviews, we looked to evaluate our virtual interview processes. Programs are challenged to both help applicants assess program fit and evaluate applicant fit on a virtual platform. This study found that virtual open houses can improve applicants’ perceptions and desire to apply to a program. It is important to continue assessing the value and efficacy of newly developed experiences, with particular consideration of the stress and obligation associated with events. Furthermore, virtual interviews should provide new information i.e. not available using other resources and avenues for conveying the resident culture and camaraderie. These insights will remain important as we constantly strive to improve and adapt our resident recruitment to the changing environment of the application process.

Declarations of Competing Interest

None.

Funding

At the time of this work, Dr. Ho was supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health under Award Number T32HL094293-13 in the form of partial stipend support. The content of this publication is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.