Impact of the COVID-19 pandemic on the 2020 pediatric anesthesiology fellowship application cycle: A survey of program directors.

BACKGROUND: The COVID-19 pandemic created a situation with an urgent need to produce a virtual system for the 2019-2020 pediatric anesthesiology fellowship cycle. With fellowship interviews beginning in April 2020, there was minimal time for programs to adapt. Each program rapidly developed its own platform, expectations, materials, and process for interviews-all while under the stress of managing the unfolding patient care, financial, and leadership crises of the unfolding pandemic. AIMS: The aim of this survey-based study was to help identify changes compared with previous traditional cycles, obstacles encountered, and program director attitudes toward this, and possible future, virtual application cycles. We separately report the results of an applicant-based survey about the 2020 virtual interview cycle.
METHODS: A 50-question survey was developed utilizing Qualtrics. An anonymous survey link was sent to all pediatric anesthesiology program directors with an invitation for one program leader (either the program director or assistant/associate program director but not both) from each program to complete the voluntary survey. RESULT: Thirty respondents completed the survey, for a response rate of 50% based on the 60 ACGME-approved pediatric anesthesiology fellowships. Compared to previous traditional cycles, almost half of respondents (14/29, 48.3%) reported no increase in the number of applications received, and a majority of respondents (16/29, 55.2%) reported that they offered the same number of interview spots for the year. The virtual interview process appears to have gone smoothly, as a majority of programs reported they never or rarely had technical problems across a number of different domains. Importantly, the majority of respondents were either somewhat satisfied or extremely satisfied (21/28, 75%) with the overall virtual interview process. Given the choice of preferred interview format going forward, half of the respondents (14/28) would offer a combination of traditional and virtual interviews.
CONCLUSIONS: These results demonstrate that virtual interviews are a viable alternative to traditional interviews and can be completed without many technological complications. PDs were overall satisfied with virtual interviews, despite not having appropriate time to plan and coordinate, and would be interested in continuing this option into the future. There are areas where improvements can be made and with appropriate planning could look to mitigate some of the areas where virtual interviews fall short of traditional interviews. In combination with our previously published survey of applicants (Paediatr Anaesth. 2021;31(9):968-976), these data form a more complete picture of virtual interview best practices going forward.

What is already known about the topic

Graduate medical education interviews have traditionally been conducted in‐person, but the COVID‐19 pandemic required the rapid creation and execution of a virtual interview process. Separately published data described pediatric anesthesiology fellowship applicant preferences in a virtual interview process and how they compare to in‐person interviews. There are no published data from pediatric anesthesiology program leaders about their impressions of this unique interview season.

What new information this study adds

This study informs us about program leader perceptions of specifics from the interview season, materials they chose to provide to applicants, and information they conveyed during the interview day. It also helps to paint a picture of program director preferences and, in combination with pediatric anesthesiology applicant survey data, inform best practices for future virtual interviews.

INTRODUCTION

The COVID‐19 pandemic precipitated an unprecedented time that touched almost every aspect of both personal and professional life for physicians; the pediatric anesthesiology fellowship application cycle was no different. After the World Health Organization (WHO) declared the SARS‐CoV‐2 virus a worldwide pandemic, many states began to issue “stay‐at‐home” orders and teaching hospitals across the United States began restricting travel. As the pediatric anesthesiology fellowship interview cycle was just starting the month of the shutdowns, an urgent situation was created for program directors (PDs) to quickly develop novel virtual interview processes. As published studies of virtual interviews in anesthesiology graduate medical education are limited, there were little data to guide PDs on how to either best showcase their respective fellowship programs or identify the optimal applicants to fill fellowship positions.

The purpose of this survey‐based study was to explore changes implemented by fellowship program leaders following the conclusion of the fellowship match; we also sought to identify obstacles encountered and attitudes about the process for the benefit of potential future virtual (or hybrid) application cycles. A survey of applicants was completed following the 2020 interview cycle; those findings have been separately reported. This manuscript describes the process and results of a survey of pediatric anesthesiology fellowship program leaders in 2020.

The purpose of this work is to describe the first virtual fellowship application cycle and clarify how programs conducted this process with the hope to, when combined with previously reported applicant data, determine (1) whether virtual interviews (or hybrid interviews) should continue when they are no longer mandated, and (2) a set of best practices for virtual interviews, should they continue.

METHODS

Institutional Review Board exemption was obtained from Children's Hospital of Philadelphia.

The 50‐question survey instrument was developed utilizing Qualtrics (Qualtrics; Provo, Utah), a web‐based tool for creating and conducting online surveys, by a multi‐institutional panel including one fellow (M.D.) and several faculty (E.L.S., M.C.L., M.C., J.L.L.). The survey instrument was pilot tested for clarity by pediatric anesthesiologist GME educators prior to distribution.

Following the National Resident Matching Program (NRMP) fellowship match in October of 2020, an introductory letter was sent to all program directors of ACGME‐accredited pediatric anesthesia fellowships via the Pediatric Anesthesiology Program Directors’ Association (PAPDA) electronic mailing list. The letter included a request for one program leader (either the program director or assistant/associate program director, but not both) for each program to complete the anonymous and voluntary survey. The survey remained open for one month (through November 16, 2020), and one reminder email was sent 2 weeks after the initial message.

Data were analyzed using Qualtrics internal reporting statistics. Data are presented as percentages where appropriate.

RESULTS

A total of 30 respondents, representing 30 unique fellowship programs, completed the survey, for a response rate of 50% based on the 60 ACGME‐accredited pediatric anesthesiology fellowships nationwide (30/60).

Program data

Among the programs represented by respondents, the mean number of fellowship positions offered per program was 3.8 (SD = 3.10) with a range of 1–12 position(s) offered annually. 96.7% (29/30) of the programs placed all of their positions in the NRMP Match and the remaining 3.3% (1/30) placed some, but not all positions in the Match (out‐of‐match positions were for two‐year fellows for this program). 96.6% (28/29) of programs reported that the virtual interview transition did not affect the number of fellowship positions they offered through the NRMP Match.

Almost half of the respondents (14/29, 48.3%) reported that there was no change in the total number of applications received during this application cycle when compared to prior years. 34.5% (10/29) reported that there was an increase in the number of applications they received. Of the programs that reported an increase in applications, the majority (6/10, 60%) reported an increase of 10%–20%. Additionally, 20% (2/10) reported less than 10% increase, 10% (1/10) reported an increase of 20%–30%, and 10% (1/10) reported a 90%–100% increase in applications over previous years. 17.3% (5/29) reported that there was a decrease in the number of applications received. Of the programs that reported a decrease, all were either 10%–20% decreased (4/5, 80%) or less than 10% decreased (1/5, 20%).

The majority of program leaders (16/29, 55.2%) reported that they offered the same number of interview spots in 2020 as they did in previous years. 3.5% (1/29) of the programs reported that they offered fewer interviews this year. 41.4% (12/29) reported that virtual interviewing allowed them to offer more interviews than in previous years. Of those 12 respondents, 67% (8/12) reported that they offered <25% increase in the total number of interviews while 33% (4/12) reported that they offered >25% increase from previous years.

Interview season

In this interview season, 90% of the program leaders (27/30) reported that they exclusively held virtual interviews. Of the program leaders that held in‐person interviews, 66.7% (2/3) reported that <10% of their interviews were in‐person while 33.3% (1/3) reported that between 10% and 20% were in‐person. No data are available about the timing of the 3 programs offering in‐person interviews (i.e., before or after the pandemic started in the United States).

The most common time for program leaders to first start contacting applicants to schedule interviews was the beginning of March (6/30, 20%). The middle of January was the earliest that program leaders reported contacting applicants (2/30, 6.7%). By the middle of March, 76.7% (23/30) of program leaders reported contacting applicants. A minority initiated contact in April (13.3%, 4/30), May (6.7%, 2/30), or later (1/30, 3.3%, in September).

The most common time for program leaders to hold their first interview was the beginning of April (7/30, 23.3%). The majority of program leaders (24/30, 80%) held their first interview during either April or May, and 16.7% (5/30) of program leaders reported holding their first interview prior to April (earliest report as the middle of February). One program leader (3.3%) reported holding the first interview at the end of September, just prior to the October match date.

Half of the program leaders (15/30) reported holding their final interview during the months of August or September, with the last reported interview occurring at the end of September. The earliest a program leader reported completing a final interview was the end of May (2/30, 6.7%), and 26.7% (8/30) of the program leaders reported holding their final interview prior to the end of July.

Interview experience

All of the respondents to the survey (30/30) reported that they conducted each applicant's interviews within a single day. The mode total time commitment for applicants on the interview day was 4–5 h (12/30, 40%). Two program leaders (6.7%) reported a greater than 5‐h time commitment and 3 program leaders (10%) reported a 1–2‐h time commitment from each applicant. No program leaders reported having a total interview time commitment of less than 1 h for any applicant.

The majority of respondents (23/30, 76.7%) reported that each applicant had between 3 and 5 interviews during the interview day. The lowest number of interviews per applicant was 2 (2/30, 6.7%), and the highest was 6 (5/30, 16.7%). Every respondent (30/30) reported that all applicants were interviewed with the PD. The most common other individuals that program leaders reported applicants interviewed with were other faculty members (21/30, 70%), APD (20/30, 66.7%), Chair/Chief of the division/department (15/30, 50%), and other departmental leaders (15/30, 50%). One respondent (3.3%) reported having applicants interview with nurse anesthetists, and four respondents (13.3%) reported having applicants interview with current fellows.

The median number of applicants interviewed during a standard interview session was reported as 4 (IQR = 2). The fewest number of applicants interviewed was 1 (7/30, 23.3%), and the most were 10+ (2/30, 6.7%). Almost half of the program leaders (14/30, 46.7%) reported that virtual interviews did not change the number of applicants interviewed per interview session. Seven of the 30 respondents (23.3%) reported interviewing fewer applicants, and 30.0% (9/30) reported interviewing more applicants per session. All but one respondent (29/30, 96.7%) reported that their interviews were time‐limited with the average interview length reported as 31.8 minutes (SD = 22) with a minimum of 15 and a maximum of 120 min.

The most common supplemental experiences that program leaders provided applicants on their interview day were as follows: an introduction to the fellowship (29/30, 96.7%), an opportunity to ask questions (28/30, 93.3%), an overview/presentation about the program (27/30, 90%), and a meeting with current fellows (24/30, 80%). The least common experiences provided were a live video tour of the program/hospital (2/30, 6.7%), a pre‐recorded video tour of the hospital/program (13/30, 43.3%), and education about the program's surrounding city/geographic region (17/30, 56.7%). The most common resources provided outside of the virtual interview by program leaders to applicants were electronic documents containing an overview of the program (20/28, 71.4%), contact information for faculty (17/28, 60.7%), and contact information for current fellows (13/28, 46.4%). The least common information provided outside of the virtual interview was a pre‐recorded video about the program (7/28, 25%), print material about the program (7/28, 25%), and print material about the surrounding city/geographic region (7/28, 25%). There were 4 program leaders (14.3%) that reported not providing any additional resources outside of the virtual interview.

Fewer than half of the respondents (14/29, 48.3%) reported access to institution‐provided or generated institution‐specific materials that were provided to applicants. Even fewer (10/29, 34.5%) reported that their institution provided or helped generate program‐specific material that was provided to applicants.

The most common virtual formats used for virtual interviews were Zoom (Zoom; San Jose, CA) (26/30, 86.7%), Facetime (Apple; Cupertino, CA) (3/30, 10%), and WebEx (Cisco; Milpitas, CA) (2/30, 6.7%). There were 3 respondents (10%) that reported they changed their primary virtual platform during the interview season. One program reported they had initially used Zoom but had to change due to an institutional ban and a second program reported that they changed platforms due to applicant feedback.

The majority of program leaders reported they never or rarely had technical problems during virtual interviews. Areas assessed included audio dropout (23/39, 79.3%), video dropout (22/26, 84.6%), dropped connection (24/26, 92.3%), screen freezing (22/26, 84.6%), audio distortion (21/26, 80.8%), links not working (23/24, 95.8%), slow bandwidth (21/26, 80.8%), unwanted participants (24/25, 96%), or lack of familiarity (21/26, 80.8%) (Figure 1). When technical problems arose, the most likely individual(s) to correct the problem were the program coordinator (20/29, 69.0%) or the PD (7/29, 24.1%). The most common backup systems in place in the event of technical difficulty were providing the applicant mobile phone number to the interviewer(s) (16/28, 57.1%), providing the interviewer mobile phone number to the applicant (13/28, 46.4%), and sharing e‐mail contact information (11/28, 39.3%).

FIGURE 1

Technical problems experienced during virtual interviews

Interview evaluation

Program leaders were asked to evaluate several aspects of interviews and compare virtual to traditional interviews; results of this comparison are displayed in Figure 2. Key findings include that few respondents rated virtual interviews as slightly or much worse than traditional interviews in terms of cost (2/25, 8%), ability to free faculty from clinical responsibilities for the interviews (3/25, 12%), ability to meet multiple candidates on the same day (4/25, 16%), and confidentiality of interactions (4/27, 14.8%). Conversely, respondents rated virtual interviews as slightly or much worse than traditional interviews in their ability to judge the personality of the applicant (20/27, 74.1%), ability to assess applicant interest in the program (18/27, 66.7%%), and “naturalness” of the interview (18/25, 72%). Program leaders felt that, compared to in‐person interviews, interviewees appeared either somewhat or extremely comfortable (18/28, 64.3%) during virtual interviews.

FIGURE 2

Comparing virtual to traditional interviews

The majority of program leaders were either somewhat or extremely satisfied (21/28, 75%) with the overall virtual interview process. Only one respondent (3.6%) reported being extremely dissatisfied. Despite their satisfaction with virtual interviews, the majority of program leaders reported that they prefer (14/28, 50%) or highly prefer (3/28, 10.7%) traditional in‐person interviews. Of the programs that preferred virtual interviews, 10.7% (3/28) highly preferred and 14.3% (4/28) preferred them. If given the choice of preferred interview format going forward, half of the programs (14/28) would do a combination of traditional and virtual interviews. Other preferences would be to go back to in‐person interviews (4/28, 14.3%), and virtual interviews with the option of an in‐person second look (6/28, 21.4%). None of the respondents indicated that they would choose to continue virtual interviews exclusively, at the time of the study.

All programs reported saving money during this virtual interview cycle (Figure 3). The most commonly reported range of savings is “$3000‐$5000” (6/27, 22%) with a range of responses from “$0–$500” (3/27, 11%) through “Greater than $10,000” (1/27, 3.7%).

FIGURE 3

Program money saved during virtual interviews

Comments

A free text area was provided to respondents to capture their biggest concern going into a virtual interview season. These were reviewed by the study team and thematic duplicates were eliminated. Representative unique comments are shown in Table 1.

TABLE 1

PD concerns entering virtual interview season

Technical issues/failures

Candidates would not get a good "feel" for the program

Inability to highlight city and/or institution

Ability to accurately judge personality, interest, etc. of applicants

DISCUSSION

This survey of pediatric anesthesiology fellowship program leaders elicited responses to determine common practices developed during the novel 2020 virtual interview cycle and captured contemporaneous feedback to determine the viability of, and potential best practices for, future virtual or hybrid live/virtual interviews.

Key findings include that compared to previous cycles, almost half the programs had no increase in the number of applicants they received, and a majority of programs reported that they offered the same number of interviews. The virtual interview process appears to have gone technically smooth as a majority of programs reported they never or rarely had technical problems across a number of different domains. Importantly, the majority of programs were either somewhat or extremely satisfied with the overall virtual interview process and if given the choice of preferred interview format going forward, half of the programs would offer a combination of traditional and virtual interviews. Perhaps more telling, these interviews were largely conducted at the start of the pandemic; we anticipate that by the time of publication both program leaders, as well as applicants, will have become considerably more comfortable with virtual meetings in general (and interviews, specifically) because of forced exposure over the past year.

Concerns about technical issues and failures were expressed by PDs prior to the 2020 virtual interview season (Table 1). Vining et al. reported a single‐center study surveying all the interviewers for surgical oncology fellowship undergoing virtual interviews and found that all the interviewers thought that the interview process was seamless or very seamless. The data presented here attempted to parse different technical areas that were more or less prone to technical failures; however, the majority of programs reported they never or rarely had technical problems with audio dropout, video dropout, dropped connection, freezing, audio distortion, link not working, slow bandwidth, unwanted participants, or lack of familiarity. This is encouraging; despite the rapidity of virtual interview process development, PDs were apparently able to implement systems relatively seamlessly using available technology. Having back‐up plans as well as a designated tech‐support individual will help keep interruptions to a minimum.

The ability to accurately judge the personality, interest, etc. of applicants was one of the concerns expressed by PDs in the survey (Table 1) prior to the virtual interview season. Menhaji et al.  surveyed female pelvic medicine and reconstructive surgery (FPMRS) fellowship program directors and found that a majority felt that a virtual interview format was effective for assessing and evaluating applicants (40/45, 88.9%). The data presented here expand on this by asking program leaders to assess different domains of candidate evaluation and to compare their ability to evaluate virtually versus a traditional in‐person interview. The majority of program leaders found that the ability to judge the personality of the applicant (20/27, 74.1%) and the ability to assess applicant interest in the program (18/27, 66.7%) was slightly or much worse in a virtual format when compared to a traditional interview. These data coincide with our separately published data about pediatric anesthesiology fellows who felt that virtual interviews were slightly or much worse at allowing an applicant to express their personality (24/47, 51.1%%) and demonstrated interest in the program (25/47, 53.2%). Despite the numerous advantages of virtual interviews, this is an important limitation that will need to be taken into consideration when choosing to host and take part in virtual interviews in the future. If virtual interviews become more common, we anticipate unique and creative workarounds that help overcome or minimize this obstacle; this is an area for future investigation.

Inability to highlight the program and the surrounding city was yet another concern about virtual interviews expressed by PDs entering the virtual interview season (Table 1). These are some of the most essential qualities to be able to convey to pediatric anesthesiology fellows as the most important factors they reported when making rank lists were (1) perceived goodness of fit and (2) desired geographic location. Despite this concern entering the virtual interview season, fewer than half of the programs provided an opportunity to learn about the city/geographic region during their virtual interview day. Additionally, only 25% of program leaders reported providing print materials about the program or about the city/geographic region. This lack of information provided may have contributed to a majority of pediatric anesthesiology fellowship applicants reporting that virtual interviews were much worse than in‐person interviews for learning the culture and livability of the city/region. In order to better highlight the city/geographic region of a program, it may be helpful in the future to dedicate time during the virtual interview presentation to this information as well as to provide information in an electronic format to applicants prior to their virtual interview. In addition, it may be helpful to allow a completely optional/obligation‐free post‐interview visit to give interested applicants a chance to learn about the city and program first‐hand.

Similarly, program leaders expressed concern that virtual interviews would not allow candidates to get a “feel” for the program (Table 1). In a previous study Hill et al.,  surveyed complex general surgical oncology (CGSO) fellowship PDs and found that a majority (19/30, 63%) agreed or strongly agreed that virtual interviews allowed for an accurate representation of the program. These data were not specifically queried from program leaders in this study, but the concern was well founded as a majority of pediatric anesthesiology fellow applicants felt virtual interviews were much or slightly worse at allowing them to learn the goodness of personal fit of a fellowship. This, along with learning about a geographic location, appears to be the largest and most difficult hurdles to overcome with virtual interviews. Similar to geographic location the best way to allow applicants to get a good “feel” about a program would be to have a post‐interview visit, but this is an area to focus on future improvements as well as investigation.

The majority of pediatric anesthesiology fellowship program leaders were either somewhat or extremely satisfied (21/28, 75%) with the overall virtual interview process. Previous studies looking at fellowship PD preferences and virtual interviews due to COVID‐19 have demonstrated that a majority would likely offer them in the future. , The present study similarly found that if given the choice of preferred interview format going forward, half of the programs (14/28) would offer a combination of traditional and virtual interviews. Almost half of pediatric anesthesiology fellowship applicants (20/42, 47.6%) would prefer to have a hybrid model where fellowships would offer a combination of traditional and virtual interviews, while the remainder harbored a fear that virtual interviews in a hybrid system would be differentially evaluated by programs. It is important to note that applicants may fear choosing a virtual interview in a system that allows for both, which could open the door to eliminating the positives of virtual interviews; efforts to mitigate this risk (or even the perception of it) will be essential in any future hybrid system. Nonetheless, both PDs and applicants appear to favor virtual interviews in the future.

While this study adds important information about the perspective of pediatric anesthesiology fellowship PDs in their first ever fully virtual fellowship application cycle, there are several limitations to the study. First, the picture captured in these data is of a virtual application cycle that was created out of necessity with very little to no preparation time available. Programs had a steep learning curve with little time to tackle inventing a new interview format and, often, with very little support from the home institution for the creation of supplemental materials. It could be hypothesized that with a full year of preparation and anticipation of a fully virtual interview season, combined with increased exposure to and comfort with virtual meetings, satisfaction, and attitudes toward future virtual interview seasons may be different; this is an area for future study. Another limitation is the 50% response rate from programs. It would always be ideal to have a 100% response rate but surveys add additional work and strain to very busy PDs. We believe that the distribution of respondents, including interview sessions spanning from 1 to more than 10, likely reflects a broad range of opinions from both large and small fellowship programs; however, demographic data were not included in the study. Finally, since the survey was completed months after interviews and after completion of the match process, recall bias, as well as response bias, may have skewed results depending on a program's satisfaction with match results.

Despite these limitations, this survey provides meaningful, previously unpublished data about virtual interviews from the perspective of program leaders. It demonstrates that these leaders were overall satisfied with virtual interviews, despite not having appropriate time to plan and coordinate, and would be interested in them continuing into the future. There are areas for improvement to allow a better product from both the applicant and PD perspective and hopefully, this study provides the initial data to spur new ideas and/or innovation. It may be deleterious to the pediatric anesthesiology fellowship to have a hybrid system with some programs offering in‐person interviews and others offering virtual interviews or multiple options. However, this study adds to our separately reported data from applicants about best practices, and we believe these risks may be mitigated by intentional efforts by program leaders—both individually and in a coordinated fashion through PAPDA.