The effect of virtual interviews on the musculoskeletal radiology fellowship match.

OBJECTIVE: Resulting from the COVID-19 pandemic, a virtual interview format was adopted in lieu of in-person interviews for fellowship recruitment. The purpose of this study is to analyze the potential effect that virtual interviews have on musculoskeletal radiology fellowship match results and collect opinions of the process.
MATERIALS AND METHODS: An anonymous survey was sent to 87 listed US-based musculoskeletal radiology fellowship directors, who were asked for their 2019-2020 and 2020-2021 match data: fellowship positions, interviewees, ranked interviewees, and matched ranks. They rated effectiveness of virtual interviews on a ten-point scale (1 poor-10 excellent) and provided an ideal percentage of future interviews to be conducted virtually.
RESULTS: Thirty responses were received (34.4% response rate). Primary analysis found significant increase in the mean number of candidates ranked in 2020-2021, 14.6 to 17.5 (p = 0.047). The mean rank of matched applicants increased from 6.02 to 6.43, without significance (p = 0.821). Secondary analysis, which attempted to exclude internally matched programs, found significant increases in 2020-2021 in the mean number of applicants-28.7 to 32.4 (p = 0.017), interviews conducted-17.8 to 21.3 (p = 0.007), and candidates ranked by programs-16.3 to 19.8 (p = 0.015). The mean rank of matched applicants increased from 6.39 to 7.03, without significance (p = 0.713).
CONCLUSION: With results showing an increase in applications, interviews conducted, and ranked applicants while lowering the average rank of matched candidates, musculoskeletal radiology fellowship directors should consider interviewing more applicants than they usually would in the prior in-person recruitment format.

Introduction

The COVID-19 global pandemic has altered the way academic radiology departments operate with the introduction of social distancing practices, changing clinical and administrative workflow, including trainee recruitment. This monumental change began with the 2020–2021 application cycle. Under guidance from the Society of Chairs of Academic Radiology Departments (SCARD), all trainee interviews moved to virtual interviews (V-Int), inclusive of fellowships [1].

Personal interviews have historically been a critical portion of the selection process for both applicants and programs. Considering the most recent results of the National Resident Matching Program (NRMP) survey, the factors that radiology resident applicants report being most important to them include “goodness of fit” and the “interview day experience” [2]. These impressions are formed in a significant way by the interview process. Amongst program directors, the interview continues to be a key component in ranking applicants [3].

Social distancing and travel restrictions initiated with the global pandemic led to a rapid widespread adoption of V-Int. This transition to a virtual process has been described by some as beneficial in decreasing cost and increasing flexibility [4-6]. Some of the drawbacks of the virtual process primarily revolve around a decrease in interpersonal engagement. This has been described to affect program assessment of applicant professionalism and personality as well as diminished ability of applicants to understand the culture, relationships, and location of a program [4-6]. The virtual format also limits interaction and spontaneity with a less natural interaction than in-person interviews [6]. There is also some potential for technical inequity and inefficiency [7].

While others have published opinions and results regarding the radiology residency match, there is little data describing the perception or effect of the virtual recruitment process on the fellowship match, particularly regarding the musculoskeletal radiology subspecialty. We surveyed musculoskeletal radiology fellowship directors with the goal of analyzing the potential effect that the V-Int format had on match results and to collect opinions of the process.

Materials and methods

Following Institution Review Board exemption, a voluntary anonymous survey (Microsoft Form) was sent via email to fellowship directors of the 87 US-based musculoskeletal radiology fellowships listed in the Society of Skeletal Radiology “MSK Fellowship Program List.” This was sent after the completion of the 2020–2021 NRMP Fellowship match cycle that occurred June of 2021. One month following the initial invitation, an email reminder was sent.

The survey asked for details of each program’s match for the 2019–2020 (in-person) and the 2020–2021 (virtual) cycles: number of fellowship positions offered, fellowship applications, interviews conducted, and applicants ranked. Specific ranks of matched applicants were also requested for both year’s application cycles (how far down a program matched on its rank list numerically). Fellowship directors were also asked the number of V-Int conducted in 2019–2020. They were also asked to rank how effective V-Int were in comparison to in-person interviews on a 10-point Likert scale (1 = poor and 10 = excellent) as well as a preferred percentage of interviews conducted virtually in the future. An optional free text response was also provided for comments. For reference, the survey questions can be seen as supplementary material within Table 4.

Statistical analysis

A paired sample t-test was performed to assess mean differences across all programs between 2019–2020 and 2020–2021 in the number of available fellowship positions, applications, interviews conducted, applicants ranked, percentage of applicants interviewed, and unmatched positions. Specific ranks of matched applicants were compared across all programs between 2019–2020 and 2020–2021 using a Wilcoxon signed-rank test to assess for significant difference.

Next, a secondary analysis was performed after removal of programs whose number of applicants and/or interviews equaled the number of fellowship positions in either or both years, as these were assumed to represent programs who had filled all positions by matching internal residency program candidates. This secondary analysis was rationalized in that while matching positions in this way is a valid strategy, it does not reflect the “typical” experience of matching candidates outside their home programs and creates sizable outliers. A paired sample t-test was performed in the secondary analysis group to assess mean differences between 2019–2020 and 2020–2021 regarding the number of applicants, number of interviews, applicants ranked, and percentage of applicants interviewed. Specific ranks of matched applicants and the lowest rank of matched applicants at each program were similarly compared across all programs between 2019–2020 and 2020–2021 using a Wilcoxon signed-rank test.

A subgroup analysis was performed to assess for the possible effect of unmatched positions in 2019–2020 on interviewing and ranking decisions in 2020–2021. Paired sample t-tests were performed on mean number of interviews, applicants ranked, and percentage of applicants interviewed for two subgroups: programs who fully matched all positions in 2019–2020 and programs who had one or more unmatched positions in the same year.

To characterize some of the effect that program size had on behaviors, a simple linear regression analysis was performed examining the relationship between the number of positions available and the average rank of matched applicants for both 2019–2020 and 2020–2021.

All statistical analyses were performed using Stata (version 16.1, StataCorp). An alpha value of 0.05 was used to determine statistical significance for all analyses.

Results

There were 30 responses from the 87 fellowship program directors for a response rate of 34.4% accounting for 79 positions in 2019–2020 and 80 positions in 2020–2021. No virtual interviews were reported in the 2019–2020 cycle. The primary analysis found a significant increase in the mean number of candidates ranked by programs in 2020–2021 compared to 2019–2020, increasing from 14.6 to 17.5 (p = 0.047). There was no significant difference in the number of positions, applicants, or interviews conducted, although the means were higher in 2020–2021 for these variables. The number of interviews and percentage of applicants interviewed had higher means in 2020–2021 as well, approaching significance (p = 0.054 and 0.057, respectively). No significant difference in the program ranks of matched candidates was found with a mean increase from 6.0 to 6.4 (p = 0.821). No significant difference was noted between the lowest ranked candidates with a mean increase from 7.7 to 8.7 (p = 0.807). Expanded results are summarized in Table 1. A chart of these results is shown in Fig. 1A.

Table 1

Analysis of responses regarding the 2019–2020 and 2020–2021 musculoskeletal radiology fellowship match

Variable	Total (n)	Mean (range)	95% confidence interval	p value
Primary analysis of the 2019–2020 and 2020–2021 fellowship match (n = 30)
Positions				0.326
2019–2020	79	2.63 (1–5)	2.22–3.04
2020–2021	80	2.67 (1–5)	2.27–3.06
Applicants				0.242
2019–2020	845	28.16 (2–80)	20.85–35.47
2020–2021	915	30.50 (2–76)	23.10–37.90
Interviews				0.054
2019–2020	481	16.03 (1–35)	12.65–19.42
2020–2021	563	18.77 (1–40)	14.67–22.86
Ranked applicants				0.047
2019–2020	437	14.57 (1–35)	11.24–17.89
2020–2021	524	17.47 (1–39)	13.52–21.41
Percentage of applicants interviewed				0.057
2019–2020	56.9%	64.8% (10.0–100%)	0.54–0.75
2020–2021	61.5%	72.0% (4.0–100%)	0.62–0.82
Unmatched positions				0.174
2019–2020	31	1.03 (0–4)	0.59–1.48
2020–2021	39	1.30 (0–4)	0.86–1.74
Ranks of matched candidates (Wilcoxon signed-rank test)				0.821
2019–2020	41	6.02 (1–23)
2020–2021	37	6.43 (1–39)
Lowest rank of matched candidates (Wilcoxon signed-rank test)				0.807
2019–2020	22	7.68 (1–23)
2020–2021	24	8.67 (1–39)
Secondary analysis of the 2019–2020 and 2020–2021 fellowship match accounting for presumed internal matching
Applicants (n = 28)				0.017
2019–2020	802	28.68 (4–80)	21.11–36.25
2020–2021	908	32.43 (3–76)	25.03–39.83
Interviews (n = 25)				0.007
2019–2020	446	17.84 (3–35)	14.39–21.29
2020–2021	532	21.28 (3–40)	17.28–25.58
Ranked applicants (n = 25)				0.015
2019–2020	407	16.28 (1–35)	12.81–19.74
2020–2021	494	19.76 (1–39)	15.82–23.70
Percentage of applicants interviewed (n = 25)				0.152
2019–2020	55.6%	72.2% (32.5–100%)	0.63–0.82
2020–2021	58.5%	75.9% (37.2–100%)	0.67–0.85
Ranks of matched candidates (Wilcoxon signed-rank test)				0.713
2019–2020	38	6.39 (1–23)
2020–2021	33	7.03 (1–39)
Lowest rank of matched candidates (Wilcoxon signed-rank test)				0.940
2019–2020	20	8.30 (1–23)
2020–2021	21	9.67 (1–39)

Fig. 1

A The bar chart shows a visual representation of the primary analysis of the survey results between the two match cycles. Significance is denoted by * for p < 0.05, with whiskers representing the 95% confidence interval. B The bar graph shows a visual representation of the secondary analysis of the survey results between the two match cycles. Significance is denoted by * for p < 0.05 and ** for p < 0.01, with whiskers representing the 95% confidence interval

There were a total of 31 unmatched positions reported in 2019–2020 and 39 in 2020–2021. Between 2019–2020 and 2020–2021, 14 programs reported no change in the number of unmatched positions, 8 of which reported no unmatched positions in both years. For 2020–2021, 11 programs reported an increase in unmatched positions and 5 reported a decrease. No significant difference in the mean number of unmatched positions per program was found. Expanded results are shown in Table 1.

For the secondary analysis, two programs reported applications equaling the number of available positions in 2019–2020 and/or 2020–2021, while an additional three reported interviews equaling the number of available positions. After exclusion of these programs, the secondary analysis found significant increases in 2020–2021 in the mean number of applicants—increasing from 28.7 to 32.4 (p = 0.017), interviews conducted—increasing from 17.8 to 21.3 (p = 0.007), and candidates ranked by programs—increasing from 16.3 to 19.8 (p = 0.015). No significant difference in the mean percentage of applicants interviewed was found. The mean rank of matched applicants increased from 6.4 to 7.0, but without significance (p = 0.713). Likewise, the mean lowest rank of matched applicants to a program increased from 8.3 to 9.7, but without significance (p = 0.940). Expanded results are also summarized in Table 1. A chart of these results is shown in Fig. 1B.

In 2019–2020, 16 of 30 (53.3%) programs had 1 or more unmatched spots, increasing to 20 of 30 (66.7%) in the following 2020–2021 virtual cycle. Subgroup analysis of fully matched programs in 2019–2020 and of unmatched programs in 2019–2020 showed trends but without significance. Fully matched programs reported an increase of interviews from 18.4 to 22.0 (p = 0.067), ranked applicants from 17.2 to 20.2 (p = 0.103), and percentage of applicants interviewed from 57.8 to 62.3% (p = 0.312). Non-fully matched programs reported identical movements with an increase of interviews from 14.0 to 15.9 (p = 0.353), ranked applicants from 12.3 to 15.1 (p = 0.22), and percentage of applicants interviewed from 71.0 to 80.5% (p = 0.117). Full results are located in Table 2.

Table 2

Subgroup analysis of fully matched and non-fully matched programs from the 2019–2020 musculoskeletal fellowship match

Variable	Fully matched programs2019–2020 (n = 14)		Non-fully matched programs2019–2020 (n = 16)
	Mean (range)	p value	Mean (range)	p value
Interviews		0.067		0.353
2019–2020	18.36 (2–35)		14.00 (1–33)
2020–2021	22.00 (1–40)		15.94 (2–40)
Ranked applicants		0.103		0.222
2019–2020	17.21 (1–35)		12.25 (1–30)
2020–2021	20.21 (1–35)		15.06 (1–39)
Percentage of applicants interviewed		0.312		0.117
2019–2020	57.8% (10.0–100%)		71.0% (30.0–100%)
2020–2021	62.3% (5.0–100%)		80.5% (4.0–100%)

There was positive correlation between the program size and average matched rank. For every singular increase in available fellowship position in a given program, the average rank of matched applicants rose 2.3 positions in 2019–2020 (p = 0.033) and 3.7 positions in 2020–2021 (p = 0.014). This is shown in Table 3.

Table 3

Simple linear regression of the average matched rank by number of positions available

Year	Coefficient	R2	95% confidence interval	p value
2019–2020	2.278	0.208	0.21–4.35	0.033
2020–2021	3.688	0.243	0.81–6.57	0.014

When asked to rank how effective V-Int were in comparison to in-person interviews, the average score was 5.2 across all 30 responses. Responses are displayed in Fig. 2. The average preferred percentage of interviews to be conducted in the future was 40.6% across 24 responses. Responses are displayed in Fig. 3.

Fig. 2

The bar graph shows opinions on how effective virtual interviews were in comparison to in-person interviews on a 10-point Likert scale

Fig. 3

The histogram shows responses to the preferred percentage of interviews to be conducted virtually in the future

Discussion

This is one of the first studies to explore the effect V-Int in the setting of fellowship recruitment, and to our knowledge, the first to examine its impact on the musculoskeletal subspecialty. Our survey results showed trends of increasing applications, interviews conducted, and applicants ranked based on limited survey data regarding the musculoskeletal fellowship match comparing the in-person and the virtual cycles. The increase in mean number of candidates ranked by programs was statistically significant in the primary analysis. The secondary analysis assumed 5 programs to have internally matched all positions based on review of the data. After their exclusion, the increases in mean number of applications, interviews conducted, and candidates ranked by programs were significant between the two fellowship match cycles. Furthermore, additional subgroup analysis revealed that regardless of whether a program fully matched or had unmatched positions, the trends were similar but without significance. This shows that these findings are likely attributable to the conversion to the virtual format rather than as a response to an unfilled match.

These findings support the idea that given the relative ease of online applications and attending virtual interviews, there is an artificial rise in applications to any given program, a concept that has been expressed by other authors [8-10]. This could have the effect of essentially decreasing the interview pool as more programs would interview the same subset of highly qualified applicants [10]. With regard to residency recruiting practices, some programs reported offering more interviews than usual [5]. From the resident match perspective, some resident applicants admitted to applying to more programs and accepting more interviews, as was also suspected in our study of fellowship applicants. Eighteen percent of resident respondents even admitted to applying to programs they were unlikely to rank [11].

Despite the increased number of applications, the responding programs reported an increase in unmatched positions in addition to falling lower on their rank order lists for matched fellows, although without statistical significance. It is important to note that the average ranks included in the analysis did not account for unmatched positions as these do not have an associated rank number that could be included in statistical analysis. The number of unmatched positions increased between the 2 years included in our analysis, supporting the notion that the outcome of the match was less desirable after implementing V-Int. This finding is in contradistinction to residency survey results showing that 92% of rank lists performed the same or better using V-Int [10]. Our findings support the notion postulated by Rozenshtein et al. that V-Int leads to a smaller effective rank pool that could, in turn, lead to unfilled positions [9].

Regarding the entire musculoskeletal radiology fellowship match and extrapolation of our findings to the subspecialty-wide process, this is not straightforward due to the incomplete data. According to the published NRMP Program results, there was no change in the number of unmatched musculoskeletal radiology fellowship positions (62) between 2020 and 2021. Between those years there was an increase in offered positions from 213 in 2019–2020 to 218 in 2020–2021 [12]. These combined data suggests that this could be due to a redistribution of fellowship position matches amongst different programs, and thus locations. Given the confidential nature of the survey, our data contains no specific information on the details of this redistribution. Our data set contains 16 of 30 (53.3%) unfilled programs in 2019–2020 and 19 of 30 (63.3%) unfilled programs in 2020–2021, suggesting a mixed population of responses, not necessarily biased by reflecting program directors disproportionally left with unmatched positions. In view of these results, musculoskeletal fellowship directors should consider interviewing more candidates than they historically have during implementation of a virtual format.

Furthermore, in an open letter, the AAMC suggested a misdistribution of interviews, favoring top tier applicants, with a result of all applicants applying to more programs [13]. It has been further suggested that applicants may be using the additional interviews at their less desired programs as preparation for other programs they held in higher regard [11]. We believe the data supports this opinion as there was an increase in the number of interviews across responding programs accompanied by an increase in unmatched positions rather than a decrease.

If fellowship directors decide to alter the size of their program, they should keep in mind the reported survey results between 2019–2020 and 2020–2021. There was a significant positive relationship between the size of a program and the average matched position. In 2019–2020, every increase in position offered resulted in an increase in the average matched position of a program of 2.3. This increased to 3.7 during the virtual match year. Although this is a simple model, it shows that if a program decided to increase their available positions, there would be a greater than expected fall down the rank list.

The overall impression of V-Int was mixed across this group of musculoskeletal fellowship directors. The effectiveness of V-Int was scored a 5.2 with scattered responses approaching both “poor” and “excellent” in a bimodal distribution, demonstrating wide variability on different institutions’ experiences. Individual free text comments also varied with positive, neutral, and negative responses regarding V-Int. This result differs from other recently reported surveys of residency directors who found V-Int more favorable and successful [5, 10]. Respondents in our study were mixed about the future of V-Int. The averaged preference of musculoskeletal radiology fellowship directors would be to have them comprise 40.6% of interviews in the future under ideal conditions. This finding is similar to the reticence to continue a full virtual format in other surveys of residency directors [5, 10].

Limitations

There are limitations to the study. As there was a 34.4% response rate from the survey, the data is incomplete. The sample size is limiting with respect to statistical power as well. More specific details that would have been useful in analysis regarding the individual programs were not collected to preserve anonymity. Although there could be response bias of unfilled programs to complete the survey, the data show a mixed population of respondents in terms of program size and include programs that filled entirely and some that did not. There was an attempt to control for internal matching within our dataset, but it was not possible to account for partially internally matched programs. These were treated as equivalent to standard programs in this analysis. While no statistically significant change in the program-list ranks of matched candidates was found, this quantitative analysis did not account for the qualitatively different but highly important impact of unmatched positions. We were also not able to control for the marked variability in tactics and priorities of the different programs with regard to their fellowship positions.

In conclusion, the virtual interview format appeared to result in an increase in applications, interviews conducted, and number of ranked applicants while lowering the average rank of matched candidates for the musculoskeletal radiology fellowship match. Fellowship directors should consider this in their recruitment process and should contemplate interviewing more applicants than they usually would in the prior in-person recruitment format. Overall, the new process received mixed feedback from responding musculoskeletal radiology fellowship directors with uncertainty of the future role of virtual interviews.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 13 KB)