Adapting to the Era of Virtual Recruitment: Radiology Departmental Website Response to COVID-19 and Portrayal of the Resident Experience.

RATIONALE AND
OBJECTIVES: The COVID-19 pandemic has transformed radiology recruitment into a virtual affair and placed an even stronger emphasis on the importance of departmental websites. In this study, we evaluate residency websites in detailing the response to COVID-19 as well as initiatives which help describe the resident experience.
MATERIALS AND METHODS: Program websites for diagnostic radiology residencies listed in the 2022 Electronic Residency Application Service (ERAS) program list were evaluated for 31 criteria related to departmental response to COVID-19, online outreach, and resident wellness.
RESULTS: Of 184 programs, 182 had functioning websites for review. One program was excluded from analysis as the website was almost entirely video-based. In response to COVID-19, ≤1% described resident redeployment, vaccination information, departmental response to ABR Core Exam changes, or regular administration updates. Six (3.3%) described revised read-out protocols, four (2.2%) mentioned supplementary non-clinical education, and 14 (7.7%) indicated changes to educational conferences. The majority of websites (122, 67.4%) offered an informational or tour video, while 44 (24.3%) described expectations for virtual interviewing, and 20 (11.0%) had virtual "open-houses." Departmental social media, primarily Twitter, was linked for 60 (33.1%) programs. A total of 134 (74.0%) websites described community highlights. More than a quarter mentioned meal stipends (72, 39.8%), paid sick time (54, 29.8%) and healthcare resources (57, 31.5%). Although social activities were described by 44 (24.3%) programs, some specifically indicating changes to COVID-19, formal resident mentoring (25, 13.8%) and wellness committees (28, 15.5%) were less common. These criteria were found more commonly at the largest third of residency programs (chi square, p < 0.00625).
CONCLUSION: Programs rarely described work flow changes to COVID-19, and websites could improve in virtual outreach. Compared with prior literature, departmental websites have improved in describing wellness initiatives and related measures.

INTRODUCTION

The COVID-19 pandemic has brought unprecedented challenges for radiology residency programs. Unique hurdles included the delay of the 2020 ABR Core Exam, which caused significant disruption to PGY4 residents preparing for the exam (1), as well as clinical redeployment, which heavily impacted junior residents (2). To address COVID-19 challenges, programs adopted social distancing, virtual curricula, and remote-read out techniques (3, 4, 5). Despite these changes, a recent survey of program directors still suggests an overall negative impact on educational activities and decreased resident morale (6). Given COVID-19 remains an active concern globally and with uncertain times still ahead, prospective residents are also stakeholders in how a program has responded to COVID-19.

In response to the COVID-19 pandemic, the 2020–2021 residency application cycle was performed completely virtually per recommendations from the Accreditation Council for Graduate Medical Education (7). In June of 2021, the Association of Program Directors in Radiology issued a statement recommending continued virtual interviews for the 2021–2022 residency application cycle (8). In this virtual era, studies within and outside of radiology have reported generally positive feedback among faculty and applicants to the electronic interview structure (9, 10, 11, 12).

One of the main challenges to virtual interviewing is the limited exposure that applicants have to each program along with its culture and geographic location. In place of in-person interactions, program websites and other online resources have increased in importance in describing the resident experience to applicants and their decision making for program ranking (9,13). Departmental websites vary widely in the level of information they provide (14, 15, 16, 17). Prompt website updates may be particularly important at this juncture to highlight a program's response to these unprecedented times to applicants (2, 3, 4, 5, 6,18, 19, 20).

Literature within and outside of radiology suggest that virtual interviews may persist beyond the era of heightened COVID-19 precautions (9,10,21). While it is too early to evaluate the long-term impacts from virtual interviewing, there are a number of immediate benefits which may argue for maintaining their use even if not necessitated by a pandemic. Cost and time savings can be tangible, with estimates of average interview travel expenses costing $4,000–$4,500, often requiring additional loans or compelling applicants to forego a portion of interviews (22, 23, 24). Cost savings have helped increase interview attendance (13) and likely improved equitable access (22). Environmental cost savings are also substantial, with literature estimating the carbon footprint of travel at 0.21-0.49 metric ton of CO2 per interview (11,12). If extrapolated across the number of diagnostic radiology ranks per US MD/DO applicant (25,26), the carbon footprint in 2020 amounted up to 5,092 metric tons of CO2. This carbon offset is equivalent to powering 925 homes for 1 year (27). With unknown perpetuity of virtual interviewing, programs should consider continuing to improve their online presence and efforts in virtual recruiting.

The purpose for this study was to evaluate available radiology residency websites for the presence of criteria pertaining to a department's COVID-19 response and its virtual outreach, as well as factors which may reflect the resident experience at a program. This data may help guide programs in improving the transparency and completeness of their online presence.

METHODS

A list of diagnostic radiology residency programs within the United States was obtained from the 2021–2022 Electronic Residency Application Service (ERAS). If departmental websites were incorrectly hyperlinked, the residency program was found via Google search. Two authors tabulated the presence or absence of 31 criteria on program websites during September of 2021. Criteria were developed by the authors, ranging from resident trainees to program director and faculty with a variety of experience in educational leadership, both before and during the COVID-19 pandemic. A subset of criteria were also selected based upon prior literature. Table 1 lists the surveyed criteria. To reduce interobserver variability, the two website reviewers revisited tabulated data after the initial 20 websites were assessed to maintain consensus. Criteria were also marked present if program websites directly linked to a pertinent webpage in a single hyperlink. Institutional Review Board approval was not needed as data was publicly available online.

Table 1

Categories and Criteria Evaluated

Category	Criteria	Description/Example
COVID-19 Response	Redeployment	Radiology residents working outside of their home department related to COVID-19 needs
	Vaccination Information	COVID-19 vaccine provision, requirements
	Read-Out Changes	Timing, frequency, or social distancing changes due to COVID-19
	Remote Clinical Work	Availability/use of off-campus clinical work stations
	Remote Non-Clinical Education	Asynchronous education not directly impacting patient care
	ABR Core Exam Changes	Testing or study accommodations
	Educational Conference Changes	Timing, frequency, or social distancing changes due to COVID-19
	Regular Administrative Updates	Description of regular program or departmental communications to trainees
Virtual Outreach	Virtual Open House	Advertisement or mention of online events to provide program information to prospective applicants
	Virtual Interview Description	Interview day timeline, expectations, virtual software utilized, as well as pre/post interview events
	Program Video	Informational videos, either GME or specific to the radiology department
	Social Media	Online media platforms and accounts specific to the radiology department
Program Information	Meal Stipend	Funds provided to residents to purchase hospital food
	Free Food	Free meals or snacks provided to residents on a regular basis, catered conferences
	Book Fund	Discretionary education funds over four years
	AIRP Fund	Funds provided for AIRP in addition to tuition
	Provided Technology	Provided tablet or computer devices
	PGY2/R1 Call	PGY2 after-hours clinical responsibilities
Personal Wellness	Paid Sick/Wellness Time	Paid time off per year for illness or wellness
	Gym	Available fitness facilities for trainees
	Wellness/Non-clinical Curricula	Non-interpretive curriculum such as wellness, resiliency, financial literacy, business practice
	Healthcare/Mental Wellness Resources	Primary care or mental health support for trainees
	Partner/Childcare Resources	Support resources for significant others or children
Work-Life Wellness	Resident Space	Dedicated non-clinical space for trainees, either GME or specific to the radiology department
	Community Highlights	Introducing social activities or city highlights outside of the hospital institution
	Retreats	Designated social activities at various time points in the year: resident welcomes, holiday parties
	Socials	Regular department-sponsored social outings
	Resident Wellness/Social Committee	Formal workgroups tasked with promoting trainee wellness or social activities
	Wellness/Social Webpage	Online information specifically for resident wellness or social activities
	Pictures of Social Gatherings	Media of residents in non-clinical, social activities
	Resident Mentoring	Formal “buddy” or “family” systems to provide resident support

AIRP, American Institute for Radiologic Pathology; GME, graduate medical education.

The number of residents at each program was determined on program websites and was inclusive of integrated interventional radiology residents within PGY2–PGY5 years. Program size was grouped into equal thirds (small, ≤18 residents; medium, 19–32 residents; large, ≥33 residents) for chi-square analysis. Chi-square analysis was performed within Microsoft Excel to determine if criteria varied in prevalence according to program size. Similar to prior literature (16), eight criteria related to work-life wellness were analyzed with chi-square analysis. Data without program size information was censored from chi-square analysis. Due to the number of criteria included within the chi-square, Bonferroni correction for multiple comparisons was applied to analysis (significant p = 0.00625).

Educational book funds and stipends beyond tuition for the American Institute for Radiologic Pathology (AIRP) were quantified, if available. Descriptive statistics and 95% confidence intervals were provided to describe variance and to provide a comparison with prior literature. The presence of criteria was not evaluated within program videos, due to the wide variability in the length and structure of videos.

RESULTS

Website Availability and Program Size

Out of 184 programs participating in the 2021–2022 ERAS, 182 had websites available for review. Two programs lacked websites, and one program was excluded from analysis as the entire website was primarily comprised of a multitude of videos. Therefore, criteria data was available for 181 programs. Program size was determined for 180 programs, with a mean of 26.4 residents.

COVID-19 Response

In response to COVID-19, only select programs described resident redeployment(1, 0.6%), vaccination information (2, 1.1%), or regular updates from departmental administration (2, 1.1%). No programs described the departmental response to the delay in the 2020 ABR Core Exam. No programs mentioned any clinical work performed remotely off campus. Six (3.3%) described revised read-out protocols, which consisted of descriptions for virtual read-outs using telecommunication software as well as two accounts of socially distanced in-person read-outs. Four (2.2%) mentioned remote non-clinical education: two were online virtual curricula accessible from home and two described a simulation imaging work list. A total of 14 (7.7%) indicated changes to educational conferences, primarily describing lectures broadcast electronically for at least a portion of the resident classes.

Virtual Outreach

Virtual outreach initiatives are summarized in Figure 1 . The majority of websites (122, 67.4%) offered a program video. These varied in length and content, involvement of residents and faculty, as well as in overall style. For example, a number centered on tours, while others focused on resident and faculty testimonials. Out of the 122 websites with a program video, 26 (21.3%) displayed a generic graduate medical education (GME) video and did not have media specific to the radiology residency.

Figure 1

Prevalence of criteria related to program virtual outreach among departmental websites.

Forty-four programs (24.3%) mentioned virtual interviewing procedures, and 20 (11.0%) had information about a virtual “open-house.” Departmental social media was provided for 60 programs (33.1%). The most common social media outlet was Twitter (52/60, 86.7%), followed by Instagram (34/60, 56.7%), and YouTube (22/60, 35.7%).

Program Information

Seventy-two out of 181 programs (39.8%) mentioned meal stipends on their websites. Most described stipends per call shift, while some described a set amount per meal or per year. A free food criterion, which encompasses a broad spectrum across sponsored snacks or drinks in a resident lounge to regularly catered lunches for conferences, was described on 26 websites (14.4%). Educational book funds were present for 100 programs (55.2%), of which 73 programs provided a quantitative amount averaging $3782 (SD $2261, 95% CI ± $519) over 4 years. AIRP funds were described in 93 programs (51.4%), of which 47 programs provided quantitative data averaging $2347 (SD $1054, 95% CI ± $301) over 4 years. Department sponsored technological tools, most commonly described as an iPad or tablet, were present for 31 programs (17.1%). The criteria that may impact financial considerations in the resident experience are summarized in Figure 2 .

Figure 2

Prevalence of criteria with program details which may impact financial considerations for residents among departmental websites.

PGY2 resident after-hours call responsibility was described on 89 program websites (49.2%), but only explicitly defined in 68 of these, with responsibilities starting in the fall of PGY2, spring of PGY2, or the beginning of PGY3 in fairly equal proportion (Figure 3 ).

Figure 3

Prevalence of mentioning junior resident after-hour call responsibilities and, if available, the percentage of overall programs describing when call duties start.

Personal Wellness

Paid sick days or wellness days were explicitly mentioned on 54 department websites (29.8%). Sixteen programs (8.8%) specifically mentioned wellness days at the discretion of trainees, averaging 3.1 days per year. A wellness or non-clinical curriculum, which encompasses non-interpretive skills such as building of resiliency, learning about personal finance, and/or practice administration was present for 41 websites (22.7%). At least one quarter of programs mentioned fitness facilities or healthcare and mental health resources, with fewer describing partner or childcare support. The prevalence of personal wellness criteria online is displayed in Figure 4 .

Figure 4

Prevalence of personal wellness resources on departmental websites.

Work-Life Wellness

A dedicated resident lounge space was mentioned on 40 websites (22.1%). This criterion did not differentiate between a radiology specific or house staff specific space. Community highlights and available activities were posted for 134 programs (74.0%).

Criteria pertaining to resident wellness and social interactions were variably present. The most commonly available item was media demonstrating resident social activities (67, 37.0%), and the least common was description of a resident “family” or mentoring group (25, 13.8%). Criteria analyzed by chi-square are described in Figure 5   (observed data provided in Appendix Table A1 ). Formal retreats (p = 0.0162) and a wellness or non-clinical curriculum (p = 0.00780) were observed more often than expected within programs in the top third of residency size, but this was not statistically significant after Bonferroni correction.

Figure 5

Prevalence of wellness criteria on radiology residency websites. *Found at larger programs (≥33 residents) and at ^mid-sized programs  (19–32 residents) more often than expected after Bonferroni correction for multiple comparisons  (chi-square, P<0.00625).

Table A1

Observed Data Evaluated by Chi-Square Analysis to Determine if Criteria Appeared More Often Than Expected Depending on Program Size Grouped Into Equal Thirds  (Small, ≤18 Residents; Medium, 19–32 Residents; Large, ≥33 Residents). Shaded Boxes Indicate Statistical Significance After Bonferonni Correction for Multiple Comparisons (p < 0.00625).

		Retreats	Socials	Pictures of Social Gatherings	Wellness/Non-clinical Curriculum	Resident Mentoring	Resident Wellness/Social Committee	Wellness/Social Page	Resident Space
Program Size	Small	6	3	10	8	0	2	11	8
	Medium	14	12	22	10	10	8	21	14
	Large	21	24	35	23	15	17	21	17
Sum Observations		41	39	67	41	25	27	53	39
Expected Observations per Program Size if Null		13.7	13	22.3	13.7	8.33	9	17.7	13
Chi-square p-Value  (statistically significant p = 0.00625)		0.0162	0.000196	0.000912	0.0078	0.000912	0.00178	0.152	0.199

DISCUSSION

Departmental responses to COVID-19 are in various stages of implementation depending on local institutional policies and COVID-19 prevalence. Websites may offer information to prospective residency applicants interested in evaluating a program's level of response to COVID-19 challenges. Transparency and close communication with program administration can be critical for resident wellness (19). Although there is a plethora of radiology education literature regarding COVID-19 and anecdotally at least some changes among all programs, we find a marked disconnect with the paucity of information available online, specifically on residency program websites.

Without in-person interactions, prospective residents rely on virtual resources to learn about programs. Virtual open houses have been described as an effective method for applicants to learn about a program culture, its educational offerings, and what to expect in virtual interviews (28). A recent survey indicated that 57% of a program's applicants who attended a virtual open house said the event impacted their application decision (13). Despite this importance, virtual open houses were advertised on only 11.0% of residency program websites. Candidates may learn of open houses on social media (28), but we find only one-third of programs linked their social media accounts on their websites. Although open houses may be most impactful at the start of an ERAS application season, improving applicant exposure to current residents and faculty as well as bolstering avenues to connect on social media may still impact rank list decisions (13).

Program videos, virtual tours, and testimonies likely play an important role in introducing applicants to a program's culture and its resident experience (9,20). Out of the programs that displayed a video on their website, 21.3% (14.4% of all residency websites) had only a generic video from the institution's GME. These videos may provide a general sense of the overarching institution and location, but generic house-staff videos are unlikely to adequately convey factors unique to radiology residency and its trainees. If budgeting allows, departments should consider developing a program specific video.

The Resident Experience

Department websites are great avenues to communicate clinical curricula and educational programs, and these have been described in prior literature (14,15,17). We also aimed to evaluate criteria describing program information, personal wellness, and work-life wellness which may impact a resident's overarching experience at a program. Multiple elements factor into resident welfare, but we specifically selected a subset of criteria previously evaluated in April of 2020, and subsequently published (16), to investigate how program websites have responded to COVID-19 and the 2020–2021 virtual interview season. All criteria compared with previous data have increased in prevalence on departmental websites.

Both availability of meal stipends and book funds showed increases greater than 10% (14.4% and 13.6% respectively). No significant difference was observed between mean book funds ($3782, 95% CI ± $519, versus $3762) or mean AIRP funds ($2347, 95% CI ± $301, versus $2204). With AIRP presented entirely virtually until August of 2022, and at least a portion of virtual options extending to November of 2023 (29), book and AIRP funds may be subject to change in the coming years.

Junior resident call responsibilities were mentioned in slightly greater frequency than previously reported (49.2% versus 47.6%). There continues to be similar proportion of programs describing call duties starting in the fall of PGY2, spring of PGY2, or the beginning of PGY3. In the initial COVID-19 surge, PGY2 residents were described as having the largest decrease in imaging interpretation volume, likely due to clinical redeployment. This may have downstream effects on radiology resident call preparedness (2). The timing of commencement of after-hour call responsibilities, which impacts work-life wellness, may be susceptible to changes with COVID-19 surges and imaging volume.

The criteria pertaining to personal wellness with the greatest improvements in prevalence on departmental websites were: healthcare and mental health resources (31.5% versus 13.5%), as well as wellness and non-clinical curricula (22.7% versus 8.1%), followed by sick and wellness days (29.8% versus 20.0%). The increase in website information for healthcare and mental health resources is a positive development, and likely reflects a recent report that program directors felt there has been adequate access for residents (6). Days specifically allotted for wellness were identified at fewer programs than before (8.8% versus 10.8%), but this is likely attributable to the current study's technique of evaluating departmental websites only (and not institutional GME sites). Overall, increased visibility and availability of the assessed criteria on program websites may reflect heightened program initiatives to focus on resident wellness during the pandemic.

Work-life wellness continues to play an important role in training and is even more important with added COVID-19 stressors; therefore, some have suggested innovative activities to implement in the era of social distancing (18,19). In addition to the increase in program websites describing work-life wellness criteria, some described innovative techniques such as: virtual happy hours, virtual painting activities, movie nights, and work-out challenges. Chi-square analysis of eight criteria previously evaluated shows that department-funded socials and pictures of social activities remain more prevalent than expected at the largest third of programs, in addition to resident mentoring as well as wellness and social committees. On the other hand, the other criteria (formal retreats, wellness and non-clinical curricula, and dedicated wellness or social pages) are no longer statistically significantly associated with the largest programs, though still increased in online representation. These changes suggest that there has been shifts in website information and possibly work-life wellness offerings among programs. Programs should continue their progress in improving the thoroughness of their websites and reflecting on the support they provide to their residents.

Limitations

A true resident experience can be difficult to glean from online resources. However, utilizing the presence or absence of website criteria as a proxy offers a program-to-program comparison accessible to prospective applicants. Program videos, especially thorough ones, may offer great insight into a program. A limitation of this study is that we did not specifically include the content of the videos in evaluating criteria, though acknowledge that there may be information provided in video form and not otherwise documented on a website. In the same vein, we did not attend any virtual open houses, which is likely another effective avenue to distribute information to applicants.

Although we noted the number of trainees at a program, we did not record information about resident diversity. Subjectively, some programs did include diversity, equity, and inclusion statements on webpages, but further study is necessary to evaluate program offerings.

Intra- and inter-observer variability is also a possible limitation. We attempted to mitigate this by performing a systematic and exhaustive search of all available webpages and links on program websites, as well as assessing consensus early in the review period.

Programs vary in frequency and thoroughness in updating their websites. On our review, there were websites which were clearly outdated, for example with graduated resident classes listed as current residents, or interview instructions describing local restaurant expectations in pre-pandemic interview socials. These websites were included in review to offer a broad evaluation of program websites. Hopefully this data provides encouragement for programs to be mindful of the accuracy of their online resources.

CONCLUSIONS AND FUTURE DIRECTIONS

This study evaluated radiology residency program websites for information related to changes from COVID-19 and program initiatives in virtual recruitment, as well as portrayal of the resident experience. Despite substantial impacts from COVID-19, programs rarely offered online information about their workflow changes. Virtual recruitment initiatives could be upgraded, and this should be a strong consideration given the uncertainty of whether virtual interviews will persist on account of time, cost, and carbon footprint savings. In summary, between April 2020 and September 2021, program websites have improved in describing wellness initiatives and the overall resident experience. However, opportunities exist to make additional improvements and updates.

Until more data becomes available and given lack of standardized guidelines, programs should consider soliciting feedback from their residency class starting radiology in July of 2022 regarding the content and effectiveness of their websites. These trainees were first to experience virtual recruitment and may have crucial insight for website improvement. Programs can also consider utilizing their websites to provide updated information and offerings for their current trainees, which would then also be visible to prospective applicants, serving a dual purpose.