Incorporation of a Social Virtual Reality Platform into the Residency Recruitment Season.

RATIONALE AND
OBJECTIVES: The Covid-19 pandemic ushered a sudden need for residency programs to develop innovative socially distant and remote approaches to effectively promote their program. Here we describe our experience using the social virtual reality (VR) platform Mozilla Hubs for the pre-interview social during the 2020-2021 radiology residency virtual recruitment season, provide results of a survey sent to assess applicants' attitudes towards the VR pre-interview social, and outline additional use-cases for the emerging technology.
MATERIALS AND METHODS: A VR Meeting Hall dedicated to the pre-interview social was designed in Mozilla Hubs. To assess applicants' impressions of the Mozilla Hubs pre-interview social, applicants were sent an optional web-based survey. Survey respondents were asked to respond to a series of eleven statements using a five-point Likert scale of perceived agreement: Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree. Statements were designed to gauge applicants' attitudes towards the Mozilla Hubs pre-interview social and its usefulness in helping them learn about the residency program, particularly in comparison with pre-interview socials held on conventional video conferencing software (CVCS).
RESULTS: Of the 120 residency applicants invited to the Mozilla Hubs pre-interview social, 111 (93%) attended. Of these, 68 (61%) participated in the anonymous survey. Most applicants reported a better overall experience with Mozilla Hubs compared to CVCS (47/68, 69%), with 10% (7/68) reporting a worse overall experience, and 21% (14/68) neutral. Most applicants reported the Mozilla Hubs pre-interview social allowed them to better assess residency culture than did pre-interview socials using CVCS (41/68, 60%). Seventy-two percent of applicants reported that the Mozilla Hubs pre-interview social positively impacted their decision to strongly consider the residency program (49/68).
CONCLUSION: Radiology residency applicants overall preferred a pre-interview social hosted on a social VR platform, Mozilla Hubs, compared to those hosted on CVCS. Applicants reported the use of a social VR platform reflected positively on the residency and positively impacted their decision to strongly consider the program.

INTRODUCTION

The Covid-19 pandemic ushered a sudden need for residency programs to develop innovative socially distant and remote approaches to effectively promote their program. These efforts included updating residency program websites, producing promotional and informational videos, utilizing social media platforms, developing remote away rotations and other educational curricula, and preparing for an interview season held over video conferencing software (1, 2, 3). Overlooked or underemphasized in this transition, however, was designing an opportunity for applicants to interact with the current resident cohort.

Among the suite of activities that a residency program offers during a typical interview season, the time-honored tradition of the “pre-interview social” is often one of the few opportunities for residency applicants to meet with current residents en masse. These events provide applicants with an opportunity to interact with residents in a less formal setting and have candid conversations about the program. Moreover, applicants get a chance to familiarize themselves with the overall culture of the residency and judge whether their sensibilities fall in line with the existing social and academic environment (read: assess their “fit”) (4). This is important because “fit” is often regarded as the critical intangible factor that applicants must seek out to find the residency program where they will be most likely to thrive (5,6).

Conventional video conferencing software (CVCS), such as Zoom (Zoom Video Communications, Inc., San Jose, CA) and Webex (Cisco Webex, Milpitas, CA), is suboptimal for facilitating the sort of social interaction conducive to assessing “fit.” Conversations held over CVCS are linear and adynamic. Users are hesitant to participate and off-screen distractions are common (7). While the “pre-interview social” is meant to be one of the more light-hearted, low-stress parts of the residency interview experience, CVCS risks making the experience feel like “yet another interview” from the perspective of applicants. Further, the experience is taxing on the cohort of residents who need to maintain enthusiasm for multiple such events throughout an interview season. This leaves all parties desiring a low-cost, accessible solution, such as that found with a social virtual reality (VR) platform like Mozilla Hubs.

Social VR platforms are an emerging, diverse group of multiuser online applications that enable people to communicate and collaborate in a common virtual space (8,9). These platforms vary widely in their target use-cases, design, and functionality. For our purposes, Mozilla Hubs integrated a number of features that addressed the limitations of CVCS and provided an opportunity to innovate. Users enter a private VR “Meeting Hall” designed and decorated with photographs of the radiology facilities, resident social events, and clickable hyperlinks to items of interest, such as the residency website and social media profiles. Users are embodied by 3D avatars that they use to move around the VR Meeting Hall. Uniquely, Mozilla Hubs features “spatialized audio” which gives users the impression of a sound source within the 3D environment, increasing or decreasing users’ speech volume depending on their distance from each other. This feature allows for dynamic conversations that evolve as users come and go while exploring the VR environment and permits the candid, natural conversations with residents that are critical in a medical student's assessment of a residency program.

Here we describe our experience using the social VR platform Mozilla Hubs for the pre-interview social during the 2020-2021 radiology residency virtual recruitment season, provide results of a survey sent to assess applicants’ attitudes towards the VR pre-interview social, and outline additional use-cases for the emerging technology.

METHODS

A VR Meeting Hall dedicated to the pre-interview social was designed by the residency program chief residents in Mozilla Hubs using Mozilla's open-source Hubs environment creation platform, Spoke (10). Of the many premade environments offered on Spoke, “Gathering Hall” was selected for its open format, clean aesthetic, and because it featured multiple rooms which could be designed to present different elements of the residency program (Fig 1 ). Using this prefabricated environment, the walls and spaces were outfitted with photographs of the city and medical center. Special themed rooms were created with a focus on “Resident Life,” displaying photographs of residents together outside of the workplace, and the “Residency Program,” with photographs of residency leadership and links to the website and social media pages (Fig 2 ).

Figure 1

Bird's eye view of the Mozilla Hubs VR Meeting Hall designed for the pre-interview social from the (a) isometric and (b) top-down perspectives. (Color version of figure is available online.)

Figure 2

Users’ perspective of the VR Meeting Hall designed for the pre-interview social. (a) View of the welcome area, where users logging in to the pre-interview social would appear and be greeted by residents. (b) View of the Main Room, featuring a welcome banner positioned over a 3D model of a city skyline, images of the medical center, and entrances to the Resident Life and Residency Program rooms. (c) View of the Resident Life room, featuring photographs of resident social events, graduation day ceremonies, extracurricular gatherings, and a map of the city with clinical sites annotated for ease of reference. (d) View of the Residency Program room, featuring the names and photographs of the residency program leadership and clickable links to the residency program website and social media pages. (Color version of figure is available online.)

Upon accepting their invitation to interview at our program, applicants were sent a notice that the pre-interview social would be taking place in Mozilla Hubs. On the morning of the event (one day prior to their interview day), applicants were sent an email reminding them of the Mozilla Hubs pre-interview social that would be taking place that evening. This email quickly reviewed the features of Mozilla Hubs, our rationale for using it, and a time-limited link to the private Mozilla Hubs room, to be accessed that evening for the event. Additionally, this email contained a “Mozilla Hubs How-To” page—a detailed set of instructions for setting up a Mozilla Hubs account, creating and customizing a 3D avatar, and verifying that Mozilla Hubs had the necessary access to their device's microphone and speakers to allow for a proper VR Meeting Hall experience (supplemental Figure 1).

The Mozilla Hubs pre-interview social was scheduled for 1 hour and 15 minutes. All sessions were attended by 8-10 applicants and 9-14 residents. The first 10 minutes allowed time for all attendees to arrive and familiarize themselves with navigating the VR Meeting Hall. After 10 minutes, a chief resident delivered a formal welcome by live streaming their web camera into the Main Room, thanking the applicants for joining and encouraging them to use the event to learn about the program. Following the welcome, all attendees were allowed to resume exploring and conversing. After 1 hour and 5 minutes, with 10 minutes remaining, a chief resident gathered everyone together and broadcasted a trivia session to share facts about the institution and city, after which the session was formally concluded (Fig 3 ).

Figure 3

Fig. 3. Example images of the VR Meeting Hall during use (usernames blurred for anonymity). (a) Multiple small groups of 2-3 users each forming and conversing in the welcome area as applicants log in and residents greet them. (b) A large group of users forming a circle in the Main Room, having coalesced from smaller substituent groups to discuss of topic of collective interest. (c) Users participating in the Trivia session in the Main Room. Note that all users are looking toward the large display temporarily positioned in the Main Room for all to see and follow along. (Color version of figure is available online.)

To assess applicants’ impressions of the Mozilla Hubs pre-interview social, applicants were sent an optional web-based survey on Google Forms (Alphabet Inc., Mountain View, CA) via email. Applicants were assured that their responses were anonymous and their privacy would be protected by not recording the email addresses associated with each survey response. The survey was administered once and included all applicants invited to interview at the residency program. The survey was sent February 1, 2021—following the conclusion of the residency program's interview season and prior to the National Residency Matching Program's Rank Order List Certification Deadline. The survey period was open for two weeks.

Survey respondents were asked to respond to a series of eleven statements using a five-point Likert scale of perceived agreement: Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree. Statements were designed to gauge applicants’ attitudes towards the Mozilla Hubs pre-interview social and its usefulness in helping them learn about the residency program, particularly in comparison with pre-interview socials held on CVCS (Table 1 ). An optional twelfth open-response question was included in the survey to better understand what experiences and/or pieces of information applicants found most impactful during the Mozilla Hubs pre-interview social.

Table 1

Five-point Likert Scale Statements of Perceived Agreement (Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree) Designed to Gauge Applicants’ Attitudes Towards the Mozilla Hubs Pre-interview Social and its Usefulness in Helping Them Learn About the Residency Program, Particularly in Comparison with Pre-interview Socials Held on Conventional Video Conferencing Software

	Survey Question
1	My overall experience of the Mozilla Hubs pre-interview social was positive.
2	My overall experience of pre-interview socials using conventional video conferencing alternatives (e.g., Zoom, WebEx) has been positive.
3	My overall experience of the Mozilla Hubs pre-interview social was better than my experience of pre-interview socials using conventional video conferencing alternatives (e.g., Zoom, WebEx).
4	Mozilla Hubs was straightforward to use, either with or without the help of the provided “How To” instructions page.
5	Conversations during the Mozilla Hubs pre-interview social felt more natural than those during pre-interview socials using conventional video conferencing alternatives (e.g., Zoom, WebEx).
6	The Mozilla Hubs pre-interview social allowed me to better assess the culture of the residency program than did pre-interview socials using conventional video conferencing alternatives (e.g., Zoom, WebEx).
7	The Mozilla Hubs pre-interview social allowed me to ask questions I did not feel comfortable asking during pre-interview socials using conventional video conferencing alternatives (e.g., Zoom, WebEx).
8	The Mozilla Hubs pre-interview social allowed me, in general, to become more familiar with the resident(s) I interacted with than did pre-interview socials using conventional video conferencing alternatives (e.g., Zoom, WebEx).
9	The Mozilla Hubs pre-interview social allowed me to see more resident-resident interactions than did the pre-interview socials using conventional video conferencing alternatives (e.g., Zoom, WebEx).
10	The use of Mozilla Hubs for the pre-interview social reflects positively on the residency program.
11	The Mozilla Hubs pre-interview social, or any interaction that occurred during, positively impacted my decision to strongly consider the residency program.

RESULTS

Of the 120 residency applicants invited to the Mozilla Hubs pre-interview social, 111 (93%) attended. Of these, 68 (61%) participated in the anonymous survey. Survey results are tabulated in Table 2 .

Table 2

Applicants’ Perceived Agreement with Statements Regarding the Mozilla Hubs Pre-interview Social in Comparison with Pre-interview Socials Held on Conventional Video Conferencing Software

	Strongly Agree	Agree	Neutral	Disagree	Strongly Disagree
Survey Question	Sum Agree, n (%)		n (%)	Sum Disagree, n (%)
1	30	28	8	0	2
	58 (85%)		8 (12%)	2 (2.9%)
2	14	37	11	6	0
	51 (75%)		11 (16%)	6 (8.8%)
3	28	19	14	5	2
	47 (69%)		14 (21%)	7 (11%)
4	16	35	14	3	0
	51 (75%)		14 (21%)	3 (4.4%)
5	28	20	12	7	1
	48 (71%)		12 (18%)	8 (12%)
6	13	28	17	10	0
	41 (60%)		17 (25%)	10 (15%)
7	15	14	26	12	1
	29 (43%)		26 (38%)	13 (19%)
8	22	23	17	6	0
	45 (66%)		17 (25%)	6 (8.8%)
9	15	19	18	13	3
	34 (50%)		18 (26%)	16 (24%)
10	34	23	10	1	0
	57 (84%)		10 (15%)	1 (1.5%)
11	19	30	17	1	1
	49 (72%)		17 (25%)	2 (2.9%)

Applicants reported positive experiences with the Mozilla Hubs pre-interview social (58/68, 85%) and pre-interview socials using CVCS (51/68, 75%). Three percent (2/68) of applicants reported negative experiences with Mozilla Hubs and 9% (6/68) reported negative experiences with CVCS. Most applicants reported a better overall experience with Mozilla Hubs compared to CVCS (47/68, 69%), with 10% (7/68) reporting a worse overall experience, and 21% (14/68) neutral. Seventy-five percent (51/68) of applicants reported finding Mozilla Hubs technically straightforward to use.

Most applicants reported the Mozilla Hubs pre-interview social allowed them to better assess residency culture than did pre-interview socials using CVCS (41/68, 60%), with 15% (10/68) disagreeing, and 25% (17/68) neutral. Applicants reported that on Mozilla Hubs, conversations felt more natural (48/68, 71%) than on CVCS and that they were able to become more familiar with the residents with whom they interacted (45/68, 66%). Forty-three percent (29/68) of applicants reported that the Mozilla Hubs pre-interview social allowed them to ask questions they did not feel comfortable asking on CVCS pre-interview socials. Fifty percent (34/68) of applicants reported that Mozilla Hubs pre-interview social allowed them to see more resident-resident interactions than did CVCS pre-interview socials.

Eighty-four percent (57/68) of applicants agreed that the use of Mozilla Hubs for the pre-interview social reflected positively on the residency program, with 1.5% (1/68) disagreeing, and 15% (10/68) neutral. Seventy-two percent of applicants reported that the Mozilla Hubs pre-interview social, or any interaction that occurred during the social, positively impacted their decision to strongly consider the residency program (49/68), with 3% (2/68) disagreeing.

Twenty-nine percent (20/68) of applicants responded to the optional twelfth open-response survey question. Two responses mention benefiting from intermittent opportunities for one-on-one conversations with residents. Four responses mentioned technical difficulties diminishing the experience of the Mozilla Hubs pre-interview social, with three referring primarily to difficulty hearing other users due to “background noise” from the other conversations taking place elsewhere in the VR Meeting Hall.

DISCUSSION

This survey of applicants to a radiology residency program during the 2020-2021 residency application season, amidst the Covid-19 pandemic, demonstrated a general preference for a pre-interview social hosted on a social VR platform, Mozilla Hubs, as compared to CVCS. The majority of applicants reported they were able to assess residency culture better on social VR than on CVCS, that the use of a social VR platform reflected positively on the residency, and that the pre-interview social positively impacted their decision to strongly consider the program.

The pre-interview social is an important part of the residency interview season. Beyond official hospital department webpages and promotional media, the pre-interview social provides applicants the opportunity to interact with the living embodiment of the residency—the residents. This interaction is critical for assessing the social environment of the program and gauging whether the current residents’ attitudes toward the program correspond with those portrayed in official correspondence from the program leadership. Applicants can assess whether current residents’ learning styles, career goals, and overall professional ideals mirror their own. Applicants can probe at a residency program's commitment to goals like diversity and resident wellness, and seek data on how the program has handled events such as residents starting families. These types of conversations are fundamental to applicants assessing the “feel” of a program—that nebulous quality that many agree is paramount in making an informed choice in selecting a residency program and has largely been the aim of on-site residency interviews. However, the Covid-19 pandemic and remote residency application season, hosted overwhelmingly on CVCS, has severely limited applicants’ opportunity to have these meaningful exchanges.

CVCS, designed and effective for one-way communication and presentations, is not equipped to facilitate dynamic conversations for larger groups. More recent features added to some CVCS platforms, such as “break out rooms,” permit larger groups to divide into smaller, more intimate groups. However, a “moderator” it still required to make decisions on who is paired with whom, introducing artificial starts and stops in conversation and the same limiting format.

Social VR platforms, such as Mozilla Hubs, are an emerging, diverse group of multiuser online applications that are collectively exploring the frontiers of communication in mixed/VR, particularly with the use of a VR headset. (11) Such platforms vary widely in their target use-cases, design themes, aesthetic, and functionality, and the degree of freedom they grant their users. For our purposes, Mozilla Hubs integrated a number of features that addressed the limitations of CVCS and provided opportunity to innovate. Fully customizable virtual environments, such as our “VR Meeting Hall,” decorated with photographs of the residency's home city, clinical sites, reading rooms, conference halls, and workstations recovered some of what was lost by applicants not being able to visit physically. Individual rooms in the VR Meeting Hall were themed to feature resident life and wellness and social events. And beyond two-dimensional photographs, the space was decorated with clickable hyperlinks to the residency program webpage, social media pages, and 360° photos of points of interest for the residency program.

Custom 3D avatar creation allows for applicants to simply show their professional headshot, display some of their personal taste by selecting from hundreds of stock 3D avatars, or exhibit some creativity by creating one of their own. In our case, applicants were sent a guide for how to create a basic avatar that featured their ERAS application photograph as a part of the “Mozilla Hubs How-To” instructions page. Some applicants chose to go above and beyond by further personalizing their avatars with pictures of themselves taken from all angles. Other applicants chose one of many premade avatars without featuring their ERAS photograph, granting a degree of privacy that has not previously been possible during in-person gatherings. New users opting for a premade avatar can click on the link for our VR Meeting Hall and be participating in under one minute.

Foremost among Mozilla Hubs’ features that made it appealing for the pre-interview social was its “spatialized audio.” This gives users the impression of a sound source within the 3D environment, with each users’ audio arising from within their 3D avatar. This feature increases or decreases users’ speech volume depending on their distance from each other, allowing for dynamic conversations that evolve as users come and go exploring the VR environment, just as would happen in-person. For example, one applicant emphasized in their free response that they were strongly impacted by “the willingness for residents to approach you and start a conversation and answer questions. In a video format, usually everyone waits to see if someone has questions,” drawing a contrast with the level of engagement that is possible on CVCS. And while we were drawn to the way the spatialized audio feature would facilitate dynamic small-group conversations, two applicants specifically mentioned that it was the intermittent one-on-one conversations that would occur as groups formed and dissolved that made them “more sure I would fit in well with the residents.”

Additional features include the ability to stream one's web camera into the VR environment, which was used for a formal welcome to the applicants at the beginning of the social, as well as stream any internet tab or computer application into the environment, which was used to host a brief trivia session, sharing fun facts about the history of the city, hospital, and residency program. Regarding security, Mozilla Hubs allows the host to create private links that can be distributed at will and revoked when necessary, so that users cannot access the VR environment off schedule. Importantly, we found Mozilla Hubs to be the most forgiving social VR platform with regards to its technical requirements—compatible with most devices (e.g., smartphones, tablets, laptops, and VR headsets), accessible over most internet browsers, and free to use. For those interested in creating their own virtual environments, Mozilla's free 3D scene-editing platform, Spoke, has an intuitive user interface allowing those with no prior 3D modeling experience to build their own Hubs environments.

Social VR platforms are an emerging media and its applications will only continue to grow. Pending and future directions for our use of social VR platforms such as Mozilla Hubs include the creation of new VR Meeting Halls for the fellowship interview season, guided and independent virtual tours of our department and clinical sites for remote visiting professors and Grands Rounds lecturers, and designing VR rooms to share “A Day in the Life of a Radiologist” for medical students unable to shadow or complete an away rotation due to social distancing policies. With the future of in-person residency recruitment being hotly debated and massive forecasted growth in the mixed/VR industry, social VR platforms may represent an approaching new normal in residency recruitment (12,13).

The VR pre-interview social was not completely free of technical issues. While Mozilla Hubs is one of the most forgiving social VR platforms with respect to its technical requirements, there were rare applicants who encountered technical limitations that could not be overcome. We attempted to minimize this by providing thorough “How-To Mozilla Hubs” instructions, easing the Mozilla Hubs account creation, 3D avatar design, and device preparation steps. However, the myriad variables at play in how any one user interfaces with the internet (e.g., device hardware, device software, internet browser, internet router and firewall, data limits, etc.) introduces a level of complexity that at times cannot be troubleshooted without special expertise and prevented the rare applicant from participating in the pre-interview social. For those who were able to participate, the most common complaint was the presence of background noise arising from conversations happening elsewhere in the VR Meeting Hall occasionally preventing users from hearing others close to them. This was primarily an issue with some users having poorly placed microphones—making their in-Hubs voices relatively quiet—however this may represent a point of improvement for the Mozilla Hubs’ “spatialized audio” feature or could perhaps be fixed by giving hosts the ability to adjust the strength of the “spatialized audio” feature at will in the Room Settings menu.

There are limitations in the design and implementation of the survey. Likert scales are subject to acquiescence bias, in which respondents tend to choose positive options, and social desirability bias, in which respondents tend to report views that will be regarded favorably by others (14). These biases may have been compounded by the timing of the survey—held before the National Residency Matching Program Rank Order List deadline—as applicants may have been hesitant to report anything negative about a residency program despite our efforts to assure them of their anonymity.

CONCLUSION

Radiology residency applicants overall preferred a pre-interview social hosted on a social VR platform, Mozilla Hubs, compared to those hosted on CVCS. Due in part to its spatialized audio feature, applicants reported that their conversations felt more natural, that they became better acquainted with the residents they met, and that they were better able to assess residency culture. These factors led to applicants reporting the use of a social VR platform reflected positively on the residency and positively impacted their decision to strongly consider the program. Being free to use and with a relatively low barrier to entry from the perspective of technological savvy, coupled with the future of exclusively on-site residency interviews in question, residency programs should consider incorporating social VR platforms such as Mozilla Hubs into their suite of interview season offerings.

Sources of Support

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

IRB Statement

The work was not human subjects research and thus did not required institutional review board approval.