Virtual grand rounds as a novel means for applicants and programs to connect in the era of COVID-19.

BACKGROUND: COVID-19 has disrupted the 2020-2021 residency application cycle with the cancellation of away rotations and in-person interviews. This study seeks to investigate the feasibility and utility of video conferencing technology (VCT) as an opportunity for applicants to interact with faculty from outside programs.
METHODS: 18 prospective urology applicants were randomized to 6 urology programs to give a virtual grand rounds (VGR) talk. Presentations were recorded and analyzed to determine audience engagement. Students were surveyed regarding perceived utility of VGR. Faculty were surveyed to determine system usability of VCT and ability to evaluate the applicant.
RESULTS: 17 students completed the survey, reporting a 100% satisfaction rate with VGR. A majority felt this was a useful way to learn about outside programs. 85 physicians completed the faculty survey, with nearly half feeling confident in their ability to evaluate the applicant. Video transcription data shows sessions were interactive with minimal distractions.
CONCLUSIONS: VGR can be a useful means for medical students to express interest in programs as well as an additional marker for faculty to evaluate applicants.

Introduction

COVID-19 has caused historic changes to medical education and the 2020–2021 application cycle. The Coalition for Physician Accountability, which is comprised of the Association of American Medical Colleges (AAMC), Accreditation Council for Graduate Medical Education (ACGME), and other governing and licensing bodies for medical education, officially recommended on May 11, 2020 that in-person away rotations and sub-internships be cancelled for the 2020–2021 application cycle, with rare exceptions. Furthermore, the group recommended interviews be conducted virtually. Specific to urology, the Society of Academic Urologists (SAU) reaffirmed these guidelines on May 22, 2020.

Given these recommendations, programs and applicants have explored innovative ways to connect. Many programs have offered virtual open houses and town halls for students to interact with residents and learn more about programs. Social media, particularly Twitter and Instagram, have been used as a medium to interact with applicants and to highlight programs. However, in the absence of in-person sub-internships and away programming, it is difficult for applicants to express interest and programs to learn about the applicants, resulting in considerable uncertainty for both applicants and programs.

The SAU has recommended virtual sub-internships as a way for applicants to gain exposure at outside programs. However, implementation can be difficult depending on individual institutional support and resources. It is currently unknown what proportion of programs are planning to offer virtual sub-internships. Furthermore, due to pauses in clinical rotations starting in March 2020, many students are balancing graduation requirements with scheduling virtual clerkships. Virtual grand rounds (VGR) offers an alternative, easy to implement option and requires a limited time commitment for both students and programs. Students can demonstrate their urologic knowledge, presentation skills, and ability to respond to questions while interacting with faculty and residents at specific programs. This study aims to investigate the feasibility and utility of video conferencing technology (VCT) in the evaluation of urology applicants.

Methods

This study was determined to be exempt by the Institutional Review Board of the University of Maryland, Baltimore (HP-00091215).

Overview

18 prospective urology applicants from 5 medical schools were randomized to 6 different urology programs to participate in VGR. Programs were geographically diverse and ranged in size from 1 to 5 residents per class. Students were given the option to rank their top programs and were randomized within their top three whenever possible. Applicants were asked to prepare a 10-minute presentation on a topic of their choice and required to download the appropriate VCT based upon program. VGR sessions were audio and video recorded using Zoom or WebEx video conferencing platform. Presentations occurred during scheduled grand rounds at the included institutions and all faculty and residents were encouraged to attend. Programs could host multiple students at one session or individual students at multiple sessions. No information, personal or academic, was forwarded to the urology programs. Following the sessions, a survey was administered to prospective applicants and a separate survey was given to all attendings, fellows, and residents attending VGR.

Survey design

The student survey was designed to assess student concerns about urology applications due to COVID-19 and satisfaction with VGR using a 5-point Likert scale. The faculty and resident survey was designed to evaluate system usability of the video conferencing platform and ability for program personnel to evaluate applicants. The system usability section was adapted from the system usability scale, which is an abbreviated questionnaire utilized to measure perceptions of usability. , Specific measures to evaluate the applicants were taken from ACGME core competencies. ,

Data analysis

Survey analysis was done via Google forms, and Microsoft Excel was used to create descriptive statistics.

Recordings were reviewed by two independent graders (DA and JK), who watched VGR footage, noting questions and phrases pertinent to the discussion, and categorized them into 4 broad themes of questioning, encouraging, commenting, and explicit distractibility, using an adapted version of a scale used to evaluate video based coaching. Questioning included both open and closed-ended questions, encouraging involved general confirmation or specific positive and negative feedback, and commenting focused on any statements extraneous to the discussion at hand. Explicit distractibility was noted by any obvious or visible distractions during the recording. This was further divided into three categories of external disruptions such as technical difficulties, phone usage as visible on the recording screen, and completion of other tasks while on camera. Evidence of active listening, noted by the mention of specific details from the student’s presentation in comments and questions, were recorded. Host program personnel who contributed to the discussion were quantified as number of responders. Initial intergrader concordance was 91.3%, and any discrepancies were reconciled after discussion and repeat evaluation of video recordings.

Results

Student survey demographics

18 prospective applicants were randomized for VGR at 6 different urology programs between May 6, 2020 and May 27, 2020. 10 (55.6%) were male, 8 (44.4%) were female. All student data came from completed surveys (response rate 94.4%). All students had a home urology program, but only 9 (50%) had completed a rotation in urology. All students were planning on doing at least one away rotation in urology. 16/17 (94.4%) students reported feeling very or fairly stressed or anxious due to the upcoming application cycle as well as the perceived impact of being unable to do away rotations on their application.

Student feedback on VGR

Results of the student survey are reported in Fig. 1 . All students reported satisfaction with VGR. 11 (64.7%) reported feeling very or fairly confident in establishing rapport with the program personnel. When asked if the virtual experience could replace traditional student-faculty interactions, 9 (52.9%) strongly or somewhat agreed, 3 (17.6%) were neutral, and 5 (29.4%) either somewhat or strongly disagreed. When asked about the utility of VGR as a medium to learn about another program, 14 (82.4%) responded that it was either very or fairly useful.

Fig. 1

Student survey responses after participating in VGR (n = 17). Surveys assessed student concerns about urology applications due to COVID-19 and satisfaction with VGR using a 5-point Likert scale.

Qualitative student feedback

Selected positive feedback from the open-ended student survey questions fell under two common themes: learning about another program and having the opportunity to display their knowledge. Some direct quotes included, “Useful to learn about another program but doesn’t replace in-person interviews and visiting the city/hospital”, “Incredible opportunity. Appreciated the chance to display my knowledge”, "Great opportunity to showcase to another institution. Nice to chat with faculty members. Overall, positive experience. I would be happy to participate again”, and “I really loved being able to present and get exposure to another program!”

Negative feedback included the desire for a more standardized way of learning about the program and would have appreciated if the faculty and residents introduced themselves prior to the VGR. Selected quotes include, “Great exposure... Would be nice to hear from faculty or residents about program”, “Good overall. Would like dedicated time to chat with residents about the program” and “Difficult to read the room over zoom. Interactive moments fell flat over the computer”, “Would have been nice if people showed video and introduced themselves” and “Would have liked residents/attendings to keep cameras on to make it more engaging.”

Video transcription data

Video transcription and coding was completed on 17 student presentations. 1 student video was unable to be analyzed due to technical difficulties. Results are shown below in Table 1 . Student presentation lengths averaged 10–12 min and discussions following these presentations averaged 5–6 min. Each student received input from around 3–4 responders, with an average of 4 questions and 5–6 comments of feedback. Feedback included general confirmations and specific positive and negative commentary. Evidence of active listening ranged from 0 to 3 occurrences per session with an average of 1.29.

Table 1

Video transcription data.

	Average time (mins)	Range (mins)
Student Presentation Time	11.51	7.8–19.5
Discussion Time	5.86	2.7–13.1
	Average number per presentation	Range
Number of responders	3.4	2–8
	Average number of occurrences per presentation	Range
Questions	3.8	1–10
Closed	1	1–3
Open	2.9	1–7
Feedback	5.71	1–12
Active Listening	1.29	0–3
Explicit Distractibility	1.00	0–3
External Disruptions	0.47	0–3
Phone Usage	0.23	0–3
Doing Other Tasks	0.29	0–2

Distractibility

Incidences of explicit distractibility averaged ∼1 per student discussion. Table 1 shows the distribution, which was divided between external disruptions such as technical difficulties (47%), phone usage (23%), and completion of other tasks by faculty members (29%).

Breakdown by program

Data was separated by host program labeled A-F in Fig. 2 . The number of responders between programs ranged from 2 to 5, with a median of 10.5 questions and/or comments by program faculty, fellows and residents. Between programs, explicit distractibility had a median of 0.5 incidences/presentation, with an outlier of 3.33 in program A.

Fig. 2

Breakdown of video transcription data by average number of responders, average number of occurrences for questions and feedback, as well as explicit distractibility for participating programs A-F. Programs A, C, D, E, and F all had 3 student presenters, program B had 2 students.

Faculty survey demographics

Of the 6 programs that hosted VGR, there were 85 survey responses (71.7% attending physicians, 28.3% residents and fellows). Because some programs hosted talks on multiple dates, some respondents may have completed the survey multiple times. The majority (98.8%) of faculty did not have any prior knowledge of the presenting applicant. Respondents included 69.4% males and 27.1% females, with 3.5% declining to answer. Median age was between 40 and 49. Median years out of training was between 0 and 9 years. 75 (88.2%) reported using VCT at least more than once per week, and 94.1% found the software easy to use. Nearly all (96.4%) had expectations that others should find VCT easy to use. Most (58.8%) either strongly agreed or agreed that a virtual meeting is just as comfortable to participate in as a traditional in-person meeting (16.5% were neutral, 23.5% disagreed, and 1.2% strongly disagreed).

Faculty feedback on VGR

After VGR, 70.3% of faculty reported confidence that they gained knowledge of the applicant, with 36.9% feeling confident in their ability to establish rapport with the applicant. Nearly half (47.6%) felt fairly confident with their ability to evaluate the applicant. When broken down into the specific assessments of ACGME core competencies, 41.7% were confident in assessing medical knowledge, 71% were confident in assessing communication with other healthcare professionals, 40.5% were confident in assessing communication with patients and family members, 15.8% were confident in assessing compassion and respect for others, 16.7% were confident in assessing cultural sensitivity, and 36.9% were confident in assessing the applicant’s ability to identify his/her own strengths and weaknesses.

When asked what the likelihood they would invite the applicant for an interview based on VGR, 67.9% had decided that they would or would not invite the candidate (17.9% highly likely, 45.2% likely, 4.8% unlikely vs 32.1% undecided). Fig. 3 includes a complete list of survey responses.

Fig. 3

Faculty and resident survey responses after participating in VGR (n = 85). Surveys assessed respondent confidence in utilizing VCT as well as ability to evaluate the applicant using a 5-point Likert scale.

Discussion

COVID-19 has drastically changed residency applications for medical students across the United States. Traditional methods for medical students to gain urologic knowledge and experience, learn about other programs, and express interest in urology have been limited due to efforts to reduce non-essential travel and spread of the virus. , Many students expressed anxiety about the impact of COVID-19 on their applications. While not a replacement for in-person experiences, virtual experiences provide students and programs with some insight and may serve as another data point in the residency interview selection process for both the applicant and the program. Student-led grand rounds have previously been utilized as a means for students to demonstrate their knowledge with high satisfaction. This study showed similar rates of satisfaction despite grand rounds being virtual.

Video transcription data reveals interactive sessions with questions and feedback from an average of 3–4 responders. Questions also showed evidence of active listening and engagement on behalf of faculty. Minimal distractions were also noted, with approximately 1 incidence per student session.

Sessions were compared across programs to determine generalizability of VGR as a platform for future urology applicants to interact with faculty from outside programs. Overall, data was similar across the 6 programs as demonstrated in Fig. 2. This indicates that levels of participation were comparable and suggests that results are generalizable across programs. Demographics of faculty and residents surveyed at the 6 programs were fairly representative of demographics across urology programs in the US. Programs were also located in geographically diverse regions. The greatest variability was in explicit distractibility; however, not all participants had their cameras on and only explicit distractions were able to be measured, limiting ability to count and compare distractions.

Most faculty and residents felt comfortable using Zoom or WebEx as a virtual platform. With the cancellation of in-person conferences and the movement of education to virtual platforms, many have become proficient in using VCT. While a minority (25%) reported not feeling as comfortable participating in VGR as compared to a live grand rounds conference, almost 60% reported confidence in engaging virtually. We found an average of 3–4 unique responders per student session and 70% of survey respondents felt as though the video session improved their knowledge of the applicant. Interestingly, there was a discrepancy in perceived ability to establish rapport. Nearly 65% of students felt confident in their ability to establish rapport with program personnel through virtual means. However, less than 40% of attendings and residents agreed that they were able to establish rapport with the applicant.

Applicants were evaluated based upon ACGME core competencies for residents. Given that the ACGME has moved towards competency-based education and assessment through the Milestones program, it is reasonable for programs to evaluate prospective candidates using these methods. In general, the majority of faculty and residents felt comfortable assessing the applicant’s communication with health professionals after the session. However, less than half felt confident in assessing applicants’ communication with patients and family members as well as applicants’ medical knowledge and ability to identify their own strengths and weaknesses. Even fewer felt comfortable assessing compassion and respect for others and cultural sensitivity. While VGR can be helpful in assessing certain aspects of the core competencies, others may be difficult to glean in this setting. Overall, nearly half of faculty and residents felt comfortable assessing the applicant after VGR. When evaluating comfort by rank of participant, residents and fellows felt significantly less comfortable assessing the applicant compared to attending physicians (29.2% vs 55%). It is possible that attending physicians have greater experience assessing applicants and may feel more comfortable making a decision based upon a virtual interaction.

Feedback collected from both student and faculty surveys generally revealed positive comments. Many agreed that this was a valuable experience for students and faculty to interact with and learn more about each other. Negative comments or challenges centered around the lack of standardization surrounding the structure of VGR. For example, multiple comments specifically addressed that not all participants joined with video, which limited participation. This could be addressed by outlining an agenda for each talk to maximize participant engagement.

This study did have several limitations. One of the largest limitations was that VGR was not standardized and there was high variability between how talks were structured. Some faculty included a presentation about their program or a session after the talks where applicants were encouraged to interact with residents to learn more about the program. Other programs limited the interaction to the talk itself. This lack of standardization may have affected how some faculty and residents perceived the students and vice versa. However, it is likely that even with standardization and set agendas, there will be variation in how programs decide to structure their talks. Furthermore, VGR participants were not required to have their cameras on, which may have affected level of interaction and video transcription data. As we measured explicit distractibility through observation of on-camera actions, this was unable to recorded and analyzed for those who joined without video. Another limitation of this study is related to technical difficulties. Several participants experienced technical difficulties, leading to poor quality of video, inability to share their presentations, and one talk not being audio recorded and analyzed.

Student and faculty surveys were not piloted prior to dissemination, so it is possible that participants may have interpreted and responded to questions in a manner not intended by survey authors. Although there is risk for potential bias in surveying students after VGR, we believe perceived impact of survey completion and responses on matching at specific programs was low. Applicants were contacted for survey completion by medical student investigators and reminded that their participation in this study was completely voluntary and that survey responses were anonymous. Survey responses were aggregated prior to presentation of data, further reducing the ability to identify specific students and responses.

Prior to COVID-19, several studies investigated feasibility of virtual residency interviews to reduce costs and time burden for both applications and programs.12, 13, 14 Residency applications and interviews require significant time and resource commitment, with many students making interview decisions based upon limitations in time and money. VGR can offer an opportunity for students to explore and make connections at more geographically diverse programs that previously would have been time and cost-prohibitive. Given its relative ease of implementation, programs may also utilize VGR to gain additional knowledge about a greater number of qualified candidates.

Conclusion

In the era of COVID-19, residency programs and applicants are turning to virtual means to connect. VGR are a relatively low time commitment option for students to express interest in specific programs and for programs to implement. Even once travel restrictions are lifted, VGR may continue to be a useful means for students to explore more geographically diverse programs. Most participants felt comfortable using a virtual platform such as Zoom or WebEx to interact. VGR were interactive with minimal distractions. The majority of students were satisfied with VGR and felt as though it would be a useful addition to their residency application. Based solely on VGR, 70% of the faculty had determined the likelihood of inviting or not inviting an applicant for an interview, demonstrating that VGR may help faculty gain insight into applicants. VGR has the potential for applicants to demonstrate interest in different urology programs across the US and may be helpful to urology programs in deciding which applicants to interview.