Conducting a synchronous virtual multiple mini-interview using Webex for medical school admissions.

COVID-19 pandemic restrictions abruptly changed the way interviews for medical school admissions have been conducted. This study is unique as it highlights the first successful virtual synchronous multiple mini interview (MMI) in Canada. Our low technical incident rate, troubleshooting strategies and approach may reassure other medical schools considering conducting a virtual MMI. Success was achieved with collaboration, a strong organizational and communication strategy, learning along the way and a priori contingency plans. Virtual interviewing in academic medicine is likely here to stay, and future work to highlight the impact on applicants will help to build on the diversity mission in undergraduate medicine admissions.

Introduction

The University of Saskatchewan College of Medicine (U of S COM) uses the multiple mini interview (MMI)[1] to assess its applicants. Prior to COVID-19 restrictions,[2] virtual interviewing in academic medicine was rare, occurring in select post-graduate programs.[3] The purpose of this paper is to highlight the methodology used by U of S COM to conduct a successful synchronous virtual MMI in their medical school admission process, the first school in Canada to do so, with the goal of having other schools learning from its approach.

Innovation

A Plan-Build-Test-Deploy method was used to ensure that the videoconferencing platform, Webex, was adequate to support the planned virtual MMI. Given the short timeline in pivoting to the virtual MMI based on COVID-19 restrictions and concerns, Webex was chosen because it was the only supported platform available for use by the U o f S COM and it was familiar to our IT personnel. Due to time constraints, a pilot interview was not conducted. Pertinent information about the virtual MMI was emailed to applicants and assessors. Interviews were accessed through meeting codes given to applicants prior to the interview and the waiting room function ensured only one applicant was in an interview at any time. A buzzer system was used for timing of the MMI stations. Interview scenarios were presented using placards through the videoconferencing camera during the three-minute reading time. Each interview lasted ten minutes, then applicants entered another meeting code to complete their next interview, and this pattern was repeated for all four interviews.

Utilizing strict COVID-19 safety protocols, assessors (n = 36) and staff (n = 20) were on site while applicants (n = 288) connected virtually. Tactics utilized included: cohorting personnel, private rooms for assessors, reducing the number of stations and building in flex time, having a strong communication strategy with training materials and testing platforms, and contingency plans such as using WhatsApp for connectivity issues. This study was granted an exemption by the University of Saskatchewan Behavioural Research Ethic Board as per Article 2.1 of the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans – TCPS 2 (2018).

Outcomes

Synchronous mini-interviews (n = 1152) were conducted over two days. The technical incident rate was 3.2% (n = 37) with a declining trend indicating ongoing improvements (Table 1). All incidences were resolved through our contingency plans and applicants experiencing a technical incident were not negatively disadvantages after review of their scores.

Table 1

Technical incident report with resolutions during MMI

Technical Incident	Saturday am(n = 72)	Saturday pm(n = 72)	Sunday am(n = 72)	Sunday pm(n = 72)	Resolution
Virtual Connection Issues (Audio/Visual/Both)	6 (2.1%)	5 (1.7%)	6 (2.1%)	5 (1.7%)	Transition interview to WhatsApp (15)Disconnect and reconnect (4)Interview moved to end of circuit (3)
Timing of Interviews (initiated early or delayed)	7 (2.4%)	3 (1.0%)	0	1 (0.3%)	Delaying entire circuit (5)Specific timing adjustments off the main circuit (4)Communicating buzzer procedures with assessor (1)Communicating logging in procedure with assessor (1)
Applicant Instruction Issues (Applicants attempting to enter into wrong interview at the wrong time)	3 (1.0%)	1 (0.3%)	0	0	Applicants not admitted into the interview by assessor and communicating with applicants over phone to remind them of their passcodes and the order of their stations. Sent additional communication to remaining applicants to make them aware of this issue.

Overall, 95.7% (n = 66) of surveyed assessors (95.8% response rate) strongly agreed or agreed that they were provided enough information to adequately assess the applicants and 81.5% (n = 208) of surveyed applicants (88.5% response rate) strongly agreed or agreed that the virtual method allowed them to adequately communicate with the assessor.

The G coefficient (reliability of applicant scores) of the virtual MMI was acceptable at 0.61 and comparable to previous MMI administrations (2019 = 0.68, 2018 = 0.76, 2017 = 0.68).

Compared to our traditional MMI, the virtual format was approximately one third the cost to the U of S COM and also saved applicants the cost of travel and accommodations.

Next steps

Virtual interviewing has many advantages, including increased convenience, flexibility and decreasing the need for travel, which may lead to improved access to a broader and more diverse applicant pool while decreasing the costs for all.[4],[5] Although we used Webex, other similar software applications would likely also be successful. Improving access and decreasing costs are key when addressing equity and enhancing diversity in admissions.[6] Further study is needed to assess the impact of virtual interviewing on student selection and its potential role in decreasing the barriers of medical school admissions.