Adaptation of internet-based multiple mini-interviews in a limited-resource medical school during the coronavirus disease 2019 pandemic.

The coronavirus disease 2019 pandemic has affected many aspects of life, including medical education. The selection of students using multiple mini-interviews (MMI) should be adapted promptly as internet-based MMI (iMMI) in the current setting. The current transition from MMI to iMMI is supported by experience in conducting conventional MMI in previous years; hence, the station and assessor preparation could be completed without significant issues. The greater challenges to the use of iMMI are due to technical issues, including poor internet connections and adaptations necessary for assessing candidates' nonverbal expressions. This paper aims to describe the experience of conducting iMMI in the current pandemic situation, with further highlights on contextual challenges in a limited-resource setting.

Introduction

The coronavirus disease 2019 (COVID-19) pandemic, which began in February 2020, has necessitated adaptations in numerous aspects of life around the world, including in medical education. Medical schools should formulate and enact strategies to move learning and assessment fully online, defer other face-to-face and hands-on activities to later dates, prevent undergraduate medical students from conducting their clinical rotations (due to safety reasons), enrich the curriculum with COVID-19 subjects, and attempt to support medical and healthcare professionals working on the front line [1,2]. Amid uncertainties and challenges, especially in assuring that medical students achieve clinical competencies, medical schools must ensure that the strategies being implemented still facilitate high-quality learning and consider all stakeholders, including current and future students.

Multiple mini-interviews (MMI) have been used extensively around the world for conducting admission to various majors and levels of study [3-5]. Participants (candidates) are required to complete various activities at each station, varying from completing a certain task, conducting a role play based on a given scenario, making decisions, interacting with other participants or the interviewer, or answering questions from the interviewer as in a standard interview session [3,4,6]. In general, the reported reliability of MMI has ranged from moderate to high, with a Cronbach’s α of 0.69–0.98 and a G coefficient reaching 0.81 on generalizability analysis [4]. In the Faculty of Medicine, Universitas Indonesia, this method has been applied as part of the undergraduate international-class medical student admissions process since 2013. This method has been an integral part of the process, along with the standard admission written examination, which allows assessors to evaluate applicants’ non-cognitive attributes.

The similar concept of conducting MMI through online videoconferencing platforms was conducted at the University of Sydney in 2011, as studied by Tiller et al. [7]. No significant difference in scores equivalence was found between conventional MMI and internet-based MMI (iMMI) [7]. The reliability coefficient demonstrated by generalizability analysis also showed satisfactory results [7]. Regarding satisfaction with iMMI, both interviewers and candidates rated it positively, showing acceptability among participants. This method is also considered cost-friendly, giving institutions significant budget savings. Based on these results, iMMI is considered an important innovation that can be readily applied by other institutions [7].

In this article, we aim to share our experience in preparing and conducting our version of iMMI as part of the medical student admissions process at the Faculty of Medicine, Universitas Indonesia, during the COVID-19 pandemic. This alternative was indeed practical and necessary, especially in this pandemic situation, which requires that people minimize social interaction—conducting conventional in-person interviews is not possible. We also aim to describe challenges in conducting our version of iMMI—given limited resources—especially regarding internet bandwidth in Indonesia, compared with other countries in Asia [8].

Methods

1. Preparation

Preparations for this iMMI were made over approximately 1 week, from the decision to adopt iMMI to the day of the interviews. The preparations focused primarily on revising scenarios, preparing technical implementations, selecting and informing interviewers, and informing candidates about the change of plan.

1) Notifications

Approximately 2 days before the interview dates, notifications were sent via email and online messaging platforms to interviewers and candidates. Candidates were also contacted directly through online messaging platforms and invited to join a group for the coordination process. They were told to prepare a desktop or laptop computer with the online video-conferencing software installed and were not allowed to use phones or tablets during the iMMI. The camera angle needed to be adjusted so that the upper half of candidates could be seen on-camera during the interview. Candidates were also required to have a stable internet connection for the interview. During the application process, candidates were asked to send a photo ID with their application; this photo ID was also used to identify the candidate during the briefing and interview process. No significant problems were raised by the candidates.

Interviewers were contacted individually through online messaging platforms to obtain consent and to ensure their understanding of the iMMI process. We used an online videoconferencing platform that was quite commonly used throughout our institution during the pandemic period. To prevent confusion to interviewers and candidates, technical control of the application was largely handled by the IT team; participants only needed to join in the room and follow the instructions displayed on-screen.

2) Scenarios

Our iMMI stations consisted of four scenario stations, one writing station, and one traditional semi-structured interview station, with a total of 20 prompts—similar to our previous MMI. The stations assessed effective communication, critical thinking, ethical decision making, altruism, writing skills, and motivation. The scenarios were modified to the current COVID-19 pandemic situation, considering that face-to-face activity is not possible in this setting. Due to limited human resources, we also had to modify the effective communication station to accommodate the lack of actors. An example of the modifications made to the effective communication scenario can be seen in Table 1.

Table 1.

Example Scenario from Traditional MMI and iMMI

	MMI	iMMI
Example scenario	You are a third-year high school student who has been having trouble completing your scientific paper. The scientific paper is a pre-requisite to graduate from high school. Your supervisor has been contacting you for the past 6 months and you did not respond properly since you feel that you are really behind the schedule. You think that you will only update your supervisor when there is progress on your project. However, you have been stuck and you do not know what to do. After discussing with your academic advisor, you finally decided to come to your supervisor, and you want to explain what has been happening with your scientific paper.	You are a first-year medical student and you have a year-end project writing a thesis proposal. The thesis proposal is a pre-requisite to pass the first year of medical school. You usually meet with your supervisor every month to update her on your progress. The last meeting was conducted a month ago, just before the COVID-19 situation emerged. Due to the COVID-19 situation, this month’s meeting will be held via an online application. During the duration of your study from home, you have been having trouble concentrating and writing your thesis proposal.
	You are meeting with your supervisor to explain your reason for not updating him/her on your scientific paper for the past 6 months.	You are now going into the monthly meeting with your supervisor via the online application.
	Note: An actor will be playing the role of the supervisor.

MMI: Multiple mini-interviews, iMMI: Internet-based MMI, COVID-19: Coronavirus disease 2019.

3) Stations

Each station was set up using the break-out room feature on an online videoconferencing platform, in which each break-out room consisted of the interviewer (except in the writing station), the candidate, and a co-host displaying the scenario and countdown timer for the station. The co-host station was manned by the information and technology (IT) department, which also provided technical support as needed. Each station had an interview duration of 7 minutes, with an additional 1–2 minutes for station transition, during which the candidates were moved to another break-out room/station and the interviewers scored the candidates. All interview processes were recorded; the participation of the interviewers and candidates was considered consent for the recording. The writing station was not attended by an interviewer. To ensure that the short essay written was the candidate’s own work, in addition to recording the candidate at the writing station, we developed a form for the candidates to write their short essay, with a countdown timer in which the form closes after 7 minutes. The short essays were then assessed by faculty, who also comprised the iMMI committee.

4) Scoring

The most compatible online assessment system available to our institution at the time was our objective structured clinical examination (OSCE) scoring system, in which the candidates are scored from 0 to 3 instead of 1 to 4. We modified the scoring system to the OSCE system (0: unsatisfactory, 1: satisfactory, 2: good, 3: excellent). The global rating score was also adapted to the OSCE system, which consists of “did not pass” (previously “no”), “borderline” (previously “not sure”), “pass”, and “superior” (previously “yes”).

5) Simulation

To avoid technical problems, a simulation was conducted prior to the iMMI, using three stations. The simulation was held using the online videoconferencing platform that would be used during the actual interview (Zoom). Three faculty acted as the interviewers, three alumni acted as the candidates, and the IT department served as the co-host displaying the scenarios. The simulation was conducted in real-time, in which the candidates tried to answer the scenarios and complete the form prepared for the writing station. The candidates were also moved from one station/break-out room to the next. Potential pitfalls were identified.

Results

1. Implementation

1) Briefing

Only the event coordinator and several IT department staff were present in the faculty on the interview day. With social distancing and pandemic safety protocols in place, the event coordinator and the IT department staff accessed the interview from the faculty’s computer lab, which normally can accommodate 120 computers. Thirty computers were utilized for the co-hosts at each station, displaying the scenario and countdown timer for the station. Interviewers and interviewees accessed the meeting remotely using their personal computers.

One hour before the iMMI, two briefing sessions were conducted simultaneously for both the interviewers and candidates. The briefings were held in two break-out rooms on Zoom. The briefing explained the iMMI mechanism, stations, and timing, with an emphasis on the expectations for the candidates and the scenarios for the interviewers. During the briefing, interviewers were informed of their assigned stations and were given the password to access the stations’ scenario. They were also briefed on how to access and complete the online assessment system.

All of the interviewers had experience as an MMI interviewer; thus, they were familiar with the scenarios and prompts. Training for student assessment and the OSCE was a prerequisite for teaching staff to participate in MMI interviewer training. In our previous conventional MMI process, after the briefing and before the interview, interviewers at the same station would convene to discuss the scenario and reach an agreement on scoring the prompts. Due to time constraints, this process could not be carried out. Interviewers and candidates were directly transferred to different breakout rooms on Zoom by the IT department.

2) iMMI

Fifty candidates and 36 interviewers were involved in the iMMI, divided into two sessions, with the first and second sessions consisting of six circuits and four circuits, respectively. Each circuit consisted of six candidates and five interviewers. During the interview process, aside from Zoom, the interviewers, the MMI committee, and the IT team also coordinated through a separate online messaging platform (WhatsApp) to address any technical problems that might arise. Personnel from the IT department manned a computer lab in which the computers served as the co-host in each station. The timers on the co-host computer proved highly useful in helping the interviewer and the candidate adhere to the allocated time for each station, as there were technical difficulties in moving the candidates to the next break-out room. The differences between iMMI and MMI are summarized in Table 2.

Table 2.

The Differences between iMMI and MMI in the Current Setting

	MMI	iMMI
Preparation time	Usually, 2 weeks are allocated to simultaneously prepare and review the scenarios, notify the candidates and interviewers, and prepare the stations.	This is relatively similar to MMI, although more technical training for the assessors would be beneficial.
	MMI interviewer training is conducted at a separate time.
Scenario	Constructed with the assumption that face-to-face activity is possible, there is an actor in the station.	Modified for the COVID-19 situation, no actor is in the station. The interviewer also plays this role if necessary.
Station	There are six evaluation stations and all stations have interviewers.	There are six evaluation stations and no interviewer is present at the writing station.
Scoring	1 to 4 global rating scale of no, not sure, and yes	0 to 3 global rating scale of did not pass, borderline, pass, and superior
Simulation	No simulation was conducted as interviewers were already familiar with the conventional MMI process.	- A simulation is conducted 1 day before the interview day to prevent technical problems.
		- Only three stations are set for the simulation.
Briefing	- Face-to-face briefing	- Before the interview day, notifications are sent to interviewers and candidates via e-mail and online messaging platforms.
	- After the briefing, interviewers at the same station discuss and agree upon the scoring at the station.	- On the interview day, briefings are conducted via an online videoconferencing platform.
		- No discussion sessions between interviewers are held after the briefing.
Interview	- Walk-in interviews are conducted.	- Interviews are conducted via online videoconferencing.
	- Physical stations are used; an established objective structured clinical examination center is usually required.	- ‘Virtual stations’ are used, requiring a computer lab and the support of IT department staff.
	- Interviewers are already at each station before the candidates are allowed in.	- After briefings, interviewers and candidates are automatically pushed to different break-out rooms by the host.
	- All stations are attended by at least one interviewer.	- No interviewer is available for the writing station.
	- Upon finishing a station, 2 minutes of interval time are set for the candidates to move to the next station.	- Upon finishing a station, 2 minutes of interval time are set, but candidates are automatically pushed to the next break-out rooms.
	- Timers are announced through the speaker system.	- Timers are shown on the co-host screen.
Cost	- Requires transportation to and from the interview location for candidates, interviewers, and the committee.	- Transportation required only for the event coordinator and IT department staff.
	- Entails consumption costs for candidates, interviewers, and the committee	- No consumption costs, although the internet bandwidth used is a potential hidden cost.
	- Unit cost: IDR 623,724/candidate	- Unit cost: IDR 517,500/candidate

iMMI: Internet-based MMI, MMI: Multiple mini-interviews, COVID-19: Coronavirus disease 2019, IT: Information and technology, IDR: Indonesian rupiah.

2. Challenges

The preparation stage posed only minimal difficulties, as there was sufficient time to plan and coordinate the changes to be applied to the iMMI. In terms of cost, the iMMI required less budget allocation compared to MMI due to significant cost reduction for transportation and consumption. The implementation stage, however, posed myriad problems, such as inadequate human resources, poor internet connectivity, and the faculty’s limited technological expertise.

The briefing session for the interviewers did not begin on-time, as some interviewers experienced problems in joining Zoom. Two interviewers also were suddenly unable to participate in this interview; thus, last-minute substitutions needed to be made. Due to these disruptions, the interview session was delayed for approximately 1 hour.

The most significant problem during the interview was poor internet connectivity, both for the interviewers and the candidates. No disconnection problems occurred with the interviewers, but two candidates were disconnected completely from the beginning of the interview for more than 15 minutes, causing them to miss two stations. The assessments for these two candidates were annulled and make-up sessions were conducted on a different day with a different set of scenarios.

There were also issues regarding station change. Several times, the station change took more than 2 minutes because the IT department had to move 20–30 candidates at once to the next designated break-out room. This issue delayed the entire interview process—each session lasted for approximately 90 minutes when they should have lasted no longer than 1 hour. Although the interviews lasted longer than anticipated, these delays did not significantly disrupt the interview process.

Discussion

This paper aims to elaborate the preparation and implementation of iMMI, which was conducted in response to the needs of student selection during the COVID-19 pandemic in a medical school in Indonesia. The school has been conducting face-to-face MMI for the past 6 years. iMMI was conducted in different settings, both as the original student selection method [7] and as an adaptation during the pandemic [9]. Our report supports the feasibility of iMMI implementation, despite challenges with internet connectivity and familiarity with the use of videoconferencing technology for selection purposes. The cost efficiency of iMMI also supported the feasibility, although further technical preparations and the internet bandwidth might pose additional cost.

We learned that more detailed technical preparation and a pilot study for such a high-stakes selection assessment should be conducted. This should involve more technical training for the assessors and the IT support team to prevent delays due to unanticipated technical glitches. Overall, because Indonesia has systemic challenges in internet connectivity, the decision to conduct iMMI in our setting was subject to numerous considerations, including a contingency plan to overcome technical problems. The recommended internet bandwidth for the online videoconferencing service that we used (Zoom) is 1.5 Mbps (up/down). This bandwidth is well within the capability of Indonesia’s internet connection, although stability differs widely. In the future, the recommended bandwidth information should be sent to all candidates and interviewers, along with guidelines on how to check the internet bandwidth using a speed test, in order to minimize technical problems during the interview. We realized that, despite the technical challenges, iMMI was the best possible way to conduct MMI for student selection while still ensuring implementation of COVID-19 transmission-prevention measures [9].

The station blueprint for iMMI was the same as for the “conventional” MMI that we had conducted before the COVID-19 pandemic. This ensured that the MMIs had comparable content. Several adjustments were made for iMMI in our current setting—the interviewer also became an actor performing roleplay at the communication skills station, no interviewer/supervisor was present at the writing station, and the use of Zoom for technical meetings and the iMMI itself was optimized. The adjustments implemented did not jeopardize the validity of the iMMI stations. The total number of stations and the duration of each station were within an acceptable range of best practices for MMI [4]. Some stations also utilized the COVID-19 pandemic as contextual material, which made the candidates’ experience at the iMMI stations more relevant to the current global challenges. The assessors previously conducted conventional MMI; therefore, they were well-trained in assessing MMI. However, as mentioned previously, it became apparent that the transition to iMMI required further planning to prepare the assessors to use the iMMI system [10]. In addition to the assessors’ readiness, the iMMI requires adequate preparedness by the support and IT staff, who play significant roles in ensuring the “smoothness” of iMMI implementation. While conventional MMI also requires such technical support, the support of IT staff is more crucial for iMMI. The IT staff must be able to navigate the online platform being used and to manage hardware, network, and software issues simultaneously.

To assure the reliability of assessments in similar stations across iMMI circuits, the assessors were allowed to discuss the relevant scenario, questions, and scoring to reach a similar understanding, which is key to ensuring the reliability of the MMI [4]. One challenge during iMMI was preparing the assessors and candidates to complete the virtual interview at each station. While assessors seemed able to assess the candidates’ critical thinking, reasoning, judgments, and verbal expressions, it was quite difficult to observe non-verbal expressions and the candidates’ comfort in a new environment, which was a reported challenge in conducting iMMI [9]. Nevertheless, given evidence that MMI has good fairness and does not favor candidates with certain cultural, gender, or socio-economic backgrounds [4,6], we expect that issues regarding limited observation during iMMI did not compromise fairness.

iMMI in the current setting was necessitated by the COVID-19 pandemic, which obliges social distancing and other preventive measures; conventional MMI could not be completed. The use of iMMI was feasible and supported by adequate validity and reliability. In the future, faculty should decide whether to use iMMI or the conventional MMI format. The ultimate aim of conducting MMI as a means of selecting the best candidates should be thoroughly considered. Evidently, iMMI and conventional MMI reportedly produce comparable results and save resources [7,10]. In addition, the limitations of iMMI in assessing non-verbal expressions and stress tolerance in a new environment require further adaptations [9]. The adaptation level of assessors and candidates in using virtual interviews should be assessed; this must be a consideration in using iMMI. In a limited-resource setting, especially regarding internet access and quality, technical issues due to network problems may reduce iMMI’s advantages. Therefore, whether or not iMMI becomes common practice after the pandemic depends on multiple factors. It is more important that the MMI format being used is valid, reliable, fair, and feasible.

We are fully aware that this report has several limitations. First, despite the validity of iMMI, as evidenced by the brief evaluation and feedback received from the assessors, candidates, and organizers during separate debriefings following iMMI, this report is not supported by a comprehensive satisfaction evaluation from all stakeholders. Further evaluation of iMMI should be conducted to inform future implementation in the current setting. Second, since our focus is on elaborating the preparation and implementation of iMMI as a shift from conventional MMI, we have not reported more detailed results from the selection process, which may be necessary to strengthen the validity and reliability of iMMI. Details on preparation and implementation should illuminate the face validity and feasibility of iMMI in a limited-resource setting such as that in our study. Finally, future studies on iMMI can focus on the assessment of non-verbal expression and adaptation in new environment which might require further development of iMMI stations and the use of more sophisticated technology.

In conclusion, we have described the experience of conducting iMMI in a limited-resource setting. All aspects of the preparation process—the scenario, stations, scoring system, simulation before interview day, briefing for candidates and assessors, and all technical support— should be detailed in advance to anticipate and mitigate risks during the iMMI. Because internet connections can pose a great challenge in the current setting, a contingency plan for internet connectivity problems should be discussed in advance and enacted promptly. Despite the challenges and limitations, the Faculty of Medicine at Universitas Indonesia has succeeded in conducting its first iMMI while preserving its validity and reliability. Future evaluation is necessary to reassess the format for future MMI.