A Multiple-Mini Interview (MMI) for Emergency Medicine Residency Admissions: A Brief Report and Qualitative Analysis.

INTRODUCTION: A multiple-mini interview (MMI) is a type of structured interview, which may assess many non-cognitive domains in residency applicants. There are few studies on MMI during the emergency medicine (EM) residency admissions process in the United States. We sought to determine the strengths, weaknesses, and acceptability of a pilot MMI for EM residency admissions.
METHODS: We piloted a five-station MMI with nine residency applicants. Following the MMI, we surveyed all participants, using 15 open- and closed-ended questions. Using grounded theory analysis, we coded the responses to the post-intervention survey to uncover the strengths and weaknesses of the MMI for EM residency admissions.
RESULTS: All nine students completed the survey. A positive theme that emerged from the survey was that the MMI was a positive, unexpected experience (all respondents, n=9). Candidates felt they were able to showcase unique talents, which would not be observed during a traditional interview (n=3). A negative theme that emerged from the survey was that the experience was intimidating (n=3). Candidates felt that the MMI left out important aspects of a typical interview day (n=3); such as, time for the candidate to become more familiar with the program.
CONCLUSIONS: An MMI may be a positive experience for candidates, but may also induce more anxiety. The MMI may omit an important piece of the interview day: an opportunity for the applicants to familiarize themselves with the residency program.

Introduction

A program director (PD) considers many aspects of an applicant’s file when determining his or her position on the rank order list. These factors include clerkship and basic science grades, standardized and non-standardized letters of recommendations, membership in Alpha Omega Alpha (AOA), interview performance, United States Medical Licensing Examination (USMLE) scores, emergency medicine rotation performance, extracurricular activities, other awards and honors, and the medical school attended. Ultimately, the ideal applicant has excellent aptitude for all of the Accreditation Council for Graduate Medical Education (ACGME) core competencies. Cognitive domains, such as medical knowledge, might be assessed through grades and test scores. However, non-cognitive domains, such as professionalism and interpersonal and communication skills, may be more difficult to assess through the standard admissions packet. The interview is thought to provide insight into some of these non-cognitive domains. EM program directors regard the interview as an important factor in making ranking decisions (1,2). However, Hayden, et al. (3) found that there was a poor correlation between the interview and future residency performance in EM. Many authors suggest that structured interviews add better predictability and score reliability, and reduce bias in the residency and medical school selection process (4-6). Unstructured (informal, conversational, non-standardized questions) interviews are commonplace in EM.

A multiple-mini interview (MMI) is a type of structured interview technique with multiple, focused encounters (7). In each interview station, an applicant participates in a scenario or activity or responds to a standardized question in an Objective Structured Clinical Examination (OSCE) type exercise. The MMI may provide a better, more reliable assessment of the candidates’ abilities and aptitudes in the non-cognitive domains and may predict future success in the health professions (8,9). The MMI has been described in undergraduate medical, pharmacy, veterinary, physician assistant, nursing, and dental admissions, as well as in the interview process for residency programs in most specialties (7,9-18).

There are few studies on the MMI for selection for emergency medicine residency training (14-17). Some studies found that students prefer traditional interviews (TI) or mixed TI-MMI over the MMI alone (16,17). To the best of our knowledge, no studies have explored the strengths and weaknesses of the MMI for emergency medicine applicants. Therefore, we sought to determine the strengths and weaknesses of an MMI for EM residency admissions through a qualitative pilot study.

Methods

The University of California, Irvine Residency Program interviews approximately 80 candidates per interview session. The residency selection committee (RSC) ranks the candidates based on the standard factors described in the introduction. The interview score comprises approximately 20% of the final ranking score. Candidates who have already rotated through the ED are well known to the RSC. For these candidates, the interview day is largely a formality and makes little difference on the candidate’s position on the rank order list.

We piloted the MMI during a single interview day for nine candidates who had previously rotated clinically through our department. At the time of the study, there were no studies on the MMI for United States Emergency Medicine residency selection. We attempted to test the following competencies: professionalism (ethical decision making, professionalism, ethical thought process), systems-based practice (use of resources, multi-tasking ability), and interpersonal and communication skills (team-building, leadership, communication with colleagues) through five interview stations (Table 1). After blueprinting the above constructs, we developed new testing stations (stations 2, 3, and 5) or adapted previously published methods (stations 1 and 4) to test our constructs. Each station was novel or adapted; we did not have validity or reliability for the stations or the assessment tools. One or two members of the RSC moderated each station and completed a standardized assessment form for each candidate based on their performance at that station. Since the candidates were well known to the RSC through previous interactions, we did not use the results of this pilot MMI to affect their rank list position.

Table 1

Stations and competencies tested

Station	Title, Description	Competency/Skill tested
1	Multitasking ability test (MTAT)	Multi-tasking ability, prioritization, efficiency, systems-based practice
2	Building Blocks: Candidates are observed while creating structures with modular connecting building blocks; small groups of candidates choose their own structural goals.	Team-building, interpersonal and communication skills, leadership, flexibility
3	Situational interview question: Candidates are given a patient care scenario, regarding imaging selection in trauma patients, and asked how they would handle it.	Interpersonal and communication skills, systems based practice, decision making, integrity
4	Parking garage: Candidates interact with an actor, following a simulated scenario where conflict has arisen.	Professionalism, interpersonal and communication skills
5	Situational interview question: Candidates are given a patient care scenario and asked how they would disclose a medical error.	Communication skills, ethical decision making, integrity, taking responsibility

After the candidates participated in the MMI, we asked them to complete an anonymous survey of their opinions of the MMI. This was a mixed-methods study, using both open-ended and structured survey questions. We used constant comparative data analysis methods to develop grounded theory in a study of medical students' perspectives about the use of the MMI in the EM admissions process. We asked seven open-ended and eight closed-ended questions (Appendix 1). Candidates recorded their answers on the written survey. The university institutional review board approved this study. Informed consent was waived, as the survey and experience was performed as a residency innovation, rather than a research study with waived informed consent.

A single reviewer, with experience in qualitative methods, coded the responses to the seven (open-ended) questions. Each separate idea listed by the candidates was coded separately. After a first pass, similar themes were grouped together to identify the themes from the candidates’ opinions of the MMI.

We also asked the candidates for their opinion on what qualities were being assessed through the MMI and triangulated these themes with the competencies that were actually being tested [professionalism (ethical decision-making, professionalism, and ethical thought process), systems-based practice (use of resources, multitasking ability), and communication skills].

Finally, we performed member checking. Candidates reviewed the themes that were identified by the reviewer and shared their opinion on an online anonymous single question survey: “Do you agree with the analysis of your survey responses? Do you have anything to add?”

For the closed-ended questions, we reported the frequency of the themes among the nine candidates.

Results

All nine participants in the MMI completed the survey. Several themes emerged from the open-ended questions (questions 1-7). The themes from questions 1 and 2 are displayed in Table 2. Overall, the students felt that the MMI was a positive, unexpected experience (n=9). Candidates felt they were able to showcase unique talents, which would not be observed during a standard interview (n=3). A few candidates felt the experience was intimidating (n=3). Some felt that their behavior during the MMI might have been different if they were being interviewed at an unfamiliar program (n=5).

Table 2

Emergent themes (codes) from qualitative analysis of questions 1 and 2

	Emergent themes	Number of comments related to this theme	Number of candidates giving such comments	Representative quotations from survey
Questions 1 & 2: What did you think of the interview experience? Strengths and weaknesses?	Positive/fun experience	11	9	“Overall good experience”“Fun interview day”
	Unexpected and different experience	7	6	“Different and unexpected”
	Candidates able to showcase unique talents	7	3	“Allowed us to show our unique strengths”
	Behavior could have been affected by being in familiar institution	6	5	“I felt comfortable because the department [knows me]”
	Spontaneous, not forced	4	4	“Allowed for spontaneous conversation that did not feel forced”
	Felt something was missing from the interview day	4	3	“Throws out the one important part of the standard interview - the “getting a feel” for a program”
	Intimidating/stressful	3	3	“Slightly intimidating”

Some candidates noted that they felt something was missing from the interview day (n=3). Specifically, candidates felt that the MMI neglected the “getting to know you” aspect of the interview day. Some mentioned that the interview was “one-sided” and candidates lacked the ability to discover if the program was a good fit for them. Emergent themes from questions three and six supported this notion. Many candidates felt that aspects of the standard interview should be added to the MMI (n=4); the interview day would have been ideal as a combination of MMI and standard interviews.

Regarding whether candidates would want all of their interviews in the MMI format, most would enjoy all of their interviews in the MMI format (n=5); the remaining candidates would not want all interviews in the MMI format (n=2) or were unsure (n=2).

Candidates felt that the MMI was testing intellectual factors (n=7) even though knowledge or intelligence was not something we aimed to test. Candidates accurately identified the majority of competencies being tested. They realized that we were assessing their ethical decision-making (n=6), professionalism (n=2), communication skills (n=5), teamwork (n=4), and other personality factors (n=7). Only one student recognized that we were testing the use of resources (systems-based practice).

For the closed-ended question responses, the majority of candidates (n=5) felt they were able to accurately portray themselves during the interview; the remaining candidates were unsure. The majority of candidates (n=7) stated that the use of an MMI would not prevent them from applying to a program. However, the majority felt that an MMI would cause more anxiety than a standard interview (n=8). An MMI would not require more preparation than a standard interview (n=6). Of the five stations, the most positively reviewed one was the building blocks (station 2), with eight of nine candidates stating that they enjoyed this station.

Four of nine students responded to our member checking query and felt their opinions had been accurately analyzed.

Discussion

For undergraduate admissions, the MMI has been shown to predict future OSCE and clerkship performance and clinical-decision making scores (9,21). The MMI has good score reliability in post-graduate admissions, including internal medicine, obstetrics and gynecology, and pediatrics (13). While we constructed our MMI to assess the constructs not addressed by other components of the standard residency application, we did not study the reliability, validity, or predictability. In this pilot study, our goal was to explore the strengths and weaknesses of the MMI experience from the candidates’ perspective.

We found that the candidates enjoyed the MMI experience. Many felt that the experience would have been improved as a mix of MMI components and standard interview components. Adding portions of the standard interview experience would have better allowed the candidates to determine whether the program was a good fit for them. This is supported by other published studies that have shown that candidates prefer a mix of TI and MMI to MMI alone (16,17).

Our students felt that the MMI would cause more anxiety than the standard interview; this is different from the McMaster University results, when the MMI was used for undergraduate admissions (7). The MMI is a type of simulation that tests the performance, rather than the ability to answer somewhat predictable interview questions. Students accurately identified many of the competencies we were assessing.

Our study found that the majority of candidates felt they could accurately portray themselves, enjoyed the experience, were able to showcase talents not typically discovered during standard interviews, and would not be deterred from applying to programs that utilize the MMI. Future research with a larger sample size should determine the validity and reliability of the MMI to predict the residency success in EM, especially in the non-cognitive domains.

The major limitation of this study was the small sample size. All students were from our home institution and were familiar with the interviewers and the program. Because of the resources and time, our MMI consisted of only five stations; previously published literature has demonstrated a good score reliability for eight to twelve station MMIs (7,19). We did not perform any reliability or validity analysis during this small pilot study. We simply explored the strengths and weaknesses of the MMI from the candidates’ perspective. With the exception of two questions, the questions were somewhat specific and may have hindered the qualitative processes. Additionally, only a single reviewer identified the qualitative themes.

Conclusion

Overall, students enjoyed the MMI but felt that it lacked important aspects of the traditional interview, specifically that candidates did not have the ability to familiarize themselves adequately with the residency program. An MMI would not deter most candidates from interviewing at a program.