Using Multiple Mini-Interviews for Students' Admissions in Pakistan: A Pilot Study.

BACKGROUND: A significant challenge continuing to face medical educators is the development of an effective method for student admissions into medical school. Conventional interviews have been commonly used for assessment of non-cognitive skills; however, they are subject to different biases and lack of standardization. The present study aims to determine the validity, reliability, feasibility, and acceptability of implementing Multiple Mini-Interviews (MMI) as a selection method for undergraduate medical students.
METHODS: MMI consisting of 8 stations was developed and implemented. A variance component analysis was computed to identify different sources of variance and estimate overall reliability. Content validity was established by seeking consensus on non-cognitive attributes desired by stakeholders. Acceptability and feasibility were determined by a post-MMI questionnaire.
RESULTS: A total of 381 out of 450 (84.66%) candidates participated in the study. The G coefficient was 0.70. Participants found MMI as an acceptable and feasible method of assessment, with 75% of candidates and 95% of assessors preferring MMI to traditional interviews.
CONCLUSION: MMI as an assessment method for students' admissions demonstrated reliable findings. Stakeholders found it feasible and acceptable. Individual institutions can tailor the stations towards selection of the characteristics that are most valued within their local context.

Introduction

A significant challenge continuing to face medical educators is the development of an effective method for students admission into the medical school.1 Student selection for a medical school is mostly based on multiple criteria including academic grades attained from high and secondary school, personal statements, letters of professional references, aptitude tests and interviews.2,3 Conventional interviews have been commonly used for assessment of non-cognitive skills,4 however, they are subject to different biases, lack of standardization and reported to have low reliability.3,5,6 Multiple mini-interviews are a reliable and valid method of selection of candidates for admission, as they allow to gain an accurate insight of each person’s strengths and limitations.1,5,7 In the interview process, candidates rotate through a series of short, timed, and structured mini–interviews, or stations in which an interviewer asks questions based on a scenario using standardized scoring descriptors. Each station is designed to measure one or more non-cognitive attributes.8

A number of studies have been conducted about MMI as an admission criterion in various disciplines.5,9,10 Hofmeister et al11 conducted a study for selection of international graduates into family medicine residency while Barbour’s and Sandy's12 study focused on selection of students for dentistry. Cameron and Mackeigan8 conducted a study in which pilot testing of MMI was conducted for pharmacy admission, and Perkins et al13 used MMI as a selection tool for undergraduate nursing program.

In institutions where the predominant mode of selection of medical students is academic scores, introducing a different method of selection can raise issues in terms of acceptability and feasibility to its stakeholders. In addition, the psychometric performance of an MMI can vary as it is likely to be contingent upon the approach used for station development and testing.2 Therefore, the aim of the study was to explore the validity, reliability, feasibility, and acceptability of MMI as a selection method for undergraduate medical students.

Is MMI a valid and reliable method of selection for admissions of undergraduate medical students?

Does the candidates and faculty consider it an acceptable and feasible method of assessment for admission in medical schools?

Methods

The study was conducted in one of the Medical Colleges (Shalamar Medical and Dental College, Lahore) affiliated with the University of Health Sciences, Lahore, Pakistan. The ethical approval for this study was obtained by the Institutional Review Board (IRB). For the year 2016–2017, the medical college received many applications of which only those students who had secured >80% marks in their premedical years, and the medical college entrance test were invited for interviews.

Previous to this year only academic scores were used for admission to the medical school. This comprised of their O level (or equivalent), A level (or equivalent), and their medical college admission test (MCAT) scores. The MCAT is a state (provincial) admission test that is content-based consisting of multiple-choice questions from the subjects of Physics, Chemistry, Biology, and English. A 10% weightage was given to the O levels, 40% to the A level scores and 50% to the MCAT scores in finalizing their merit for admission.

We decided to include interviews in the admission process to assess some of the attributes that we as an institution felt were important for a medical student and future doctors. The attributes selected are listed below. MMI was selected as an interview modality over conventional interviews because of their greater validity and reliability. The MMI scores were given a 4% weightage which was added to their academic scores for admission.

Content for Each Station

The content for each station was selected by reviewing the literature and taking on the attributes deemed important for future doctors. This committee included the dean, chairpersons, section heads, consultants, professors, and lecturers of basic and clinical sciences. After several iterations, the committee reached consensus on eight attributes. These eight attributes were formulated into 8 MMI stations.

Communication

Ethical decision-making

Admitting fault

Problem-solving

Integrity

Teamwork

Motivation

Knowledge about existing medical schools

Specific content and scoring rubrics for each station were also refined by the committee.

Interviewers

Interviewers were faculty members from both basic and clinical sciences including Medicine, Surgery, Gynaecology, Eye, ENT, Radiology, Anatomy, Biochemistry, Pathology, Physiology, Pediatrics, Community Health Sciences, and Psychiatry. A total of 90 interviewers were invited to participate in the study. All these interviewers had the experience of conducting conventional interviews also. A week prior to MMI, two hours mandatory workshop was conducted in which all the interviewers were shown sample interviews/ questions and the rating scale, and were given opportunities to clarify any queries regarding MMI. Interviewers did not have access to any information about candidates and were not informed of interview scores at any other stations.

Interview Process

The interviews were conducted over a period of one week. Letters were sent to the candidates inviting them for the interview and informing them briefly about the purpose and process of the interview. Written consent was obtained from all participants of the study. All the candidates consented to participate in the study. On the day of the interview, candidates were briefed about MMI prior to the interview.

On the day of the MMI, interviewers were provided with the objectives of the station and background information about the case scenario and the prompt questions for each station. Each mini-interview took place in a separate room and lasted for eight minutes. Between each station, candidates were given two minutes duration in which they read the scenario for the next station. Interviewers were instructed to allow the candidates to talk at length without interrupting them and use prompt questions only if necessary. Each interviewer scored the candidate on a 10 point global scale on the overall performance in each station. Station scores were summed to obtain a score for each candidate based on a maximum possible score of 80. Towards the end of the MMI, interviewers and candidates were also asked to provide anonymous feedback regarding the MMI by using a semi-structured questionnaire.

Data Analysis

Descriptive statistics for both the socio-demographic data and each station were tabulated. To determine whether each station was measuring a single domain, correlation between stations was examined. We expected stations to assess different attributes and arrive at different scores A Generalizability study (G-study) was calculated to identify the different sources of variances and a decision study (D study) was also performed to determine reliability by varying the number of interviewers and stations. The design of the present study was crossed and random. This design allowed us to explore variance due to overall student’s ability; overall interviewers’ stringency and leniency; stations (level of difficulty); students*stations (students' station-specific performance); students*interviewers (interviewers' student-specific marking), interviewers*stations (interviewers' station-specific marking). Pearson correlation was used to examine the concordance between students overall school academic grades and MMI scores. ANOVA was used to determine differences between males and females at each station. Quantitative and qualitative analysis of post-survey questionnaires determined the feasibility and acceptability of the MMI.

Results

Demographics

A total of 381 (84.66%) out of 450 students participated in the study. Of these, 175 (46%) were males and 206 (54%) were females, while a total of 75 (44 males and 31 females) faculty members were involved in conducting the interviews.

Scores

Overall, all students (males and females combined) scored highest at station 8 (knowledge about existing medical schools) with mean value of (6.71±1.391); followed by station 7 (ethical decision-making)(6.48±1.297); station 1 (admitting fault)(6.4±1.398); station 5 (motivation) (6.24±1.668); station 4 (integrity) (5.95±1.624); station 2 (communication) (5.93±1.451); station 3 (teamwork) (5.678±1.561); and lastly station 6 (problem solving) (5.51±1.605). None of the students failed the MMI, i.e., scored less than 50%. Scores achieved by male and female students at each stations and group comparison by one-way ANOVA is summarized in Table 1. Values are given as mean ± standard error of mean.

Table 1

Group Comparison and Scores Achieved by Male and Female on Each Station

Stations	Scores Achieved by Males (Mean± Standard Deviation)	Scores Achieved by Females (Mean± Standard Deviation)	F	p-value
1 (admitting fault)	6.431±1.467	6.374±1.338	0.161	0.689
2 (communication)	5.87±1.489	5.99±1.419	0.613	0.434
3 (problem solving)	5.551±1.611	5.785±1.513	2.124	0.146
4 (integrity)	5.93±1.649	5.97±1.606	0.043	0.836
5 (motivation)	6.1±1.797	6.35±1.544	2.076	0.15
6 (teamwork)	5.26±1.685	5.72±1.507	7.757	0.006
7 (ethical decision making)	6.32±1.335	6.61±1.251	4.834	0.029
8 (knowledge existing medical schools)	6.48±1.376	6.9±1.377	8.723	0.003

Reliability Analysis

The range of station/ total score correlations between 0.006 and 0.387 indicated that each station was measuring a different domain. The overall Generalizability coefficient (G coefficient) for 8 MMI stations (questions) was 0.70. The highest source of variability (37%) was due to students’ performance on different stations, which allowed gaining insight into each student’s strengths and limitations. This was followed by overall students’ performance on a station which was 21%. Overall interviewers’ stringency/ leniency was 16% while interviewer student-specific variation was 11.78% indicating that it could be further minimized by rigorous training. Variability due to interviewers station-specific marking was 8.19% and station difficulty was 4.38%.

The result of D study indicated projected reliability with different number of stations and interviewers. For the present study, we had 8 stations with 1 interviewer, increasing the number of interviewers to 2 does not significantly increase reliability. However, for future, increasing the number of stations (questions) to 11 and having two interviewers, will increase the reliability to 0.80.

Concordance Between Total School Academic Marks and MMI Interview Scores

No correlation (−0.065) was found between the marks obtained in the MMI interview and total marks (MCAT+12th Grade+10th Grade) of the students.

Post-MMI Surveys

Table 2 illustrates the responses given by the candidates regarding their views of the MMI experience. Overall candidates gave a positive feedback regarding the MMI process. In addition to these questions, candidates were also asked four open-ended questions: In response to the question, “What did you like most about the MMI”, majority of the candidates found MMI to be a fair process for assessing the students. Examples of some of the comments are: MMI checks a student’s general knowledge, his/ her confidence and intelligence which are not very easily or properly checked just by their academic scores. As compared to other interviews, MMI gives and provides a better opportunity to a student to express and explain his/her views and it is a fair process of assessing as well.

Table 2

Candidate Feedback to Post-MMI Survey

	Statements for Feedback Assessment	Strongly Agree	Agree	Disagree	Strongly Disagree
1	The MMI process and purpose was clearly explained	196 (51.44%)	175 (45.93%)	9 (2.36%)	1 (0.26%)
2	The day was well organized	231 (60.63%)	142 (37.27%)	8 (2.1%)	0 (0%)
3	Compared to traditional interview MMI caused more anxiety/stress	6 (1.19%)	11 (2.88%)	214 (56.16%)	150 (39.37%)
4	Instructions before each station were clear	197 (51.71%)	161 (42.26%)	19 (4.99%)	4 (1.05%)
5	The MMI is a fair process of assessing	161 (42.26%)	191 (50.13%)	25 (6.56%)	4 (1.05%)
6	MMI is better than traditional interviews	199 (52.23%)	171 (46.45%)	11 (2.88%)	0 (0%)
7	MMI should become continuous part of admission selection	159 (41.73%)	216 (56.69%)	6 (1.57%)	0 (0%)
8	The number of MMI stations were sufficient	161 (42.25%)	183 (48.03%)	33 (8.66%)	4 (1.04%)
9	Time for each station was sufficient	115 (30.18%)	219 (57.48%)	40 (10.49%)	7 (1.83%)

In response to the question, “What did you like the least about the MMI,” candidates’ responses focused primarily around logistics issues such as timing and waiting period for students. Examples of some of the comments are less time on a single station to discuss the issues properly. Fast-paced questioning. The duration of explaining our points and answer at every station is quite less, more time like 15 mins should be given.

In response to the question of how MMI could be improved for the next year, candidates did not suggest any significant improvements. Some candidates suggested adding more stations.

Table 3 illustrates the responses given by the interviewers regarding their views of the experience. Interviewers were also asked the same open-ended questions. In response to the question of what did you like most about the MMI, interviewers found MMI to be useful for assessing the students. Examples of some of the comments are, a useful means to have insight of capabilities of the candidate. Candidate has full opportunity to express his/ her thoughts. You have multiple stations to assess the candidate in multiple ways.

Table 3

Assessor Feedback to Post-MMI Survey

	Statements for Feedback	Strongly Agree	Agree	Disagree	Strongly Disagree
1	The MMI process and purpose was clearly explained	34 (45.33%)	39 (52%)	2 (2.67%)	0 (0%)
2	The training for my station was clear and sufficient	38 (50.67%)	33 (44%)	3 (4%)	0 (0%)
3	The evaluation form for applicants was clear and easy to use	40 (53.33%)	35 (46.67%)	0 (0%)	0 (0%)
4	The number of MMI stations was sufficient	23 (30.67%)	46 (61.33%)	6 (8%)	0 (0%)
5	The day was well organized	40 (53.33%)	31 (41.33%)	2 (2.67%)	2 (2.67%)
6	It was possible to determine the candidates attributes in the allotted time	25 (33.33%)	48 (64%)	2 (2.67%)	0 (0%)
7	The instructions given to candidates before your station were clear	28 (37.33%)	47 (62.67%)	3 (4%)	0 (0%)
8	The MMI is a fair process of assessing	28 (37.33%)	40 (53.33%)	7 (9.33%)	0 (0%)
9	MMI is better than traditional interviews	39 (52.0)	32 (42.66)	3(4%)	1(1.33%)

In response to the question of what did you like least about the MMI, interviewers' responses focused primarily around time. Examples of some of the comments are; it is a very long exercise, time, and candidates should be minimized. Length of interviewing process is long.

In response to the question of how MMI could be improved for the next year, interviewers suggested involvement of more faculty and stations. Examples of some of the comments are, more faculty members' training, and involvement. Involve large number of interviewers.

Discussion

The aim of the study was to explore the validity, reliability, feasibility, and acceptability of using MMI as an assessment method for student selection in Pakistan. MMI is globally used as a method of assessment to discern the attributes of candidates for admission purposes.14–16

The findings of the present study can be compared with the findings of other similar studies2,11,17 where reportedly the reliability G coefficient was between 0.65 and 0.85. In the present study, the G coefficient was 0.70 which is considered acceptable for MMI. The highest source of variance was due to student*station which implies that candidate’s ability is influenced by each station. This finding corroborates with Eva et al17, and is imperative in fulfilling the very purpose of MMI that is, providing insight into each candidate’s strengths and limitations, and differentiating between outstanding and mediocre candidates. However, interviewer’s stringency/ leniency was 16%, which could be minimized by providing more specific training to the interviewers, to ensure that all interviewers have the same interpretation of the rating scale, and the stations.

Correlation between stations was minimal which indicates that each station was measuring a different domain. This finding is supported by other studies in which reported correlation between stations were from 0.057 to 0.363.2 Evidence reports negligible correlation between MMI and other pre-interview selection measures.2,11 The findings from the present study also report no correlation between MMI and other pre-interview selection measures. This is not surprising considering that MMI aims to measure non-cognitive attributes while other pre-interview measures are focused on measuring cognitive abilities.

In the present study, both interviewers and candidates preferred MMI as compared to the conventional interviews. The majority reported it as an acceptable method of assessment. This demonstrated to the other faculty members in the medical school that conducting MMI is feasible.

The only suggestion from faculty members for improving MMI was to make it less time-consuming. For future MMI, this could be addressed by training more faculty members to conduct the interviews which would reduce the number of hours per faculty member.

A vast majority of candidates scored 50% or above in all stations which suggested that candidates were able to comprehend the scenarios in each station. In two stations (problem solving and teamwork) the majority of the students scored considerably low which may imply that the content of the station needs to be reviewed for better understanding. In terms of improvement, only some candidates suggested increasing the time duration between stations so that they could explain their answer but majority reported it as adequate.

For the present study, faculty members’ consensus was sought on the non-cognitive attributes which they deemed pertinent for future doctors, keeping in view the institution’s vision and its curriculum tenets. Although, the attributes list could have been compiled from the existing literature alone but faculty opinion ensured the attributes deemed important by the host faculty were assessed in the stations set. Eva et al17 also advocated that medical schools engaging in MMI should consider the blueprint technique to ensure that the attributes they value in students are assessed at selection. In addition, considering MMI was introduced for the first time in the college, it was imperative that key stakeholders take collective responsibility and ownership for its development, implementation, and decision on its continuous use as an assessment method.

The sample size for our study was 381, which is relatively small sample; however, evidence indicates that reliable findings for MMI can be demonstrated with smaller sample size of 115 candidates and 26 candidates.5,18 Limitations of the study include (a) Construct and concurrent validity were not established, and (b) Results are from single institution only. However, considering that the culture and ethos is similar across other institutions, there is no reason why the candidates interviewed through this process should not be broadly representative of all applicants.

The results of the present study are encouraging, but additional work is needed to determine whether the students selected through MMI perform better in medical school than those students who were previously selected through conventional methods. We will be following the selected students over the next six years, ie, five years of medical school and one year of mandatory internship. During this period, we will be monitoring their performance and see if the MMI scores correlate with their academic performance. In addition, we will be getting feedback from faculty, students, and staff about the attributes of students during their medical school and especially during their internship. Attributes such as ethics, admitting fault, integrity, teamwork, and problem solving will be assessed relative to their interview scores. We would be interested in seeing if the scores in the station on motivation were related to their future performance in the medical school. In addition, there is a need to build interventions in areas where the students’ scores were on the lower side, eg, communication skills, teamwork, and problem-solving. Timely interventions during the next six years will help to overcome these weak areas and hopefully produce even better doctors with a better skill set to serve the society in a more befitting manner. Although validity, reliability, feasibility, and acceptability of MMI has been established, this study is an addition in further confirming these attributes of MMI.

Conclusion

The present study reports the experience of using MMI as an assessment method for student selection in Pakistan. Primarily the aim was to determine the validity, reliability, acceptability, and feasibility of the MMI. MMI is a moderately reliable method of selection for admissions of undergraduate medical students. Despite concerns that MMI may not be acceptable to the stakeholders, the interviews were well accepted by the faculty members and candidates alike, and demonstrated reliable findings. Individual institutions can tailor MMI stations towards selection of the characteristics that are most valued within their local context.