The validity of the assignment book-based structured interview in predicting academic performance in medical schools: a retrospective cohort study.

PURPOSE: Structured interviews have become essential in the medical schools admission selection because structured interviews predict academic achievement after admission. The purpose of this study was to determine validity and fairness of the new structural interview technique, assignment book-based structured interview (ABSI), in predicting future academic achievement of the medical students.
METHODS: The validity of this new interview technique and academic achievement was evaluated based on the data of all the applicants and successful applicants who applied for on-time admission between the year 2011 and 2014.
RESULTS: The ABSI technique showed a significant correlation and predictive validity for academic achievement in the medical school. The retention group received significantly lower T-scores of ABSI compared with the superior student group.
CONCLUSION: The results indicate that ABSI is a feasible, reliable, fair and valid admission selection tool. The ABSI may be meaningful and fair method for predicting academic achievements, and it could be incorporated as one of the contents in the multiple mini-interview.

Introduction

To develop competent health professionals, medical schools and colleges have been using a combination of several methods to select applicants with intellectual and non-cognitive attributes as core competencies, professional attributes, or skills. Traditionally, intellectual level and academic ability was measured through undergraduate grade point averages (GPAs) and official entrance tests while other evaluation methods such as structured interviews, was used to evaluate non-cognitive skills. In the last 20 years, multiple mini-interview (MMI), objective structured clinical examination style, and structured interview format have been developed to assess applicants’ non-cognitive skills during medical school admission globally [1-3]. Recent systemic reviews revealed that MMI represented a non-biased selection tool for applicants based on factors such as gender, age, socio-economic status, or personal background, except for English proficiency for international applicants [4,5]. Applicants have been evaluated on their ability to analyze information, think critically, and solve the problem. Nevertheless, when looking at the impact of individual differences for each competency, prior knowledge could affect instruction or conceptual development of critical thinking and therefore affect critical thinking evaluation [6].

Critical thinking is the mental process of conceptualizing, analyzing, and evaluating information, and applying it to guide one’s actions and beliefs [7]. Through critical thinking, information processing style obtained from observation, reflection, experience, learning, communication, and reasoning becomes the cornerstone of our decisions [8]. When evaluating critical thinking through situations or presentations, applicants’ prior knowledge and experience may be extremely varied that they might affect the evaluation. Also, responding to medical ethical situations could be influenced by their major or prior education. Therefore, references related to the evaluation topic should be selected and studied in advance to minimize information asymmetry.

Reading a book can activate a reader’s cognition to deduce, comprehend and transfer ideas from the source domain to the target domain, and it is assumed to be the ability to construct meaning and thinking through integrating the reader’s background knowledge with the information [9]. Even though reading improves critical thinking, content knowledge is essential because if a new piece of information in an unfamiliar field, reader can only accept it at face value.

In this study, we investigated the impartiality of the interview questions development and evaluation process and if the process is based on the selected books to provide content knowledge. In addition, we examined whether the assignment book-based structured interview (ABSI) method is affected by other selection variables and analyzed its validity in predicting academic achievement after admission.

Methods

1. Participants

Four cohorts of applicants who enrolled from the years 2011 to 2014 were selected among potential candidates, using several assessment tools such as South Korean aptitude test (Medical Education Eligibility Test, MEET), undergraduate GPA (uGPA), English ability test, and structured interviews. In brief, 77, 76, 83, and 80 candidates participated and 126 (29, 32, 32, and 34) students were selected each year from the years 2011–2014 (Table 1).

Table 1.

Demographic Characteristics of the Participants and Candidates Who Joined the Medical School in 2011–2014

Characteristic	Year				Total
	2011	2012	2013	2014
Candidates (applied)
No. of participants	77	76	83	80	316
Age (yr)	25.6±3.0	26.5±2.8	24.8±3.3	24.7±2.5	25.4±1.1
Gender (%)
Male	53.2	56.6	60.2	67.5	59.5
Female	46.8	42.1	39.8	32.5	40.2
Participants (admitted)
No. of participants	29	32	32	33	126
Age (yr)	25.1±2.9	27.1±3.2	25.8±4.2	25.0±2.4	25.7±3.3
Gender (%)
Male	51.7	59.4	56.3	63.6	57.9
Female	48.3	40.6	43.8	36.4	42.1

Data are presented as number or mean±standard deviation, unless otherwise stated.

2. Development and procedures of ABSI

First, the development center committee chooses the main subject and then selects a book that could support basic knowledge or background to perform critical thinking. The selected books were “Critical reasoning in ethics” by Anne Thomson, “Why morality?” by Michael J. Sandel, “Justice” by Michael J. Sandel, and “Philosophy of medicine: an introduction (2nd ed)” by Henrik R. Wulff, Stig Andur Pedersen, and Raben Rosenberg in year 2011, 2012, 2013, and 2014, respectively. To develop the problem, we selected short reading materials and questions that co-related the selected book (briefly one sample in Supplement 1). The assessment sheet was highly structured to support the guidelines of each checkpoint. All interviewers had practiced several times together with simulated candidates the day before the interview, to refine the words and standardize their scores. The interview was structure in a way that one interviewer evaluated one candidate in a single room.

In brief, a month earlier, the concerned committee assigned the selected book to candidates to prepare in advance. The ABSI process was as follows: preparation, presentation with Q&A, and evaluation step, respectively. First, the candidate entered in the preparation room with a book to read the given materials and prepare a presentation for 30 minutes. Then the candidate moved to the interview room and did a presentation with Q&A for 8 minutes. A perfect score was set as 50 points, and fail score was less than 20 points. Each candidate’s score was standardized to a T-score.

3. Statistical analysis

All data were analyzed with IBM SPSS Statistics ver. 21.0 for Windows (IBM Corp., Armonk, USA). One-way analysis of variance following least significant difference (LSD) post hoc tests, Pearson’s correlations, multiple stepwise linear regression analysis, and k-means clustering analysis were performed to determine whether the ABSI predicted academic achievement in the medical schools. Values were considered statistically significant when p<0.05.

4. Ethical approval

This study was approved by the institutional review board of Kangwon National University for both review and informed consent exemption according to the South Korea’s law of Bioethics and Safety Act (KWNUIRB-2015-003).

Results

In this study, selection was done using aptitude tests (MEET), uGPA, English ability tests, and structured interviews including ABSI. Factor analysis was performed to determine whether each evaluation factor measured similar evaluation capabilities (Fig. 1A). Based on the principal component analysis, the overall correlation matrix for each factor was not suitable for the factor analysis (Kaiser-Meyer-Olkin=0.498), and varimax rotation showed that the four factors were independent.

Fig. 1.

The Predictive Validity of the ABSI in Predicting Academic Performance in Medical Schools

(A) Representation of four groups of evaluation variables after multiple factor analysis. (B) Multiple regression between evaluation variables and final grade point average (GPA) in medical school. (C) Mean GPA scores of academic achievement groups clustered by k-means clustering analysis. (D) Assignment book-based structured interview (ABSI) T scores of academic achievement groups. MEET: Medical Education Eligibility Test, uGPA: Undergraduate GPA. *p<0.05. **p<0.01.

Analysis on the correlation between each factor showed significant correlations between the medical school GPA, and the uGPA and the ABSI, 0.348 and 0.226, respectively (Fig. 1B). There was also a significant correlation between uGPA and ABSI (p<0.05). Based on the significant correlation between the admissions tools and GPA, next, we performed multiple stepwise linear regression analysis to determine whether ABSI and uGPA scores can predict competency in the medical school. Our results showed that ABSI score (β=0.185, p<0.05) and uGPA score (β=0.31, p<0.01) were positively correlated with the GPA in the medical schools (Fig. 1B). As a result of a detailed analysis of the correlation between ABSI and academic achievement by grade, ABSI score showed significantly positive correlations with GPA in grades 2 (0.232, p=0.008), 3 (0.205, p=0.021), and 4 (0.223, p=0.013) except for the first grade (0.173, p=0.52) in the medical school.

The k-means clustering analysis was performed to divide participants into homogenous groups according to their academic achievements as measured by the four grade’s GPA. First, we assigned students with GPA of 2.0 or less in each semester to the retention group; there were 17 participants in the retention group. After excluding the grade retention participants, two clusters were listed from the k-means clustering analysis. Based on the GPA’s score, the highest scoring cluster was named as superior group, and the next cluster as moderate group. The GPA score (mean±standard deviation [SD]) was 3.49±0.23, 2.78±0.25, and 2.32 ±0.29 in the superior, moderate, and retention groups, respectively. There were significant differences among the groups after the LSD post hoc test (F=197.7, degrees of freedom [df]=2, p<0.001) as shown in Fig. 1C. As shown in Fig. 1B, our findings suggested that ABSI score could predict the GPA in the medical schools. Therefore, we compared the ABSI score in the superior, moderate, and retention clusters; the ABSI T-score (mean±SD) was 53.23±10.21, 50.04±7.84, and 47.70±7.63 in superior, moderate, and retention groups, respectively. There was a significant difference between the superior and retention groups (F=3.196, df=2, p<0.05) (Fig. 1D).

Discussion

In the medical schools admission selection process, several evaluation factors should be factored in order to make a comprehensive judgment by evaluating several individual competencies. Therefore, it is important to design an evaluation method that analyzes each evaluation factor independently or with minimal relationship. In the present study, factor analysis was used to analyze the four evaluation factors independently. Furthermore, according to post-ABSI surveys for candidates, the response for the validity of the ability assessment in the interview technique was 3.95±0.76 (mean±SD), and the acceptance of the interview technique as a fair process was 4.12±0.77 (mean±SD) on a 5-point Likert scale (data not shown). That means ABSI is an acceptable method of assessment in the selection to the candidates.

There was a significant correlation between uGPA and ABSI. Notably, GPA score is a result of a long-term process. Several studies showed that students with high conscientiousness had an excellent academic performance [10]. Also, critical thinking has shown a significant correlation between academic achievement [11]. Since we assigned the book a month before the interview day, the participants may have had enough time to read and understand. In order to get a high score within the limited time, the participants might have to read the book conscientiously and analyze the handouts and problems in the short-time period. Therefore, the significant correlation between uGPA and ABSI might be the reason for the conscientiousness and critical analysis outcomes of the uGPA and ABSI. Based on the evaluation period, uGPA reflects the integrity over the long-term, whereas the ABSI reflects the integrity of work performance in a relatively short period. Therefore, if there is a bias to evaluate uGPA, ABSI could also be a complementary selection method.

There are many reasons why doctors should have and learn critical thinking skills. It is a crucial component in independent, and fast problem-solving, and making decisions. Nevertheless, people cannot think critically about topics for which they have little knowledge. Therefore, critical thinking should be viewed as a domain-specific construct that evolves as an individual acquires domain-specific knowledge [12]. For instance, most ordinary people have no basis for prioritizing patients in the emergency department to be shifted to the only bed available in the intensive care unit. This implies that medical professionals who could be thinking critically in their discipline would have difficulty thinking critically about problems in other fields. Therefore, “targeted domain” critical thinking training and evaluation might benefit the targeted domain since it is specific, i.e., the medical profession. There are various methods of critical thinking assessment. The common way to assess the critical thinking ability is multiple choice form of summative tests. However, those kinds of assessment generally test content knowledge and do not assess higher-order thinking skills, such as generating argumentation, analyzing case-based articles, constructing the graph, and explaining with a maturity of judgment [3]. ABSI could assess higher-order thinking skills because it needs to solve the problem within the case study that presents a specific situation according to reading materials. Briefly, the candidate is given questions that help gauge their problem solving, prioritization, ethical responses, and assessment abilities. In the present study, to minimize the deviation of prior knowledge, we availed the book to the applicants in advance, and created a problem based on the contents of the book. This approached enabled us to engage the “targeted domain” thinking and promote fair and accurate critical thinking assessment.

Our study showed that the uGPA and ABSI were significant positive predictors of academic achievement in the medical school. Especially, ABSI methods showed a good predictive validity. Comparing superior and retention groups clustered by GPA, low ABSI T score might increase the risk of retention while attending medical school. However, the findings from this study must be interpreted cautiously because there were several limitations. First, the sample size of each year was relatively small and therefore, inadequate to test variable contributions in the regression equation, and the direction of influence is unclear. Second, although students’ past performance, GPA, is not the criterion for success and evaluation of “good” and “poor” students, GPA may only reflect students’ academic achievement in this study. Third, the correlation between ABSI and academic achievements can be described as moderate to low, even though it has been suggested that measures with modest predictive validity considerable value to the selection system in medical school [13]. Fourth, we did not assess the ability of critical thinking independently.

Although our research confirms the impact of ABSI, which is based on critical thinking ability on future academic achievement, it is meaningful to the fairness of the selection process, to minimize the deviation of the prior background intellectual level in the student selection process. Furthermore, ABSI could be one of the contents in MMI, because it directly evaluates the attitude as well as the problem-solving ability of the applicants by asking a situation-based problem.

We confirmed that uGPA and ABSI are good pre-admission predictors for academic success in the medical schools. Particularly, ABSI may be a meaningful and appropriate method for predicting academic achievements. However, more research on the correlation between changes in critical thinking ability and the outcome of ABSI during medical school enrollment is necessary.