The Predictive Value of Full-length Practice Exams for the New MCAT Exam for Premedical Students.

OBJECTIVE: To investigate whether full-length practice exams could predict performance in the new MCAT exam in a pilot group.
METHODS: A dataset of the first group of students who enrolled in the premedical post-baccalaureate program, established at Khalifa University of Science and Technology in the 2018-2019 academic year, were used. Nineteen students from the group were subsequently admitted to the newly launched Doctor of Medicine Program at KU in the 2019 fall semester.
RESULTS: Performance in the full-length practice exams may significantly predict the MCAT score (β = 0.74, t = 6.50, P = 0.000), independent of English proficiency.
CONCLUSION: These results are the first to provide direct empirical evidence supporting that doing full-length practice exams before the MCAT test day is a good strategy for preparation. Given the size of the cohort and the amount of time the program has been in existence, further studies are thus required to support this initial result.

Introduction

The Medical College Admission Test (MCAT) is a standardized, multiple-choice examination administered by the Association of American Medical Colleges (AAMC). It has been widely used as one of the requirements for medical school admission in North America, as well as many other countries. In 2019, Khalifa University of Science and Technology (KU) in the United Arab Emirates (UAE) took the initiative to introduce US-style medical education in the nation by establishing the College of Medicine and Health Sciences (CMHS). An admissible MCAT score is thus required to enter KU’s Doctor of Medicine (MD) program for students in this region. KU’s premedical post-baccalaureate was instituted 1 year earlier to the launch of the MD program to prepare students for the MCAT exam and potential entry into the inaugural MD class of 2019.[1]

The MCAT aims to measure readiness for medical school. To do so, it has been designed to include a wide range of questions since the very beginning when the MCAT was administered in the 1920s.[2] There have been several versions of this exam since then. In April 2015, the AAMC launched a new version of the MCAT, which incorporates even more natural and social science subjects such as biochemistry, psychology, and sociology.[3] These changes bring new challenges for examinees. Not only are there more subjects to review before the exam, but the testing duration was almost doubled, from 200 minutes for the previous version of MCAT to 375 minutes for the new version of MCAT. With a longer testing duration, the score is more likely to be affected by non-cognitive factors such as tiredness, stress, and anxiety.

These challenges could be even more difficult for students in the UAE to overcome who are presumably less prepared for MCAT in their undergraduate education, as compared to their North American peers.[4] First, most of them are not native English speakers. Second, the MD in KU is the first US-style medical program in the UAE that requires the MCAT. Most students in this region do not prepare for this test during their undergraduate studies. However, to pursue their goal to become a physician through KU, they had to prepare for the MCAT in a short period.

To bridge the gap between the preparedness of students and the new requirements of KU’s MD program, this institution developed a post-baccalaureate premedical program called the Pre-Medicine Bridge Program (PMB). The PMB program provides intensive training with formal lectures and review sessions on all 4 sections of the MCAT exam, and requires that students complete a minimum number of full-length practice exams, which were administered by its partner Kaplan, Inc. and the AAMC.[1]

This curriculum is well supported by the advances of psychological studies on learning in the past two decades.[4,5] Traditionally, exams were only considered as a means to evaluate the outcome of learning. However, cognitive psychological studies provided converging evidence suggesting that retrieval practice, which usually occurred during exams, could enhance learning even better than rereading materials.[4] Furthermore, all the full-length practice exams were administered in conditions that were similar to those on the test day. For example, students completed the exams at workstations with the option to use noise earplugs and/or headsets to assist noise reduction. Students were also required to complete the 7.5-hour exam in one session and take scheduled breaks outlined by the AAMC. We believed that doing the practice exams in similar testing conditions would help the students get used to the required environment, which would help minimize test anxiety. At least students would have fewer distracting thoughts about the testing environment and concerns about the testing time during the test day.[6]

To further test the effectiveness of this practice-based curriculum, the current study investigated whether the performance of full-length practice exams could predict performance in the new MCAT exam. Especially, we tested how the predictive value of practice exams was compared to students’ level of English proficiency. The MCAT is administrated only in English, and it is thus likely that English proficiency could affect MCAT performance to a large extent. For instance, one study showed that non-native English speakers scored significantly lower than native English speakers, particularly in the verbal reasoning section in the previous version of MCAT.[7] However, the MCAT is designed to test foundational science knowledge rather than merely English proficiency. One’s performance on full-length practice exams could reflect their level of preparation for science subjects covered in the exam. Thus, we expect that full-length practice exam performance could predict the MCAT performance independent of English proficiency. The results of the study would provide insights for both individual examinees and MCAT preparation programs to develop a better MCAT preparation plan, particularly within non-native English-speaking regions.

Methods

The current study was based on a dataset of 19 students who attended the PMB program across the 2018-2019 academic year and were subsequently admitted to the inaugural MD class at KU in the fall of 2019. During this period, 26 students entered KU’s premedical program. In comparison, the overall class size of this MD class was 30. Access to the data was approved by KU’s Institutional Review Board.

For all the students, we obtained their International English Language Testing System (IELTS) scores, undergraduate grade point average (UGPA), MCAT scores, and scores of each full-length practice exam. Full-length practice exam performance was summarized by the following 3 measures: median score, maximum score, and most recent score. The median score was used instead of the arithmetic mean because the students completed a relatively small number of exams, and different students finished different numbers of exams. Using the median score could help minimize the influence of the number of exams and potential outliers. The maximum score could help determine whether the students were able to maintain their top performance on the MCAT test day. The most recent score could define whether there was any recency effect. Meanwhile, we also investigated whether the number of full-length practice exams had an effect. The full-length practice exams comprised a combination of exams taken from the commercial MCAT prep package of Kaplan, Inc and the MCAT official prep practice package by AAMC.

Data analysis was done mainly with IBM SPSS Statistics for Windows (Version 25.0). First, the Pearson correlation analysis was employed to measure how all the variables were related to each other. Then, a multiple linear regression analysis was carried out with the MCAT score as the dependent variable, and UGPA, English proficiency (defined by a student’s IELTS Score), the number of practice exams students completed, and various measures of full-length practice exam performance as regressors. Two regression models were tested sequentially to investigate to what extent English proficiency and practice exam performance contribute to MCAT score independently. Model 1 included only background information, UGPA, and English proficiency. In addition to Model 1, Model 2 also included the measures for the practice exams, the number of exams, and different measures of full-length practice exam performance. Two of the 3 measures, the median and maximum scores, were used in the final model because the most recent score showed a very strong positive correlation with the former 2 variables and could potentially induce a collinearity issue for the regression model. Figure 1 illustrates the 2 models. The R-square (R2) and R-square change were used to evaluate and compare the goodness-of-fit for the 2 models.

Figure 1.

The 2 models of the multiple linear regression.

Model 1:

Model 2:

Results

Descriptive analysis

The number of full-length practice exam the students completed ranged from 1 to 11 exams, and on average, they finished a total of 7 exams. Figure 2 shows the average performance on each full-length practice exams and the average MCAT score. As expected, performance improved with more practice exams, especially for the first 4 full-length practice exams. The average MCAT score on the test day was close to the performance level that the students reached after 4 full-length practice exams, but was lower than the group maximum score (paired t-test results: t(16) = 3.18, P = 0.006). The average score on the MCAT was closer to the maximum score on the Kaplan-based practice exams than to the AAMC-based practice exams.

Figure 2.

Group average score for each full-length practice exam and the average MCAT score. The black square indicates the scores for Kaplan-based practice exams; the black triangle indicates the scores for AAMC-based practice exams respectively; and the red circle indicates the MCAT score. The number at the right bottom corner of the markers indicates the number of students who took the exam. Error bars represent the standard deviations.

Pearson correlation analysis

We found that not only English proficiency but also the performance on the full-length practice exams positively correlated with MCAT score (see Table 1). Students with higher levels of English proficiency scored better on the actual MCAT exam (r = 0.77, P = 0.000). Students with better scores in the full-length practice exam, regardless of the types of measure, scored higher on the MCAT as well (r = 0.60, P = 0.006 for the maximum score; r = 0.92, P = 0.000 for the median score; r = 0.79, P = 0.000 for the most recent score). We also found the number of full-length practice exams that students completed correlated with the most recent score (r = 0.46, P = 0.048). The more practice the students took, the better they performed in their final practice exam. However, the amount of practice per se did not significantly affect the performance on the test day (r = 0.24, P = 0.32). Unexpectedly, we did not find that undergraduate GPA correlated with any aspect of students’ performance in either full-length practice exams or the actual MCAT exam.

Table 1.

Correlation coefficients with P values in the parentheses.

	MCAT score	UGPA	English proficiency	Number of exams	Maximum score	Median score
UGPA	0.04 (0.87)
English proficiency	0.77 (0.000)	0.07 (0.77)
Number of exams	0.24 (0.32)	−0.11 (0.67)	0.05 (0.83)
Maximum score	0.60 (0.006)	0.16 (0.51)	0.48 (0.04)	0.24 (0.32)
Median score	0.92 (0.000)	0.18 (0.46)	0.62 (0.004)	0.21 (0.40)	0.65 (0.003)
Most recent score	0.79 (0.000)	0.06 (0.81)	0.53 (0.02)	0.46 (0.048)	0.72 (0.000)	0.82 (0.000)

Multiple linear regression analysis

The Pearson correlation analysis showed that the 3 measures of practice exam performance, the median score, the maximum score, and the most recent score, correlated with each other strongly. The most recent score was excluded from the multiple linear regression analysis to avoid multicollinearity, because this measure most strongly correlated with the other 2 measures. The multicollinearities of the models were measured using the variance inflation factor (VIF). And for the models reported here, all variables had VIF smaller than 3.3, which indicated a neglectable level of multicollinearity.[8]

Table 2 shows the results of the multiple linear regression analysis. Both Model 1 and Model 2 were significantly different from the null hypothesis (F(2,16) = 11.94, P = 0.001 for Model 1; and F(5,13) = 32.98, P = 0.000). More importantly, a comparison between Model 1 and Model 2 suggested that including the measures of practice exams significantly improved the regression model (Model 2, F change = 19.46, P = 0.001). When both English proficiency and full-length practice exam performance were included in Model 2, the standardized regression coefficient β was 0.34 for English proficiency, but 0.74 for the median score of practice exams, indicating that practice exam performance had a more substantial contribution than English proficiency for predicting MCAT scores.

Table 2.

Results from the multiple linear regression analysis predicting MCAT score.

Models and predictors	Coefficients of predictors					Goodness-of-fit of models
	b	SE b	β	t	P	R 2	F Change	P for F Change
Model 1 background info						0.60	11.94	0.001
Constant	442.34	13.73		32.22	0.000
UGPA	−0.28	2.90	−0.02	−0.10	0.93
English proficiency	7.31	1.50	0.78	4.88	0.000
Model 2 practice exam performance						0.93	19.46	0.000
Constant	37.63	57.03		0.66	0.52
UGPA	−1.86	1.42	−0.10	−1.31	0.21
English proficiency	3.18	0.92	0.34	3.46	0.004
Number of Exams	0.19	0.22	0.07	0.87	0.40
Median score	0.92	0.14	0.74	6.50	0.000
Maximum score	−0.03	0.08	−0.04	−0.37	0.72

Abbreviations: F Change, F-test statistics for R2 change between 2 models; SE b, Standard error for b.

Discussions

The current study aimed to investigate whether full-length practice exams could predict performance in the new MCAT exam and how much the practice exam performance could predict MCAT score independently of English proficiency. The results showed that students improved gradually over time with more practice exams. More importantly, we found that performance in the full-length practice exams can predict MCAT score, independently of English proficiency; and compared to English proficiency, full-length practice exam performance showed stronger predictive value for the MCAT score.

Since the MCAT exam is administrated in English, we expected to find that English proficiency predicts MCAT scores. In fact, English proficiency may continue to predict clinical performance in medical school for non-native English speakers.[9] These findings support the emphasis of English proficiency criterion for admission to premedical programs and medical schools.

Importantly, our results highlighted that the performance in full-length practice exams could also significantly predict MCAT score, with an even higher predictive value than English proficiency, as revealed by the standardized regression coefficients in Model 2. This finding suggests that while it is necessary to have a certain level of English proficiency to score high in MCAT, the major contributing factor to performing well on the MCAT may not be language proficiency, but knowledge related to specific science subjects. In fact, these knowledge, as reflected by the performance in the new MCAT exam, may continue to predict medical school performance as reported by a recent study.[10] Our result suggests that students’ performance in full-length practice exams could also be a valid evaluation on their scientific knowledge. We believe that such results may provide admission committees with additional evidence to support student entry during the current pandemic when taking the MCAT exam is not feasible.

While several existing articles have suggested that examinees do full-length practice exams for MCAT preparation,[11,12] to our knowledge, the current study is the first documentation to show the benefit of full-length practice exams empirically. Contrary to the suggestion to use full-length practice exams to drill exam functionality (e.g., highlight, strikethrough),[12] we strongly recommend that examinees use full-length practice exams as a learning resource throughout the whole MCAT preparation stage. The benefits of testing on learning have been observed not only in cognitive science laboratories[4,5] but also in classrooms.[13] There are many benefits, including, but not limited to, that repeated retrieval through testing could enhance long-term retention; post-test feedback could enhance learning; and practice exam scores could improve study plans. Consistent with these studies, we found that our students’ performance improved with practice exams, and their performance on these exams, as measured with the median score, could strongly predict how well they would perform on the MCAT test day.

Additionally, we did not find the maximum score of the practice exams could predict the MCAT score. This finding may indicate that students could not keep up with their top performance during the practice on the test day. One main reason could be test anxiety.[14] According to Stöber’s study,[14] there are at least 4 aspects of test anxiety that could impair students’ performance during a test, which are worry, lack of confidence, emotionality, and interference. And only 2 aspects, worry and lack of confidence possible, are likely to decrease with better task-oriented preparation. Despite the effort of our practice-based curriculum, the level of preparation obtained from full-length practice exams may help to relieve certain aspects of test anxiety, but not all of it. How to manage all different aspects of test anxiety, which may influence MCAT performance, need to be addressed in future studies to help the examinees better cope with this competitive exam.

Conclusion

In summary, the current study aims to provide first direct empirical evidence supporting that doing full-length practice exams regularly during MCAT preparation is a good strategy. This study has several limitations. We have a relatively small sample size due to the pandemic situation. And the majority of the students in the sample had engineering backgrounds rather than a pre-medicine background, which may explain why we did not find any correlations between UGPA and MCAT score. Despite these limitations, we still found a substantial predictive value of the full-length practice exam. The results could be specifically inspiring for minority examinees who speak English as a second language and come from a non-science background.