Can we predict failure in licensure exams from medical students' undergraduate academic performance?

BACKGROUND: In 2015, the Medical Council of Canada increased the minimum pass level for the Medical Council of Canada Qualifying Examination Part I, and students had a higher rate of failure than in previous years. The purpose of this study was to predict students at an increased odds of examination failure to allow for early, targeted interventions.
METHODS: We divided our dataset into a derivation cohort and two validation cohorts and used multiple logistic regression to predict licensing examination failure. We then performed receiver operating characteristics and a sensitivity analysis using different cutoffs for explanatory variables to identify the cutoff threshold with the best predictive value at identifying students at increased odds of failure.
RESULTS: After multivariate analysis, only pre-clerkship GPA was a significant independent predictor of failure (OR 0.76, 95% CI [0.66, 0.88], p < 0.001). The probability of failure increased steeply when the pre-clerkship GPA fell below 80% and 76% was found to be the most efficient cutoff for predicting failure (OR 9.37, 95% CI [3.08, 38.41]).
CONCLUSIONS: Pre-clerkship performance can predict students at increased odds of licensing examination failure. Further studies are needed to explore whether early interventions for at-risk students alter their examination performance.

Introduction

The best way to predict the future is to study the past, or prognosticate

- Robert Kiyosaki

Successful completion of medical licensure examinations is a pre-requisite to practice medicine in the United States and Canada. Furthermore, high failure rates on these examinations is felt to reflect poorly upon the quality of education offered at various institutions. Thus, improving performance on licensure examinations is desirable for both students and their Undergraduate Medical Education (UME) programs. In order to improve performance on a licensure examination, we must first identify students who are at increased risk of failure on this examination so that we can then target them for academic mentoring. But how and when can we best identify students at increased risk of failing a licensure examination?

Previous studies have identified several early (pre admission)[1]-[6] as well as later (post admission)[5]-[10] variables that associate with student performance on licensure examinations. Pre admission variables with significant association have included overall scores on the Medical College Admission Test (MCAT),[1],[3] scores on the biologic science[2],[11] and verbal reasoning[6] section of the MCAT, undergraduate Grade Point Average (GPA),[1] science GPA,[2] multiple mini interview scores,[4] race,[2] age,[2],[12] gender,[6],[12] and college selectivity.[2] Post admission variables have included performance in individual first year courses,[7],[9] overall performance in the first[1],[6],[7],[9] and second[1],[6] years of undergraduate medical programs, clerkship in-training evaluation reports[10], national board of medical examiners (NBME) scores,[5] block scores,[5] clinical performance,[1],[8] and performance on a final Objective Structured Clinical Examination (OSCE).[6] Unfortunately, the majority of these studies attempted to predict performance on the United States Medical Licensing Examination (USMLE)[1]-[3],[7],[8],[13],[14] and not the Medical Council of Canada Qualifying Examination (MCCQE) Part I. Less is known about predicting licensing examination outcomes in the Canadian context.

Many variables that predict licensing examination performance in the United States have not shown to associate with outcomes on the MCCQE Part I, including MCAT total score, MCAT verbal reasoning score, GPA, and MMI ratings.[11] When it comes to predicting MCCQE Part I scores, although a number of models have been developed, broad application of these models to identify students in need of intervention has been limited by the types of variables included, such as NBME scores[5] (which are not used at all Universities) and the proximity of these variables to the MCCQE Part I.[6] Models that show good predictive ability but that incorporate variables that occur close to the MCCQE part I leave limited time for intervention for students deemed to be at risk of exam failure. Given the available literature, it is not possible for all medical schools to identify a group of students at-risk of MCCQE Part I failure early on during their medical training. If this gap were addressed, schools could implement programs that target these students for pre-emptive academic mentoring designed to reduce the odds of licensure examination failure.

In this study, our goal was to identify simple academic performance variables that can predict students at increased odds of failure on the MCCQE Part I Licensure examination. We considered three easy to identify indicators of academic performance as our explanatory variables: prior examination failure, pre-clerkship GPA, and clerkship GPA. We created a derivation cohort and two validation cohorts from our dataset. In our derivation cohort, we used multiple logistic regression to identify predictors of students failing the licensure examination and performed a sensitivity analysis to identify the cutoff threshold with the best predictive value. We then applied this cutoff threshold to our two validation cohorts to test the robustness of our predictors and cutoff threshold.

Methods

Study design

The purpose of this study was to predict students at increased odds of MCCQE Part I failure to allow for early, targeted interventions. We used a retrospective derivation-validation cohort design. Our study was approved by the Conjoint Health Research Ethics Board at the Cumming School of Medicine, University of Calgary.

Setting

This study was completed at the Cumming School of Medicine in Calgary, Alberta. At the time of this study, we had a three-year undergraduate program that includes two years of pre-clerkship comprising seven combined, system-based courses in addition to longitudinal courses in Population Health, Ethics, Evidence Based Medicine, and Medical Skills. The final year of our curriculum is a clinical clerkship, of which there are two versions: a rotation-based clerkship that includes mandatory rotations in Anesthesia, Emergency Medicine, Family Medicine, Internal Medicine, Obstetrics and Gynecology, Pediatrics, Psychiatry, and Surgery; and a longitudinal integrated clerkship where students spend nine months in a primary care setting in addition to hospital-based rotations in Internal Medicine, Pediatrics, and Surgery. In both versions of clerkship students also had twelve weeks of clinical electives in addition to longitudinal clinical skills training.[15] Regardless of which version of clerkship the students were enrolled in, all students completed the same mandatory clerkship examinations during this final year of training. The curriculum was stable during this study period and the assessments were standardized.

Sample size and sampling methods

Our participants were medical students who completed the undergraduate curriculum at the Cumming School of Medicine and then sat the MCCQE Part I examination for the first time in the spring of 2013 (n = 159), 2014 (n = 173), and 2015 (n = 167).

Study protocol

We used two sources of data in our study. The first was performance on summative assessments that are part of our undergraduate curriculum, and the second was performance on the MCCQE Part I. Canadian medical students write the MCCQE Part I after their final year of UME training. This examination, which at the time included 3.5 hours of Multiple Choice Questions (MCQs) and 4 hours of Clinical Decision Making (CDM) questions, assesses students’ knowledge in the seven Canadian Medical Education Directives for Specialists (CanMEDS) domains. In 2015, the Medical Council of Canada (MCC) appointed a standard-setting panel to develop a new scale for scoring the examination. Before this standard setting procedure, the examination had a fixed Minimum Pass Level (MPL) of 390 and standard deviation of 100. After this procedure, scores had a range of 50 to 900, with a pre-set mean of 500, standard deviation of 100, and MPL of 427. This new MPL corresponded to a score of 440 on the previous scoring scale, which is higher than the previous MPL of 390.[6]

From our pre-clerkship summative assessments, we calculated the pre-clerkship GPA, which was the mean of scores on all pre-clerkship summative examinations (Appendix A). We calculated the clerkship GPA in a similar way by averaging scores on summative MCQ examinations for the eight mandatory clerkship rotations. Finally, we also noted if students had ever failed a summative examination during their undergraduate training.

For the 2015 cohort we dichotomized students into pass vs. fail depending upon whether their score on the MCCQE Part I examination was >= 427 or < 427, respectively. For students from the 2014 and 2013 cohorts, rather than using the previous MPL of 390, we applied the revised MPL (which equated to 440 on the old scoring scale) and dichotomized students into pass vs. fail using a cutoff of 440.

Outcome measures and data analysis

We divided our dataset into a derivation cohort (Class of 2015) and two validation cohorts (Class of 2014 and 2013). In our derivation cohort we used multiple logistic regression in which our dependent variable was MCCQE Part I examination failure (coded as failure = 1, pass = 0) and our potential explanatory variables were pre-clerkship GPA, clerkship GPA, and prior summative examination failure. We also considered all two-way interactions between our explanatory variables and performed backward elimination in our regression model, beginning with the interaction terms.[16],[17] From the results of our logistic regression we then generated probability of MCCQE Part I examination failure using the equation: probability = ea+bx/1+ea+bx.[17]

To evaluate the predictive ability of our explanatory variables we performed receiver operating characteristic (ROC) analysis where our outcome was MCCQE Part I examination failure. We then performed a sensitivity analysis using different cutoffs for explanatory variables to identify the cutoff threshold with the best predictive values and that we felt was most efficient at identifying students at increased odds of MCCQE Part I examination failure. We used a two-sample t-test to compare means and Cohen’s d to estimate effect size.

Having identified the optimal cutoff threshold for our derivation cohort we then applied this cutoff threshold to students from the Class of 2013 and 2014 to evaluate the predictive performance of this threshold cutoff in these validation cohorts. We used STATA® Version 11.0 (StataCorp, College Station, Texas) for our statistical analyses.

Results

For the derivation cohort (class of 2015), 17 students (10.2%) failed the MCCQE Part I. In the two validation cohorts, 13 (8.2%) would have failed in 2013 and 17 (9.8%) would have failed in 2014 had the new MPL been applied in those years. Demographic data, pre-clerkship GPA, and clerkship GPA for these three cohorts are shown in Table 1.

Table 1

Demographic data and GPAs for derivation and validation cohorts

Year	n	Mean age (SD)	Percent female	Mean preclerkship GPA (SD)	Mean clerkship GPA (SD)
2015	167	29.7 (4.2)	54.4 %	82.1% (4.7)	81.8% (3.7)
2014	173	30.5 (3.7)	54.0%	81.7% (5.0)	82.5% (4.3)
2013	159	30.8 (3.8)	52.1%	80.6% (4.5)	82.2% (4.0)

Derivation cohort

For the derivation cohort (class of 2015), each of the potential explanatory variables (prior summative examination failure, pre-clerkship GPA, and clerkship GPA) were associated with MCCQE Part I examination failure by univariate logistic regression (p < 0.05 for each). When we entered all three variables into our multiple logistic regression model we found no significant interaction between variables and only pre-clerkship GPA remained as a significant independent predictor of MCCQE Part I examination failure (odds ratio 0.76, 95% confidence interval [0.66, 0.88], p < 0.001). This means that for every unit increase in pre-clerkship GPA (here, this equals a 1% increase) the odds ratio for failing the MCC Part I falls by 24%. When treated as a continuous variable, pre-clerkship GPA was a “good predictor” of MCCQE Part I examination failure (area under the ROC curve 0.83 [0.74,0.91]) (Figure 1).

Figure 1

Predictive performance of pre-clerkship GPA on MCCQE Part I examination failure for undergraduate medical students at the University of Calgary (graduating class of 2015)

When we plotted the relationship between pre-clerkship GPA and MCCQE Part I examination failure, we observed that the probability of failure increased steeply when the pre-clerkship GPA fell below 80% (Figure 2). Based upon this graph, we decided to focus on the predictive ability of different pre-clerkship GPA cutoff thresholds between 75 and 80.

Figure 2

Association between pre-clerkship GPA and probability of failing the MCCQE Part I examination for undergraduate medical students at the University of Calgary (graduating class of 2015)

The positive and negative predictive values for the different thresholds and the odds ratio for MCCQE Part I examination failure for cohorts selected based upon these thresholds is shown in Table 2.

Table 2

Odds of MCCQE Part I examination failure and predictive values of pre-clerkship GPA at different cutoff thresholds for undergraduate medical students at the University of Calgary (graduating class of 2015)

GPA cutpoint	n below	OR for failure [95% CI]	PPV	NPV
75	10	4.65 [1.08, 20.02]	0.30 [0.10, 0.61]	0.92 [0.86, 0.95]
76	21	9.37 [3.08, 28.41]	0.38 [0.20, 0.60]	0.94 [0.88, 0.97]
77	28	6.18 [2.13, 17.88]	0.29 [0.15, 0.48]	0.94 [0.88, 0.97]
78	36	5.38 [1.90, 15.19]	0.25 [0.13, 0.42]	0.94 [0.88, 0.97]
79	41	5.76 [2.03, 16.35]	0.24 [0.13, 0.40]	0.95 [0.89, 0.98]
80	50	5.50 [1.90, 15.86]	0.22 [0.12, 0.36]	0.95 [0.89, 0.98]

Based upon the data shown in Table 2, we considered the most efficient cutoff for pre-clerkship GPA to be 76. At this threshold we identified a cohort of 21 students, of which 8 (38.1%) would go on to fail the MCCQE Part I examination. When we compared clerkship GPA and MCCQE Part I examination scores for this cohort to the 152 students with pre-clerkship GPA ≥76 we found that scores for the cohort with pre-clerkship GPA <76 were significantly lower (with a large effect size) on both (77.9 (SD 3.2) vs. 82.4 (3.4), p < 0.0001, d = 1.36 for clerkship GPA, and 444.7 (SD 56.5) vs. 533.5 (73.7), p < 0.0001, d = 1.35 for MCCQE Part I examination).

Validation cohort

When we applied the pre-clerkship GPA cutoff of 76 to students from the class of 2014 and 2013 we identified 27 and 26 students, respectively. Of these, 10 in each year would have failed the MCCQE Part I examination if the pass mark had been set at the 2015 standard (representing 37% of the identified students in the 2014 cohort and 38.5% of the 2013 cohort). The odds ratio for MCCQE Part I examination failure for cohorts identified by the pre-clerkship GPA cutoff of 76 is shown in Table 3, along with the positive and negative predictive values.

Table 3

Odds of MCCQE Part I examination failure and predictive values of pre-clerkship GPA cutoff of 76 in derivation and validation cohorts

Assessment	Derivation cohort (class of 2015)	Validation cohort 1 (class of 2014)	Validation cohort 2 (class of 2013)
Odds ratio of cohort failing MCCQE Part I examination	9.37 [3.08, 28.41]	11.68 [3.93, 34.72]	27.08 [6.74, 108.83]
Positive predictive value	0.38 [0.20, 0.60]	0.37 [0.21, 0.56]	0.38 [0.22, 0.58]
Negative predictive value	0.94 [0.88, 0.97]	0.95 [0.90, 0.98]	0.98 [0.93, 1.00]

Discussion

Our goal was to identify students at high risk of failing the Canadian licensing examination and, ideally, to do so at a time point that allows for mentoring interventions aimed at improving examination performance. We found that using a pre-clerkship GPA cutoff of 76%, we could accurately identify students at increased odds of failure more than one year prior to the licensure examination. The high odds of licensure examination failure associated with this cut-off was confirmed in our two validation cohorts, suggesting reliability of this finding.

We are not the first group to use mathematical modeling to predict licensing examination performance in undergraduate medical students. Unlike previous studies that analyzed demographic and preadmission variables[1]-[6] we intentionally excluded these from our analysis. A previous study by Raman et al[11] explored the impact of many of these variables (including MCAT sub-scores for the biological sciences, physical sciences, and verbal reasoning sections, undergraduate GPA, and MMI scores) in the same cohort of students (Cumming School of Medicine graduates from the classes of 2013 through 2016) and found that beyond a barely significant (p = 0.04) association with the MCAT biological science score (from a version of the MCAT that is no longer used) there was no impact of any of these variables on MCCQE Part I performance.

Other groups have similarly looked at medical school performance indicators and found that these can predict licensing examination outcomes.[5]-[10] In the Canadian context, Roy et al. performed linear regression to predict Medical Council of Canada Qualifying (MCCQE) Part 1 scores and derived a model that gave a good estimate of student scores.[5] However, in that study, the majority of the variation in the MCCQE Part 1 (59.2%) was accounted for by the national board of medical examiners scores, which are subject examinations that are not routinely administered at other Canadian medical schools, limiting the usefulness of these findings. Another Canadian study used hierarchical generalized linear models to create predictive equations at various time points (admission, year 1, year 2, and pre-MCCQE part 1) along with receiver operating characteristic curves to determine the predictive ability of these models for historical and prospective cohorts.[6] Unfortunately, they found that the only model that was able to predict performance in a prospective cohort was the pre-MCCQE part 1 model, which included distal curricular variables, limiting the time available to intervene to assist students at risk of exam failure.

Using a pre-clerkship GPA cutoff to identify students at increased odds of licensure examination failure allows for early intervention for students who may previously not have been identified to be at risk. At our centre, students are promoted from pre-clerkship to clerkship if they pass all evaluation components in the pre-clerkship curriculum. Given that we are a pass-fail school, a cumulative pre-clerkship GPA is not calculated or used. Prior to this study, students were identified as struggling in the curriculum if they failed a pre-clerkship course. Once this occurred, they were offered a meeting with an Assistant or Associate Dean to discuss the failure and identify any precipitating events. If this student went on to fail a second pre-clerkship course, they were reviewed at our Student Academic Review Committee, who would determine if an intervention (such as repetition of a pre-clerkship year) was necessary. The same approach was taken during clerkship, where students were reactively identified for intervention after one or more rotation failures. The use of a cumulative GPA cut-off, although not usually calculated in a pass-fail system, has the advantage of identifying both students that are overtly struggling and those that are passing all of their courses, but repeatedly getting grades that are close to the MPL. As others have noted,[9],[18] as the majority of undergraduate medical students pass their examinations the use of a pass-fail system can lead to non-recognition (by both faculty members and the students themselves) of the need for intervention. The use of a medical student GPA cutoff that identifies students who are at higher odds of licensure examination failure but aren’t overtly failing in their UME curriculum can identify previously unrecognized students who are struggling and may benefit from additional support.

In our derivation cohort, pre-clerkship GPA was the only significant independent predictor of MCCQE Part I examination failure. Pre-clerkship GPA was likely a better predictor of licensure examination outcomes than clerkship GPA because, at our institution, pre-clerkship examination content is better matched to the content and format of the MCCQE Part I, including content on population health and ethics. In addition to this, pre-clerkship GPA incorporates student performance during two academic years, which may provide a more reliable assessment of students’ knowledge than clerkship GPA, which incorporates performance from only one year of training. Pre-clerkship GPA also incorporates both MCQ and OSCE examination scores, whereas clerkship GPA incorporates only MCQ examinations. A previous study has shown that for internal medicine residents, OSCE progress test scores not only correlate with written national licensing examination scores, but that they can help identify those at in increased risk of examination failure.[19] Failure of an individual examination was likely a poor predictor of licensure examination performance for a number of reasons. First, students who fail an examination will often have a clear precipitating factor such as a health concern or life event that then resolves. Second, at our centre, an examination failure triggers academic counseling, which does not occur for students who are passing with a GPA just above MPL. Lastly, students who are below the MPL on an examination may be more motivated to change their learning behaviour than students just above MPL.

While our study is not the first to identify variables associated with licensure examination performance, we feel that our findings make a meaningful addition to the existing literature. Similar to previous studies, we started by using a regression model to identify variables associated with examination performance. However, after identifying a single variable that was independently associated with licensure examination performance (pre-clerkship GPA), we identified a discreet cut point in this continuous variable that consistently identified students with high odds of failure (range in odds ratio of 9.37-27.08 over the three cohorts studied). When we combine the reliability of this finding with the timing of the pre-clerkship GPA result (more than one year prior to the licensure examination), we feel that the dichotomized pre-clerkship GPA result may provide a straightforward and reliable predictor of students at increased risk of failure on the licensure examination and who may, therefore, benefit from interventions aimed at reducing the risk of failure.

There are some important limitations of this study that we should highlight. First, as the scale for scoring the MCCQE Part I changed in 2015, we had to apply the new scale and MPL to our historical validation cohorts retrospectively in order to predict which students would have failed the MCCQE Part I using the new MPL (rather than students who did fail based upon the historical MPL). Unlike other studies, we did not include any pre-admission variables in our regression model, and it would be interesting to explore whether pre-admission variables remain as independent predictors when combined with pre-clerkship GPA. This was a single centre study and, since our curriculum differs from many others due to its structure (132 curricular weeks condensed into a three year time frame) and curricular format (clinical-presentation based), our results may not generalize to other centres. However, a similar process could be applied at other centres to identify local performance indicators that predict increased odds of MCCQE Part I failure. Finally, our results are based upon retrospective data rather than prospective validation. Our justification for this is that we have now introduced an academic mentoring program based upon our pre-clerkship GPA. That we now intervene when the pre-clerkship GPA is less than 76 introduces a performance bias that reduces the validity of our GPA threshold as a predictor of licensure examination performance. Future research will compare the relative risk of MCCQE Part I failure between students involved in our academic mentoring program and historical controls and explore the intended and unintended consequences of introducing this program.

Conclusion

Using early performance indicators to predict performance on licensure examinations is challenging, but offers potential benefits. Here we have described the process that we use to identify students at high risk of failure on the Canadian licensure examination. Given the differences between undergraduate curricula at different medical schools, our results are not applicable to other centres. However, the process whereby we identified predictor variables and cut-off thresholds should be. Further studies are needed to confirm these findings from other centres and explore whether or not early identification and mentoring of students at increased risk of licensure examination failure alters their examination performance.

Assessment	Format	Content	Weight in preclerkship GPA	% Coefficient of Variation (calculated from class of 2012 to 2018)
Course 1	Type A MCQ items	Hematology, Gastroenterology, Infectious Disease	0.091	1.93
Course 2	Type A MCQ items	Rheumatology, Orthopedics, Dermatology	0.091	1.69
Course 3	Type A MCQ items	Cardiology, Respirology	0.091	2.65
Course 4	Type A MCQ items	Nephrology, Endocrinology	0.091	1.96
Course 5	Type A MCQ items	Neurology, Geriatrics	0.091	2.00
Course 6	Type A MCQ items	Obstetrics, Gynecology, Pediatrics	0.091	1.07
Course 7	Type A MCQ items	Psychiatry	0.091	3.71
Population Health	Type A MCQ items	Population Health and Ethics	0.091	Insufficient data to calculate CoV
Evidence-Based Medicine	Type A MCQ items	Evidence-based medicine	0.091	Insufficient data to calculate CoV
Medical skills I	OSCE	Clinical skills	0.091	Insufficient data to calculate CoV
Medical skills II	OSCE	Clinical skills	0.091	Insufficient data to calculate CoV