Institutional differences in USMLE Step 1 and 2 CK performance: Cross-sectional study of 89 US allopathic medical schools.

INTRODUCTION: The United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge (CK) are important for trainee medical knowledge assessment and licensure, medical school program assessment, and residency program applicant screening. Little is known about how USMLE performance varies between institutions. This observational study attempts to identify institutions with above-predicted USMLE performance, which may indicate educational programs successful at promoting students' medical knowledge.
METHODS: Self-reported institution-level data was tabulated from publicly available US News and World Report and Association of American Medical Colleges publications for 131 US allopathic medical schools from 2012-2014. Bivariate and multiple linear regression were performed. The primary outcome was institutional mean USMLE Step 1 and Step 2 CK scores outside a 95% prediction interval (≥2 standard deviations above or below predicted) based on multiple regression accounting for students' prior academic performance.
RESULTS: Eighty-nine US medical schools (54 public, 35 private) reported complete USMLE scores over the three-year study period, representing over 39,000 examinees. Institutional mean grade point average (GPA) and Medical College Admission Test score (MCAT) achieved an adjusted R2 of 72% for Step 1 (standardized βMCAT 0.7, βGPA 0.2) and 41% for Step 2 CK (standardized βMCAT 0.5, βGPA 0.3) in multiple regression. Using this regression model, 5 institutions were identified with above-predicted institutional USMLE performance, while 3 institutions had below-predicted performance.
CONCLUSIONS: This exploratory study identified several US allopathic medical schools with significant above- or below-predicted USMLE performance. Although limited by self-reported data, the findings raise questions about inter-institutional USMLE performance parity, and thus, educational parity. Additional work is needed to determine the etiology and robustness of the observed performance differences.

Introduction

The United States Medical Licensing Examination (USMLE) is a 3-step examination required for medical licensure in the United States. The first two exams, USMLE Step 1 and Step 2 Clinical Knowledge (CK), assess medical students’ mastery of basic biomedical principles and their clinical applications [1,2]. About 40,000 trainees take each exam annually, of which over 35% are non-US/Canadian medical students [3]. Both exams are high-stakes parameters of medical student performance critical for advancement [4], residency applicant screening and selection [5,6], and future board certification [7]. Multiple studies have demonstrated correlations between individual factors–including Medical College Admission Test (MCAT) score [8], undergraduate grade point average (GPA) [9], and study behaviors [10]–and USMLE performance. However, little is known about institutional USMLE performance variation. One group analyzing data from the 1990s demonstrated that institutional variables, including curricular differences, did not predict USMLE performance [11,12]. A recent study using one year of national data found some evidence of inter-institutional USMLE performance differences, but the short study duration precludes definitive conclusions [13].

In this exploratory, institution-level study, we analyze institutional variation in USMLE Step 1 and Step 2 CK performance relative to mean matriculant GPA and MCAT. Our primary objective was to identify institutions with above-predicted USMLE performance, which may indicate educational programs successful at promoting students’ medical knowledge.

Methods

This observational study was conducted in accordance with the STROBE guidelines for observational studies in epidemiology [14].

Data sources

We manually tabulated self-reported institutional data–aggregate percentages and means representing yearly medical student cohorts at single institutions–from the annual US News and World Report “Best Graduate Schools” publication (2008–2016 editions) [15] and the Association of American Medical Colleges (AAMC) Medical School Admission Requirements publication (2008–2012 editions) [16] for all 131 US allopathic medical schools. Osteopathic institutions were excluded from this study, as osteopathic students typically take the COMLEX licensing examination rather than the USMLE and very few US osteopathic institutions reported USMLE performance data. A sample size calculation was not performed because we obtained available data for a census of US allopathic medical schools during the study period. National averages for all allopathic matriculants and examinees were obtained from official AAMC [17] and USMLE sources [18,19]. Institutional Review Board approval was not required as no human subjects or identifiable data were involved.

Primary outcome measures and predictor variables

The primary outcome measures were institutional mean USMLE Step 1 and 2 CK scores, averaged over the 3-year study period 2012–2014. Predictor variables included students’ prior academic performance (institutional mean undergraduate GPA and MCAT, averaged over 3 years) and demographics (percentage non-traditional students, minority students, undergraduate biological sciences or humanities majors), and medical school factors (acceptance rate, public/private status, faculty-to-student ratio, National Institutes of Health research funding, graduates entering primary care). MCAT scores represented total scores computed as the sum of the average institutional scores on all 3 sections (biological sciences, physical sciences, verbal reasoning). Institutional USMLE scores were matched to institutional GPA and MCAT averages from two or four years prior (for Step 1 or 2 CK, respectively) to account for the typical lag between matriculation and USMLE testing.

Statistical analysis

All analysis was at the institution level. We performed ordinary least squares linear regression analysis, with test of Pearson’s r for bivariate correlations. Conditions of linearity, nearly normal residuals, and homoscedasticity were checked [20]. Institutions with 3-year average USMLE performance outside a 95% prediction interval (regression residual ≥2 standard deviations, SD, from predicted) were identified [21]. Hypothesis tests were 2-sided with α = .05; ANOVA was used to confirm overall significance of multiple regressions. Statistical analysis was done using SPSS version 25.0 (SPSS Inc., Chicago, Illinois).

Results

In total, 89 (54 public and 35 private) of 131 US allopathic medical schools reported complete USMLE scores over the 3-year study period (68% reporting rate), representing 39,615 and 39,252 Step 1 and 2 CK examinees, respectively. Among reporting institutions, the institutional mean USMLE Step 1 score was 229.7 (SD 5.5) and Step 2 CK score was 238.3 (SD 4.7) (Table 1). GPA and MCAT scores showed minimal heterogeneity across the study years (data not shown). USMLE scores increased across the study years, which was also observed nationally. The average GPA, MCAT scores, and USMLE Step 1 scores for the 89 reporting institutions were slightly higher than national averages for all matriculants/examinees. Complete GPA, MCAT, and USMLE data for reporting institutions and nationally are provided in S1 Table.

Table 1

Average GPA, MCAT, and USMLE Step 1 and 2 CK score among 89 US allopathic medical schools and nationally.

	All Schools (n = 89)Average (SD)	Public (n = 54)Average (SD)	Private (n = 35)Average (SD)	P value*	National Average
GPA 2010–2012	3.70 (0.08)	3.68 (0.07)	3.73 (0.08)	ns	3.67
MCAT 2010–2012	31.9 (2.2)	30.9 (1.7)	33.5 (2.0)	< .001	31.1
USMLE Step 1
2012	227.6 (6.1)	225.4 (5.3)	230.9 (5.6)	< .001	227
2013	230.4 (5.8)	228.1 (4.9)	233.9 (5.3)	< .001	228
2014	231.1 (5.6)	229.0 (4.6)	234.4 (5.6)	< .001	229
2012–2014 (combined)	229.7 (5.5)	227.5 (4.4)	233.1 (5.2)	< .001	228.0
USMLE Step 2 CK
2012	235.6 (5.5)	234.7 (5.1)	237.0 (5.8)	ns	237
2013	238.8 (5.3)	237.6 (4.8)	240.7 (5.5)	< .01	238
2014	240.5 (4.5)	239.3 (4.1)	242.4 (4.4)	< .01	240
2012–2014 (combined)	238.3 (4.7)	237.2 (4.2)	240.0 (4.9)	< .01	238.3

GPA, Undergraduate Grade Point Average; MCAT, Medical College Admission Test score; USMLE, US Medical Licensing Examination; CK, Clinical Knowledge; ns, not significant at P < .05 threshold

* Two-tailed t-test comparing public to private

Predictors of institutional USMLE performance

The strongest predictor of institutional USMLE scores was prior student academic performance, including undergraduate GPA (Step 1, Pearson’s r = .64; Step 2 CK, r = .53; both P < .001) and MCAT score (Step 1, r = .84; Step 2 CK, r = .62; both P < .001). Numerous student body demographic and institutional factors had moderately strong correlations with institutional USMLE scores in bivariate regression; however, when controlling for GPA and MCAT, these correlations were weak and no longer significant (Table 2). For example, private institutions were correlated with higher USMLE Step 1 scores (r = .51, P < .001), but this correlation vanished after controlling for GPA and MCAT (r = .12, P = .42), as private institutions recruit students with higher MCAT scores compared to public institutions (mean 33.5 vs. 30.9, difference 2.7, 95% CI 1.9–3.5; P < .001).

Table 2

Linear regression between various institutional characteristics and institutional USMLE performance, without and with control for average institutional GPA and MCAT.

USMLE Step 1	Pearson’sr	Partialρ†	USMLE Step 2 CK	Pearson’sr	Partialρ‡
Institutional GPA	.64**	--	Institutional GPA	.53**	--
Institutional MCAT	.84**	--	Institutional MCAT	.62**	--
USMLE Step 2 CK	.56**	.05	USMLE Step 1	.56**	.06
Minority Students	.46**	.16	Minority Students	.25*	-.03
Biological Science Majors	-.36**	-.07	Biological Science Majors	-.27*	-.12
Humanities Majors	.13	.07	Humanities Majors	.10	.11
Non-Traditional Students	.01	-.13	Non-Traditional Students	-.03	-.07
Acceptance Rate	-.30**	-.14	Acceptance Rate	-.17	-.05
Private Institution	.51**	.12	Private Institution	.30**	-.06
Faculty:Student Ratio	.44**	.01	Faculty:Student Ratio	.35**	.06
NIH Funding	.58**	-.13	NIH Funding	.47**	-.01
Primary Care Grads	-.31**	-.12	Primary Care Grads	-.10	.17

GPA, Undergraduate Grade Point Average; MCAT, Medical College Admission Test score; USMLE, US Medical Licensing Examination; CK, Clinical Knowledge; NIH, National Institutes of Health.

* P < .05

** P < .01

† Partial correlation controlling for GPA and MCAT (2010–12)

‡ Partial correlation controlling for GPA and MCAT (2008–10)

The final regression model utilizing GPA and MCAT achieved an adjusted R2 of 72% for Step 1 (standardized βMCAT 0.7, βGPA 0.2, model P < .001) and 41% for Step 2 CK (standardized βMCAT 0.5, βGPA 0.3, model P < .001). GPA added significant but incremental validity evidence over MCAT alone (Step 1, ΔR2 2%, P = .009; Step 2 CK, ΔR2 4%, P = .02); accordingly, for visualization, institutional USMLE was regressed on MCAT score alone (Fig 1).

Fig 1

Regression analysis of institutional MCAT versus USMLE performance.

(A) Regression analysis of institutional average matriculant Medical College Admission Test (MCAT) score (2010–2012) versus institutional average US Medical Licensing Examination (USMLE) Step 1 score (2012–2014) across n = 89 US allopathic medical schools, representing 39,615 examinees. (B) Regression analysis of institutional average matriculant MCAT score (2008–2010) versus institutional average USMLE Step 2 Clinical Knowledge (CK) score (2012–2014) across n = 89 US allopathic medical schools, representing 39,252 examinees. For both plots, each bubble represents 3-year average at one institution, with bubble size reflecting number examined at each institution. Ordinary least squares best fit line (solid) and 95% prediction interval (dashed lines) are shown, with colored data points highlighting institutions outside the prediction interval.

Institutions with above- or below-predicted USMLE performance

Using the GPA and MCAT regression model, we identified a subset of institutions with 3-year average institutional USMLE scores statistically above or below predicted (Table 3).

Table 3

US allopathic medical schools with above- or below-predicted institutional USMLE Step 1 or Step 2 CK performance, 2012–2014.

USMLE Step 1
Institution	Average Score (SD)	Score Deviation from Predicted, Points	Standardized Residual, SD	Examinees, No.
University of Hawaii–Manoa	234 (3.2)	+8.4	+2.9	182
University of Missouri	236 (4.6)	+8.2	+2.8	296
Baylor College of Medicine	241 (1.0)	+5.9	+2.0	517
Institution Xb	220 (3.2)	-5.9	-2.0	504
USMLE Step 2 CK
Institution	Average Score (SD)	Score Deviation from Predicted, Points	Standardized Residual, SD	Examinees, No.
Emory University	250 (2.6)	+10.1	+2.8	424
University of Virginia	248 (2.1)	+7.1	+2.0	449
Institution X	228 (5.3)	-7.3	-2.0	481
Institution Y	228 (9.7)	-9.2	-2.6	507
Institution Z	230 (3.2)	-12.0	-3.4	305

USMLE, US Medical Licensing Examination; CK, Clinical Knowledge; SD, standard deviation.

a Based on regression models incorporating institutional average Medical College Admission Test (MCAT) score and undergraduate grade point average (GPA) of entering students, as follows: Institutional USMLE Step 1 score = 122 + 1.7 * MCAT + 14.1 * GPA; Institutional USMLE Step 2 CK score = 149 + 1.0 * MCAT + 15.6 * GPA.

b The names of institutions with below-predicted institutional USMLE performance were withheld due to the sensitive and exploratory nature of this data.

Discussion

In this exploratory study of 89 US allopathic medical schools, we identified 5 institutions with above-predicted institutional USMLE performance based on the described model. The etiology of these institutions’ relative success (or the 3 unnamed institutions’ below-predicted performance) is unclear; we can only say that numerous demographic and institutional factors we assessed did not account for this variation. We hypothesize that unmeasured student factors that vary systematically between institutions (e.g., through admissions processes) or institution-specific factors (e.g., alignment of curricula with USMLE content) may explain these institutional differences. For example, medical schools that provide commercially available Step 1 question banks [22] or where students take Step 1 after the core clerkships [23] have reported improved institutional scores, demonstrating that unique institutional strategies can promote students’ USMLE success. Further study is needed to understand if the 5 institutions identified here have unique factors that promoted their students’ success on these exams.

We found that institutions’ average student GPA and MCAT accounted for substantial variation in institutional average USMLE Step 1 and Step 2 CK scores, which was expected based on prior studies at the individual [8,9] and institutional level [9,11,12]. Importantly, institutional demographic factors (such as percent minority students or biological sciences majors) were correlated with institutional USMLE performance in bivariate regression but were not significant after controlling for GPA and MCAT. National Institutes of Health research funding, which had been previously shown to correlate with institutional USMLE performance [13], was similarly not significant when controlling for GPA and MCAT.

Institutions with a propensity for matching students in the primary care specialties family medicine, pediatrics, and internal medicine–which have lower USMLE screening thresholds for residency interviews than other more “competitive” specialties [24]–tended to recruit students with lower GPA and MCAT scores, and thus lower institutional USMLE scores. As with other institutional factors, however, institutions’ primary care specialty rate was not associated with differential USMLE performance beyond its association with GPA and MCAT.

Such findings highlight the critical importance of controlling for prior academic performance when attempting to explain USMLE performance differences. However, we doubt that pre-medical students–a key consumer of the annual US News and World Report data–consider these covariates when interpreting institutional USMLE scores and identifying medical schools of interest. Indeed, undergraduates might conclude (erroneously) that private medical schools outperform public schools on the USMLE, when in fact students attending private schools have higher test scores at matriculation. There may be a role for better contextualizing this data so that pre-medical students can be informed consumers. The National Board of Medical Examiners (NBME), who produce the USMLE, are positioned to more rigorously explore the relationship between institutions and exam performance.

Limitations

This study relied on self-reported institutional data via a third-party publication, as the NBME does not publish institutional score performance. Misreporting is possible, although we validated the reported scores from several institutions. US News and World Report provides their methodology for data collection with each annual release [25], but do not state specifics related to how data is validated or standardized within- or between-schools. For example, it is unclear if institutions have discretion in how they formulate their institutional MCAT average, including how individuals with multiple test results are handled, which can introduce bias into the relationship between MCAT and USMLE performance [26].

Although we assessed numerous student and medical school factors, some potentially important covariates–such as percent of students with advanced degrees, curricular structure, timing of USMLE examinations, and school age–were not incorporated into this study but are important areas for future investigation. For example, some institutions have moved the USMLE Step 1 test window to after core clinical clerkships [23], with small benefits in scores and reduced failure rates [27].

Moreover, only 89 of 131 US allopathic medical schools (68%) reported complete data; non-reporters may differ in important ways. We found that reporting institutions, as compared to an average of all students nationally, had slightly higher average GPA and MCAT scores, with an associated 1.5-point higher average USMLE Step 1 score. Statistical comparisons of these differences are not advisable given the different units of reporting (institutions vs. individuals); yet the very small differences suggest that the reporting institutions were nationally representative. The relatively short 3-year study period does not preclude that the observed institutional outliers may represent random variation; replication with longer observation is needed. Finally, our study was ecological; no inference can be made that institution-level findings translate to individual students (i.e., the ecological fallacy), and indeed only institutional averages and counts were reported (without any measure of student-to-student variability). Nevertheless, the purpose of this study was only to compare institutions.

Conclusions

We found that institutional average GPA and MCAT scores correlate strongly with institutional USMLE performance. Numerous student demographic and institutional factors were insignificant when controlling for GPA and MCAT. We identified several institutions with significant above- or below-predicted USMLE performance, raising questions about inter-institutional USMLE performance parity. Methods to assess institutions’ overall performance on knowledge-based exams may offer a parameter to evaluate medical schools and their curricula, while providing prospective students with valuable data regarding these high-stakes exams. Additional study is needed to explore the etiology and durability of the observed performance differences, and to incorporate other student and institutional factors that may be important predictors of performance.

Average GPA, MCAT, and USMLE Step 1 and 2 CK for 89 US allopathic medical schools reporting complete data from 2012–2014, with associated national averages.

(XLSX)

Click here for additional data file.

Transfer Alert

This paper was transferred from another journal. As a result, its full editorial history (including decision letters, peer reviews and author responses) may not be present.

5 Sep 2019

PONE-D-19-20613

Institutional differences in USMLE performance: Cross-sectional study of 89 US allopathic medical schools

PLOS ONE

Dear Dr. Burk-Rafel,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by Oct 20 2019 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Andrew Carl Miller

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide.

3. Thank you for stating the following in the Competing Interests section:

"I have read the journal's policy and the authors of this manuscript have the following competing interests: Dr. Burk-Rafel reports working as a research consultant for ScholarRx, a digital learning platform that includes USMLE preparation services, during the late stages of writing this manuscript. ScholarRx was not involved in this study in any way. All other authors declare no competing interests."

a. Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

b. Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

4. Please amend either the title on the online submission form (via Edit Submission) or the title in the manuscript so that they are identical.

Additional Editor Comments:

Please ensure that your manuscript adheres to the STROBE guidelines and provide citation:

von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. PLoS Med. 2007;4(10):e296. PMID: 17941714

Please indicate the study’s design with a commonly used term in the title or the abstract (eg. cohort, observational, retrospective, etc.)

Was a sample size calculation performed? If so, please provide.

Is the same data available for osteopathic medical schools? If so, why were they excluded?

42 allopathic schools were excluded from the analysis for not providing complete USMLE schools over the 3-year period. It would be nice to know more about those schools. Are there any commonalities amongst those schools that make them different from the other 89? Were they more likely to be new schools, struggling schools, etc. How do you think that this impacts the results?

How does school age impact the results (e.g. first 5 years vs longer established)?

Does the percentage of international students affect the results?

Do schools with a higher percentage of students with other advanced degrees (masters, PhD, etc) perform better (or worse)?

Is there a way to carry this forward to see the specialties that these students choose? Is there any relationship to schools having a higher or lower percentage of students matriculating into highly vs. less competitive specialties for residency?

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: It would be helpful to have a Table in the supporting document list the mean USMLE scores and mean MCAT scores for each institution since those data were the data used for the analysis. All of these data are publicly available why not list them.

Reviewer #2: Thank you for the opportunity to review this manuscript. This addresses an important topic in continuing to explore the impact of medical school curricula and other factors that contribute to USMLE interinstitutional variation. This large scale study does have major limitations which deserve greater discussion. Specific comments are listed below.

1. Page 5: describe where data will be available.

2. Methods: Further information is needed to better understand the data limitations of the self-reported data. For example, is anything known about consistency of reporting of the MCAT score? Is this is superscore of the last 3 takes or single best take? Is there a consistent reporting structure between institutions or is this reported completely at the institution’s discretion? The variability in how this data was reported represents a major potential limitation of the study.

3. Undergraduate gpa is a frequently discussed variable for MCAT performance. However, an increasing number of students have graduate degrees. Does percentage of students with graduate degrees and graduate gpa warrant a discussion in this manuscript?

4. Please describe how the MCAT score data was defined. Was this related to a single section of the MCAT or an average of the 3 sections?

5. In the results section, for overall interpretation of the results of this study, it would be important to understand the national USMLE average scores for the time period investigated. Is this sample representative of all medical schools?

6. During the past couple of decades, most medical schools have undergone significant curricular changes related to increased emphasis on active learning and decreasing lecture time. Many medical schools and students feel that the emphasis on USMLE performance competes with a curriculum focused on clinical performance. In future studies, it will be important to explore if certain types of curricula are more or less successful in preparing students for USMLE examinations.

Thank you again for the opportunity to review this manuscript. Despite the limitations, I feel that this type of larger study is needed to better understand USMLE performance at the institutional level.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: PLOS ONE Review.docx

Click here for additional data file.

16 Oct 2019

Response to Reviewers (see attached 'Response to Reviewers' document for formatted version)

RE: PONE-D-19-20613

“Institutional differences in USMLE Step 1 and 2 CK performance: Cross-sectional study of 89 US allopathic medical schools”

In response to the reviewers’ comments, we addressed each point.

1. Please amend either the title on the online submission form (via Edit Submission) or the title in the manuscript so that they are identical.

The title of the online submission form and manuscript is “Institutional differences in USMLE Step 1 and 2 CK performance: Cross-sectional study of 89 US allopathic medical schools.” We changed the title on the online submission form and manuscript to be identical.

2. Please ensure that your manuscript adheres to the STROBE guidelines and provide citation.

We have reviewed the STROBE guidelines and cited the citation into our manuscript. We attest that the manuscript adheres to the STROBE guidelines and we emphasize that our manuscript is purely an exploratory observational study meant to stimulate and generate a hypothesis based on reported data from prior years. The methodology of our study should still be generalizable to future studies as new data is reported.

3. Please indicate the study’s design with a commonly used term in the title or the abstract (eg. cohort, observational, retrospective, etc.)

To further indicate the study’s design, we used the term “observational” in the abstract. The title already specifies it as “cross-sectional,” a type of observational study.

4. Was a sample size calculation performed? If so, please provide.

A sample size calculation was not performed because we sought to obtain a complete census of all US allopathic medical schools. A statement to this effect was added to the methods.

5. Is the same data available for osteopathic medical schools (DO schools)? If so, why were they excluded?

Osteopathic institutions were excluded from this study, as osteopathic students historically took the COMLEX licensing examination rather than the USMLE and very few US osteopathic institutions reported USMLE performance data. It is important to note as well that at the time of data collection only 25 DO institutions had at least a first admitting class year, of these approximately 15 osteopathic schools were formed from 2002-2012. The rapid development and the continuous changes in curricula as typically seen in new institutions were additional reasons for excluding the small number of DO institutions that did report data. Theses reasons for excluding osteopathic institutions are now included in the methods.

6. 42 allopathic schools were excluded from the analysis for not providing complete USMLE schools over the 3-year period. It would be nice to know more about those schools. Are there any commonalities amongst those schools that make them different from the other 89? Were they more likely to be new schools, struggling schools, etc. How do you think that this impacts the results?

We have added national average performance data to Table 1 and S1 Table, and included a statement in results: “The average GPA, MCAT scores, and USMLE Step 1 scores for the 89 reporting institutions were slightly higher than national averages for all matriculants/examinees.” We go on in the discussion to discuss: “We found that reporting institutions, as compared to an average of all students nationally, had higher average GPA and MCAT scores, with an associated 1.5-point higher average USMLE Step 1 score. Statistical comparisons of these differences are not advisable given the different units of reporting (institutions vs. individuals); yet the very small differences suggest that the reporting institutions were nationally representative.” Overall, we are pleased that the reporting institutions very nearly matched national averages, suggesting good representativeness. Unfortunately, we do not have information regarding the non-reporting institutions to characterize their reasons for not reporting to US News and World Report.

7. How does school age impact the results (e.g. first 5 years vs longer established)?

We do not have this data. We have added this to the discussion as an important limitation and area for further investigation.

8. Does the percentage of international students affect the results?

Unfortunately, schools did not report the percent of international students within their classes. This would be an interesting covariate for future study.

9. Do schools with a higher percentage of students with other advanced degrees (masters, PhD, etc) perform better (or worse)?

Schools' percent of non-traditional students (who took >1 year to do something between undergraduate and medical school) was not associated with differential performance, when controlling for MCAT and GPA. We do not have data on the percent of advanced degrees when entering medical school, but have added this as a limitation and area for future investigation.

10. Is there a way to carry this forward to see the specialties that these students choose? Is there any relationship to schools having a higher or lower percentage of students matriculating into highly vs. less competitive specialties for residency?

Unfortunately, the main residency match data does not include match information at the level of schools and schools did not report distribution of specialties at match in this data set. However, the data set did include the percent of graduates entering primary care specialties (here defined as family medicine, pediatrics, and internal medicine), which have lower USMLE screening thresholds and are thus less “competitive”. Although having more primary care grads had a significant negative correlation with USMLE scores at baseline, it was not significant after controlling for GPA and MCAT. We have included several sentences in the discussion on this topic.

11. It would be helpful to have a Table in the supporting document list the mean USMLE scores and mean MCAT scores for each institution since those data were the data used for the analysis. All of these data are publicly available why not list them.

We have created a supporting table which lists the mean GPA, MCAT, and USMLE scores for all 89 institutions. This table is referred to in the results portion of our paper and included in the Supporting Information.

12. Describe where data will be available.

Data will be available in the Supporting Information table.

13. Further information is needed to better understand the data limitations of the self-reported data. For example, is anything known about consistency of reporting of the MCAT score? Is this a superscore of the last 3 takes or single best take? Is there a consistent reporting structure between institutions or is this reported completely at the institution’s discretion? The variability in how this data was reported represents a major potential limitation of the study.

We do not know how schools reported their MCAT scores. We have added this to the limitations’ discussion of the paper and understand its importance. In a study conducted by Zhao et al, four different approaches were investigated when attempting to predict USMLE Step 1 scores using MCAT scores. It was concluded that averaging MCAT scores for an individual is the best predictor for Step 1 scores, highlighting the importance of how MCAT scores are gathered. However, it is reassuring that the average scores (GPA, MCAT, and USMLE) for all reporting institutions very nearly matched national averages for the study period, suggesting good fidelity of reporting.

14. Undergraduate GPA is a frequently discussed variable for MCAT performance. However, an increasing number of students have graduate degrees. Does the percentage of students with graduate degrees and graduate gpa warrant a discussion in this manuscript?

We do not have data concerning the percentage of students with graduate degrees and graduate GPA. We did note that the schools' percent of non-traditional students (who took >1 year to do something between undergraduate and medical school) was not associated with differential performance, when controlling for MCAT and GPA. We do not have data on the percent of advanced degrees when entering medical school, but added this as an important factor for future research. A literature review was performed to further investigate students with graduate degrees and their graduate GPA. No papers were found discussing the relationship between graduate GPA and medical school examinations, highlighting this as an important area for further investigation.

15. Please describe how the MCAT score data was defined. Was this related to a single section of the MCAT or an average of the 3 sections?

The MCAT score data was defined as the average institutional scores for each of the 3 sections (biological sciences, physical sciences, verbal reasoning) to create a total score institutional average. We included this description of the MCAT score data to the methods.

16. In the results section, for overall interpretation of the results of this study, it would be important to understand the national USMLE average scores for the time period investigated. Is this sample representative of all medical schools?

We have included the national average GPA, MCAT, and USMLE scores for the study period in Table 1 and S1 Table for reference. We have also added reflection in the results and limitations section regarding the representativeness of the sample, highlighting that the sample had very similar scores compared to national averages, suggesting a nationally representative sample.

17. During the past couple of decades, most medical schools have undergone significant curricular changes related to increased emphasis on active learning and decreasing lecture time. Many medical schools and students feel that the emphasis on USMLE performance competes with a curriculum focused on clinical performance. In future studies, it will be important to explore if certain types of curricula are more or less successful in preparing students for USMLE examinations.

We are aware that some medical schools have undergone significant curricular changes. More specifically, there has been a trend in increasing clinical experience and decreasing time in the classroom. Studies completed by Daniel et al and Jurich et al have discussed the pros and cons experienced with this fundamental change in curriculum. We have added this information to the discussion portion of our study and recommended future studies explore which types of curricula are more or less successful in preparing students for USMLE examinations.

18. The one suggestion is to clearly state in the conclusion that institutional mean GPA and mean MCAT scores correlate strongly with USMLE performance whereas student body demographic and other institutional variables were weak or insignificant when controlling for MCAT and GPA.

We updated the conclusion to clearly state that the institutional mean GPA and mean MCAT scores correlate strongly with USMLE performance whereas student body demographic and other institutional variables were weak or insignificant when controlling for MCAT and GPA.

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

21 Oct 2019

Institutional differences in USMLE Step 1 and 2 CK performance: Cross-sectional study of 89 US allopathic medical schools

PONE-D-19-20613R1

Dear Dr. Burk-Rafel,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Andrew Carl Miller

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Thank you for the important and interesting article.

Reviewers' comments:

23 Oct 2019

PONE-D-19-20613R1

Institutional differences in USMLE Step 1 and 2 CK performance: Cross-sectional study of 89 US allopathic medical schools

Dear Dr. Burk-Rafel:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Andrew Carl Miller

Academic Editor

PLOS ONE