Should osteopathic students applying to allopathic emergency medicine programs take the USMLE Exam?

INTRODUCTION: Board scores are an important aspect of an emergency medicine (EM) residency application. Residency directors use these standardized tests to objectively evaluate an applicant's potential and help decide whether to interview a candidate. While allopathic (MD) students take the United States Medical Licensing Examination (USMLE), osteopathic (DO) students take the Comprehensive Osteopathic Medical Licensing Examination (COMLEX). It is difficult to compare these scores. Previous literature proposed an equation to predict USMLE based on COMLEX. Recent analyses suggested this may no longer be accurate. DO students applying to allopathic programs frequently ask whether they should take USMLE to overcome this potential disadvantage. The objective of the study is to compare the likelihood to match of DO applicants who reported USMLE to those who did not, and to clarify how important program directors consider it is whether or not an osteopathic applicant reported a USMLE score.
METHODS: We conducted a review of Electronic Residency Application Service (ERAS) and National Resident Matching Program (NRMP) data for 2010-2011 in conjunction with a survey of EM residency programs. We reviewed the number of allopathic and osteopathic applicants, the number of osteopathic applicants who reported a USMLE score, and the percentage of successful match. We compared the percentage of osteopathic applicants who reported a USMLE score who matched compared to those who did not report USMLE. We also surveyed allopathic EM residency programs to understand how important it is that osteopathic (DO) students take USMLE.
RESULTS: There were 1,482 MD students ranked EM programs; 1,277 (86%, 95% CI 84.3-87.9) matched. There were 350 DO students ranked EM programs; 181 (52%, 95% CI 46.4-57.0) matched (difference=34%, 95% CI 29.8-39.0, p<0.0001). There were 208 DO students reported USMLE; 126 (61%, 95% CI 53.6-67.2) matched. 142 did not report USMLE; 55 (39%, 95% CI 30.7-47.3) matched (difference=22%, 95% CI 11.2-32.5, p<0.0001). Survey results: 39% of program directors reported that it is extremely important that osteopathic students take USMLE, 38% stated it is somewhat important, and 22% responded not at all important.
CONCLUSION: DO students who reported USMLE were more likely to match. DO students applying to allopathic EM programs should consider taking USMLE to improve their chances of a successful match.

INTRODUCTION

Board scores represent an important aspect of a medical student’s application for an emergency medicine (EM) residency position.1,2 United States Medical Licensing Examination (USMLE) scores have been shown to correlate with overall success in EM residency3 and with in-training scores.4 Therefore, residency directors use these standardized tests to objectively evaluate an applicant’s potential and help decide whether to interview a candidate.1,2

While allopathic students take the USMLE as part of their licensing process, osteopathic students take the Comprehensive Osteopathic Medical Licensing Examination (COMLEX). Since allopathic and osteopathic students take different examinations, it is difficult to compare board scores between these applicants. This may make it difficult for allopathic programs to decide whether to offer osteopathic students an interview. Although previous literature has suggested a correlation between COMLEX and USMLE scores,6 and proposed an equation to predict USMLE scores based on COMLEX scores, more recent analyses suggested that this conversion may no longer be accurate.7–9 A recent study of osteopathic applicants to one allopathic EM residency program showed no correlation between COMLEX-1 and USMLE Step I scores.10 Therefore, to overcome this potential disadvantage, some osteopathic students will also take the USMLE. A recent survey of graduating osteopathic medical students found that 59.5% reported taking at least one step of the USMLE, of which 35.3% stated the primary reason they took USMLE was “to enhance my chances of getting into an allopathic residency.”11 In the same survey, 70.2% of graduates also recommended that students take at least one step of the USMLE.

The first objective of this study was to compare the likelihood of osteopathic applicants who reported a USMLE score to match in an allopathic EM residency to osteopathic applicants who did not report a USMLE score. The second objective was to clarify how much emphasis is placed by program directors on whether an osteopathic applicant reported a USMLE score.

METHODS

This study had two components: 1. A review of ERAS and NRMP data, and 2. A survey of all EM residency program leadership. This study was approved by our local institutional review (or human subjects) committee. Waiver of informed consent was granted.

Review of ERAS and NRMP data

Application and match data were collected for allopathic and osteopathic applicants to all allopathic EM residency programs for application season 2010–2011. This season began September 1, 2010 and continued through match day March 17, 2011. Application and match data are published annually by the National Resident Matching Program (NRMP) in “National Resident Matching Program, Results and Data,” (www.nrmp.org). This publication was reviewed to obtain the number of allopathic and osteopathic applicants and the percentage of successful match. Additional unpublished data was obtained through NRMP and the Electronic Residency Application Service (ERAS). NRMP representatives queried their database and supplied the number of osteopathic and allopathic students who ranked allopathic EM programs, the number of osteopathic and allopathic students who successfully matched in an allopathic EM program, and whether the osteopathic students reported a USMLE score. ERAS representatives queried their database and provided the number of osteopathic students who applied to allopathic EM programs and how many reported a USMLE score.

Survey

To clarify how much weight is placed by program directors on whether or not an applicant reported a USMLE score, we sent a questionnaire to all allopathic EM residency programs accredited by the Accreditation Council for Graduate Medical Education (ACGME). Each program was allowed one response to the survey. Osteopathic EM residency programs approved by the AOA were excluded. Programs with dual ACGME and AOA accreditation were not excluded. The survey is presented in Figure 1. It was sent electronically (www.surveymonkey.com) to all EM programs via the Emergency Medicine Association of Residency Coordinators (EMARC) Listserv. Program directors were subsequently contacted individually by e-mail to encourage completion of the questionnaire.

Figure 1

Survey.

To prevent duplicate responses from the same program, we encouraged program directors to complete the survey together with their program coordinators. In addition, internet protocol (IP) addresses of respondents were reviewed. Where IP addresses were the same, duplicate responses were excluded. Where these responses differed from each other, the “worst case scenario” response was kept. For example, if two responses were received from the same IP address, one responded “Not important at all” and one responded “Extremely important”, the “Not important at all” response was kept and the “Extremely important” response was excluded.

Outcome measures

The primary outcomes were the percentage of osteopathic students who matched in allopathic EM residencies, and percentage of osteopathic applicants who reported a USMLE score matching in an allopathic EM residency in application year 2010–2011. The secondary outcome was the percentage of allopathic EM residency program directors responding to the survey who feel it is important for osteopathic applicants to take USMLE.

Data Analysis

Categorical data are presented as percentages with 95% confidence intervals (CI). The percentages of osteopathic and allopathic students matching in allopathic EM residency programs and the rates of osteopathic students with or without a reported USMLE score who matched in an allopathic EM program were compared by Fisher’s exact test. Statistics were calculated using GraphPad InStat (Version 3.05, for Windows 95/NT, GraphPad Software, San Diego California USA, Copyright 1992–1998 GraphPad Software Inc.).

Sample size analysis was performed. In previous match years, the overall match rate for EM applicants was approximately 85%. Assuming this match rate, to show a 10% decrease for osteopathic applicants 131 subjects would be required (90% power, alpha 0.05).

RESULTS

ERAS and NRMP data

In application year 2010–2011, 153 allopathic EM programs participated in NRMP for 1,626 positions.13 The number of allopathic and osteopathic applicants, how many reported a USMLE score, and how many matched is displayed in Figure 2. Among allopathic students, 86% ( 95% CI 84.3–87.9) successfully matched compared to 52% (95% CI 46.4–57.0) of osteopathic students (difference=34%, 95% CI 29.8–39.0, p<0.0001).13

Figure 2

Flow of applicants.

Among osteopathic students who reported a USMLE score, 61% (95% CI 53.6–67.2) matched compared to 39% (95% CI 30.7–47.3) of those who did not report a USMLE score (difference=22%, 95% CI 11.2–32.5, p<0.0001).

One hundred thirty survey responses were received. Duplicate IP addresses were identified for 16 responses and an additional 16 responses were from osteopathic programs. These were excluded. The remaining 98 programs formed our data set, and represent 64% of all allopathic programs. Programs responding to the surveys had a total of 106 (59%) osteopathic applicants who successfully matched through NRMP and 14 pre-matched applicants. Demographics of the programs are presented in Table 1.

Table 1

Demographics of programs responding to survey.

Hospital type	n (%)
Urban	75 (77%)
Suburban	21 (21%)
Rural	2 (2%)
Geographic location
Northeast	36 (37%)
South	22 (22%)
West	13 (13%)
Central	27 (28%)
Residency length
1-2-3	75 (77%)
1-2-3-4	20 (20%)
2-3-4	2 (2%)
No response	1 (1%)
Residency size	number
Minimum	6
Maximum	60

Six programs (6%) stated they do not consider applications from osteopathic students. One program stated they ask osteopathic students to complete a transitional year before applying to their program. Seventy-seven programs (79%) had osteopathic residents in their residency program.

Programs were asked, “When applying to your program, how important is it that osteopathic students take USMLE?” and 91 responses were received. Survey results are reported in Table 2 and Figure 3. Thirty-nine percent of program directors reported that it is extremely important that osteopathic students take USMLE, 38% stated it is somewhat important, and 22% responded not at all important.

Table 2

Survey results.

Does your program consider applications from osteopathic students?	n (%)
Yes	92 (94%)
No	6 (6%)
When applying to your program, how important is it that osteopathic students take USMLE?
Extremely Important	36 (39%)
Somewhat Important	35 (38%)
Not At All Important	20 (22%)
No Response	1 (1%)
Comments	n
Somewhat Important
USMLE Step I is required, Step II preferred	1
“We highly recommend taking USMLE but do not make it mandatory”	1
“If an osteopathic student expects to compete with an allopathic student they should take USMLE”	1
USMLE allows them to better compare applicants	6
Prefer USMLE	4
A strong USMLE score is more meaningful than a strong COMLEX score	2
Not At All Important
We don’t require it but it sure is helpful	1

USMLE, United States Medical Licensing Exam; COMLEX, Comprehesnive Osteopathic Medical Licensing Exam

Figure 3

Importance of the United States Medical Licensing Exam.

DISCUSSION

In the 2010–2011 application season, allopathic students were more likely than osteopathic students to match in an allopathic EM program. Osteopathic students who reported a USMLE score were more likely to match into an allopathic EM residency position than those who did not.

In our survey sample, 94% of allopathic programs consider applications from osteopathic applicants and 79% have osteopathic residents in their program. Seventy eight percent stated it is somewhat important or extremely important that osteopathic students take USMLE. Even among those who said “somewhat important,” several commented that they recommend or prefer it.

Residency directors evaluate many aspects of an applicant’s file when deciding how high to place them on their rank list. Whereas course grades may have different meaning in different medical schools, USMLE scores reflect performance on a standardized examination taken by all students across the country. This allows program directors to better compare students across the entire spectrum of applicants. This may be the reason that program directors feel it is so important for osteopathic students to take USMLE.

It would be helpful for allopathic residency directors to have a conversion between COMLEX and USMLE scores. Previous literature reported such a conversion factor,6 however subsequent studies suggested this conversion factor is not accurate.8,9 In a letter to the editor, the president and chief executive officer of the National Board of Osteopathic Medical Examiners stated, “it is not possible—or even desirable—to make a direct numerical comparison between the scores of the COMLEX-USA examination series and those of the USMLE.”7

Osteopathic students applying to allopathic EM programs frequently ask if they should take USMLE to be more competitive. The findings in this study will help advisors answer that question.

LIMITATIONS

Our study represents data from a single application season. It is possible that the likelihood of an osteopathic student to match varies from year to year. Future studies can examine several years of match data.

Many factors other than board scores affect a student’s likelihood of matching. This study did not look at the actual score the students received on USMLE or COMLEX since these were not available to us. It is possible that the students who took USMLE also had better overall applications and interviewed better. Similarly, it is feasible that some osteopathic students took USMLE but did not report their scores via ERAS. Students who did not report their scores may have had weaker overall applications. Thus, it is possible that we are observing association rather than causation.

Some programs have dual accreditation from both the ACGME and the AOA. These programs were not excluded from the ERAS and NRMP data. In addition, some of these programs may have been included in our survey. Respondents were asked, “Is your residency an osteopathic residency program?” If they responded yes, they were excluded. Programs with dual accreditation may not have answered yes and would have been included in our data. However, we believe this makes our results even stronger. Programs with dual accreditation would be more comfortable with COMLEX scores and would be more likely to accept applicants without USMLE scores. So, including these programs in our study would make it more difficult to show that applicants with only COMLEX scores were less likely to match and would skew our results in the direction of less program directors stating it is important for osteopathic applicants to report a USMLE score. Despite including these programs in our study group, we still showed that applicants with only COMLEX scores were less likely to match and a majority of program directors still felt it is important for applicants to report a USMLE score.

We assessed the number of osteopathic students who matched as a percentage of those that ranked EM programs in NRMP. We did not include osteopathic students who obtained positions outside the match. In the 2011 match, osteopathic students were eligible to accept positions directly from program directors prior to the match (known as “pre-match”). From the sample set of our survey respondents, approximately 13% of osteopathic students who began allopathic residency in 2011 pre-matched. Osteopathic students may also have obtained unfilled positions after the match (known as “the scramble”). In 2011, only five EM positions at two programs were available in the scramble period. However, we were specifically studying the effects of reporting a USMLE score on the chances of that student to obtain a position in the NRMP match. Future studies can include applicants who pre-matched and those who obtained positions in the scramble period.

It is possible that some students ranked other specialties higher than EM and matched in those specialties. We included all applicants who ranked EM positions, not only those for whom EM was their first choice. In the 2010–2011 application season, approximately 90% of applicants who ranked EM ranked it as their only choice or their first choice. 17

The survey was sent out via the EMARC Listserv. It is possible that some residency programs are not members of EMARC. However, in academic year 2010–2011, all 153 allopathic EM residencies were members of EMARC.18 In addition, all program directors were contacted directly by e-mail, thus ensuring that all programs had the ability to complete the survey. The response rate to our survey was 64%. It is possible that programs that do not consider osteopathic applicants were less likely to complete the survey. It is also possible that programs who did not respond to the survey do not consider osteopathic students who have not taken USMLE.

CONCLUSION

In the 2010–2011 application season, allopathic students were more likely than osteopathic students to match in an allopathic EM residency. Osteopathic students who reported a USMLE score were more likely to match into an allopathic EM residency than those who did not. Osteopathic students applying to allopathic EM programs should consider taking USMLE to improve their chances of a successful match.