Correlation of the emergency medicine resident in-service examination with the American Osteopathic Board of Emergency Medicine Part I.

INTRODUCTION: Eligible residents during their fourth postgraduate year (PGY-4) of emergency medicine (EM) residency training who seek specialty board certification in emergency medicine may take the American Osteopathic Board of Emergency Medicine (AOBEM) Part 1 Board Certifying Examination (AOBEM Part 1). All residents enrolled in an osteopathic EM residency training program are required to take the EM Resident In-service Examination (RISE) annually. Our aim was to correlate resident performance on the RISE with performance on the AOBEM Part 1. The study group consisted of osteopathic EM residents in their PGY-4 year of training who took both examinations during that same year.
METHODS: We examined data from 2009 to 2012 from the National Board of Osteopathic Medical Examiners (NBOME). The NBOME grades and performs statistical analyses on both the RISE and the AOBEM Part 1. We used the RISE exam scores, as reported by percentile rank, and compared them to both the score on the AOBEM Part 1 and the dichotomous outcome of passing or failing. A receiver operating characteristic (ROC) curve was generated to depict the relationship.
RESULTS: We studied a total of 409 residents over the 4-year period. The RISE percentile score correlated strongly with the AOBEM Part 1 score for residents who took both exams in the same year (r=0.61, 95% confidence interval [CI] 0.54 to 0.66). Pass percentage on the AOBEM Part 1 increased by resident percent decile on the RISE from 0% in the bottom decile to 100% in the top decile. ROC analysis also showed that the best cutoff for determining pass or fail on the AOBEM Part 1 was a 65(th) percentile score on the RISE.
CONCLUSION: We have shown there is a strong correlation between a resident's percentile score on the RISE during their PGY-4 year of residency training and first-time success on the AOBEM Part 1 taken during the same year. This information may be useful for osteopathic EM residents as an indicator as to how well prepared they are for the AOBEM Part 1 Board Certifying Examination.

INTRODUCTION

The Basic Standards for Residency Training in Emergency Medicine of the American College of Osteopathic Emergency Physicians (ACOEP) requires all osteopathic emergency medicine (EM) residents to annually participate in the Resident In-Service Examination (RISE).

Residency training programs in other specialties have demonstrated correlations between their specialty in-service examinations and passing future board certification examinations.1–14 In 2009, the American Osteopathic Board of Emergency Medicine (AOBEM) began offering the option of taking Part 1 of the certifying examination to eligible EM residents in their fourth postgraduate year (PGY-4) of EM residency training. Prior to 2009 only EM residency graduates were permitted to participate in the examination.

We wish to demonstrate a correlation between the percentile score on the RISE with corresponding scores and the dichotomous outcome of passing or failing on the AOBEM Part 1 exam. We also sought to find a point whereby the likelihood of passing the AOBEM Part 1 was greatest. This will help residency program directors and residents gauge the progress a resident is making towards board certification.

METHODS

We obtained data from the National Board of Osteopathic Medical Examiners (NBOME). The NBOME is an organization that independently grades and performs statistical analyses on both the RISE and AOBEM Part 1. The hospital's institutional review board approved the project. In the United States there are 45 osteopathic EM residency programs comprised of a total of 1,777 EM residents during our study period. Of this, there was an average of 221 PGY-4 EM residents who participated in the RISE.

We studied the correlation of RISE percentiles with the scores and pass rate of the AOBEM Part 1 when both examinations were taken during the same year. We used RISE percentile rather than the RISE raw score since raw scores varied from year to year. The AOBEM Part 1 scores are equated from year to year.

We measured the performance of all fourth-year osteopathic EM residents who took both the RISE and AOBEM Part 1 examinations in the same year from 2009 to 2012. The number of PGY-4 osteopathic EM residents that took the RISE and AOBEM examinations in the same year are listed by year in Table 1.

Table 1.

The distribution of American Osteopathic Board of Emergency Medicine (AOBEM) Part 1 test takers by post graduate year-4 residents who took the Resident In-Service Examination (RISE) from 2009 to 2012. These residents (n=409) were the subjects on whom data were reported.

We used the following units of measurement: RISE percentile (of all residents at every level of training taking the examination), AOBEM Part 1 score, AOBEM Part 1 Pass/Fail.

Data Analysis

We calculated Pearson's correlations (r) of RISE percentiles with AOBEM Part 1 scores.15,16 A receiver operating characteristic (ROC) curve was generated to compare the RISE percentile with the probability of passing or failing the AOBEM Part 1. We performed statistical analysis using SPSS Version 12.0.

RESULTS

From 2009 to 2012, 409 (46.3%) of the 884 PGY-4 residents who took the RISE also took the AOBEM Part 1. (Table 1)

There was a good correlation between both the RISE percentiles and the AOBEM Part 1 scores for each year (Table 2).15,16 The overall correlation between the RISE percentile and the AOBEM Part 1 score was 0.61 (95% confidence interval [CI] 0.54 to 0.66). The scatter plots and correlation for each year are listed in Figure 1.

Table 2.

The Pearson correlation between Resident In-Service Examination (RISE) percentiles and American Osteopathic Board of Emergency Medicine (AOBEM) Part 1 scores for post graduate year-4 residents who took both exams in the same year from 2009 to 2012.

Figure 1.

. Scatter plots for the Resident In-Service Examination (RISE) Percentiles vs. the American Osteopathic Board of Emergency Medicine (AOBEM) Part 1 score for each of four years and all four years combined. Correlation coefficients and 95% confidence intervals are in Table 2.

An ROC curve was generated with RISE percentile and AOBEM Part 1 Pass/Fail (Figure 2).9 The ROC curve is a graphical plot that illustrates the performance of a binary classifier system as its discrimination threshold is varied. If the curve reached the left upper corner, where sensitivity=1 and specificity=0, then the prediction is perfect (100% correct). Practically speaking, the point on the curve, which is closest to the left upper corner, would be considered as the best cut-off with the greatest sensitivity and specificity. The sensitivity and the specificity of different percentiles as cut-off points for predicting pass or fail on the AOBEM Part 1 are listed in Table 3. These data demonstrate that lower percentile scores had higher sensitivity and lower specificity while higher percentile scores produced lower sensitivity and higher specificity. It was found that the 65th percentile was the best cut-off point, which maximized the sensitivity (0.81) and specificity (0.88) together.15 The area under the curvewas 0.885 (95%CI=0.834 to 0.936).

Figure 2.

The receiver operating curve analysis. When 0.65 was selected as the optimum cut-off point the area under curve was 0.89 (95% confidence interval 0.75 to 0.85).

Table 3.

The sensitivity and the specificity of different Resident In- Service Examination percentiles as cut-off points for predicting pass or fail on the American Osteopathic Board of Emergency Medicine Part 1 for the pooled data on the 409 subjects. The 65th percentile (in bold) was the cut-off point, which maximized the sensitivity (0.81) and specificity (0.88).

The analysis of RISE percentile vs. AOBEM Pass/Fail is presented in Table 4 as descriptive statistics. The average percentile by the Pass group was significantly higher than that by the Fail group by independent T test (78.0 vs. 43.4; p<0.001).

Table 4.

. The mean percentile scores of the residents that failed the American Osteopathic Board of Emergency Medicine (AOBEM) Part 1 were significantly lower than the scores of the residents who passed the AOBEM Part 1 (p<0.001).

The 22 of 26 residents who failed the AOBEM Part 1 scored below the 65th percentile on the RISE exam. The group that passed the AOBEM Part 1 had more residents distributed in the higher percentile area than in the low percentile area (Figure 3).

Figure 3.

The American Osteopathic Board of Emergency Medicine (AOBEM) Part 1 fail and pass rates for each decile score on the Resident In-Service Examination (RISE).

The ROC analysis (Figure 2) demonstrated that the 65th percentile on the RISE was the most sensitive inflection point for predicting a resident's outcome of pass or fail on the AOBEM Part 1. The residents whose RISE scores were at the 7th decile and above (≥60th percentile) had a pass rate of greater than 95%. The passing rates for the 7th, 8th, 9th and 10th deciles were >95% for each decile. The passing rate for the 6th decile (51st - 60th percentile) dropped to 84.4%, which was substantially higher than the lower deciles (Table 5).

Table 5.

The American Osteopathic Board of Emergency Medicine (ABOEM) passing rate for each decile on the Resident In-Service Examination (RISE). The greatest change in association between RISE percentile and ABOEM pass rate occurred from the 50th to 70th percentiles.

Figure 4 uses logistic regression analysis to generate a graph that can be used by residents and program directors as a rough estimate of the probability of passing the AOBEM Part 1 based on RISE percentile. This estimate is based on the past data that we had available to us.

Figure 4:

A logistic regression analysis curve predicting the probability of passing the American Osteopathic Board of Emergency Medicine (AOBEM) Part 1 based on Resident In- Service Examination percentile according to the analyzed data.

DISCUSSION

Other medical specialties have found that correlations exist between their resident in-service examinations and resident performance on board certification examinations.1–14 These specialties for the most part have shown that moderately strong correlations exist between scores on their in-service exams and their board certifying specialty exams.

Our results show correlation that an osteopathic EM resident's RISE percentile during their PGY-4 year of residency training correlates strongly with their AOBEM Part 1 score.

Our results also correlate that the rate of passing the AOBEM Part 1 increases with higher percentile scores on the RISE. By describing the RISE percentiles as deciles, we showed correlation that the pass rate on the AOBEM Part 1 generally increases by decile. For our study period, scoring in the top decile on the RISE virtually guaranteed first-time success on the AOBEM Part 1 certifying examination. This information can potentially be very useful to fourth-year residents who are debating whether or not they are ready to sit for the AOBEM Part 1 before they graduate residency. Based on the results of this study, PGY-4 residents who score in the top 4 deciles on the RISE exam could be encouraged to take the AOBEM Part 1 before finishing residency. This recommendation is based on historical data, and statistics may change from year to year.

Although most people in the upper 8 deciles on the RISE do pass the AOBEM Part 1, the pass rate generally improves with each higher decile.We also used the ROC curve to identify the RISE percentile score that could best predict success or failure on the AOBEM Part 1. Using this analysis, the 65th percentile was determined to be the most significant percentile as a breakpoint in performance prediction. According to our data, examinees who scored greater than this percentile passed the AOBEM Part 1 approximately 99% of the time while examinees who scored below this percentile on the RISE passed the AOBEM Part 1 only 76% of the time. This is a metric that program directors and examinees may use to determine examinee preparedness for the AOBEM Part 1.

The majority of the examinees who took the AOBEM Part 1 before graduating residency scored in the upper 2 deciles on the RISE, while 22 of 26 residents who failed scored below the 65th percentile on the RISE exam.

When examining the data, we showed correlation that the overall pass rate for the AOBEM Part 1 was higher for the PGY-4 residents when compared to the overall exam pass rate. The PGY-4 residents who opted to take the exam may have chosen to take it at a time when they felt most prepared. Another possible reason for this result is that included in the overall exam pool are examinees who had previously failed the AOBEM Part 1. There are other possibilities that may have influenced a resident's decision to postpone taking the AOBEM Part 1.

We chose to focus our analysis on PGY-4 residents who took the AOBEM Part 1 because the RISE and AOBEM Part 1 are offered only several weeks apart and reflect the most consistent knowledge base. Additional learning or forgetting of concepts would be minimized by this short time span. Additionally, this information would be most useful for residents as a predictor of their need to further prepare themselves for the AOBEM Part 1. Program directors can also use this information to modify training programs to better prepare their residents for first-time success on the AOBEM Part 1 exam. Future studies could analyze other post-graduate years and provide performance information earlier in the residency training period.

LIMITATIONS

This was a retrospective study, which permits only associations rather than cause and effect. Since only PGY-4 residents who opted to take the AOBEM Part 1 during the same year were used, this limits the generalizability to other postgraduate years.

Osteopathic PGY-4 EM Residents have only been allowed to participate in the AOBEM Part 1 during their residency training since 2009. Each year an increasing number of PGY-4 residents have chosen to take the AOBEM Part 1. Future results may not have the corresponding predictive power as more PGY- 4 residents opt to take the AOBEM Part 1.

The number of failures (26) of total examinees (409) was relatively low. Even though we included the entire population, the future predictive value of any percentile score on the RISE would at best be approximate.

We only analyzed correlations between the RISE percentiles and AOBEM Part 1 performance and did not look at other variables such as age, sex, race, and size of training program. We did not include people who took the test more than once. Therefore this correlation may be limited to first time test takers only.

The study group was a convenience sample and captured less than 50% of graduating resident performance. Selection bias may exist in residents choosing to take AOBEM early.

CONCLUSION

The RISE is a useful tool for both osteopathic EM residents and program directors to gauge a resident's preparedness for the AOBEM Part 1 as shown by a strong correlation between performances on both exams. The number of residents who opted to take the AOBEM Part 1 as a PGY-4 increased each year of our study period. Continued analysis of subsequent exams should be performed. Future studies can provide residents and program directors with ongoing analysis so they may effectively use the RISE as a tool to gauge a resident's future performance on the AOBEM Part 1 Board Certifying Exam.