Evaluation and enhancement of medical knowledge competency by monthly tests: a single institution experience.

BACKGROUND: In-training examination (ITE) has been used as a predictor of performance at the American Board of Internal Medicine (ABIM) certifying examination. ITE however may not be an ideal modality as it is held once a year and represents snapshots of performance as compared with a trend. We instituted monthly tests (MTs) to continually assess the performance of trainees throughout their residency.
OBJECTIVE: To determine the predictors of ABIM performance and to assess whether the MTs can be used as a tool to predict passing the ABIM examination.
METHODS: The MTs, core competencies, and ITE scores were analyzed for a cohort of graduates who appeared for the ABIM examination from 2010 to 2013. Logistic regression was performed to identify the predictors of a successful performance at the ABIM examination.
RESULTS: Fifty-one residents appeared for the ABIM examination between 2010 and 2013 with a pass rate of 84%. The MT score for the first year (odds ratio [OR] =1.302, CI =1.004-1.687, P=0.04) and second year (OR =1.125, CI =1.004-1.261, P=0.04) were independent predictors of ABIM performance along with the second-year ITE scores (OR =1.248, CI =1.096-1.420, P=0.001).
CONCLUSION: The MT is a valuable tool to predict the performance at the ABIM examination. Not only it helps in the assessment of likelihood of passing the certification examination, it also helps to identify those residents who may require more assistance earlier during their residency. It may also highlight the areas of weakness in program curriculum and guide curriculum development.

Introduction

The American Board of Internal Medicine (ABIM) since its establishment in 1936 has worked toward improvement of the US health care system. The ABIM certification implies that the physician has demonstrated adequate clinical knowledge, judgment, and skills to render appropriate patient care.1 Board certification not only ensures better patient care2,3 but is also preferred by the public from their health care providers.4

The performance at the ABIM certification examination is of particular importance to residency programs. It not only helps in maintenance of their American Council of Graduate Medical Education (ACGME) accreditation5 but also necessitates the programs to recruit residents of higher caliber. This enables programs to build upon the medical knowledge of their residents throughout training and allows for better performance on subsequent certification examinations.6

The ABIM pass rates for the certification examination has shown a declining trend in the 3-year rolling average recently, with pass rates being 91% in 2008 and 85% in 2012.1 Hence, the residency programs want to identify residents who are at risk of being unsuccessful at the ABIM certification examination and need remediation. In-training examination (ITE) has been utilized as a predictor of performance of trainees in the ABIM examination in their first attempt.7,8 This tool not only helps to provide feedback on resident performance but also helps to develop intervention strategies. However, ITE may not be the ideal modality of identification of at-risk individuals as it is held annually and represents snapshots of the performance as compared with a trend. The examination is too far apart to assess actual improvement and trends in performance. Hence, there is a need for a complementary method of assessment of a resident’s medical knowledge frequently over a period of time. This led us to develop a monthly test (MT) to enable the program to continually assess the performance of the residents throughout their residency.

Our curriculum is composed of two components. This test is based on the curriculum comprised of the ABIM core objectives known as the core curriculum. The other curriculum component is tailored appropriately to address areas of weakness identified by the residents’ performance in the previous year’s ITE; this is known as the dynamic curriculum. The dynamic curriculum is developed in a recurring cycle of 18 months.

The test was instituted in 2007 and is given in the third week of the month with a feedback session the following week.

The residents are tested once a month followed by a critique session and formative feedback of the multiple choice questions (Figure 1). The sources of the questions for the MT are standardized questions obtained by the program and are not a proprietary assessment tool to our program. The test session consists of 20 questions selected from an item pool of ~3,000 questions. The questions are selected by the program director and the chief resident based on the objective topics discussed in the didactic sessions of the previous month. Moreover, a semi-annual summative test is administered and consists of all objectives from the previous five MTs and consists of 40 questions per session, held in December and June of every year.

Figure 1

Flow diagram of the administration of the monthly tests.

Abbreviations: ACGME, American Council of Graduate Medical Education; ABIM, American Board of Internal Medicine; ITE, in-training examination.

With the importance of performance in the internal medicine boards both for the residents and the program, there is need for identification of residents who may be at risk for failure at the ABIM examination. There is no study which has evaluated MTs as a tool to assess performance at the ABIM examinations.

Objective

The objectives of our study are 1) to determine the predictors of ABIM performance and 2) to assess whether the MTs administered by the program can be used along with the ITE examination to predict passing the ABIM examination.

Methods

The cohort consisted of 51 residents who successfully graduated from the internal medicine residency at the University of Toledo between 2010 and 2013. This consisted of consecutive batches after 2007, the year when the MTs were introduced. Data were collected from the records of the residency program and were de-identified. The study was approved by the local institutional review board.

Data were extracted on a predefined data collection form. Extracted data included the following: previous higher degree, United States Medical Licensing Examination (USMLE) step 1, 2, clinical knowledge, and step 3 scores. We did not have USMLE step 3 scores for all the residents as it is not a requirement to be reported to the program. In addition, information on ITE scores (percentage correct) in the postgraduate years (PGYs) 1, 2, and 3 and noon conference attendance were also collected.

The six ACGME core competencies were assessed for all the graduates by the faculty on a scale of 1–9 using New Innovations Evaluation System®, and a cumulative score was developed for the entire training period. MTs were initiated at the University of Toledo from 2007. The yearly average of these test scores in the PGY1, PGY2, and PGY3 was calculated. A score based on the number of MTs missed was also calculated (monthly missed test score: number of tests missed/total number of tests). The score took into account that residents may miss MTs when they were on vacation or night float. The semi-annual summative examination scores were not included in the analysis because it may have been a confounder in the analysis as it included the questions from the earlier half of the year.

Analysis

Continuous variables were expressed as a mean ± standard deviation, and qualitative data were expressed as a percentage. Differences between groups were evaluated by unpaired t-tests or chi-square analysis, as appropriate. The effects of baseline parameters were assessed on ABIM pass rate. Univariate association between ITE and the MTs were assessed by Pearson’s correlation.

Multivariable logistic regression analysis was applied to identify the determinants of ABIM examination performance. In this model, the independent predictors used were ITE scores, yearly average of the MT scores, MT missed score, and the cumulative core competencies score. Forward stepwise selection was used to select the subset of predictors, which contribute most to the ABIM examination performance. Statistical significance was defined as P<0.05. Analysis was performed using SPSS Version 20.0 (IBM Corporation, Armonk, NY, USA).

Results

A total of 51 graduates appeared for the ABIM certification examination since the start of the MTs. Out of 51 graduates, 43 (84%) passed the ABIM examination on the first attempt and the remaining passed in their second attempt. All the residents had taken the annual ITE and the MT. There was no correlation found between features such as age, sex, time lag to start residency, and ABIM pass rate. The ITE and the MT scores improved for both the groups. The residents who passed the ABIM on the first attempt had higher evaluations by the faculty in terms of core competencies score as well as higher ITE scores (PGY1: 57.62±7.84 vs 49.85±6.12; PGY2: 65.65±8.21 vs 51.00±12.46; PGY3: 67.49±7.37 vs 58.32±10.17) and MT average scores (PGY1: 56.21±8.93 vs 47.80±4.55; PGY2: 64.16±9.26 vs 54.88±6.80; PGY3: 69.34±8.54 vs 65.63±5.28; Table 1). These residents also had higher USMLE scores as compared with the residents who failed to pass the ABIM examination in the first attempt.

Table 1

Faculty evaluation and resident’s performance stratified on the basis of American Board of Internal Medicine certification examination results

Characteristics	ABIM pass (n=43, 84%)	ABIM fail (n=8, 16%)
ACGME core competencies
PC	7.44±0.40	7.35±0.35
MK	7.46±0.54	7.17±0.29
PBLI	7.21±0.33	7.01±0.26
ICS	7.51±0.72	7.35±0.33
Prof	7.60±0.29	7.29±0.34
SBP	7.31±0.37	6.94±0.37*
Total cumulative score	45.81±7.76	43.14±7.64
Monthly tests
PGY1	57.62±7.84	49.85±6.12**
PGY2	65.65±8.21	51.00±12.46***
PGY3	67.49±7.37	58.32±10.17***
Average	64.01±6.16	53.44±3.28***
In-training examination
PGY1	56.21±8.93	47.80±4.55**
PGY2	64.16±9.26	54.88±6.80***
PGY3	69.34±8.54	65.63±5.28
Average	62.81±8.24	50.25±5.28***
USMLE step 1	230.44±17.33	213.00±11.78***
USMLE step 2 CK	233.86±19.57	210.38±23.48***

Notes: P-value:

<0.05;

<0.01;

<0.001. Data presented as mean ± standard deviation.

Abbreviations: ABIM, American Board of Internal Medicine; ACGME, American Council of Graduate Medical Education; PC, patient care; MK, medical knowledge; PBLI, practice-based learning and improvement; ICS, inter-communication skills; Prof, professionalism; SBP, system-based practice; PGY1, postgraduate year 1; PGY2, postgraduate year 2; PGY3, postgraduate year 3; USMLE, United States Medical Licensing Examination; CK, clinical knowledge.

MTs had a good correlation with the ITE examination. The average of the first-year MTs showed a moderate correlation with second-year ITE (r=0.568, P<0.001; Table 2). Similarly, the second-year MTs were correlated with third year ITE (r=0.603, P<0.01; Table 2).

Table 2

Correlation between the monthly tests and the in-training examination scores

Monthly tests	In-training examination
	PGY1		PGY2		PGY3
	r	P	r	P	r	P
PGY1	0.583	<0.001	0.568	<0.001	0.565	<0.001
PGY2	–	–	0.503	<0.001	0.603	<0.001
PGY3	–	–	–	–	0.592	<0.001

Note: – Indicates the relevant correlation was not performed.

Abbreviations: PGY1, postgraduate year 1; PGY2, postgraduate year 2; PGY3, postgraduate year 3.

The logistic regression model was developed to predict the result of the ABIM examination. The model included the subjective assessment of the ACGME core competencies by the faculty and the objective assessment of the medical knowledge by the ITE and the MT scores. The Hosmer– Lemeshow goodness of fit for the model was not significant (P>0.9) indicating that the model was well calibrated. The MT scores for the first year (odds ratio [OR] =1.302, CI =1.004–1.687, P=0.04) and second year (OR =1.125, CI =1.004–1.261, P=0.04) were independent predictors of ABIM performance along with the second-year ITE scores (OR =1.248, CI =1.096–1.420, P=0.001). Missed MT score was also independent predictor of ABIM certification (OR =1.237, CI =1.042–1.469, P=0.01; Table 3).

Table 3

Logistic regression analysis showing independent predictors of ABIM certification

Characteristics	OR	Confidence interval	P-value
Monthly test scores
PGY1	1.302	1.004–1.687	0.04
PGY2	1.125	1.004–1.261	0.04
ITE test scores
PGY2	1.248	1.096–1.420	0.001
Missed MT score	1.237	1.042–1.469	0.01

Abbreviations: ABIM, American Board of Internal Medicine; OR, odds ratio; PGY1, postgraduate year 1; PGY2, postgraduate year 2; ITE, in-training examination; MT, monthly test.

Discussion

The findings of our study indicate that MT can be used along with the ITE examination as a tool to assess medical knowledge and predictor of ABIM examination performance. The ACGME core competencies scores were not found to be predictive of the performance in the ABIM certification examination. Though studies have reported correlation between USMLE scores and performance during residency,9 we did not include USMLE scores as they reflect background medical knowledge rather than knowledge gained during residency. The number of tests missed had a negative association with ABIM pass rates with missing more tests indicative of being unsuccessful at the boards.

The results of the second-year ITE highly correlated with ABIM pass rates on the first attempt.10 Moreover, the ITE was primarily developed to assess the knowledge of residents in the middle of the residency. Hence, it was not developed to predict a resident’s performance based on the first-year ITE results. Moreover, the first-year ITE is taken too early in the residency to assess the knowledge or skills gained. By the time the second-year ITE results are known, little time may be left for intervention strategies to have the desired effect. MTs have an advantage as they help early identification of at-risk residents and consequently remedial measures could be instituted early during the course of their second year. These measures include, but are not limited to faculty supervised intensive board review sessions, dedicated topic reviews and attending board review courses. They also help to monitor the trend and effectiveness of the remedial measures instituted and to adapt them according to the needs of the focused group. It has been shown that the effects of repeated testing on learning are more than repeated studying.11 It is because it involves retrieval of information from memory, which leads to long-term retention as compared with repeated studying.12 The MT including the semi-annual cumulative testing along with the critique sessions are designed to help consolidate the information gained, which further strengthens the memory for long-term retrieval.12

The ITEs alone, which provide a snapshot at a time, may not be an adequate tool to assess the performance of residents who are at the borderline.8 The MT may be able to correctly identify these individuals, as a longitudinal trend would be better as compared with an annual exercise. These are the individuals who gain most from an intervention. The MT and subsequent critique sessions will allow the residents to evaluate their strengths and weaknesses and compare them to their peers on a monthly basis. This also gives the program an opportunity to assess its curriculum and didactic methods.

In our study, we did not find any correlation with the faculty assessment of the competencies and ABIM examination performance. MTs may also be a good tool to aid the faculty in assessment of the core competencies. This may play an important role in view of the new accreditation system implementation where emphasis is placed on innovation with tools to assess the six educational competencies, with clear performance-based endpoints and not numerical rating scales.13

Our study has several limitations. As the MTs were started relatively recently, our sample size was small, which may have affected our analysis. We evaluated the effect of this intervention in only one residency program. It may be helpful to validate this in other residency programs as well. We did not have the scores for ABIM examination available to us, and the results were taken as either pass or fail. It would be helpful to know the scores, especially those at the borderline, in order to further assess the remedial measures. We did not have complete data on the attendance at didactics. This would have shown some insight into the attitude and commitment of the residents toward education. We did not include USMLE scores, which may have given an insight on the medical knowledge of the residents. However, the scores would have been representative of the background medical knowledge instead of the residency training. Lastly, being a retrospective analysis, it has all the limitations associated with such an analysis.

Conclusion

The MT is a tool to enhance the formative feedback in medical knowledge and assess the performance of residents over a period of time. It may not only help in the evaluation of medical knowledge of the residents, but also supplement the ITE to help predict residents at risk of failing the ABIM. Looking at the usefulness of the monthly examination in conjunction with the annual ITE, it may be meaningful for other residency programs to replicate similar initiatives. To have a more definite conclusion, a comparative study to assess the impact of MTs and to study targeted optimal remediation regimens for residents with poor performance is needed.