A national survey: use of the National Board of Medical Examiners® basic science subject exams and Customized Assessment Services exams in US medical schools.

PURPOSE: The National Board of Medical Examiners® (NBME®) offers basic science subject exams and exams developed using the Customized Assessment Services (CAS) program. The scope and utilization of these exams by US medical schools has not been reported. Therefore, this survey aimed to measure US medical school usage of NBME® basic science subject exams and exams administered using the CAS program.
METHODS: In May 2016 a survey was sent to 139 US medical schools with provisional or full accreditation by the Liaison Committee on Medical Education®. Schools were asked if they administered NBME® basic science subject exams or exams developed using the NBME® CAS program. Follow-up questions were asked if the school responded "Yes" to administering the exams.
RESULTS: Responses were received from 46-48 schools (33%-35% response rate). Of the schools responding, 7-13 schools (15%-28%) administer an NBME® basic science subject exam and 14 schools (30%) administer an exam developed using the CAS program. Of the schools that administer an NBME® basic science exam, 7-12 (83%-100%) use the exam for summative assessment for a course and 6-11 (77%-100%) use the exam results for curricular revision. Twelve schools (86%) that administer an NBME® exam developed using the CAS program use the exam for summative assessment for a course. Time of administration within the curriculum for NBME® basic science subject exams varied between 1 month prior to >6 months prior to students taking the United States Medical Licensing Examination® Step 1.
CONCLUSION: According to our data, most US medical schools responding to the survey do not administer NBME® basic science subject exams or exams developed using the NBME® CAS program. Schools that do administer NBME® basic science subject exams use the exams for summative assessment for a course and curricular revision, while exams administered using the CAS program are used predominately for summative assessment for a course.

Introduction

US medical schools strive to prepare all students to become licensed physicians. Part of the process for becoming a licensed physician is completion of the United States Medical Licensing Examination® (USMLE®) Step 1 examination which measures basic science knowledge.1 Although this single external exam is a top factor for ranking applicants for the National Resident Match Program®,2 each program is unique in its delivery and assessment of basic science content.

As we have previously reported, a National Board of Medical Examiners® (NBME®) comprehensive basic science exam (CBSE) is used by US medical schools to identify students performing below expectations;3 however, the NBME® offers basic science subject exams and clinical science subject exams “for the purpose of assessing the educational achievement of individuals in specific subject areas.”4 The program was developed in the 1960s to measure basic and clinical science.5 In an effort to allow programs to develop exams consistent with changing medical school curricula, the NBME® created the Customized Assessment Services (CAS) program in 2007 which allows faculty to create integrated basic science or clinical science subject exams.5

There are many studies reporting the utility of the NBME® clinical science subject exams in undergraduate medical education.6–11 In addition, Haist et al5 reported that in 2015 “greater than 550,000 assessments were provided through the subject examination program, NBME self-assessment services, and CAS.” It was also reported that the number of exams administered using the NBME® CAS has steadily increased since its inception.5 Although the number of exams provided has been reported, the scope and utilization in US medical schools for NBME® basic science subject exams and exams developed using the NBME® CAS program is limited.

The purpose of this study was to determine the extent to which NBME® basic science subject exams and exams developed using the NBME® CAS are used 1) in preclinical medical education, 2) as part of summative or formative assessments, and 3) for curricular revision.

Methods

Survey methods

Survey methods have previously been described.3 Briefly, 139 Liaison Committee on Medical Education® schools with provisional (4 schools) or full (135 schools) accreditation as of February 29, 2016, were identified. An e-mail message and link to the anonymous survey, hosted at SurveyMonkey®, was sent to the Associate Dean for Curriculum, or equivalent on May 3, 2016. A follow-up reminder was sent 1 week later and the survey ended on May 16, 2016. The aim of the study was presented in the e-mail message.

Ethical approval

A review by the University of South Carolina institutional review board was conducted and it was determined that this was not human subject research.

Results

There were 46–48 responses (33%–35% response rate) regarding the use of NBME® subject exams for basic science content. The first question regarding NBME® subject exams was answered by 48 schools; however, all schools did not answer the remaining questions regarding NBME® subject exams. There were 46 responses (33% response rate) regarding the use of exams developed using the NBME® CAS program.

Data presented cover only schools that administer the NBME® basic science subject exams or exams administered using the CAS program

Table 1 provides the data regarding the number of schools that offer NBME® basic science subject exams. In addition, Table 1 provides the number of schools that use the NBME® basic science subject exams for curriculum revision, formative assessment for a course, and summative assessment for a course. Of the schools that responded, the histology and cell biology subject exam is administered by the lowest number of schools (15%) and the pathology subject exam is administered by the highest number of schools (28%) (Table 1). Of the schools that offer the NBME® basic science-specific subject exams, most use the exams for curricular revision (range 77%–100%) and summative assessments for a course (range 83%–100%).

Table 1

Utilization of NBME® basic science subject exams for schools responding

NBME® (specific) subject exam	Question 1			Question 2			Question 3			Question 4
	TR	Yes	No	TR	Yes	No	TR	Yes	No	TR	Yes	No
Behavioral sciences	48	10 (21%)	38 (79%)	10	10 (100%)	0	10	1 (10%)	9 (90%)	10	10 (100%)	0
Biochemistry	47	10 (21%)	37 (79%)	9	9 (100%)	0	9	0	9 (100%)	9	9 (100%)	0
Gross anatomy and embryology	47	12 (26%)	35 (74%)	12	11 (92%)	1 (8%)	12	0	12 (100%)	12	12 (100%)	0
Histology and cell biology	47	7 (15%)	40 (85%)	7	6 (86%)	1 (14%)	7	0	7 (100%)	7	7 (100%)	0
Neuroscience	47	12 (26%)	35 (74%)	12	10 (83%)	2 (17%)	12	2 (17%)	10 (83%)	12	10 (83%)	2 (17%)
Pathology	47	13 (28%)	34 (72%)	13	10 (77%)	3 (23%)	13	0	13 (100%)	13	12 (92%)	1 (8%)
Pharmacology	47	10 (21%)	37 (79%)	10	10 (100%)	0	10	0	10 (100%)	10	10 (100%)	0
Physiology	46	11 (24%)	35 (76%)	11	10 (91%)	1 (9%)	11	0	11 (100%)	11	11 (100%)	0

Notes: Curriculum leadership was asked question 1 for each of the different NBME® basic science subject exams listed. If the response to question 1 was “Yes,” curriculum leadership was prompted to respond to questions 2–4. Question 1= does your institution administer the NBME® (specific) subject exam prior to the USMLE® Step 1 exam? Question 2= does your institution use data from the NBME® (specific) subject exam to revise the curriculum? Question 3= is the NBME® (specific) subject exam used as an aspect of formative assessment for a course? Question 4= is the NBME® (specific) subject exam used as an aspect of summative assessment for a course? Number of respondents (percentage of respondents).

Abbreviations: NBME, National Board of Medical Examiners; TR, Total responses; USMLE, United States Medical Licensing Examination.

Most pharmacology, pathology, and behavioral sciences subject exams are administered 3 months or less prior to students taking the USMLE® Step 1 while most biochemistry, gross anatomy and embryology, histology and cell biology, neuroscience, and physiology exams are administered >6 months prior to students taking the USMLE® Step 1 (Table 2).

Table 2

Administration of NBME® (specific) subject exams in relation to students taking USMLE® Step 1

When is the NBME® (specific) subject exam administered relative to the USMLE® Step 1 exam?
NBME® (specific) subject exam	TR	Within 1 month prior	1–3 months prior	4–6 months prior	>6 months prior
Behavioral sciences	10	3 (30%)	3 (30%)	1 (10%)	3 (30%)
Biochemistry	9	0	0	0	9 (100%)
Gross anatomy and embryology	12	0	0	0	12 (100%)
Histology and cell biology	7	1 (14%)	0	0	6 (86%)
Neuroscience	12	0	0	2 (17%)	10 (83%)
Pathology	13	3 (23%)	9 (69%)	0	1 (8%)
Pharmacology	10	4 (40%)	5 (50%)	0	1 (10%)
Physiology	11	1 (9%)	0	0	10 (91%)

Note: Number of respondents (percentage of respondents).

Abbreviations: NBME, National Board of Medical Examiners; TR, total responses; USMLE, United States Medical Licensing Examination.

The number of schools administering exams using the NBME® CAS program is presented in Table 3. Of the schools that administer exams using the NBME® CAS program, most use the data for summative assessment for a module (Table 3).

Table 3

Utilization of an NBME® CAS exam for basic science content by schools responding.

NBME® CAS exam	Question 1			Question 2			Question 3			Question 4
	TR	Yes	No	TR	Yes	No	TR	Yes	No	TR	Yes	No
CAS	46	14 (30%)	32 (70%)	14	5 (36%)	9 (64%)	14	3 (21%)	11 (79%)	14	12 (86%)	2 (14%)

Notes: Curriculum leadership was asked question 1 regarding utilization of the CAS program. If the response to question 1 was “yes”, curriculum leadership was prompted to respond to questions 2–4. Question 1= does your institution administer an NBME® exam developed through the CAS prior to the USMLE® Step 1? That is, members of your institution select the content areas or exam items that are administered. Question 2= does your institution use the data from an NBME® exam developed through the CAS prior to the USMLE® Step 1 exam to revise the curriculum? Question 3= does your institution use an NBME® exam developed through the CAS prior to the USMLE® Step 1 exam as an aspect of formative assessment for a course? Question 4= does your institution use an NBME® exam developed through the CAS prior to the USMLE® Step 1 exam as an aspect of summative assessment for a course? Number of respondents (percentage of respondents).

Abbreviations: CAS, Customized Assessment Services; NBME, National Board of Medical Examiners; TR, total responses; USMLE, United States Medical Licensing Examination.

Figure 1 presents the percent of schools that administer the NBME® CBSE which also administer an NBME® subject exam. The pathology subject exam is administered by the highest percentage (34%) of schools that offer the NBME® CBSE, while the histology and cell biology subject exam is administered by the lowest percentage (17%) of schools that offer the NBME® CBSE. We have previously reported the number of schools that administer the NBME® CBSE.3

Figure 1

Percent of schools that administer the NBME® CBSE that also administer an NBME® subject exam prior to students taking the USMLE® Step 1.

Notes: Schools that administer the NBME® CBSE (n=37). *CAS = exams developed using the CAS program.

Abbreviations: CAS, Customized Assessment Services; CBSE, Comprehensive Basic Science Exam; NBME, National Board of Medical Examiners; USMLE, United States Medical Licensing Examination.

Figure 2 presents the percent of schools that administer an NBME® subject exam that also administer the NBME® CBSE. All schools that administer behavioral sciences subject exam also administer the NBME® CBSE (highest percentage), while only 86% of schools that administer the histology and cell biology subject exam also administered the NBME® CBSE (lowest percentage).

Figure 2

Percent of schools that administer an NBME® subject exam and the NBME® CBSE prior to students taking the USMLE® Step 1.

Notes: Number of schools that administer a subject exam: behavior sciences (n=10), biochemistry (n=10), gross anatomy and embryology (n=12), histology and cell biology (n=7), neuroscience (n=12), pathology (n=13), pharmacology (n=10), physiology (n=11), CAS (n=14). *CAS = exams developed using the CAS program.

Abbreviations: CAS, Customized Assessment Services; CBSE, Comprehensive Basic Science Exam; NBME, National Board of Medical Examiners; USMLE, United States Medical Licensing Examination.

Discussion

To our knowledge, this is the first report describing US medical school use of NBME® basic science subject exams and exams administered using the CAS program for preclinical medical education. On average, less than a quarter of the schools responding to the survey administer an NBME® basic science subject exam. Schools that do offer an NBME® basic science subject exam use the exams predominately for summative assessment for a course and to help revise curriculum (77%–100% depending on exam) (Table 1). In addition, only 30% of schools that responded to the survey administer exams using the CAS program, with most schools using the exams administered for summative assessment for a course (Table 3), while only 36% of schools that administer exams using the CAS program use the data to help revise curriculum (Table 3). Of the schools that administer an exam through the CAS program, 29% (4 out of 14 schools) also administer an exam using one or more of the NBME® basic science exams. Most schools that administer an NBME® basic science subject exam also administer the NBME® CBSE (Figure 2).

The finding that <30% of respondents administer NBME® basic science subject exams and only 30% administer exams created using the NBME® CAS program for preclinical assessment of medical student basic science knowledge is surprising given the fact that the NBME® basic science subject exams offer high-quality questions over a broad range of basic science content with the “majority of subject test items being used previously on the USMLE.”5 Although there is continued growth in assessments developed using the CAS program,5 offering exams using NBME® basic science exam or exams administered using the CAS program would provide exposure of USMLE® board style questions to students throughout their preclinical training. Use of externally validated exams may be an alternative for programs as it has been reported that internally developed exams are low quality, with the exception being for exam items developed by formally trained question writers.12 Additionally, it has been reported that multiple preclinical NBME® basic science subject exams correlate with performance on the NBME® surgery subject exam, and when a multiple regression analysis was performed the NBME® pathology exam remained the strongest single predictor of NBME® surgery exam performance.13

The fact that most schools use the NBME® basic science subjects for curriculum revision is not surprising. These exams provide targeted feedback of student performance in specific basic science content areas. However, what is surprising is the fact that only 77% of the schools administering the pathology NBME® basic science subject exam use the results for curricular revision. It is not clear why more schools do not use the pathology basic science subject exam for curricular revision.

The CAS program was introduced by the NBME® in 2007 with assessments administered through the program continuing to increase.5 However, the reported scope and utilization by US medical schools is limited. We report 30% of respondents use the CAS program. This program allows schools with integrated curricula to develop assessments tailored to the integrated courses provided. Interestingly, fewer schools (36%) use exams administered through the CAS program for curricular revision. The reason for this difference compared to NBME® basic science exams is unclear. Exams developed and administered using the NBME® CAS program are mostly used for summative assessments for a course (86%). Our findings report that only 21% of schools administer an exam using the CAS program for formative assessment for a course, with 2 schools stating that they use exams administered using the CAS program for both formative and summative assessments for a course. Use of exams developed using the NBME® CAS program for formative assessments for a course may be of interest as it has recently been reported that these exams correlate with performance on the USMLE® Step 1 and is a tool used to identify students at risk for low performance on the USMLE® Step 1.14

Further research into the performance of students attending schools that offer externally developed exams compared to schools that administer only internally developed exams would be of interest. In addition, determining why more schools do not offer externally developed exams is needed as this provides additional test item exposure to students preparing to take external board exams (ie, USMLE® Step 1). Additional areas of future research are to determine student and faculty perceptions of externally developed exams for courses as compared to internally developed exams.

A strength of this survey is the ability to gather data regarding the use of external exams in preclinical medical education using a low-cost method. A limitation of the survey is the survey response rate. A second limitation of the survey is that follow-up questions regarding outcomes data for the schools that use the external exam for curricular change were not included in the survey.

Conclusion

In conclusion, we report that most US medical schools responding to our survey do not offer NBME® basic science subject exams (15%–28% of schools offering an exam depending on the exam) or exams administered using the NBME® CAS program (30% of schools offering an exam) for preclinical medical education. However, of the schools responding most schools that offer an NBME® basic science subject exam use the exam for summative assessment for a course and for curricular revision, whereas of the schools that administer an exam using the NBME® CAS program most use the exam for summative assessment for a course while fewer use it for curricular revision.