Influence of assigned reading on senior medical student clinical performance.

OBJECTIVE: This Institutional Review Board-approved, prospective, observational study compared the clinical performance of senior medical students in an emergency medicine (EM) clerkship using a clinical behavioral evaluation tool in which one group had mandatory, topic specific readings and the other did not.
METHODS: The study took place in an urban, tertiary referral center emergency department treating 43,000 patients annually and supporting medical student clerkships and an EM residency. The grades of two groups of senior medical students participating in an elective EM clerkship were compared. Those students during the 2002-2004 academic years were not assigned mandatory, topic-specific reading for the clerkship, while those during the 2004-2007 academic years were. The groups were compared on baseline demographic information, prior academic performance, and EM clerkship grade distributions using appropriate statistical techniques, including multinomial logistic regression, chi-square tests, and Fisher's Exact tests.
RESULTS: The control and experimental groups each had 83 subjects and were similar in baseline characteristics, except for the control group performing better than the experimental group during the basic science training of medical school (years 1-2; p=0.01). The experimental group had statistically significant more members in the EM Interest Group (EMIG; p=0.0001) and more members who went on to match in an EM residency (p=0.0007). The difference in grade distributions between the control group and experimental group was not statistically significant (p=0.40). Of note, those student members of the EMIG (p=0.0005) and those later matching to an emergency medicine residency (p<0.0001) were more likely to earn a grade of "honors" for the clerkship.
CONCLUSION: The addition of uniform, topic-specific reading assignments to an EM senior medical student curriculum does not improve the overall clinical performance of those students as measured using a clinical behavioral evaluation tool.

INTRODUCTION

Many medical student clinical clerkships required for graduation assign students a specified reading assignment as part of coursework. This requirement is designed to improve both the student’s factual knowledge, especially about disease processes uncommonly encountered in most clinical medical practices, and clinical performance. While it has been demonstrated that factual knowledge improves with implementation of required reading material, it has not been demonstrated that this translates into an improvement in clinical performance. Several educational studies have suggested that the format in which new material is presented is of less importance to acquiring medical skills than the number of repetitions in which that information is presented.1–10 If this is indeed true, then the addition of required reading materials to a clinical clerkship might serve as a means to increase the number of times a student is exposed to information about a specific disease process. More importantly, however, it would be useful to know whether this increased exposure to educational materials impacts clinical performance and ultimately patient outcomes.11 If required reading material improves clinical performance, then it could be argued that topic- specific readings should be included as a part of all clinical clerkships in medical school to supplement the student’s clinical experience.

Our senior medical student emergency medicine (EM) clerkship during the academic years of 2002–2003 and 2003–2004 did not require the reading of any EM topics, nor provide any suggested readings on EM topics. At that time the clerkship was purely a clinical experience in an urban emergency department (ED). Starting with the 2004–2005 academic year, the senior clerkship required students to read a textbook dealing with topics specific to senior medical students of emergency medicine. The book was written with the specific goal of providing students with information that would be useful clinically while working in an urban ED. The text, First Exposure to Emergency Medicine Clerkship, was published in June 2004.12 The student evaluation tool used to determine the student’s final grade has remained constant since the 2002–2003 academic year. This implementation affords the opportunity to determine whether required textbook readings are influential in improving student clinical performance by comparing grade distributions by year for the clinical portion of the senior EM clerkship. The null hypothesis of this study is that student grade distributions based on clinical behavioral evaluation measures will remain unchanged after the addition of required readings during the clerkship and the knowledge of required written, multiple choice testing at the end of the clerkship.

MATERIALS AND METHODS

Study Design

This study is a single center, prospective, observational study comparing the grade distributions of a consecutive sample of senior medical students assigned topic-specific readings during an EM clerkship and a historical control group of senior medical students without assigned topic-specific readings. This study was reviewed and approved by the Institutional Review Board under exempt status.

Study Setting and Population

This study was conducted at an urban university medical center ED staffed by American Board of Emergency Medicine-certified physicians. The ED has an annual patient census of approximately 43,000 patients comprised of both pediatric and adult populations. A four-week EM clerkship is available as an elective to senior medical students during July through April of each academic year. Study subjects were enrolled consecutively during the 2002–2007 academic years. For any given clerkship month, approximately four senior medical students attending the College of Medicine choose to participate in this elective clerkship. During this clerkship, the students are required to work 15 shifts of nine hours duration distributed equally among day, evening, and night hours. At the completion of a clinical shift, the student presents the supervising physician with a clinical evaluation form, utilizing an anchored Likert scale. The evaluating physician completes and returns this form to the student clerkship director. Each category of these evaluation forms is then averaged to arrive at an overall clinical grade for the clerkship of “Honors, High Pass, Pass, Marginal, or Fail.” While the students do not have mandatory didactic sessions, they are encouraged to attend the resident lectures. They do participate in suturing and splinting laboratories, but these activities do not directly impact their clinical grade other than providing them with experiences that may be utilized during clinical shifts. At an orientation session designed to familiarize the students with the ED on the first day of the rotation, the students are encouraged to use the text during ED clinical shifts to help guide their patient management.

The paperback text measures six inches by nine inches so that it may be carried in the pocket of a laboratory coat during clinical duties. It may then be used to correct knowledge deficits during the shift. The text is 455 pages in length and was written by five ABEM-certified emergency physicians at an academic hospital that teaches medical students and resident physicians. The text is divided into four sections. Section I discusses the paradigm of emergency care, the elements of a high-quality patient presentation to a supervising physician, and issues related to emergency medical services. Section II focuses on common emergency procedures, including their indications, contraindications, stepwise approach, and possible complications. Section III describes the generalized evaluation and treatment of patients presenting with chief complaints that are commonly encountered clinically. Section IV discusses specific diseases organized by organ system. The text in its entirety addresses all of the subcategories listed on the student evaluation form.

Study Protocol

Potential subjects were identified by reviewing the class list of senior medical students completing the EM clerkship for the 2002–2007 academic years. This information was available to the principal and secondary investigators in their roles as student clerkship directors for the EM clerkship. After the completion of the student clerkship and the assignment of a final grade, students were sent a cover letter explaining the study and data to be collected as well as a brief survey inquiring to the approximate percentage of the reading that was completed during the clerkship and whether they believed the readings to be a useful adjunct to their clinical performance during the clerkship experience. These surveys were returned by mail to the study investigators.

The investigators also provided the Associate Dean of the College of Medicine with a list of students from the 2002–2007 academic years who completed the elective senior EM clerkship. Demographic and prior medical school academic performance characteristics of the experimental and control groups were gathered from the College of Medicine for a baseline comparison of the groups. These data were returned to the investigators without identifiers and in aggregate form according to the academic year in which the clerkship was completed. Baseline comparison characteristics of the groups included the following: number of students completing the clerkship, number of students regarding English as their preferred language, number of under-represented minority members, gender, age at the time the clerkship was completed, number of Alpha Omega Alpha (AOA) honor society members selected during the M3 year of undergraduate medical training, number of members of the Emergency Medicine Interest Group (EMIG), and an overall comparison of academic achievement for the M1–M3 years of training based on a scaled student ranking (Z-scores) provided anonymously by the Associate Dean of the College of Medicine.

This study did not mandate an alteration of the physician interactions with or observation of the medical students during the clerkship. Each student’s clinical performance was evaluated based on criteria set forth on the medical student evaluation form. This evaluation form has been utilized since the 2002–2003 academic year continuously through the entire study period. The student’s grade for the clerkship was determined by averaging the scores given in each category of the evaluation form for each given shift. Each category was given equal weight. Although the score of a 50-item, written, multiple choice examination based on the information contained in the required reading is weighted as one-third of the student’s grade reported to the College of Medicine, this portion of the student’s grade was excluded for the purposes of this study’s comparison between the group with assigned reading and those completing the clerkship prior to the addition of the assigned reading and written examination. The grade distributions of the two groups were compared to determine whether a statistically significant difference in grade distribution exists between the 2002–2004 academic years group, before the initiation of required reading, and the 2004–2007 academic years group, after the initiation of the required reading.

Measurements

Data were collected by the study investigators and stored without identifiers in aggregate form according to the academic year in which the study subject completed the EM clerkship. The data that was collected represented information on study subject demographics, prior academic performance in medical school, and the clinical grade earned during the EM clerkship.

Data Analysis

Data were analyzed using a multinomial logistic regression model, chi-square test, and Fisher’s Exact test. The primary comparison was between students assigned textbook material and students not assigned textbook material. All of the previously mentioned covariates were included in the model to account for possible differences between the groups. For this multinomial design, sample size was calculated using the S-plus code from Frank Harrell, following the method of Whitehead.13 With an alpha level of 0.05, to have 80% power to detect an odds ratio of 2.3, 82 subjects in each group were required. In other words, the odds of having a grade of honors in the textbook group is 2.3 times that of the non-textbook group. This odds ratio could be obtained from increasing the percentage in the honors group by 20% and decreasing the pass and high pass groups both by 10%.

Additionally, Chi-square or Fisher’s Exact test were used to compare the study groups’ grades in each of the subcategories used to compile a given student’s overall grade for the clerkship. These subcategories were History and Physical Exam, Patient Presentation, Knowledge Base, Patient Management, Procedural Skills, and Effort/Reliability. When the p-value from the overall test was significant, three 2×2 tables were analyzed (for each pair of grade level). These p-values were adjusted using Bonferroni. In addition, the pass and high pass grades were combined and compared to the honors grades, and the high pass and honors grades were combined and compared to the pass grades. The statistical level of significance used in all analyses was 0.05.

RESULTS

The control and experimental groups each had 83 subjects and were similar in most baseline characteristics of interest (Table 1). While the control group performed better than the experimental group during the basic science training of medical school (years 1–2; p=0.01), the groups were similar in their performance during the first year of clinical training (year 3; p=0.41). In addition, the experimental group had statistically significant more members in the EMIG (p=0.0001) and more members who went on to match in an EM residency (p=0.0007). The group differences did not reach statistical significance in the areas of the number of AOA junior medical student members, the number of individuals without English as their first language, the number of ethnic minority members, gender differences, or median age at the time the EM clerkship was completed.

Table 1

Baseline Comparison of Groups

	‘02-’04 Control (Pre-Intervention)	‘04-’07 Experimental (Post-Intervention)	p-value
Mean MS1-MS2 Z-Score	0.1733	−0.2150	0.01
Mean MS3 Z-Score	0.1467	0.0257	0.41
# AOA Members	0 (0%)	0 (0%)	---
# without English as 1st Language	1 (1.2%)	0 (0%)	1.00
# Minority	10 (12.1%)	4 (4.8%)	0.09
# Gender
Male	60 (72.3%)	53 (63.9%)
Female	23 (27.7%)	30 (36.1%)	0.24
Median Age	26.0	26.0	0.97
# EMIG Member	11 (13.3%)	33 (39.8%)	0.0001
# EM Match	9 (10.8%)	27 (32.5%)	0.0007

MS, medical student; AOA, Alpha Omega Alpha Medical Honor Society; EMIG, Emergency Medicine Interest Group; EM, emergency medicine

The difference in the overall grade distributions between the control group and experimental group was not as large as expected (p=0.40); therefore, there was not enough power to detect statistical significance (Table 2). Statistical significance was not reached when the medical students receiving a grade of “Pass” and “High Pass” were combined and compared to the students receiving a grade of “Honors” (p=0.47) or when the medical students receiving a grade of “High Pass” and “Honors” were combined and compared to the students receiving a grade of “Pass” (p=0.19).

Table 2

Group Grade Comparisons

	‘02-’04 Control (Pre-Intervention)	‘04-’07 Experimental (Post-Intervention)	p-value
Overall Grades
Fail	0	0
Marginal	0	0
Pass	22 (27%)	15 (18%)
High Pass	43 (52%)	46 (55%)
Honors	18 (22%)	22 (27%)	0.40
Pass + High Pass			0.47
High Pass + Honors			0.19
Grades for History and Physical Exam Component
Fail	0	0
Marginal	0	0
Pass	17 (24%)	9 (11%)
High Pass	48 (67%)	54 (66%)
Honors	7 (10%)	19 (23%)	0.02
Pass vs. High Pass			0.29
Pass vs. Honors			0.02
High Pass vs. Honors			0.19
Pass + High Pass vs. Honors			0.03
High Pass + Honors vs. Pass			0.04
Grades for Procedural Skills Component
Fail	0	0
Marginal	0	0
Pass	19 (26%)	9 (11%)
High Pass	43 (60%)	50 (61%)
Honors	10 (14%)	23 (28%)	0.01
Pass vs. High Pass			0.13
Pass vs. Honors			0.01
High Pass vs. Honors			0.33
Pass + High Pass vs. Honors			0.03
High Pass + Honors vs. Pass			0.01

Grade distributions of “Honors,” “High Pass” and “Pass” for each categorical variable were also compared. Notably, those student members of the EMIG (p=0.0005) and those later matching to an EM residency during the same academic year they completed this EM clerkship (p<0.0001) were more likely to earn an overall grade of “Honors” for the clerkship. These results are represented graphically in Figure 1 and Figure 2.

Figure 1

Figure 2

The difference in the subcategory grade distributions between the control group and the experimental group reached statistical significance in the subcategories of History and Physical Exam (H&P) (p=0.02; refer to Table 2) and Procedural Skills (p=0.01; refer to Table 2). However, statistical significance was not reached in the other subcategories of Patient Presentation (p=0.10), Knowledge Base (p=0.49), Patient Management (p=0.08), or Effort/Reliability (p=0.07).

The survey inquiring about the volume of the assigned readings that were actually completed by the student and the student’s opinion regarding the clinical utility of the readings was returned by 26 of the 83 subjects in the experimental group (31%). Seven of the 26 respondents (27%) read between 81% and 100% of the assigned readings, six (23%) read between 61% and 80%, eight (31%) read between 41% and 60%, three (12%) read between 21% and 40%, and two (8%) read between none and 20%. All of the respondents stated that the assigned readings they completed were helpful in guiding the evaluation and management of patients that they were caring for while they worked clinically in the ED.

DISCUSSION

In a systematic review by Oxman, et al.,14 it appears that passive distribution of information is an ineffective means of impacting behavioral changes. However, active information distribution and utilization is moderately effective in changing health professional behavior and health outcome. More recently, Costa, van Rensburg, and Rushton15 found that interactive teaching styles are preferred by students and that knowledge retention is better for teaching orthopedic topics when compared to the traditional didactic lecture format. Our results support those of Oxman and Costa in that no difference was observed between those students with and without assigned topic-specific reading in an elective EM clerkship. Although the reading material was contained in a book small enough to be easily carried in the pocket of a lab coat and referred to during clinical shifts, not all the students used it in this manner. While the passive reading material presented clinical information to the readers, it did not include any interactive activities in which the readers could use the information in the readings for hypothetical problem-solving activities. It is possible that a difference in group performance might have been observed if the assigned readings also included problem-solving activities.

These data do demonstrate that those students participating in the EMIG and those students who later that same academic year matched in an EM residency do outperform their peers in terms of clinical performance as judged using a behavioral evaluation tool. This observation is likely a function of heightened attention to the subject matter and increased relevance to the student contemplating the specialty of EM as a career. Certainly, adult learners retain information better if they are attentive to the material, and the material is relevant to the individual learners. Despite the experimental group having more EMIG members and students later matching to an EM residency, a difference in clinical performance between the control and experimental groups was still not demonstrated in this study (Table 2).

The data also support that the experimental group who were supplied the text performed better in the subcategories of H&P and Procedural Skills (Table 2). The text specifically discusses an EM- based approach to both of these tasks that likely varies from the experience of the students on prior required rotations. The importance of brevity and efficiency in taking an initial H&P without sacrificing completeness is stressed in the text. This may be markedly different than the detailed H&P endorsed by other medical specialties, such as Internal Medicine and Pediatrics, to which the medical student would likely have had significantly more exposure up to this stage of their training when they would have been participating in an EM rotation. Additionally, the text has a section dealing with emergency procedures that students may be allowed to perform. Perhaps students having reviewed this section were better prepared for performing a given emergency procedure when offered the opportunity.

LIMITATIONS

One limitation of this study is that the student evaluators varied according to the specific shift being worked and the academic year in which the student participated in the EM clerkship. The attending physician complement was constant during the period of time the control group participated in the clerkship (2002–2004 academic years). However, during the clerkship participation time of the experimental group (2004–2007 academic years), four student evaluators (attending physicians) were added and three student evaluators were lost due to turnover of employment positions. Consequently, the investigators could not ensure an equal distribution of shifts of students working with a specific evaluator (attending physician). In addition, there is no way of accurately accounting for all student absences. However, each individual student was allowed only a single shift absence during the clerkship. Otherwise, the clerkship was not considered complete, and the student had to schedule “replacement shifts” to complete the clerkship and receive a grade, a condition that was met by all the students during the study period.

Another study limitation is that the clinical evaluation tool’s reliability and validity is unproven. Because of this, we cannot be certain that the data obtained through the evaluation form is reliable and valid. However, the evaluation form used in this study is similar to other institutional evaluation forms used to derive student grades for other clerkships, suggesting the form has face validity.

An accurate assessment of the volume of the assigned readings actually completed by all the subjects in the experimental group cannot be made without having had an investigator witness the reading for each study subject. The only available means of estimating this quantity is by self-report through the returned surveys, and only 26/83 (31%) of those forms were returned for inclusion in the study results. It may be more likely that those individuals returning the survey were also the individuals who were more likely to have completed the assigned readings. Consequently, it is possible that the 57 study subjects not returning the survey form (69%) might not have read any of the assigned readings for the clerkship.

A final study limitation is that because this study did not employ a randomized design there could be differences between the groups for which the model will not account. The performance evaluation could be relative to the year of the clerkship, so that even if students perform better in a given academic year, there might not be a difference in grade distribution. The textbook assignment and year of the clerkship are confounded so a significant difference could represent a year difference instead of an assigned reading difference.

CONCLUSIONS

The addition of uniform, topic-specific reading assignments to an elective EM senior medical student clerkship curriculum does not improve the overall clinical performance of those students as measured using a clinical behavioral evaluation tool. It does appear that EMIG members and those students who later match to an EM residency do perform at a higher level clinically during an EM clerkship when compared to their peers. These results and conclusions may not be generalized to EM senior medical student clerkships at other institutions with a dissimilar clinical experience or evaluation method.