Components of Mid-clerkship Feedback in a Neurology Clerkship and their Impact on Subsequent Student Performance.

Feedback during clinical rotations can be valuable in improving medical student education, but it is unclear what effect such feedback has on subsequent clinical performance and also which feedback topics are the most important in student growth and education. We compared medical student clinical performance before and after mid-clerkship feedback in a Neurology clerkship, with evaluators at the mid-clerkship and at the end blinded to the others' comments. We found that the most important areas holding back student clinical performance were communication, interpersonal interactions, and work ethic rather than textbook knowledge, or the ability to take a history, and do a physical and neurologic examination. Further, students who had concerning comments at the mid-clerkship feedback session usually continued to have the same problems after the feedback (in an admittedly short clerkship). The results suggest that more attention should be given to communication and other interpersonal skills and involvement in the clinical service during feedback sessions and that feedback continue over a much longer period than during a relatively brief Neurology clerkship alone.

Introduction

Feedback during a clerkship or other clinical assignment has long been considered valuable in medical student education [1]. For over a decade, such mid-clerkship feedback has been carried out in our medical center, as required in the US by the Liaison Committee on Medical Education: “each medical student is assessed and provided with formal formative feedback early enough during each required course or clerkship to allow sufficient time for remediation. Formal feedback occurs at least at the midpoint of the course or clerkship [2].”

The Neurology rotation or clerkship at our medical school is short (4 weeks total), and significant faculty time is devoted to rendering helpful feedback. Nevertheless, it is unclear what effect such communication has on student performance, at least during the remainder of the clerkship. In order to learn which factors were most important and to assess the effect of mid-clerkship feedback on subsequent performance, we compared the feedback for students who had the most concerning comments at that point with their final evaluations at the end of the clerkship.

Methods

A 4-week Neurology clerkship is required at all sites at Harvard Medical School. About one-third of each class takes the clerkship at Beth Israel Deaconess Medical Center (Boston, MA). Students rotated for 2 weeks each on one of two inpatient General Neurology or Stroke ward services or on a consultation service.

For the 2 years of the study, the Associate Clerkship Director (ACD) conducted feedback sessions at the midpoint of the clerkship for all students, after gathering information from at least three faculty members and residents who had worked with those students. Feedback was shared with students and their own impressions were discussed with the ACD. A standardized feedback form, including the comments assembled from the residents and faculty, was shared with the student and recommendations for improvement were offered. In order to avoid biasing faculty and residents who would work with students in the second half of the clerkship, the information from mid-clerkship feedback forms and sessions was not shared with the clerkship director or with other faculty members or residents.

At the end of the clerkship, the final evaluation was created, and a grade determined, by the clerkship director based on three components: 70% of the evaluation came from an assessment of students’ clinical work on the wards and consultation services, with three inputs: (1) checked “benchmarks” on history, communication, etc., (2) an overall numerical score for becoming a physician, relative to the average student, and (3) single phrase descriptors of clinical work quality (with a list of adjectives such as “excellent”… “fair”) by a forced choice. Each component was converted to a numerical score (each with a very similar mean and distribution) and averaged to form an overall clinical score. In each 2-week rotation, the student worked with one or two faculty members and three to five residents. Evaluations were collected from everyone who worked with the student for 1 week or more and collection was complete (ie, “return” was 100%), guaranteeing that each student had at least nine evaluations by the end of the clerkship, in an attempt to guard against excessive influence by “easy” or “hard” graders. Also, numerical scores considered significant “outliers” were moved toward the mean in order to avoid skewing of scores by any individual evaluator.

Another 15% of the final grade was based on the NBME “shelf” examination (which were not used as a cut off for any particular grade level). The final 15% was based on a bedside history and neurologic examination, with scoring and feedback, as observed by a faculty member in the last few days of the clerkship. To determine the final grade level, a grand average of these scores was computed (weighted: 70% overall clinical score; 15% shelf exam; 15% observed history and physical). This grand average was then compared to the distribution of such scores from the clerkship over the previous 12 years – which had remained stable. This grand average score (not the comments) determined the grade level. Following the guideline of the medical school at the time, students with grand average scores in the top 25-30% were awarded the grade “Honors with distinction,” the next ~60%, “Honors,” and the bottom 10-15%, “Pass.” The ACD was never informed of the final scores or grades. (Note: our medical school switched more recently to a “Satisfactory,” “Unsatisfactory” assessment system.)

Comments on the later evaluation forms were used to supply the Summative comments for the students’ final evaluation and to give advice for improvement in the Formative comments on the final evaluation.

Using 2 years of mid-clerkship feedback data, the ACD (blinded to the final grades) was asked to predict which students’ grades would fall in the lowest (“Pass”) category based on the comments from the mid-clerkship feedback forms – assuming that this would apply to 10 to 20% of students. The nature of any problems identified during the mid-clerkship session was tabulated, as were any problems noted in the Formative comments of the final evaluation by the clerkship director (who was blinded to the mid-clerkship data).

Chi-square analysis was used to determine the statistical significance of a single 2 x 2 table.

The Institutional Review Board of Beth Israel Deaconess Medical Center determined that this study was exempt from IRB approval as it was not considered engagement in human subject research.

Results

Over a 2-year period, 97 students did their Neurology clerkships at our medical center, were given mid-clerkship feedback, and received final evaluations, with grade levels and Summative and Formative comments. Table 1 gives examples of mid-clerkship comments shared with students, including both positive and concerning reports. Based on review of the mid-clerkship feedback, the ACD determined that there were 11 students whose mid-clerkship forms raised significant concerns about the quality of their clinical work and another nine with lesser concerns.

Table 1

Examples of Comments Discussed at Mid-clerkship Feedback Session

Area of evaluation	Positive comment	Concerning comment
Knowledge	“Great knowledge base, often reading about patients and bringing interesting diagnoses to add to the differential.”	“He clearly does not read at all and could never answer the most basic questions, including those we prepped him to expect the attending to ask him.”
Presentation	“Very good presentations; did a very good job presenting at the chairman’s conference.”	“Could use some help with her presentations. She has a lot of trouble giving us a chief complaint and focusing on the pertinent positives, and less on the minutiae.”
Note writing	“I was very impressed by her thorough and organized documentation of the multiple tests obtained on a complicated Neurology patient.”	“Notes were sometimes lacking key information and usually had to be significantly amended by the residents.”
Industry and involvement	“He is always the first to volunteer to spend extra time talking with patients, calling families, and explaining plans to patients.”	“She functions at the level of a 3rd year on a first rotation -- minus any interest. She was not proactive at all and not helpful to the team.”
Interpersonal skills	“Worked well with the team and other staff (updated nurses, etc.).”	“I found him to be very challenging to work with. His attitude comes across as over-confident and perhaps a bit entitled. I was surprised at the number of times he (wrongly) contradicted us during rounds or made unnecessary snide remarks. He was always asking for feedback, but when anything vaguely critical was said, he had a litany of excuses to explain away the concern.”
Need for supervision	“Self-motivated and efficient. Quickly adapts to new situations and is very helpful to the team.”	“One time, I asked him to evaluate a patient in the morning and went to clinic. The next morning, it turned out that he never bothered, and “was not sure” when I asked if he had seen the patient and said he “had to see her face first.”

The distribution of final grades, following the medical school guidelines at the time, is shown in Table 2. Of the 12 students (12.4%) who received a grade of “Pass,” seven were among those with significant concerns raised at the mid-clerkship session, and another two, lesser concerns. Three students who received a final “Pass” grade had had no comments of significant concern at mid-clerkship; all three were considered quiet or shy, and mid-clerkship comments were relatively sparse.

Table 2

Grade Distribution vs Concerns at Mid-clerkship

	Pass (lowest 12%)	Honor	Honors w Distinction (top 25-30%)
Concerning comments at mid-clerkship	7	4	0
Mildly concerning comments	2	7	0
No concerning comments	3	49	25
	12	60	25

Correspondingly, of the 11 students with significant concerns raised at the midpoint, seven received “Pass” grades (13% of the class). (By Chi-square test, the chance of these seven being among the 12 with “Pass” grades (assuming that grades were distributed randomly to those with concerning and unconcerning mid-clerkship reports) was well below 0.001.) The other four received an “Honors” grade (signifying work quality ranging from the 13th to the 75th percentile); three of these had final “grand average” scores in the bottom quarter for all students and none were in the top half of scores. Of the nine students with milder concerns raised at the midpoint, two received “Pass” grades; seven “Honors;” and none an “Honors with Distinction” (ie, performance in the top 30%).

For individual students receiving the Pass grade, the nature or type of concerns raised in the mid-clerkship feedback session and in the final evaluation’s Formative comments are detailed in Table 3. Concerns that appeared unchanged from the mid-clerkship feedback to the final evaluation are presented in bold font. In general, problems with communication (eg, putting together a patient “synthesis” for oral or written presentations), and attitudes or work ethic, eg, getting involved in clinical team activities (“involvement,” “industry”) were more common than problems with performing histories and physical exams. Problems with knowledge were more “in the field” than in command of pathophysiologic facts. Problems with interpersonal interactions were relatively uncommon, but the text of evaluations indicated that a few were severe.

Table 3

Comparison of Mid-clerkship and Final Formative Comments

Concerns at Mid-clerkship	Concerns in Final Formative Comments
With significant concerns at mid-clerkship:
1. Knowledge, IP, Supervis	Involv, Present
2. Involv, Present, Confidence	Involv, Present, Industry, Synth
3. IP, Present	IP (major), Present, Industry
4. Synth, Present, Involv, Supervis	Synth, Present, Involv, Industry
5. Industry, IP, Supervis	Industry, Involv, Present, Synth, Knowledge
6. Knowledge, Synth, Present, Industry, Involv, Supervis	Knowledge, Synth, Present
7. Knowledge, Synth, Industry, Involv	Knowledge, Synth, Industry, Involv, Present
Modest concerns at mid-clerkship:
8. Involv, Industry	Involv, Knowledge
9. Involv, Industry, Confidence, Supervis	Involv, Present, Knowledge
No significant concerns evident at mid-clerkship:
10. Knowledge	Knowledge, Involv, Industry, Present
11. Involv	Knowledge, Present
12. Present, Synth	Present, Industry, Knowledge

Knowledge: Medical knowledge (“in the field,” not by shelf examination)

IP: Interpersonal skills and behavior

Supervis: Need for more supervision

Involv: Involvement and participation in teamwork of the clinical services, including punctuality or inappropriate absences

Present: Presentation, either oral, on rounds, or in written notes (usually a matter of organization)

Industry: Work ethic or level of industry on the clinical services

Discussion

Over the 2 years of this study, a vigorous attempt to provide meaningful and helpful mid-clerkship feedback to students on a Neurology clerkship did not appear to effect a significant improvement in the performance of students, at least among those who were having difficulties at the midpoint. Students with the most concerning mid-clerkship feedback were generally amongst those with the lowest grades at the conclusion of the clerkship, and difficulties identified at mid-clerkship (most often communication, interpersonal skills, and level of involvement in the clinical service) very often persisted for the reminder of the clerkship.

There are, of course, several limitations to this study. First, in this study, “performance” meant the relative quality of clinical work within a cohort, not the quality of work by an individual student. (Data available at the midpoint were few enough that it was not possible to assign a single numerical score that could be compared to that at the end.) It is certainly possible that most students benefited from mid-clerkship feedback. Indeed, other parts of these evaluations, and those of other students, indicated that the large majority improved significantly in neurologic knowledge; in taking histories and doing neurologic examinations; and in synthesizing an impression of a patient’s illness and coming up with a plan for management – although that growth cannot necessarily be attributed directly to the feedback. Nevertheless, students having more problems at the midpoint tended to remain in the same relative position. Necessarily, the entries in Table 3 focus on the problems rather than on the achievements. It is also difficult to demonstrate improvement after the midpoint in a very short clerkship.

Another important limitation is the difficulty of assigning a numerical score to a student’s clinical work in an objective and fair way. Still, by gathering at least nine evaluations for each student from physicians who worked with them for at least a week, and by modifying “outlier” scores, the resultant averages (for the final evaluation) are more likely to give a fair picture of the student’s performance. Also, the distribution of scores remained stable over a 15-year period, including the time of this study. Finally, we believe that relative assessments are more likely to be meaningful in a system where nearly all students receive high scores in any assessment.

An important methodologic limitation is that the work of students before the mid-clerkship feedback session contributed to the final evaluation, dampening the influence of improvement or change following the feedback (although the mid-clerkship feedback information and the final evaluations were obtained on two different forms, collected at two separate times). This was impossible to avoid as the final evaluations assessed performance over the entire clerkship, rather than for the last two weeks alone. Nevertheless, the observed history and physical exercise and the shelf examination score were determined at the end of the clerkship, such that two thirds of the final “grand average” score was determined after the midpoint.

It is hard to determine the value of any clerkship feedback in a quantitative way, and it has been explored in relatively few publications. Students and faculty appear to agree on what qualities of feedback are important [3], and several papers have recommended ways of standardizing and improving the process [4], including proposals for more frequent feedback during the clerkship [5] and for getting more student input into the process [6]. Few papers, however, focus on the content of the feedback or its effect on subsequent clinical activities or performance. In one Surgery clerkship, structured feedback led to formulation of a learning plan for the remainder of the clerkship [7]. In that study, four of 154 students failed the clerkship, but it was unclear whether the feedback and learning plan helped to keep that number low. In a Family Medicine clerkship, mid-clerkship feedback included review of patient log entries, and 53% of students logged different types of encounters after the feedback [8]. The relationship between logging practices and overall performance, however, is not clear, and at least one study showed that more voluminous log entries may even have a negative correlation with student performance [9].

Perhaps the finding of greatest interest from this study was that it was more often communication, other interpersonal skills, and work habits, rather than factual knowledge or learning basic clinical skills that were the primary explanation for less-good performances – at least when evaluators were asked, “How good a physician is this student becoming?” Further, students with professionalism and work ethic problems did not appear to change much after the feedback sessions.

It appears worth pointing out that the reports about interpersonal interactions and work habits were not obtained in the numerical or checkbox scoring of performance, but rather in submitted comments (as in Table 1), with a guarantee of confidentiality to evaluators (and also with editing by the ACD and clerkship director to make comments as kind and helpful to students as possible). It is unclear that this information would be available on website-entered scores of reaching milestone levels or benchmarked achievements.

This study does not question the value of feedback to students. Rather, it indicates that in a short Neurology clerkship, a well-intended and carefully implemented mid-clerkship feedback session did not appear to have a significant positive influence on later student performance in the most problematic areas. It also offers data to suggest that “people skills,” including communication, interpersonal relations, and work ethic are crucial in determining the quality of a student’s clinical work. While problems with medical knowledge “in the field” (rather than on the shelf exam) and with patient presentations may be remediable during the clerkship, those “people skill” problems appear much more difficult to redirect.

It might be better to focus mid-clerkship feedback on identifying the relatively few students who appear at risk of doing poorly [4] and offering more help and redirection in the areas of interpersonal interactions, including communication, and work ethic (or involvement), not only at the midpoint but perhaps more frequently and more thoroughly at the end of the clerkship, with “feed-forward” for later clinical rotations. Fortunately, at our medical school, there is an attempt to advise and counsel students longitudinally over the clinical years, rather than just in a single short clerkship.