Standardized patient outcomes trial (SPOT) in neurology.

BACKGROUND: The neurologic examination is a challenging component of the physical examination for medical students. In response, primarily based on expert consensus, medical schools have supplemented their curricula with standardized patient (SP) sessions that are focused on the neurologic examination. Hypothesis-driven quantitative data are needed to justify the further use of this resource-intensive educational modality, specifically regarding whether using SPs to teach the neurological examination effects a long-term benefit on the application of neurological examination skills.
METHODS: This study is a cross-sectional analysis of prospectively collected data from medical students at Weill Cornell Medical College. The control group (n=129) received the standard curriculum. The intervention group (n=58) received the standard curriculum and an additional SP session focused on the neurologic examination during the second year of medical school. Student performance on the neurologic examination was assessed in the control and intervention groups via an OSCE administered during the fourth year of medical school. A Neurologic Physical Exam (NPE) score of 0.0 to 6.0 was calculated for each student based on a neurologic examination checklist completed by the SPs during the OSCE. Composite NPE scores in the control and intervention groups were compared with the unpaired t-test.
RESULTS: In the fourth year OSCE, composite NPE scores in the intervention group (3.5±1.1) were statistically significantly greater than those in the control group (2.2±1.1) (p<0.0001).
CONCLUSIONS: SP sessions are an effective tool for teaching the neurologic examination. We determined that a single, structured SP session conducted as an adjunct to our traditional lectures and small groups is associated with a statistically significant improvement in student performance measured 2 years after the session.

RCT Entities:   Population Interventions Outcomes

Neurologic conditions account for a large proportion of the global burden of medical illness and are a leading contributor to hospital admissions (1). To be effective physicians in any area of clinical practice, medical students must become proficient in the performance of the neurologic examination. Despite its importance, it is a skill that is being lost with the decline in bedside teaching and neurology training at medical schools (2, 3). Any curricular change to counter this trend requires a significant investment in faculty participation and financial resources. An evidence-based approach to use the most effective teaching methods is needed.

The neurologic examination is traditionally taught in small group and lecture format. Standardized patients (SPs) are a modality widely used for teaching and assessing clinical skills (4, 5). Medical schools have also been implementing SP sessions dedicated specifically to the neurologic examination based on the opinion of authorities, such as the pioneering educator, Howard S. Barrow; the Consortium of Neurology Clerkship Directors; and the Undergraduate Education Subcommittee of the American Academy of Neurology (1, 6). Seventy-five percent of US neurology clerkship directors also report that their medical schools have a clinical skills laboratory, and 68% would use this laboratory to teach the neurological examination. Additionally, they report that 88% of their medical schools have a third or fourth year OSCE to assess student clinical skills (7). Despite the expanding use of this modality in teaching and assessing the neurological examination, the majority of the evidence supporting the use of SP sessions is derived from other areas of medicine (8–11). There exists some limited published data on the short-term effectiveness of an SP session as a tool for enhancing medical student competence with regard to a specific portion of the neurologic examination (12). To our knowledge, no published data exist on the long-term effectiveness of an SP session as a tool for enhancing performance on any portion of the neurological examination or on the complete neurological examination.

We hypothesized that the addition of an SP session dedicated to providing structured practice and feedback on the neurologic examination during the second year of a medical school curriculum would be associated with an improvement in the performance of the neurologic examination as measured by a fourth year objective-structured clinical exercise (OSCE).

Methods

Study design

The Standardized Patient Outcomes Trial (SPOT) in neurology is a cross-sectional analysis of prospectively collected data from the graduating classes of 2008, 2009, and 2010 at Weill Cornell Medical College (WCMC). The neurologic examination was taught to three consecutive classes in small group and lecture format by the neurology faculty at WCMC as a non-graded component of ‘Brain and Mind,’ a second year preclinical course that integrates neuroscience, psychopathology, clinical neurology, and neuroanatomy. Students in the class of 2010 took part in an additional SP session focused solely on the neurologic examination at the WCMC Margaret and Ian Smith Clinical Skills Center during Brain and Mind. No other pertinent changes were made to the curriculum, allowing the class of 2008 and 2009 to serve as a control group to the class that received the intervention, the class of 2010. The course directors for the Brain and Mind course did not change over the study period, nor was there a change to the pedagogical methods in the course overall.

The SP session was designed to provide structured practice and feedback on the neurologic examination. The students assigned to the intervention were told that this session was part of their formal course curriculum and that they would be assessed by the SP based on a checklist, but that it was a non-graded exercise. Prior to the encounter, they were provided with the checklist that covers the components of a standard neurologic examination as agreed upon by the course leadership. This checklist was peer-reviewed by faculty in the WCMC Department of Neurology and Neuroscience and included the examination of mental status, cranial nerves, motor function, sensation, cerebellar function, reflexes, and gait. All SPs who participated in this session were trained using this checklist by the same neurologist and clinical skills center staff.

During the session, the students assigned to the intervention were presented with an SP and instructed to perform a complete neurologic examination. Once the examination was complete, the students received 10 min of individualized feedback from the SP regarding completion of the checklist items. The students then received 15 min of immediate group feedback from the neurologist who observed the students during the SP encounters.

The control and intervention groups’ ability to perform a neurologic examination was compared by analyzing the results of an OSCE administered during the fourth year of medical school at the Clinical Skills Center. Students in both groups were informed that although this OSCE was part of their formal curriculum, their performance would not be graded since this session was primarily designed for feedback purposes. During the OSCE, students rotated through 10 case stations. At each station, students were allotted 15 min to elicit a focused history and perform a focused physical examination on the SPs.

For each OSCE case, members of the faculty developed a scripted medical history and physical examination for the SP, as well as a checklist that outlines relevant history and physical examination items for the case. The checklist is completed by the SP after each student encounter. Faculty members train the SPs by reviewing the script and checklist of the cases and by role-playing; these training sessions are typically 2–3 h in duration. One of the 10 cases on the OSCE was specifically designed to assess the students’ ability to perform the neurologic examination. The SP at that station presented a scripted history of a transient ischemic attack (hemiparesis and aphasia that resolved prior to being brought to the hospital) that was expected to trigger the performance of a neurologic examination. Following the encounter, the SP scored the student's performance using a 20 question checklist on the physical examination. This scoring methodology has been extensively studied and validated (13–16). There were no meaningful changes in the script or checklist used by the SP to evaluate the intervention and control groups. The same OSCE case script and checklist were used for all three classes and a consistent, limited pool of SPs were trained by the same faculty member and staff. The SPs were blinded to both the intervention and study.

Based on the OSCE checklist data, a composite Neurologic Physical Exam (NPE) score was developed that took into account the cranial nerve, motor strength, sensory, reflex, coordination, and gait examination. Each of these components is worth 1 point. The checklist contains 10 items that assess the cranial nerve exam, 2 items that assess the reflex exam, and 1 item for each remaining category. These checklist items were weighted to generate the 1 point per component for a total possible 6 points (Table 1). This score was designed prior to evaluation of the data by a neurologist and educators from the Margaret and Ian Smith Clinical Skills Center.

Table 1

Calculation of NPE score

Checklist item on OSCE		Points
		Yes	No
Q1	Washed hands or used sterilizer before and after the exam	1	0
Q2	Examined me from my right	1	0
Q3	Maintained my sense of modesty and comfort throughout the exam	1	0
Q4	Listened to the pulses in my neck on both sides	1	0
Q5	Checked my vision by asking me to read an eye chart (II)	1	0
Q6	Checked my pupils by shining a light in my eyes (II, III)	1	0
Q7	Checked my eye movements (III, IV, VI)	1	0
Q8	Checked my hearing (VIII)	1	0
Q9	Asked me to clench my teeth (V motor)	1	0
Q10	Asked if I can feel a cotton swab on my forehead/cheeks/jaws while my eyes were closed (V sensation)	1	0
Q11	Asked me to raise my eyebrows/close my eyes tightly/smile showing all of my teeth (VII)	1	0
Q12	Asked me to shrug my shoulders while pressing down on them or asked me to turn my head to each side against his/her hand (XII)	1	0
Q13	Asked me to say ‘ahh’ (IX, X)	1	0
Q14	Asked me to stick out my tongue and move it from side to side (XII)	1	0
Q15	Checked the muscle strength in my arms/hands and legs/feet on both sides (Motor function)	1	0
Q16	When my eyes are closed, asked if I could feel them touching my arms/hands and legs/feet on both sides (Sensation)	1	0
Q17	Checked my reflexes in my arms and legs on both sides (Reflexes)	1	0
Q18	Checked my response to having the bottom of my foot stroked (Babinski)	1	0
Q19	Asked me to touch one of their fingers and then my nose on both sides, or asked me to move each of my heels down the opposite shin, or asked me to touch my thigh (or other hand) alternately with my palm and the back of my hand (Coordination)	1	0
Q20	Asked me to walk across the room (Gait)	1	0
Q# (1, 2, 3…) = points received for checklist item
Cranial Nerve Score (CNS) = (Q5 + Q6 + Q7 + Q8 + Q9 + Q10 + Q11 + Q12 + Q13 + Q14)/10
Motor Strength Score (MFS) = Q15
Sensation Score (SS) = Q16
Reflex Score (RS) = (Q17 + Q18)/2
Coordination Score (CS) = Q19
Gait Score (GS) = Q20
Neurologic Physical Exam (NPE) Score = CNS + MFS + SS +RS + CS + GS

Participants

Within the class of 2010, 58 students participated continuously in the standard 4-year schedule from matriculation to graduation; in the classes of 2008 and 2009, 129 students followed the standard 4-year schedule. The control and intervention groups were divided into two subgroups for further analysis: a subgroup that had not taken the neurology clerkship at the time of the OSCE and a subgroup that had started or completed the neurology clerkship at the time of the OSCE. Within the class of 2010, 23 students were excluded from the intervention group because they had taken an academic leave of absence and completed the curriculum in 5 years. These students did not receive the intervention because they completed their preclinical years with the class of 2009. Because they completed their final year during the 2009–2010 school year, their performance on the OSCE was scored by the same SPs during the same sessions as the intervention group and, thus, served as an ‘OSCE control’ group.

Statistical analysis

The primary analysis was the comparison of the composite NPE score in the intervention group versus the control group. Secondary analyses included stratified analysis according to timing of the neurology clinical clerkship and analysis of composite NPE scores in the OSCE control group.

The unpaired t-test was used to compare composite NPE scores. Assuming a two-sided alpha of 0.05 and a standard deviation of 1.1 points based on data from the classes of 2008 and 2009, with 129 students in the control group and 63 anticipated in the intervention group, we expected to have 80% power to detect a difference of 0.64 points on the composite NPE score in the primary analysis. Statistical analysis was performed using Excel 2007 (Microsoft Corporation).

Standard protocol approvals

The study received approval from the Institutional Review Board of WCMC.

Results

The intervention and control groups have normally distributed NPE scores (Fig. 1). The intervention group has a mean score of 3.5 and the control group has a mean score of 2.2 (Table 2). By the unpaired t-test, this constitutes a statistically significant difference (3.5 vs. 2.2, t=7.9, p<0.0001) (Table 2).

Fig. 1

Histogram of NPE scores for the control and intervention groups.

Table 2

Mean NPE scores in each group and comparisons via a two-tailed unpaired t-test

Data set	N	Mean	Standard deviation
Intervention	58	3.5	1.1
Control	129	2.2	1.1
OSCE control	23	2.7	1.2
Comparison	p-value
Intervention vs. control	<0.0001
Intervention vs. OSCE control	0.004
Control vs. OSCE control	0.03

The intervention group's mean score of 3.5 is significantly higher than the OSCE control group's mean score of 2.7 (3.5 vs. 2.2, t=2.9, p=0.004). This difference achieved significance even though the OSCE control and intervention groups were evaluated by the same SPs during the same OSCE sessions (Table 2, Fig. 2). The difference between the OSCE control and control group was significant (2.7 vs. 2.2, t=2.2, p=0.03) even though these two groups received the same standard preclinical curriculum, which did not include the intervention.

Fig. 2

Histogram of NPE scores for the control and OSCE control groups.

On subgroup analysis, the students who had not yet taken the neurology clerkship at the time of the OSCE had a lower mean NPE score than the students who had started or completed the clerkship by the time of the OSCE. The difference in mean scores was small (Table 3). In the control group, the mean NPE score was significantly higher in students who had started or completed the neurology clinical clerkship prior to the OSCE than those who had not (2.5 vs. 1.9, t=3.0, p=0.004). In the intervention group, the mean NPE score was also higher in students who had started or completed the neurology clinical clerkship prior to the OSCE than those who did not, but this difference did not achieve statistical significance (3.8 vs. 3.4, t=1.5, p=0.1) (Table 3).

Table 3

Mean NPE scores stratified by clerkship timing

	N	Mean	Standard deviation
OSCE before clerkship
Intervention	37	3.4	1.0
Control	74	1.9	1.0
OSCE after clerkship
Intervention	21	3.8	1.1
Control	55	2.5	1.2
	p-value
Intervention vs. control
OSCE before clerkship	<0.0001
OSCE after clerkship	<0.0001
OSCE before clerkship vs. OSCE after clerkship
Intervention	0.1
Control	0.004

The intervention group outperformed the control group on the neurologic examination in the subgroup that took the OSCE before the clerkship (3.4 vs. 1.9, t=7.2, p<0.0001) as well as in the subgroup that took the OSCE after the clerkship (3.8 vs. 2.5, t=4.5, p<0.0001), demonstrating a difference between the intervention and control groups regardless of neurology clerkship timing relative to the OSCE (Table 3).

Discussion

The students who received the intervention performed significantly better on the neurologic examination than the control group. The SP session may have been effective at improving performance on the neurologic examination because it reinforced knowledge acquired from the small group and lecture sessions during Brain and Mind. Participants reviewed the examination using the peer-reviewed checklist that outlined the components of a standard neurologic examination prior to the encounter and received immediate individualized feedback from both the SP on completion of checklist items in addition to group feedback from a faculty member based on observation of the SP encounter.

The students in the intervention group also outperformed the OSCE control group. The OSCE control group was a subset of the class of 2010 that did not receive the intervention because they followed a 5-year schedule from matriculation to graduation. The OSCE control group performed significantly worse even though they were scored by the same blinded SPs during the same OSCE session. The OSCEs have been shown to retain their validity when SPs score students according to a checklist, and this scoring methodology is used by the NBME for credentialing physicians (5).

Of interest, the mean NPE score of the OSCE control group was significantly better than the control group's mean NPE score. The reason for the difference in performance between the two groups is not entirely clear. One possibility is that the OSCE control group is unique from other students in that it is a self-selected group of individuals who have elected to take 1 year off from medical school to focus in more depth on research or other academic disciplines.

Besides differences in scoring, another potential confounding variable is the neurology clerkship. At WCMC, medical students are required to take a 4-week clerkship in neurology during their third or fourth year of medical school. After they are taught the neurologic examination as second year students, the neurology clerkship is their main opportunity to practice and receive feedback on their examination performance. Since medical students at WCMC are able to take the neurology clerkship at any time during their third or fourth year, timing of the clerkship relative to the OSCE could potentially influence the result.

Within the subgroup that took the OSCE before the clerkship, a significant difference was detected between the intervention and control group. In a similar manner, a significant difference was detected between the intervention and control groups within the subgroup that took the OSCE after the clerkship. Hence, the difference in performance between the control and intervention groups cannot be attributed to the clerkship (Table 2). The actual impact of the clerkship on student performance is relatively small. The subgroup that participated in the OSCE after the clerkship within both the intervention and control groups had slightly higher average scores than the subgroup that participated in the OSCE before the clerkship. This difference achieved statistical significance in the control group but not the intervention group for two possible reasons. The clerkship may have resulted in a greater improvement in the control group because they started from a lower baseline. Alternatively, the difference may be attributed to power since the intervention group has a smaller sample size. In either case, the improvement that results from the neurology clerkship is small compared to the improvement that resulted from the SP session. The benefits of the SP session remain significant after controlling for the effects of the neurology clerkship.

This study is limited by the single-center design. It is not known if the use of the SPs to teach the neurological examination would be effective at other medical schools. A multicenter study would be required to determine whether the results can be generalized. The study is also limited by the relatively small sample size of the intervention group, although there was enough power to detect a significant difference. Another limitation of the study is the varied range of student exposure to neurology at the time of the OSCE. This was partially accounted for in the results by analyzing the effect of the neurology clerkship specifically, but certainly students have variable exposure to patients with neurologic illnesses in other clerkships such as medicine, pediatrics, and primary care. Finally, this study is limited by the use of an OSCE itself to evaluate neurological examination skills. The OSCE is a more controlled setting, and student performance of the neurological examination on a real patient in an actual clinical setting could potentially differ.

In conclusion, students that received the SP session neurological examination session in the second year of medical school demonstrated superior performance of the neurologic examination in a simulated patient care setting during the fourth year of medical school. This difference, which was demonstrated to be statistically significant 2 years after the intervention, cannot be attributed to a difference in clerkship timing between the two groups, and is unlikely to represent a change in grading patterns by the SPs. The SP sessions can be effective as a tool for teaching the neurologic examination, an important and challenging component of the physical examination. Additional research is needed to document the impact of the SP sessions on neurologic examination skills in an actual clinical setting. Further research is also needed to determine its effectiveness at teaching specialized physical examination techniques within neurology and in other areas of medicine.