Using a formative simulated patient exercise for curriculum evaluation.

BACKGROUND: It is not clear that teaching specific history taking, physical examination and patient teaching techniques to medical students results in durable behavioural changes. We used a quasi-experimental design that approximated a randomized double blinded trial to examine whether a Participatory Decision-Making (PDM) educational module taught in a clerkship improves performance on a Simulated Patient Exercise (SPE) in another clerkship, and how this is influenced by the time between training and assessment.
METHODS: Third year medical students in an internal medicine clerkship were assessed on their use of PDM skills in an SPE conducted in the second week of the clerkship. The rotational structure of the third year clerkships formed a pseudo-randomized design where students had 1) completed the family practice clerkship containing a training module on PDM skills approximately four weeks prior to the SPE, 2) completed the family medicine clerkship and the training module approximately 12 weeks prior to the SPE or 3) had not completed the family medicine clerkship and the PDM training module at the time they were assessed via the SPE.
RESULTS: Based on limited pilot data there were statistically significant differences between students who received PDM training approximately four weeks prior to the SPE and students who received training approximately 12 weeks prior to the SPE. Students who received training 12 weeks prior to the SPE performed better than those who received training four weeks prior to the SPE. In a second comparison students who received training four weeks prior to the SPE performed better than those who did not receive training but the differences narrowly missed statistical significance (P < 0.05).
CONCLUSION: This pilot study demonstrated the feasibility of a methodology for conducting rigorous curricular evaluations using natural experiments based on the structure of clinical rotations. In addition, it provided preliminary data suggesting targeted educational interventions can result in marked improvements in the clinical skills spontaneously exhibited by physician trainees in a setting different from which the skills were taught.

RCT Entities:   Population Interventions Outcomes

Background

Medical decision-making is often complex, requiring patients and their physicians to negotiate an acceptable treatment plan. Because of this, interest in physicians' use of participatory decision-making (PDM) has increased [1]. The use of PDM, its impact on patient satisfaction, and the effect on clinical outcomes has been studied in a number of clinical contexts including general medical and surgical care [2,3], diabetes self-management [4] and asthma [5,6].

Decision making within the patient/clinician relationship where the patient participates as an informed partner with their physician in jointly choosing a course of clinical care has been labelled and defined in a number of ways. These include participatory decision making [3], informed decision making [2] and shared decision making [1] to name a few. Over the last several decades, there has been a growing interest in this approach to clinical decision making [1]. While there is little evidence that PDM improves outcomes [2,4], the use of PDM has been advocated on both ethical [7] and educational grounds [8].

Given the interest in PDM and a general consensus that patients should be informed and actively involved in making decisions about their health care, PDM skills need to be incorporated into clinical training programs. Keefe et al [9] recently described their experience implementing a curricular module addressing PDM and clinical preventive services in a third year family practice clerkship at our institution.

There is little research on the extent students spontaneously use PDM skills in routine patient care activities remote from the educational context in which they were taught. To assess students' unprompted use of skills taught in Keefe et al's curriculum, we have developed a formative, PDM-based Simulated Patient Experience (SPE) administered as part of our third year internal medicine clerkship. The following report describes the study and presents data from its pilot implementation.

Methods

Subjects

Third year internal medicine clerkship students at Michigan State University (MSU) College of Human Medicine (CHM) were offered an opportunity to participate in the study. The pilot data presented in this report were based on students in the second and third rotation of the clerkship during the 2003–2004 academic year.

Intervention

The family practice clerkship at CHM implemented an educational module to enhance skills in common preventive services and teach students how to inform and involve patients concerning those services. Students are trained to use a variation of a model developed by Braddock and colleagues [2]. Braddock's model includes seven elements of informed decision making:

1. Discussion of the patient's role in decision making

2. Discussion of the clinical issue or nature of the decision

3. Discussion of the alternatives

4. Discussion of the pros and cons of the alternatives

5. Discussion of uncertainties associated with the decision

6. Assessment of the patient's understanding

7. Exploration of patient preference

Specific decisions are also categorized into "basic", "intermediate", and "complex".

PDM is presented and modelled by faculty, discussed in small groups, and practiced using web-based case simulations. Students must pass an evaluative SPE on shared decision-making and preventive services prior to completion of the clerkship.

Measure

During the third-year internal medicine clerkship, students participate in an all day skills workshop conducted during the second week of the eight-week clerkship. The SPE station was one of five small group sessions included in the workshop. Students rotate through the small group sessions and three whole group sessions over the course of the day. The SPE station was intended to provide feedback, and it was made clear to the students that their performance would not impact on their clerkship grade.

The SPE station is structured as an office appointment for a man in his early twenties with a first time seizure. Before meeting the SP, students are given a "patient file" providing the results of a history and physical examination. To ensure adequate knowledge on the part of the student, they are provided with a fact sheet about seizures. Students are given approximately five minutes to review the material. Each student then spent up to 20 minutes discussing treatment options and the implications of the seizure with the SP while being observed by a faculty member. During the last five minutes of the session, the faculty member along with the SP provided verbal feedback to the student. The faculty member also completed a rating form based on key skills taught in the family practice module and other desirable case-specific behaviors. The rating items are listed in Table 1. Performance was rated as "poor/not attempted," "adequate," or "excellent." There were three replications of the case that were necessary to allow all the students in the workshop to complete the station. The authors as well as other faculty rated the students. The faculty raters discussed the rating criteria, however, no formal training was given in how to rate the students. It was felt that the rating criteria were clear and no additional training was needed.

Table 1

Rating scale used by faculty in evaluating the students

	Item
1.	Appropriately greets patient
2.	Establishes the purpose of the encounter
3.	Clarifies roles of the patient and physician in decision making and gains the patient's permission to continue the discussion
4.	Presents the issues in terms the patient can comprehend
5.	Reviews available test results
6.	Discusses alternatives (treatment versus no treatment) and their pros and cons
7.	Discusses safety issues related to new seizures
8.	Counsels re: driving restriction under Michigan state law
9.	Explores patient's preferences
10.	Ensures patient understands the issues
11.	Responds appropriately to affect

Rating Scale 1 – Poor/Not attempted 2 – Adequate 3 – Excellent

Study design

At CHM, the three required primary care clerkships (pediatrics, family practice and internal medicine) are taken during the first three eight-week rotations of the third year of medical school. This rotational structure forms a natural experimental design. The order in which students rotate through the clerkships is arbitrary, and we believe, unlikely to bias the results of this study in any systematic way. For practical purposes, we believe it approximates random assignment.

During the second rotation of the academic year a portion of the students taking the internal medicine clerkship have completed the family practice clerkship and the PDM module during the first rotation, while the rest completed the pediatrics clerkship. During the third rotation, all students have completed the family practice clerkship; however, a portion completed it during the first rotation, approximately 12 weeks prior to the SPE, while the others completed it during the second rotation, or approximately four weeks prior to the exercise.

The rotational structure forms the study design shown in Table 2. The contrast in performance between students in cells 2 and 3 of the design (the students in the second rotation of the internal medicine clerkship) forms a pseudo-randomized trial of the impact of completing the family practice clerkship including the PDM training module as compared with completing the pediatrics clerkship without the PDM module on the students' use of PDM techniques in counseling a patient in the SPE. The contrast between cells 4 and 5 in the design (students completing the third rotation of the internal medicine clerkship) forms a parallel design to that between cells 2 and 3. The comparison, however, assesses the impact of completing the family practice clerkship and the PDM module approximately four weeks versus approximately 12 weeks prior to the SPE. Since the faculty members rating the students did not know a particular student's rotation schedule, and the students were not informed at the time of the SPE of the specifics of the study, the design is also essentially double-blinded.

Table 2

Study design formed by the clerkship rotational schedule*

First Rotation	Second Rotation	Third Rotation
1. Did not complete PDM Module	2. Did not Complete PDM Module	4. Completed the PDM module during the first rotation
	3. Completed the PDM module during the first rotation	5. Completed the PDM module during the second rotation

*The rotation (first, second, or third) refers to the rotation in which students completed the internal medicine clerkship. Cells 1–5 indicate whether or not and when the students had completed the family practice clerkship and the PDM module at the time they were in the internal medicine rotation.

Statistical analysis

Two scales were formed from the individual items on the rating sheet. The first was a total score or sum of all 11 items. A second "PDM" score was a sum of the items that specifically reflected the elements of the Braddock model; items 1–4, 6 and 9 from Table 1.

The students' performance was rated as "poor/not attempted," "adequate," or "excellent" on a three point scale with "excellent" rated as a "3". Given the small sample size of this pilot study, we chose to test for statistically significant differences between the groups using non-parametric tests. Both the Kruskal-Wallis test for difference in ranks and the "median test" for differences in medians were used to test for differences among the four groups. When differences were found, a post-hoc analysis was performed. The tests were repeated on subsets of the data to test for differences among pairs of the groups that addressed the key research questions, (cell 2 versus cell 3 and cell 4 versus cell 5 of Table 2) as suggested by Conover [10].

Results

Twenty-two students from the second rotation of the internal medicine clerkship and 25 students from the third rotation of the internal medicine clerkship agreed to participate in the study. Five of the students from the second rotation of internal medicine had completed the family medicine clerkship during the first rotation of the year and 17 had the pediatrics clerkship. Eighteen of the students from the third rotation of internal medicine had completed the family practice clerkship during their first rotation and seven had completed the family medicine clerkship during their second rotation.

Table 3 presents the number of observations, means, standard deviations and average ranks for the total of the rating items and the subset of the items directly addressing PDM skills for each of the four groups (cells 2 through 5 of Table 2).

Table 3

Student ratings by groups

	Total Score				PDM Score
Group*	N†	Mean	SD	M. Rank‡	N	Mean	SD	M. Rank
Cell 2	16	27.8	2.6	19.4	17	15.2	1.6	21.7
Cell 3	5	30.0	2.3	31.4	5	16.4	1.7	30.9
Cell 4	18	29.2	2.7	26.9	18	15.8	1.5	26.5
Cell 5	6	26.7	2.0	13.9	7	14.7	1.8	18.2

*Groups – Cells refer to cells in Table 2. e.g., Cell 2 – Internal medicine and SPE taken in the second rotation. Did not receive PDM training. Cell 3 – Internal medicine and SPE taken in the second rotation. Received PDM training four weeks prior to SPE. Cell 4 – Internal medicine and SPE taken in the third rotation. Received PDM training 12 weeks prior to the SPE. Cell 5 – Internal medicine and SPE taken in the third rotation. Received PDM training four weeks prior to the SPE. †Differences in the number of observations between total and the PDM score reflect missing ratings. ‡M. Rank – Mean rank of the observations in the group.

The differences among the groups were just statistically significant on the Kruskal-Wallis test (p = 0.05) and clearly statistically significant on the median test (p = 0.008) for the total score. Differences among the groups for the PDM items were not statistically significant (p < 0.05) for either statistical test.

The total score on the rating sheet of the students in the second rotation of the internal medicine clerkship who completed the family practice clerkship during the first rotation was compared with the students from the second rotation of the internal medicine clerkship who completed the pediatric clerkship during the first rotation (cell 2 versus cell 3) using both the Kruskal-Wallis and median test. Both tests were not statistically significant though the difference approached statistical significance for ranks (Kruskal-Wallis test) (p < 0.07).

The students in the third rotation of the internal medicine clerkship who completed the family practice clerkship during the first rotation were compared with those who completed the family medicine rotation during the second rotation (cell 4 versus cell 5) in terms of their total score on the rating sheet. Both the Kruskal-Wallis and median test were statistically significant (p < 0.044 and p < 0.016 respectively).

Discussion

Unfortunately, the students in two of CHM's six community campuses rotated through the primary care clerkships as a group, e.g. those who took family practice the first rotation all rotated through pediatrics the second rotation and internal medicine the third rotation, which resulted in an uneven distribution of students in the different cells of the design. This further reduced the power of the design that was already limited due to the modest number of students in the clerkship. The students in the second rotation of the internal medicine clerkship who completed the family practice clerkship and the PDM module performed substantially better than the students who had not completed the module. These results were expected though they failed to achieve statistical significance with the very small sample size available for conducting the inferential statistical tests.

We were somewhat surprised that the students from the third rotation of the internal medicine clerkship who completed the family practice clerkship and the PDM module during the first rotation performed substantially better than those who completed the family practice clerkship and the PDM module during the second rotation. One might expect students would be more likely to exhibit skills they were taught four weeks earlier as compared with students who where taught the same skills 12 weeks earlier. One possibility is that the students who took family practice during the first rotation used the PDM skills they were taught during the intervening pediatrics clerkship and this practice enhanced their performance on the SPE case over the performance of students who completed family practice clerkship and the PDM module during the second rotation and did not have the opportunity to practice the skills. It will be interesting to see if this finding replicates in the data that will be collected during the 2004–2005 academic year.

At this point, due to the limited amount of data, we have not formally examined the differences among the groups in terms of the students' performance on the individual rating items. With an additional year of data, these comparisons may well provide some interesting insights into the impact of the PDM module on specific aspects of how the students interact with a patient while working though a treatment decision.

Conclusions

We feel this approach to assessing the impact of educational interventions in clinical training has many potential uses and can provide valuable insights that would be difficult to achieve with more traditional evaluation designs. It also provides a means by which colleagues in different clinical disciplines can collaborate to assess the extent clinical skills taught in one clerkship are exhibited in another clerkship using a fairly rigorous quasi-experimental design.

While this in itself is a valuable and relatively unique contribution, we believe the approach also goes a step further. As noted by George Miller [11], there are qualitatively different levels of assessing clinical competency. He uses a metaphor of a pyramid where the demonstration of rote knowledge is at the bottom and the actual performance of physicians or physician trainees in real life settings is at the top. While this design does not directly measure what students will do in actual practice, assessing the extent patient counseling skills are used appropriately in a very different context weeks or months after they were taught in our view provides convincing evidence students are likely to use these skills in real clinical situations.

The rotational structure of the third year required clerkships at CHM is quite unique. In fact we know of no other medical school that conducts the required primary care clerkships in the first half of the third year and the required specialty clerkships in the second half of the third year. Having only three clerkships in the rotational structure provided this simplified and elegant design for conducting this study. The six clerkship rotational structure that is used in most medical schools in the USA creates a much more complex situation. We believe however that it is possible to develop a generalized quasi-experimental research design and analysis strategy for conducting this type of study in a traditional six clerkship rotational structure. We are in the process of working out the mathematics and logistical issues of how that might be done and hope to publish this work in the near future.

Competing interests

None declared.

Authors' contributions

DJS developed the research design and directed the project. He also performed the data collection and management, conducted the statistical analyses, rated students in the SPE and wrote a significant portion of the manuscript. HSL acted as the medical consultant for designing the SPE, wrote the SPE case and materials, rated students in the SPE, and wrote a significant portion of the manuscript. CWK helped develop and conduct the PDM training module, and reviewed and helped edit the manuscript. MET directs the family practice clerkship, helped develop and conduct the PDM training module, and reviewed and helped edit the manuscript. MMN helped develop and conduct the PDM training module, rated students in the SPE, and reviewed and helped edit the manuscript. All authors have read and approved the final manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here: