A mixed methods study of student perceptions of using standardized patients for learning and evaluation.

BACKGROUND: Educators recognize the value of using standardized patients (SPs) when teaching and evaluating clinical skills in rehabilitation entry-to-practice education programs but have published little supporting evidence and have yet to evaluate programmatic SP use from a student perspective. This study explored occupational and physical therapy students' perceptions of SP use in their professional education.
METHODS: Recruiting current and graduated students, we conducted a two-phase mixed-methods sequential-explanatory study integrating data from a quantitative survey (phase 1) and qualitative focus groups with representative students (phase 2). Quantitative data were used to direct the second phase and informed selection of a purposive sample to participate in four focus groups (N=12).
RESULTS: The 24-item online survey obtained a 32% response rate (N=167). Mean ratings were high, but significant differences were found between the four subsections of Teaching, SP Experience, Feedback, and Evaluation (P=0.000). Secondary analyses revealed significant differences based on sex, program, and age. Qualitative analysis revealed that students found SP use especially helpful earlier in their program to bridge classroom teaching and clinical practice. Students in the occupational and physical therapy programs approached SP interactions differently in terms of the authenticity, personal investment, and value of SP feedback. Educator feedback was perceived as reflective of technical skill, and SP feedback reflective of therapeutic value, which students prioritized differently. Students identified a preferential continuum of options for learning and practicing skills, ranging from peers and instructors through SPs to actual patients.
CONCLUSION: SPs were perceived as most useful early on in the professional education program, serving to bolster self-confidence and prepare students for clinical fieldwork. Discipline-specific differences impact the perception of SP use and value. Educators need to be aware of pragmatic and contextual issues when using SPs for examination purposes, including repeated exposure to the same actor.

Background

Health care educators use a variety of pedagogical methods to help students develop their interpersonal, clinical assessment and treatment intervention skills.1 Educators have emphasized teaching and evaluating skill performance, in addition to cognitive knowledge, as written tests and multiple-choice questions can demonstrate low external validity in relation to clinical aptitude.2,3 Clinical skills can be taught and evaluated in a variety of ways including clinical education and use of a standardized patient (SP), also referred to as a simulated patient. An SP refers to “a healthy person who has been carefully trained to realistically portray a patient in a reproducible fashion”.4 These actors are provided with comprehensive training and reproduce appropriate physical symptoms, relevant history, and emotional responses consistent with a structured clinical scenario.5 SPs were first used in medical education by Howard S Barrows6 in 1965 and have gradually been introduced into the field of nursing7 and allied health professions such as social work, speech and language pathology, occupational therapy (OT), physical therapy (PT), and dietetics.4,5,8–10

SPs have proven valuable in both the teaching and evaluation of students. Barrows6 outlined several advantages of using the SP method, including the ease of access at any time and in any location. The SP method offers students a transition from classroom/didactic learning to patient-based clinical learning without unnecessary concern or risk related to their clinical abilities.6 While evaluation in a clinical setting provides a valid assessment of student ability, SP use has the advantage of demonstrating robust reliability in addition to high external validity.2,11 As healthy individuals, SPs may lack some degree of authenticity in clinical presentation, but are able to “present an illness or scenario in a standardized, unvarying manner”.7

Use of SPs across health professions

Historically, SPs have been primarily employed in the education of medical students, and the majority of the literature examining SP use for education and evaluation comes from the perspective of medical students and their educators. Despite a growing interest in SP use among the allied health professions,12 a gap in this literature is apparent.4,5 SPs play a large role in the curriculum for PT and OT students at the University of Manitoba. At the time of this study, the OT program was a 24-month master’s degree, and the PT program a 3-year bachelor’s degree. While there were no shared courses between the programs, there was considerable collaboration between faculty around teaching and evaluation strategies. SPs were used for teaching interviewing/interpersonal skills, clinical assessment skills, and provision of interventions, and to examine student knowledge and skill through objective structured clinical examinations (OSCE) in PT and practical examinations in OT. With increasing application of SP use in our professional programs, the authors felt it important to explore the effectiveness of this approach to ensure students receive a high quality, evidence-based education. SP-based programs are expensive to implement and maintain, and require considerable investment in training;13,14 evaluating effectiveness is critical to justify the expense, and the knowledge gained could enhance and further improve existing programs.

Rationale

Several factors contributed to the need for this study. Despite a growing trend in SP use among OT and PT professional education programs, there is little discipline-specific evidence reported in the literature. There is a great deal to be learned from differences between professional programs as well as discerning where and when SPs are most optimally used. Furthermore, much of the literature tends to focus on either the learning process or the evaluation of outcomes; a broader understanding of the SP experience in these areas collectively could provide considerable insights into sequencing and targeting SP use. In fact, to our knowledge, there is no literature exploring the programmatic use of SPs in professional education, nor how students globally perceive the use of SPs as a teaching and evaluation tool. Several discipline-specific issues are also of particular interest. First, over the past decade, nearly all Canadian OT and PT university programs have transitioned from an undergraduate to a graduate degree, and it is important to evaluate the comprehensiveness of these new programs. Second, as professions that advocate an adult learning approach, including self-directed and reflective learning strategies, it is appropriate that educators also reflect on teaching strategies and involve students in the evaluation of educational methodologies. Third, as health care disciplines that espouse a patient-centered approach to service delivery, educators should promote a student-centered perspective to learning. To fully capture student perspectives, we need to know how they experience aspects of learning and evaluation, including the use of SPs.

Purpose

The purpose of this study was to explore the following questions: what are past and current OT and PT students’ perceptions of SP use, and how does it influence their professional education?

Methods

A sequential-explanatory mixed-methods design15 was used to first capture a comprehensive perspective of students across programs through a quantitative survey (phase 1), then obtain an in-depth perspective of representative students through focus groups configured on the basis of initial quantitative analysis (phase 2), and finally interpret these findings through integration of the quantitative and qualitative data.

Phase 1: quantitative

An electronic (online) survey design was used to capture anonymous responses from current students and graduates between 2006 and 2009 from the University of Manitoba OT and PT programs. Of the total population within these parameters (N=524), we had contact information for, and sent personal invitations to, 432 potential participants. Current students (N=200) were invited via their university email account and public advertisement on campus. Students received an initial email as well as two follow-up emails, as multiple contacts have been demonstrated to improve response rates.16 Each email identified the purpose of the study and a link to the online survey. Previous graduates were approached through an alumni email list (N=242) as well as public posters and advertisements in provincial professional organization newsletters.

Survey

The study investigators developed the survey over several months, developing content based on a review of the literature and the pedagogical questions of the investigators. After pilot testing and revision of the survey, a total of 24 questions were configured in four content sections (see Supplementary material). Section one (“Standardized Patients/SP”) dealt with global issues of SP performance, student–SP interaction, and relevance to their professional education. Section two (“Teaching/Education”) related to SP effectiveness as a strategy for learning different types of skills. Section three (“Feedback”) explored the relevance, timing, and perceived benefit of feedback from SP interactions. Section four (“Examination”) covered perception of SP use as a method of evaluation. The questions were presented as statements, using a four-point Likert scale with response options of “strongly disagree” (1), “somewhat disagree” (2), “somewhat agree” (3), and “strongly agree” (4). Questions varied with respect to positive and negative wording (eg, “SPs remain in their role throughout the duration of the interaction” versus “SPs often try to trick students”). Three questions had a “not applicable” option; one related to the use of videotaping, which PT did not incorporate into their program, and two asked about multiple encounters with the same SP, which some students would not have experienced. Demographic data was also collected related to age, sex, program (OT/PT), graduation year, and level of education prior to entering the program. Respondents also indicated whether they would be willing to be contacted to participate in the subsequent qualitative phase.

Data analysis

Demographic data were summarized using mean and standard deviation (for continuous variables) and frequencies and proportions (for nominal variables). Individual survey question responses were weighted from 1 to 4; negatively worded questions were adjusted for reverse weighting. Since each of the four content sections was composed of 5–7 items, composite scores could be calculated for each of the four content sections (total weighted score/[number of questions – not applicable responses]).17 The primary analysis involved descriptive statistics related to overall student ratings. Mean scores and standard deviation were calculated for each content section.17,18 Secondary analysis explored potential differences between programs, sex, and graduation year, and any potential association between age and respondent rating. Sex was a potential variable of interest as rehabilitation programs in Canada have historically demonstrated an imbalance related to enrollment and employment, with OT being roughly 92% female19 and PT 78%.20 Graduation year was considered because we postulated that after graduation and a period of clinical practice, students might change their perspectives of SPs, compared with being in the midst of their education. We collapsed the data in 2-year increments by graduation year (current students =2010 and 2011; recent graduates =2008 and 2009; past graduates =2006 and 2007). To create consistency, and because PT students have limited exposure to SPs in their first year, only PT students in their second and third year of study were enrolled.

Student t-tests were used to compare mean scores between groups, using the independent variables of study program (OT/PT) and sex. Analysis of variance was used to compare scores based on graduation year. Regression using the Pearson statistic was used to identify any relationship between age and respondent ratings.18

Phase 2: qualitative

A series of focus groups were conducted to investigate the student perspectives in greater detail. Results of the quantitative data analysis informed the selection of participants. Correlation between survey scores and participant age was apparent, as well as significant differences in scores between study program (OT versus PT) and participant sex. Males account for only 10% of OT and 22% of PT students, precluding sex as a primary factor for group differentiation. Therefore, four focus groups were configured to contrast both program and age: older OT, young OT, older PT, and young PT. This stratification enabled us to consider responses for each program and age category across two focus groups. Only respondents who, on the quantitative survey, had provided consent to follow-up in the qualitative phase (N=94) were eligible. To optimize contrast, these respondents were ranked by age and, beginning with the oldest, email invitations were extended until a group of five was achieved. A comparable process was conducted, beginning with the youngest respondent in each program. A research assistant (RA) administered all invitations to ensure confidentiality of participants from the authors, who were faculty members. The focus groups were conducted at the university, but with the authors off site. Each group was co-facilitated by two RAs; one a current student and the second a recent OT program graduate, and both receiving training from the study authors. An interview guide was developed based on study-predicated issues and results of the quantitative phase. Each focus group interview was audio-recorded and transcribed verbatim. One RA transcribed all interviews, while the second RA audited correspondence between audio recordings and transcriptions.

The initial data analysis focused on transcripts from the young OT and PT focus groups. All authors reviewed these transcripts several times and independently open coded. Next, one author (EMG) transferred these open codes from each reviewer into a matrix. All authors met to examine and discuss similarities and differences until consensus was reached and redundancies removed. A second author (PFW) collapsed initial codes to create a preliminary framework with 12 categories. All authors again met to discuss the coding framework and, after reaching consensus, independently recoded the transcripts based on the framework, remaining open to new/emerging concepts. The authors reconvened and reviewed the transcripts, achieving consensus on categories. Finally, all authors used the coding framework to analyze the remaining two transcripts, with three additional categories emerging.

Results

Phase 1

Participants

Email invitations were sent to a total of 432 individuals from the total complement of current students and recent graduates (N=524). We had no way of knowing how many of these invitations were viewed or how many of the email accounts were dormant. A total of 179 individuals initiated the online survey; ten respondents did not answer any questions, and two respondents answered all the questions but did not provide demographic data, precluding their use in the analyses (see Figure 1).

Figure 1

Participant enrollment.

The 167 usable datasets represent a response rate of 32% of all potential participants and 39% of targeted individuals. Response rates were higher among OT (43%) than PT (21%) students (see Tables 1 and 2). The mean difference in age between programs was 1.24 years (95% confidence interval =[−0.09, 2.57]) and was not significant. The proportion of female respondents in each program was not significantly different (χ2=1.50, df=1, P=0.221) nor was the distribution of respondents by graduation year (χ2=0.525, df=2, P=0.769). The female response rate was slightly higher in OT than PT, although not significantly, and both were comparable to Canadian employment ratios for each profession.

Table 1

Participant demographics reporting mean ± SD or frequency (percentage of responses)

Criterion	OT	PT	Total
Respondents (N)	112 (67.1%)	55 (32.9%)	167 (100.0%)
Age (years)	26.4±4.3	25.1±3.7	26.0±4.1
Sex (female)	101 (90.2%)	46 (83.6%)	147 (88.0%)
Graduation year
2006 and 2007	23 (20.5%)	14 (25.5%)	37 (22.2%)
2008 and 2009	42 (37.5%)	19 (34.5%)	61 (36.5%)
2010 and 2011	47 (42.0%)	22 (40.0%)	69 (41.3%)

Abbreviations: OT, occupational therapy; PT, physical therapy; SD, standard deviation.

Table 2

Participant response rates by number (percentage of potential respondents)

Year of graduation	Potential respondents	Survey respondents
		OT	PT	Total
2006	60	11 (36.6%)	6 (20.0%)	17 (28.3%)
2007	80	12 (30.0%)	8 (20.0%)	20 (25.0%)
2008	100	24 (48.0%)	12 (24.0%)	36 (36.0%)
2009	94	18 (38.3%)	7 (14.9%)	25 (26.6%)
2010	95	26 (54.2%)	12 (25.5%)	38 (40.0%)
2011	95	21 (44.7%)	10 (20.8%)	31 (32.6%)
Totals	524	112 (42.7%)	55 (21.0%)	167 (31.9%)

Abbreviations: OT, occupational therapy; PT, physical therapy.

Primary analysis: overall perceptions of SP use

Participant ratings for each of the four content areas are summarized in the box plot in Figure 2, and mean scores are reported in Table 3. Median scores were slightly higher than means in all four areas, with a negative skewing due to the longer whiskers in the negative direction and several low outliers. For all respondents (N=167), there was a significant difference in mean rating between content areas (F=48.9, df=3, P=0.000); post hoc analysis identified differences between all categories except Feedback and Examination.

Figure 2

Box plot of participant survey ratings.

Table 3

Mean Scores ± SD for survey content areas by response group

Response group	N	Content area scores
		SP	Teaching	Feedback	Examination
All respondents	167	3.00±0.57a,b	3.34±0.41a,b	2.93±0.51a	2.66±0.55b
Graduation year
2006 and 2007	27	3.12±0.47	3.30±0.40	3.05±0.46	2.76±0.48
2008 and 2009	61	2.90±0.57	3.32±0.40	2.85±0.50	2.60±0.60
2010 and 2011	69	3.03±0.61	3.38±0.44	2.93±0.55	2.66±0.55
Sex
Female	147	3.03±0.52	3.38±0.37	2.98±0.45	2.69±0.53
Male	20	2.79±0.84	3.05±0.56	2.54±0.73	2.49±0.68
Program
OT	112	3.09±0.52	3.38±0.41	2.97±0.52	2.75±0.52
PT	55	2.83±0.63	3.25±0.41	2.84±0.50	2.50±0.58

Notes: Bold font = statistically significant difference (P<0.05); for all respondents, mean scores sharing the same letter superscript demonstrated a significant different from one another (Tukey’s HSD).

Abbreviations: HSD, honest significant difference; OT, occupational therapy; PT, physical therapy; SD, standard deviation; SP, standardized patients.

In the Examination section of the survey, 86% of respondents indicated they somewhat, or strongly agreed, SP-based examination was an effective way to evaluate their knowledge. In comparison with other evaluation formats, 69% agreed it was more effective than written exams, and 72% felt it was more effective than written assignments. However, 89% somewhat, or strongly agreed, that interacting with an SP in an exam was anxiety provoking, and 84% disagreed that anxiety improved their exam performance.

Secondary analysis: ratings by subgroup

A breakdown of survey rating by subgroups is also provided in Table 3. Graduation year was not related to any differences in rating. Females had significantly higher ratings than males in the Teaching (t=3.49; P=0.001) and Feedback (t=3.77; P=0.000) content areas. Respondents in the OT program had higher mean ratings for all four categories; this difference was significant for the SP (t=2.88; P=0.004) and Examination (t=2.78; P=0.006) sections. Student age was positively associated with ratings in the SP category (rp=0.284, adjusted r2=0.075, P=0.000).

Phase 2

While five individuals agreed to participate in each focus group, the number who ultimately attended was variable. When attendance conflicts were identified in advance, additional recruitment was pursued using the ranked list of survey respondents. Recruitment was constrained by the number of survey respondents consenting to follow-up and the point at which differentiating age categories approached convergence. By stratifying groups, we were able to analyze responses by program (OT N=7; PT N=5) and age (older N=7; young N=5). A summary of focus group participant characteristics is provided in Table 4.

Table 4

Focus group participant characteristics

Participant ID #	Grad year	Sex	Age	Previous education
Young PT
167	2010	Female	21	Some University
51	2010	Male	22	Some University
110	2008	Male	23	Some University
Young OT
144	2010	Female	22	Bachelor of Science
31	2011	Female	23	Bachelor of Arts
Older PT
23	2006	Female	32	Bachelor of Science
41	2010	Female	33	Bachelor of Arts
Older OT
57	2009	Female	27	Bachelor of Arts
152	2011	Female	28	Bachelor of Physical Education
68	2011	Male	29	Bachelor of Science
3	2006	Female	31	Bachelor of Arts
73	2006	Female	42	Bachelor of Arts

Abbreviations: Grad, graduation; OT, occupational therapy; PT, physical therapy.

Emergent themes

In exploring the qualitative data, a variety of themes were identified in relation to the four components of the initial survey and the quantitative findings.

The SP experience

Focus group participants were overwhelmingly positive about their educational experience with SPs, including overall quality and performance. They reported the acting to be of excellent quality, and a number of students reflected on how they became “lost in the moment:” “… we had one lab which was on Parkinson’s … we totally thought that they actually had Parkinson’s until afterwards … they just showed up [and] were normal” (young PT).

One variable aspect of the SP experience that became evident was the students’ willingness to “suspend disbelief” – that they were prepared to approach the encounter as if it were a genuine clinical interaction. For example, some participants reflected on very authentic experiences, such as a young OT student who said “it was really neat how that person can just play that role over and over again and just provide that same information but in different ways … they play the role really well … they open up differently for different people.” Conversely, others alluded to their reluctance or inability to move beyond the mechanics of the encounter. A young PT provides the following insight:

I think they’re a valuable assessment tool … but other than that, I think the whole method and for me personally, I learn a lot more through interacting with actual patients … more than with standardized patients … you kinda just go through the motions.

This perspective was more apparent among PT participants, who commented that they perceived the SP as a “means to an end” or tool to demonstrate competence to their instructor, rather than investing in the interactive experience.

Appreciation for SP interaction appeared to have a temporal aspect for some. There were instances in which participants indicated that as they moved through the latter part of their program, they lost interest in the SP encounters and no longer approached them with a mindset of simulating a clinical experience.

Use of SP for teaching and education

Participants identified value in using SPs to learn and practice clinical skills. In particular, interviewing skills were highlighted as being conducive to this methodology. Broadly speaking, participants agreed that learning from and practicing with SPs was preferable to didactic strategies and was far superior to practicing with peers/colleagues and instructors. In particular, when practicing with peers or instructors, students felt their “patient” already knew what to anticipate and had comparable clinical knowledge, making it difficult to interact in a genuine manner. Furthermore, peers were inclined to facilitate the success of the encounter or intervention as they had a preexisting collegial relationship: “you get awfully comfortable doing it with just your lab partner” (older PT). Instructors provided a more desirable portrayal, as a young OT articulates: “I thought it was … better in that they gave you direct feedback … but I felt more self-conscious of myself when I was doing it with a faculty member than whenever it was with a [SP].” Participants also noted that, despite appreciating the value of SPs in these clinical learning encounters, they preferred “real” patients and saw their clinical placements as a superior venue for learning.

Students noted that practicing with SPs was particularly valuable early in their professional education. Several benefits were identified, including the sense of safety knowing they would not hurt their patient if they made an error, and having the opportunity to stop an interaction for feedback/correction or have a “redo”, which were not generally options when working with real patients in a clinical context. A number of participants reflected back on how these SP learning encounters provided a boost to their confidence, specifically in preparation for the clinic. An older OT articulates: “I forgot it was a simulated client. I was just doing my thing and [thought] oh yeah this is just like being on fieldwork. I think that it actually builds confidence in those interactions as time goes by.”

Feedback related to SP interactions

Some participants felt that feedback provided by SPs reflected the experience of a “true” patient and placed high value on these responses, whereas instructors provided more accurate technical feedback on skill performance. Those in the OT program were more inclined to see the patient’s subjective experience as relevant to them: “… as much as the instructor is giving us instruction, but it doesn’t necessarily mean it’s the right one. What’s really valuable is what the patient feels in the end of it” (older OT). This type of feedback was perceived as contributing to greater confidence, particularly as a prelude to fieldwork. A young OT describes her reaction to feedback following interview training in this way:

[SP feedback] was more important than [the instructor’s] … like the teacher’s perspectives were more like … their own experiences but from the patient’s view that is still really important because I mean you want patients that keep coming back to you in real practice so you want to know how they’re feeling too.

Participants from the PT program reported few opportunities to obtain feedback from either instructors or SPs directly, whether in learning or examination situations, and were more equivocal in their desire for SP input. In situations where feedback was provided, participants indicated SP responses were more benign and students tended to focus on their instructor’s remarks, as illustrated by this exchange among young PTs:

Facilitator: Was the [SP] feedback helpful … did you reflect on it after?

Participant 1: For me no.

Participant 2: That’s an OT thing [group laughter]

Facilitator: So was instructor feedback helpful … during your interactions with SPs?

Participant 2: Yeah, I think that’s where we got most feedback from. The lab instructors coming around to observe, they obviously prepped the patients so they know what’s supposed to happen and they’re watching you.

Use of SP for evaluation purposes

The benefits to using SPs instead of actual patients were noted in all four focus groups, including issues of student security, patient safety, consistent portrayal, and pragmatics:

So even in an exam situation, you know going in that this person is acting, that you are not really going to hurt them if you mess up and that’s why I think it’s … that safe zone (older OT).

We couldn’t do this on a real patient because they would fatigue or it would hurt (young PT).

Despite the generally positive view of overall SP performance and belief that SP-related evaluation was preferable and more reflective of student proficiency than written evaluations, participants raised the issue of fairness with respect to evaluation situations. The quality of SP portrayal took on particular importance, where inconsistent performance might benefit or handicap a student. There was concern that an error on the part of an SP could impact a student’s performance and, ultimately, their examination grade. This created a particular sense of anxiety for students, as they perceived this to be outside of their control. One young PT expressed the viewpoint of classmates in this way: “… some others felt that it wasn’t consistent … like the way the actor portrays themselves, what you have to do in your OSCE station … but because of their acting it kinda makes it harder for you to get what your task is supposed to be.”

Student anxiety related not only to SP performance specifically but also the SP evaluation format generally. Participants identified a number of factors that increased stress, including location and configuration of the examination room, familiarity with equipment and supplies, previous encounters with the same SP, and having the examiner in the room. A number of participants suggested students would feel less anxious and better able to engage with the SP without their examiner present, even if they knew they were being watched from outside the room (eg, through a one-way mirror).

Another expressed concern was deciphering whether a questionable clinical manifestation during an exam was part of the rehearsed scenario (eg, a contrived anomaly the student was expected to pick up and address appropriately), the consequence of an SP error (eg, not complying with the prescribed scenario), or the SP’s inability to accurately portray a health condition (eg, hypertonicity). There was an underlying question of laying blame for the resulting student performance deficiency, whether fault lies with the student, SP, or even the evaluator. One young PT reflected on this dilemma when a perceived SP misstep occurred during an exam: “you’re not sure if this is part of it or whether you are supposed to continue doing what you are asked to do or you have to try and change your approach because of the way the actor presented.”

One issue many participants identified, was seeing the same SP in different contexts. For example, an SP might portray a patient in a learning situation in one course and then subsequently portray a different patient and scenario in an examination in another course. This created dissonance for some students; even if they were prepared to enter the situation authentically, seeing a familiar individual in a new role created a disconnect, and participants felt this inhibited optimal performance. These multiple encounters with the same SP in different contexts also seemed to compromise verisimilitude for students in both learning and exam situations.

Discussion

Students in these rehabilitation programs were generally quite satisfied with the use of SPs in their professional education. The ratings for all four content areas were encouraging, with an overall mean around 3. Median scores were higher in all categories, indicating that a large number of responses congregated between “agree” and “strongly agree”, while a small number of respondents provided ratings at the very low end of the scale, drawing the mean downwards. There may have been a ceiling effect due to the four-point range in the survey design, as the upper limits in two content areas extended to a score of 4. No corresponding floor effect was apparent; however, a few outliers were noted with scores between 1 and 2 and likely reflect a small number of students who had a very negative perception of their SP experience. During the focus groups, it became apparent that even a singular event could impact a student’s overall perception of SP use or performance. It is possible that the outlier scores were a response to a frustrating event (eg, a poor exam) rather than the student’s global experience.

There were some significant differences in ratings between content areas. Using SPs for learning and practicing clinical and communication skills appears to be the aspect most appreciated by students. Some participants were willing and able to “temporarily suspend disbelief” – that is, to act as if they were a therapist really seeing a client. Velde et al21 reported similar findings among OT students, who believed they were consulting therapists while working with SPs. Students in many disciplines have identified a lack of believability or authenticity when practicing with peers and educators, and view SP training as an effective learning strategy, particularly with communication skills.22,23 Between the two programs in our study, there appeared to be a stronger willingness among the OT students to invest personally in the SP experience, whereas PT students found it more contrived and approached SP encounters mechanically as a means to demonstrate competence to their instructors. Baptiste and Solomon12 also reported PT students, particularly early in their program, were quite mechanistic when approaching their SP encounters. Our program uses both actors and individuals with a disability, and the survey questions were not sufficiently granular to determine whether students had a stronger preference for one or the other.

In exploring the qualitative data, a continuum of student preferences emerged. Student colleagues are often used to role-play during initial exposure to clinical and communication skills because it is pragmatic and cost-effective; however, study respondents clearly identify this as a largely undesirable option because it lacks authenticity in clinical presentation and interaction, and a successful outcome is typically enabled by the “patient”. Similar issues of unrealistic portrayal and challenges taking peer-learning seriously have been identified in OT,24 nursing,25 and medical education.26 Having educators role-play was preferable because they took the role more seriously and were more demanding and critical of student performance, but students found it challenging to entirely disengage from the preexisting relationship. SPs were identified as more authentic, and students felt the SP reacted appropriately in response to their intervention, albeit within the confines of a prescriptive scenario. However, students still preferred practicing with an individual who genuinely had a health condition versus a “healthy” SP, and ultimately felt that practice in a clinical environment with “real” patients (ie, fieldwork) was the optimal method of learning (Figure 3). There is little in the literature exploring this spectrum of portrayal preferences, although a mixed-methods study did find students rated SP cases higher than role-play as a teaching method in an OT curriculum.21 Gallimore et al27 also reported pharmacy students found SPs more useful in their learning than peers, followed by instructors and staff members, although peers were identified as least believable and authentic.

Figure 3

Learning continuum: preferred scenarios for learning and practicing clinical skills.

Abbreviation: SP, standardized patient.

There was recognition among students that, from both a pragmatic and readiness perspective, they needed to progress through these learning strategies, suggesting a second learning continuum. Participants confirmed they found role-playing and SP practice as most beneficial early in their professional education, particularly prior to their intermediate and advanced fieldwork placements. The perceived value and benefit of SP interactions, particularly for learning and practicing skills, diminished as their exposure to, and experience with, actual patients in clinical placements increased. Velde et al21 identified a desire among OT students for inclusion of SPs earlier in their curriculum. Other authors have noted that practicing with an SP early on provided a safe and supportive context to build both skills and confidence, which students felt was good preparation for the clinical setting.28,29 In particular, these early “success experiences” provided a boost of confidence going into their basic and intermediate fieldwork.

In contrast to learning scenarios, rating of SP use in Examinations was the lowest among all categories. On the one hand, students appeared to appreciate and even prefer this style of examination as better reflecting their capacity to demonstrate acquired knowledge and skills. Stehle et al30 have also found a strong positive correlation between students’ evaluation of learning and use of standardized practical examinations. However, participants in our study reported the experience to be highly stressful – a factor that did not necessarily elicit their best performance. For most students in rehabilitation, the OSCE/practical exam is a very new and intimidating experience. It is understandable that students would be more apprehensive and uncomfortable with this style of evaluation. Admission to the OT/PT programs is restricted, and applicants must achieve high academic performance to successfully gain entrance. It is typical for many students to be “high achievers” who associate substantial self-worth with their academic grades; it may be that the unfamiliarity and anxiety of practical examination (and perhaps resulting lower grades) may be factors in students’ lower ratings with this component.

Related to examination anxiety and performance, the issues of blame and responsibility emerged in the qualitative analysis. Respondents expressed uncertainty about whether a mistake on an exam (and consequently a low score) was due to student error, SP error or poor portrayal, confounding contextual factors, or examiner issues. The high premium placed on evaluation grades may exacerbate this tendency to assign external responsibility for discrepancies, which in turn may mar the SP experience. It may be worthwhile exploring whether students perceive exams differently when administered as pass/fail rather than a graded format, as was the case in our program.

Students in the OT program rated all categories higher than those in PT, particularly in the SP Experience and Examination areas. Hale et al31 have reported fairly neutral ratings of SP usefulness in a diabetes teaching module among PT students. The OT discipline has a slightly higher ratio of females, who rated all categories higher than did males, but the proportion of male to female respondents was not significantly different among our survey respondents. Our OT and PT programs use SPs in similar ways and draw from the same pool of actors; however, at the time of this study there were some differences between the programs. The OT program used SPs more frequently for teaching purposes, including a communication skills block where they interview several SPs over a period of weeks. The sessions were video recorded, and students debriefed and reviewed their evaluation with their instructor using the video. Another variation between programs was the PT use of “bell-ringer” exams with multiple SP stations in an OSCE format, while the OT program employed an extended practical exam with the same SP, where students performed several interventions in succession. It may be that differences in use and format of SP interactions between the two programs accounted for some of the difference in students’ ratings.

SP feedback was perceived to be of substantial importance to students, particularly in the OT program. Students in our OT program had more opportunities than PT students for direct SP feedback (verbal and written) because of increased SP applications in teaching labs and structured written feedback on practical exams. More study is required to determine whether difference in student perception is a result of the manner in which the SPs are used and provide feedback, or whether this is a reflection of differences between students in the two programs. Interestingly, students in the OT program placed a higher value on the SP feedback than they did on instructor feedback. They appeared to derive a better sense of success from affirming SP comments and linked a positive SP experience with their own potential for success in a clinical setting, whereas instructor feedback was perceived as more discretionary and related to performance mechanics. Conversely, PT students tended to place higher value on their instructor’s input, and suggested even the limited SP feedback they received was a consequence of how their instructor had prepared the SP. Both disciplines highlighted the importance of mastering skills of clinical reasoning through SP interactions, but OT students tended to focus on developing these skills through clinician–patient relationships, whereas PT students focused on eliciting the appropriate information required to inform decision-making. This finding is not entirely surprising given the OT conceptualization of patients as equal partners,32 whose input and perception of outcomes33 is critical to the clinical decision-making process, and the emphasis placed on students developing patient–centeredness34 in the OT program.

Student age was associated with higher ratings in the SP experience section, suggesting that older students were more likely to take these interactions seriously and less inclined to be distracted by issues such as seeing the same SP in more than one role or suspicion that the SP was trying to “trick” them during the interaction. We might speculate that with increased life and perhaps vocational experience, older students are better able to see the value of simulated practice and the relevance to educational goals. While age was not associated with any of the other content area scores, post hoc analysis did reveal an association between increasing age and lower anxiety when interacting with an SP in exams. It may be that older students have less exam anxiety in general (whether it be in practical or written format) and their rating of SP use in evaluation is merely an artifact of a different outlook on examinations generally. Alternately, as individuals move further away from their education, they may look back retrospectively on SP use in evaluation, adjusting their ratings accordingly. However, the lack of any significant relationship between graduation year and respondent rating suggests this was not the case.

Lessons learned

Several issues pertinent to our professional programs came to light through the course of this study. First, there is variability in performance quality among actors in any SP pool. Given the high premium students place on accurate and error-free SP portrayal, educators are inclined to select and use a smaller subset of actors. While this tends to increase student (and instructor) satisfaction, it also increases the risk of recurrent student encounters with the same actor in different scenarios – a situation which students identify as problematic. This produces a “catch-22” where instructors have a limited pool of SPs and must choose between diversity (with the risk of performance issues) and consistency (with the risk of repeated exposure). One approach we used at a course level was to split classes into groups and rotate the SPs through the different groups in each successive exam. However, this approach requires considerable coordination and is particularly challenging to implement across an entire program of courses.

A second area of insight was the selective use of SPs. It became clear that SPs were particularly useful early on in a professional program, providing a safe and efficient method of learning and enhancing student self-confidence prior to clinical fieldwork. As students progress and develop greater competency, SP use had to be adapted accordingly. Later in the program, SP scenarios needed to be more complex or the SP portrayal needed to be more challenging for students to continue investing in and gaining from the experience. SP activities focus more on integrative skills and clinical reasoning as students advance through their program. Furthermore, not all learning scenarios lend themselves to SP use, and instructors need to be selective about when and where they employ SP strategies.

Finally, consistency, structure, and preparation were critical to the successful use of SPs for both teaching labs and examinations. Specific guidelines have been adopted to enhance the SP interaction for students. A trained backup SP is now standard practice for exams in case a primary SP is ill or absent, rather than back-filling with an instructor or faculty member. Students are provided with access to the evaluation venue, as well as consistent training equipment, prior to SP evaluations to reduce contextually related exam anxiety. SPs receive a structured training session and comprehensive script materials prior to training for all exams. Qualitative exam feedback to students is differentiated between evaluators (ie, student performance) and SPs (ie, patient experience) and integrated into the evaluation process.

Limitations

The findings from this study are reflective of respondents from the OT and PT programs at our university and cannot be generalized beyond this cohort. While we had a reasonable response rate to our survey and focus group invitations, there is always a risk of response bias. We found considerable synergy between the quantitative and qualitative findings, particularly in regard to the OT and PT programs; however, while we were able to consider responses across focus groups (ie, program/age), some individual focus groups were small in number, which may have limited the breadth of data.

Conclusion

OT and PT students reported strong satisfaction with the use of SPs in their professional education. Students in the two programs approached SP interactions somewhat differently, with OT students perceiving the interaction as more authentic and the SP feedback having greater value; this may reflect the increased application of SPs as a learning tool and intentional use of SP feedback in the OT program. Students identified a continuum of preferred options for learning and practicing skills: peers, instructors, SPs, and actual patients. The use of SPs is most optimal early on in the professional education program and can serve to bolster self-confidence and prepare students for clinical fieldwork. SP scenarios and portrayal requires increased complexity and challenge later in the program and should focus on integration and clinical reasoning skills. Educators need to be aware of pragmatic and contextual issues when using SPs for examination purposes, including repeated exposure to the same actor.

Supplementary material

Simulated Clients

At the School of Medical Rehabilitation, Simulated Clients are used for teaching students in both the occupational therapy and physical therapy programs.

The following statements relate to your experience with Simulated Clients in the OT or PT program.

Respond to each question by marking the box that most closely reflects your opinion regarding the statement.

Respond to each question by marking the box that most closely reflects your opinion regarding the statement. Use N/A only if the statement does not apply to your SMR experience.

Teaching/Education

Simulated Clients are used in various aspects of the education process. For example, occupational and physical therapy students practice skills related to interviewing, communication, clinical assessment and interventions.

The following statements will relate to your experience with Simulated Clients as a learning tool.

Respond to each question by marking the box that most closely reflects your opinion regarding the statement.

Respond to each question by marking the box that most closely reflects your opinion regarding the statement. Use N/A only if the statement does not apply to your SMR experience.

Feedback

Feedback is an important component of the learning process. Receiving feedback allows students the opportunity to gain insight into their performance, learn from their mistakes and gain confidence in their abilities.

The following statements relate to your perceptions/experiences with receiving feedback about your Simulated Client interactions.

Respond to each question by marking the box that most closely reflects your opinion regarding the statement.

Examinations

Simulated Clients are also used for practical examination and Objective Structured Clinical Exam (OSCE) purposes. Many different components are examined, including communication and physical assessment skills.

The following statements will relate to your perceptions/experiences with Simulated Clients in exam situations.

Respond to each question by marking the box that most closely reflects your opinion regarding the statement.

Participant demographics

Age:

Sex:

Male

Female

Program:

OT

PT

Year of graduation (or expected graduation year for students):

2006

2007

2008

2009

2010

2011

Previous (Non OT/PT) Education:

Diploma (please specify in space provided below)

Some University, but no degree completed

University Degree: B.A.

University Degree: B.Sc.

University Degree: Other (please specify in space provided below)

Masters (please specify in space provided below)

Previous Education Specifics (only respond if prompted in above question):

	Strongly disagree	Somewhat disagree	Somewhat agree	Strongly agree
1. Simulated Clients remain in their role throughout the duration of the interaction.	○	○	○	○
2. Interactions with Simulated Clients are difficult to take seriously.	○	○	○	○
3. Simulated Clients often try to ‘trick’ students.	○	○	○	○
4. A negative experience with one simulated client impacted my ability to effectively use other simulated client experiences.	○	○	○	○
5. Working with Simulated Clients prepared me for working with authentic clients in fieldwork/clinical practice.	○	○	○	○

	Strongly disagree	Somewhat disagree	Somewhat agree	Strongly agree	N/A
6. Seeing Simulated Clients outside of the class/exam context makes it difficult to view them as real clients.	○	○	○	○	○
7. Seeing the same actor play different Simulated Client roles/scenarios made the interactions less realistic.	○	○	○	○	○

	Strongly disagree	Somewhat disagree	Somewhat agree	Strongly agree
8. Practicing with Simulated Clients in the classroom is an effective method for developing communication/interviewing skills.	○	○	○	○
9. Practicing communication skills with Simulated Clients is a more effective method than practicing with classmates.	○	○	○	○
10. Practicing assessment/intervention skills with Simulated Clients is a more effective method than practicing with classmates.	○	○	○	○
11. Learning/practicing new skills with Simulated Clients during class is anxiety provoking.	○	○	○	○

	Strongly disagree	Somewhat disagree	Somewhat agree	Strongly agree	N/A
12. Video-taping student/Simulated Client interactions for review later is a valuable teaching method.	○	○	○	○	○

	Strongly disagree	Somewhat disagree	Somewhat agree	Strongly agree
13. Receiving feedback about my Simulated Client interaction from an instructor was an important component of the learning process.	○	○	○	○
14. Feedback about my interactions with Simulated Clients was provided in a timely manner.	○	○	○	○
15. When practicing with Simulated Clients in class, instructor feedback provided “in the moment” was more useful than feedback provided “after the fact”.	○	○	○	○
16. Feedback provided by the Simulated Client was helpful for my learning.	○	○	○	○
17. Feedback received from Simulated Clients is more valuable than feedback received from instructors.	○	○	○	○
18. Feedback received from Simulated Clients is more valuable than feedback received from classmates.	○	○	○	○

	Strongly disagree	Somewhat disagree	Somewhat agree	Strongly agree
19. Practical examinations/OSCEs using Simulated Clients are an effective way to examine my knowledge/ability.	○	○	○	○
20. Practical examinations/OSCEs with Simulated Clients are more effective than written exams in evaluating my knowledge/ability.	○	○	○	○
21. Practical examinations/OSCEs with Simulated Clients are more effective than written assignments in evaluating my knowledge/ability.	○	○	○	○
22. Interacting with Simulated Clients during an examination/OSCE is anxiety provoking.	○	○	○	○
23. Anxiety improves performance on practical examinations/OSCEs.	○	○	○	○
24. My performance on examinations/OSCEs with Simulated Clients would improve if the instructor was not present in the room.	○	○	○	○