Comparing trained student peers versus paid actors as standardized patients for simulated patient prescription counseling.

Background: Simulation can be a useful tool for teaching and assessing clinical skills, but can also be costly and faculty-time intensive. It is defined as a technique to create an activity to portray a real experience for purpose of practicing or evaluating. Simulations can use standardized patients (SPs), which can be paid actors (PASPs), staff and faculty, manikins, volunteers, or students from higher level cohorts, also known as advanced class standardized patients (ACSPs). Objective: The objective of this study was to conduct a multifaceted analysis comparing ACSPs and PASPs, based on student performance in the assessment, student preference of SP type, and SP performance as an actor.
Methods: ACSPs and PASPs were used in a summative prescription counseling role play. For the evaluation, students counseled a SP about a new prescription medication and answered questions about taking an over-the-counter product with the new medication. The interaction was recorded and evaluated by faculty using a previously developed rubric. SP performance was evaluated by faculty using a separate rubric to determine how well the patient role was performed. A pre- and post-evaluation survey was completed by student pharmacists to gather student preferences about SPs and confidence in their counseling skills. Data were evaluated using a paired t-test.
Results: One hundred sixty-seven student pharmacists completed the summative prescription counseling evaluation. Student pharmacists performed well overall with minimal differences between SP types. Students preferred PASPs to role play the patient but felt that the actor type did not affect their performance. Conclusions: ACSPs performed the role of the SP well for a summative prescription counseling session without impacting student performance compared to PASPs and with reduced cost. However, students preferred PASPs, and PASPs were better at role playing the patient.

Introduction

Simulation is often utilized in the education of healthcare providers to reinforce classroom learning and offer students the opportunity to practice skills and experience extrinsic consequences in a risk-free setting. It has been defined as “a technique that creates a situation or environment to allow persons to experience a representation of a real event for the purpose of practice, learning, evaluation, testing or to gain understanding of systems or human actions.” The Accreditation Council for Pharmacy Education (ACPE) Standards 18.1 and 21.2 recognize the importance of simulation and recommend that colleges of pharmacy have a “sufficient number of faculty to effectively address the following programmatic needs...didactic and simulation” and that “the college or school's physical facilities include adequate access to educational simulation capabilities”.

While simulation has been a successful method of practicing clinical skills and interactions, simulation has also been explored as an assessment tool. Simulation-based assessments can be utilized with students from novices to professionals to test clinical skills and perform tasks. Simulations often utilize a standardized patient (SP), which is defined as an “individual who is trained to portray a real patient in order to simulate a set of symptoms or problems used for healthcare education, evaluation, and research”. SPs can be paid actors, staff and faculty, manikins, volunteers, or students from higher level cohorts.

Simulations used as assessment tools come in many forms. Many schools utilize Objective Structured Clinical Examinations (OSCEs) as a valid and reliable method to test clinical competence. OSCEs could include real patients, SPs, manikins, or a combination. However, the cost and time intensive nature of OSCEs have limited its use. A systematic review reported that a majority of studies used high-fidelity simulator (a manikin with full body functioning to mimic human body function at very high levels) SPs for assessment., The review found that simulation has been incorporated within many health professions educational programs, and can be effectively used for assessment; however, further research is need when used as the sole assessment modality. High fidelity simulations had high construct validity to differentiate between novice and expert but did not perform well for high stakes assessments due to reduced ability to reliably assess performance. Additionally, the review determined simulation could be an effective mode of assessment when used in combination with other methods, but more information was needed as a standalone assessment.

Role-play (RP) is a type of simulation activity where the learner is asked to perform actions in a pretend scenario. RPs range from simple to elaborate and can use a variety of individuals beyond the learner to portray a role like SP, peers and instructors. Utilization of RP with SPs could serve as methods of assessment for prescription counseling, because simulation settings could easily mimic an outpatient pharmacy interaction. A study showed that 77% of US pharmacy schools have utilized SPs. Ninety-one percent of students believed that SPs were effective. Other research has further suggested that student outcomes are improved with the use of SPs.

Cost has been a major consideration when adding SPs into the curriculum. One study with medical students recognized that the high cost for SP simulations was related to personnel costs and SP and administrative costs. Another study with student pharmacists estimated SP actor costs to be approximately $3500 for an assessment or $100.93/student incremental cost effectiveness ratio for passing the assessment., This has led to a new wave of research comparing peers to SPs within a RP. RPs have utilized students as actors in scenarios to minimize cost. Some studies have not seen student performance be affected by actor type: paid versus peers., However, Gillette and colleagues found that students who participated in the paid actor standardized patient (PASP) curriculum performed significantly better than the peer actors within RPs, with an incremental cost effectiveness ratio of $9 per student per point gained on a 30-point final assessment.

Another potential simulation method would be utilizing SPs for RP but instead of enlisting paid actors or classmates, students from an advanced class in the same program would act as the SP. Advanced class standardized patients (ACSPs) reduced the financial burden of PASPs, but questions have remained about their performance. To our knowledge, research comparing this technique to others has not been undertaken. The objective of this study was to conduct a multifaceted analysis comparing ACSPs and PASPs, based on student performance in the assessment, student preference of SP type, and SP performance as an actor.

Material and methods

The Washington State University Office of Research Assurance found the study protocol to be exempt from the need for approval by the Institutional Review Board (#15740).

Course description

The Washington State University (WSU) Doctor of Pharmacy four-year program is comprised of three didactic years and one year of experiential learning. Professional Communications is a required course in the first semester of the first year. The course introduces student pharmacists to basic communication skills including patient counseling. The course is organized as a one-hour tutorial and a corresponding two-hour laboratory session each week. The tutorial component typically uses a large group lecture format and the laboratory sessions incorporate small group learning with a mixture of RP and small group discussion. During the semester, student pharmacists utilize RP to practice prescription counseling, performing phone calls to providers, and practice asking questions (open vs closed ended questions and questions about difficult conversations). An example of a prescription counseling RP would be having a student pharmacist counsel another student who is pretending to be a patient about a new prescription for lisinopril to treat hypertension. The course is taught simultaneously on two campuses, one in Spokane, Washington, and the other in Yakima, Washington. Content was delivered live by a faculty member on each campus.

The Pharmacy Communications course utilizes teaching assistants (TAs) who are hired to assist faculty during the course and serve as ACSPs. Students are eligible to apply for the TA position if they are enrolled in the second, third, or fourth year and in good academic standing. TAs are selected based on curriculum vitaes and previous performance in the communications course. During the project period, four TAs were hired, three in Spokane and one in Yakima. Each TA worked between 5 and 10 h per week. TAs were trained to provide written and verbal feedback utilizing the counseling grading rubric. First, a video counseling example was shown to normalize feedback and expectations. Then, a course faculty member would observe the TA giving feedback to ensure feedback was consistent and constructive.

PASPs are also utilized during this course for the summative counseling assessment RP. For the class of 2020 cohort, six local paid actors were recruited through their respective campuses and paid at the campus standard rate of $20 per hour. Training for the PASPs included a training session by the SP program's coordinators at one campus and a training session with the course director on the other campus. In both cases, the training took place in the week before the activity sessions began and included an explanation of expectations. The PASPs were also provided a copy of the script by email at least one week prior to the activity. Questions and final instructions were given immediately prior to the activity.

Formative and summative counseling assessment RPs

During the Fall 2016 Pharmacy Communications course, students from the class of 2020 cohort completed two independent, formative counseling activities. For the formative counseling RP, students were assigned a medication one week before the activity and were given a copy of the grading rubric. The medication was selected from a list of medications the students were learning in a separate course. The formative RP began with the student counseling the evaluator who also played the role of the SP. The evaluator/SP was an ACSP, pharmacy resident, course faculty member, or laboratory assistant. The counseling RP was recorded and immediately replayed for the student and evaluator to watch together. The student was given feedback by the evaluator and asked to reflect on their performance.

Following the two formative counseling RPs, students completed a summative counseling RP held during week 13. Students were required to meet competency to receive a passing grade for the course. To prepare for the summative counseling RP, students were given two possible medications paired with five possible over the counter medications one week prior. Students were expected to counsel on either medication and answer questions regarding appropriateness of taking one of the over the counter medication with the new prescription. All medications were pre-assigned randomly.

For the summative counseling assessment RP, student pharmacists were pre-assigned either a PASP or ACSP. For eight student pharmacists, a laboratory assistant portrayed the patient because a PASP or ACSP was not available due to unexpected scheduling issues. Students' names were entered into an online randomizer which randomized in a 1:1 pattern of PASP to ACSP. Student pharmacists were not aware of the assignment. The students were given one to two minutes to prepare and had approximately five minutes to counsel. The summative counseling assessment RP was video recorded to be evaluated by a faculty member at a later time, but immediate feedback was provided by the SP.

Evaluation of student performance

The faculty evaluation of student performance during the summative counseling assessment RP utilized a rubric previously developed by faculty for use throughout the curriculum for patient counseling. The rubric evaluated student foundational counseling skills and Omnibus Budget Reconciliation Act of 1990 federal counseling requirements. The faculty were trained to use the rubric and completed a norming prior to use. The norming process consisted of the following: 1) evaluators met to discuss the grading rubric items and determine how to grade specific rubric items; 2) faculty independently watched a student video from a previous cohort and evaluated it using the rubric; and 3) faculty met again to discuss the results and to address discrepancies with grading the sample video. This norming process was used by faculty to ensure that grading is as equivalent as possible, regardless of who was evaluating the student.

To provide a double-check for students not meeting competency on the assessment, a second faculty evaluator would independently review and grade the video. Both evaluators then met to discuss the student's performance and determine whether competency was met. Student performance was deemed either “sufficient” or “needs enhancement” on rubric items. The student evaluation by faculty was completed within one week following the summative counseling assessment RP by video review. See Appendix A for a copy of the rubric used for this activity.

PASP and ACSP evaluation of student performance was completed using a rubric created by the faculty. The rubric contained 16 items in two domains: communication skills and professionalism. All evaluations were completed by the SP immediately after the student pharmacist exited the room. Instructions on how to use the rubric were given during the SP training. SPs rated the student performance as either “completed” or “did not complete” for each rubric item. See Appendix A for a copy of the rubric. The rubric was not used by faculty to assign a grade, but instead used to provide feedback to the student.

Evaluation of student preference for standardized patient type

In order to gather information about students' preferences for the type of SP, student pharmacists were surveyed before and after the summative counseling assessment RP. Two previously published tools were adapted into a survey instrument using a 5-point Likert scale to measure student perceptions of using SPs and to measure their confidence to effectively counsel a patient., The pre-assessment survey was distributed to students one week prior to the summative counseling activity and closed the morning the counseling activity. The post-assessment survey was distributed the day following the assessment and was closed after two weeks.

Evaluation of standardized patient performance

The project team also gathered information to determine how well the SP played the role of the patient. To evaluate the performance of the SPs, a tool was developed by the project team. It consisted of 22 items to identify how well the SP adhered to script, whether the SP prompted the students, the quality of the feedback provided to the student by the SP, and whether the SP maintained professionalism and created a realistic atmosphere. The project team faculty reviewed all of the videos and evaluated the SP performance after the assessment. See Appendix A for a copy of the tool used to evaluate the SP performance.

Collection of student demographic data

All students enrolled in the Fall 2016 Professional Communications course were included. A total of 166 students were enrolled in the course between the two campuses with 37 students on the Yakima campus and the remainder on the Spokane campus. Student information was collected including age and primary language.

Statistical analysis

Qualtrics Research Suite Software™ was used to administer the pre- and post-surveys. Examsoft®, an online learning assessment platform, was used to complete the faculty evaluation of student performance. Examsoft® evaluations were then downloaded into an Excel® spreadsheet and analyzed using the Chi-square Test. Paper evaluations were used by SPs to evaluate student performance and for faculty to evaluate SP performance. Both sets of paper evaluations were then coded and entered in an Excel® spreadsheet. Pre- and post-assessment survey data were evaluated with a paired t-test in Microsoft Excel®.

Results

One-hundred sixty-six (100%) student pharmacists completed the summative counseling RP during November 2016. The results of student performance on the counseling rubric organized by SP type are shown in Fig. 1. A statistically significant difference was found between ACSP and PASP for only two items: reviewing relevant monitoring parameters and providing pertinent information.

Fig. 1

Student performance on rubric items organized by standardized patient type.

A total of 149 (89.2%) students completed the pre- or post-assessment survey with 132 (79%) and 131 (78.4%) students completing the pre- and post-assessment surveys, respectively. The student respondent population median age was 23 years with a range of 19 to 41 years. Ninety (68%) student pharmacists reported English as their primary language. Likert scale survey results showing the percentage of students who reported they were extremely or very confident from the pre- and post-assessment surveys with p-values are shown in Fig. 2. Complete data for the pre-and post-assessment surveys are show in Appendix 2.

Fig. 2

Percentage of students who responded as extremely or very confident from the pre- and post-counseling assessment surveys.

The post-assessment survey contained two additional items that were not included in the pre-assessment survey. Responses to the statement “I prefer the following type of individual to be trained to portray a real patient actor”, are shown in Fig. 3.

Fig. 3

Student preference for standardized patient type.

Two of the students who selected the “other” category wrote in responses which included a preference for “someone who was not associated with pharmacy” and “anyone who was not a professor”. When asked on the post-assessment survey “Was your performance during the counseling exercise affected by your assigned SP type?” 95 students (74.8%) said “no”. Table 1 shows the faculty grading of the SP performance organized by SP type.

Table 1

Faculty evaluation of standardized patient performance on rubric items organized by standardized patient type.

Rubric item	ACSP (n = 73)	PASP (n = 85)
	Number of Correct Responses (%)		p-value
The SP knew the role of the patient without using notes.	60 (82%)	84 (99%)	0.0002*
The SP avoided prompting the “pharmacist” with answers or questions posed.	60 (82%)	66 (78%)	0.4785
The SP answered only the last question if multiple questions were posed together.	40 (55%)	63 (74%)	0.0110*
The SP answered yes/no questions simply with a yes/no answer without elaborating.	51 (70%)	49 (58%)	0.1122
The SP demonstrated appropriate acting skills.	68 (93%)	85 (100%)	0.0608
The SP created a comfortable/welcoming atmosphere for the “pharmacist”.	73 (100%)	84 (99%)	0.9214
The SP remained professional during the entire simulation.	73 (100%)	74 (87%)	0.0058*
The SP avoided using behaviors that distracted/disturbed the “pharmacist”.	72 (99%)	83 (98%)	0.652
The SP provided verbal feedback at the end of the session to the “pharmacist”.	68 (93%)	84 (99%)	0.0629
The SP provided feedback corresponding to the student's performance.	65 (89%)	84 (99%)	0.0082*

Abbreviations used: SP: standardized patient; ACSP: advanced class standardized patient; PASP: paid actor standardized patient; * denotes statistical significance.

Discussion

Simulation is a recognized modality for teaching learners; however, concerns have remained regarding impact on student performance during an assessment based on actor types. Our results demonstrate that variation in SP actor types did not significantly affect the student performance on a summative prescription counseling RP as evidenced by analysis of individual rubric items. Student performance results differed in only two of 47 areas described in Fig. 1: reviewing relevant monitoring parameters and providing pertinent information.

Overall students were confident both prior to and after the summative counseling RP with a trend to improved confidence in all areas. Two areas of significantly improved student confidence were in the students' ability to provide drug information and provide therapeutic rational for therapy. The increased confidence from pre- to post-assessment further supported overall comfort with the utilization of RP to assess counseling performance.

Simulation has been proven a good modality for assessment when done in an authentic and controlled setting. Authenticity could be affected by different actor types. While student perception data demonstrated a preference for a PASP (48%) over ACSP (14%), nearly 75% of student pharmacist did not feel that actor type significantly impacted performance. This preference for PASPs may have occurred, because PASPs often have little knowledge about the medications, leading to a sense of realism. Additionally, students may have known the ACSP potentially from previous experiences enhancing the difficulty to step into the RP or creating an authentic situation.

Use of student peers for RP has previously been criticized for decreasing realism. The results of this study show the PASPs were significantly at better than ACSPs at not requiring notes to remember the patient information, answering only the last question if the student asked multiple questions together, and providing feedback that corresponded to the students' actual performance, but surprisingly ACSPs were significantly better at maintaining professionalism throughout the activity. Project faculty intend to use this knowledge to provide extra training to ACSPs in the areas where there was a significant difference to enhance realism when ACSPs are used in the future.

Reliability of simulation has limited its use to during high stakes summative assessments. During the study SP actor performance was assessed and compared between actor types. Significant differences were seen in knowing the script and answering only the last question. Both of these areas are important for increased reliability when using RP in a summative assessment. However, neither actor type nor actor performance led to measurable differences in student performance.

Simulation has been demonstrated to be time intensive teaching and assessment modality. However, one of the benefits of utilizing SPs was their ability to provide feedback. SP feedback may reduce the number of items evaluated by faculty or provide a different perspective on the interaction beyond faculty grading. Anecdotally, faculty found student pharmacists to be receptive to PASP feedback and PASPs provided valuable feedback on communication delivery and student empathy guided by a rubric. PASP gave feedback that corresponded to student performance more than ACSP as seen in Table 1. Feedback quality was not assessed. Due to concerns of FERPA, feedback remained formative in nature for all SP types. Due to this PASPs provide a potential benefit over ACSPs regarding the ability to provide feedback and performance assessment.

There were several concerns that need to be considered when utilizing students as SPs for summative assessments. First, some faculty had concern about ACSPs disclosing student scores without written permission, which could be considered a violation of the Family Educational Rights and Privacy Act (FERPA). However, in this project that was not an issue since SP evaluation of the student performance was only used as feedback and scores were not included within course grades. Final grades for the counseling assessment RP were not seen by either SP type. Additionally, RP interactions with peers have previously been shown to create a sense of anxiety in the learner, which could negatively impact performance. However, the results of this project showed the actor type did not impact student performance on the assessment. Pressure to perform well or a sense of familiarity with the ACSP may lead to a lack of professionalism when a student served as the patient, but that was not observed in this case. While our team did not identify professionalism concerns, it remained an item for future consideration.

Some strengths of this project included the multifaceted study methodology, grader norming practices, and use of double checks for concerning performance. However, this study had several limitations that may limit transferability and interpretation of the results. The first limitation was that only a single year of data were collected at one institution with one cohort. This was tempered with 166 students from two campuses, but the outcomes and expectations of the students were the same. Analysis of the data was delayed due to changes in the faculty team and the COVID-19 pandemic. Additional years of more recent data and/or the addition of other institutions would increase generalizability. Another limitation was found in the training of the SPs, which differed between SP types and campuses. On the Yakima campus, PASPs were trained by the simulation program while the Spokane PASPs were trained by the course faculty. Additionally, ACSP were trained by the faculty at each location. While expectations of faculty were similar between locations, training styles could introduce variability within SPs and may have created differences in expectations or behaviors.

This project demonstrated the needs for continued research in some areas. One specific area for future focus is to explore the impact on actor types in relation to student confidence. Due to the anonymous nature of our surveys, researchers were not able to identify which students had ACSPs vs. PASPs, so we were unable to compare student confidence between actor types. Additional research is also needed to analyze the ability of various actor types to provide quality feedback. Since one of the challenges of running simulation-based assessments is the high demand on faculty time, having SPs who are able to reliably participate in the grading of student performance, even just on specific items, may enable implementation of more simulation-based assessments. Further research on these topics is needed.

Conclusions

Prescription counseling is a required element taught in all pharmacy curricula. Many methods of instruction and assessment have been used to practice and evaluate this skill; however, each method has benefits and limitations. Simulation and RP are well suited for this application. PASPs and ACSPs resulted in equivalent student performance for a summative counseling RP. PASPs performed better as actors than ACSPs and PASPs were preferred by student learners. Cost has remained a barrier for utilization SPs with many programs. Based on this study, use of trained ACSPs instead of PASPs may be an effective way to reduce costs associated with RP simulation assessments with minimal impact on student performance.

Declaration of Competing Interest

The authors have no relevant financial disclosures or conflict of interest.