Training and Validation of Standardized Patients for Assessing Communication and Counseling Skills of Pharmacy Students: A Pilot Study.

OBJECTIVE: The objective of this study is to describe the process of training valid simulated patients (SPs) for assessing communication and counseling skills of pharmacy students' performance.
METHODS: This is a cross-sectional and correlational study. Psychometric properties of checklist and SPs' portrayals and their filling of the checklist regarding assessing pharmacy students were assessed. Five SPs who were working in the simulated patient's pool were volunteered to take part in the project, which one of the SPs failed. Three scenarios, along with corresponding checklists, were developed based on the usual medications of different diseases consisting of asthma, respiratory infections, and osteoporosis. The SPs' role-play performance was video-recorded and rated independently by two experts according to an observational rating scale to assess validity. The role-play was repeated after 1 week with the same scenario and the same doctor, to assess test-retest reliability. The inter-rater agreement between SPs and experts was determined by calculating the intraclass correlation coefficient and kappa coefficient.
FINDINGS: The four eligible SPs were all women, with an average age of 37 years. The correlation between mean scores of raters and mean scores of SPs was 0.91 and 0.85, respectively. The Pearson's correlation between mean scores of raters with SPs was 0.75. The checklists' reliability, Cronbach's alpha, was calculated to be 0.72. The measured weighted Cohen's kappa for the ratings of by each SP, and the gold standard was between 0.53 and 0.57, indicating a moderate agreement. The inter-rater reliability kappa coefficient between raters was 0.75 (P = 0.01).
CONCLUSION: The authors have demonstrated the technique of using standardized patients to evaluate communication and counseling skills of pharmacy students. The findings indicated that trained SPs can be used as an effective tool to assess pharmacy students' communication and counseling skills.

INTRODUCTION

There is a universal consensus that pharmacists should be able to provide counseling to the patients, therefore, teaching such skills should be incorporated in pharmacy curriculum.[1234] The definition of communication skills as an important proficiency in pharmacy is the ability to communicate with patients, which includes; building trust, using effective verbal and nonverbal communication, considering patient beliefs and attitudes, providing relevant information with effective questioning skills, and giving therapeutic instructions and counseling.[56] Pharmacists with good communication and counseling skills can easily interact with patients, develop a patient-centered counseling and educate them about how to use the drugs.[7] In addition, effective communicating and counseling by pharmacists are the key to improve the use of medications by patients and to promote adherence to medication.[89]

Although these skills can be taught in the curriculum,[1011] they are not easily assessed by traditional methods. Effective methods for assessing communication and counseling skills include direct observation, simulated patients (SPs), objective structured video exam, surveys of patients’ experience, interviews, etc.[1213]

The use of SPs as a method for education and assessment of students’ competencies was introduced in the 1960s by Barrows; its use has increased in different domains of medical education.[14] SP is a person who has been trained to play her/his role in a standardized manner as an actual patient.[15]

Using SPs in communication skills and counseling in pharmacy still seems to be a newborn. However, studies have shown that SP is used for training and assessment purposes in recent year,[1617] but seldom used for evaluating the quality control of their performance as a valid assessment method.

According to Monaghan et al., using well-trained SPs, it would be possible to provide safe situations which they encounter with the actual patient, as well as feedback and reducing the variability of assessment among different students.[18] In addition, research on SPs has demonstrated that when SPs are used under standardized conditions, they will perform consistently and equivalent for all students.[19]

Hence, quality control of SPs is inevitable, through expanding SPs application in educational intervention in the field of pharmacy. In the current study, we focused on the techniques and outcomes of training and validation of SPs in pharmacy communication and counseling, as a part of a research project that assessed pharmacy students’ performance at Tehran University of Medical Sciences (TUMS). The aim of the current study was to assess psychometric characteristics of standardized patients’ assessments of pharmacy students’ performance at TUMS.

METHODS

This is a cross-sectional and correlational study to describe the process of creating valid SPs for assessing communication and counseling skills of pharmacy students’ performance. The study was approved by the Medical Education Research Center at TUMS (No. 133-16592). The following steps have been accomplished: (1) development of scenarios, (2) development of checklist and assessing its psychometric properties, (3) selection of SPs, (4) training of SPs, (5) video recording, (6) validation of SPs, and (7) analyzing the data.

In the present study, three scenarios and general template were developed based on the usual medications of different diseases consisting of asthma, respiratory infections, and osteoporosis. The content and face validity of the written scenarios were determined by consensus of the expert panel, including faculty members of the Medical Education and Clinical Pharmacy Departments at TUMS.

An extensive literature review was conducted to collect potentially relevant items for the checklist.[20-23] The content validity of the checklist was determined by three clinical pharmacist experts’ and two medical educationists. These items were categorized under the following headings: communicating and building a relationship, interviewing and collecting information, and counseling and delivering information. Twenty-one items were identified, and each item was to be answered on a Likert scale (0–2) where a score of 0 pointed out a poor performance and a score of 2 as an excellent performance. The correlation coefficient between SPs scores with experts’ scores was calculated to assess criterion validity. Reliability of the checklist was calculated by Cronbach's alpha, where each SP reviewed the recorded video of other SPs (3n) performance and filled out the checklist. The reason of using data of filling checklist based on the recorded video of the other SPs performance was to have sufficient sample size as a pilot study which assesses the internal consistency of the checklist.

Five SPs who were working in the simulated patient's pool at TUMS were volunteered to participate in the project, one of the SPs failed. Four females were selected. Their age range was between 22 and 53 years. They have been trained to represent patient performances realistically and to evaluate pharmacists’ communication and counseling skills. Overall, communication and counseling skills of 12 pharmacy students in their 4th year were evaluated by SPs to determine the accuracy of SPs’ ability in filling out the checklists.

The recruited SPs were trained by a pharmacist and a medical educationalist in three small group sessions, each session lasting 2 h. The training process was consisted: accurate portrayal of the patient, appropriate reactions to questions, precise observation and recall of the pharmacy student's performance, accurate completion of the checklist, and sufficient feedback to the student.[24]

During each session, SPs played their roles regarding their encounter with pharmacy students under the supervision of the trainers, and they rate SPs’ portrait by the use of the checklists. Then, each SP was received appropriate feedback by the trainers who assessed SPs’ portrayal.

In addition, SPs learned how to fill in the checklists. All the sessions were video-recorded. According to the available literature, the content (scenario, checklist, and rating scale) and process (SPs’ portrayal, reliability of their ability to fill out checklists, reliability of checklist) are the components of an SP instructions which should be considered when its validation was documented.[25]

The validity of SP’ portrayal means how well acts as a real patient or portrays correctly.[15] The SPs’ portrayals were rated by two experts. One week after the training course, the ability of the SPs in verbal and nonverbal communication were evaluated by using the previously validated observational 9-item rating scale.[25] The content validity of the rating scale was determined by two experts from the Medical Education and Clinical Pharmacy Departments. The SPs’ performance had to have better than 90% accuracy rate regarding fidelity of SPs’ role portrayal.

The encounter of each SP and pharmacy student was video-recorded. The performance of every four students was evaluated based on a similar scenario. In all, 12 encounters of SP and pharmacy student were recorded. Each encounter was rated individually by four SPs (48 ratings). Each encounter interval was around 10–15 min. Total time for test-retest recording estimated around 5 h.

For evaluating the inter-rater reliability of SPs’ ability to fill out the checklist, each SP played her role with three pharmacy students in the pharmacy skills laboratory. After each encounter, SPs rated the pharmacy students’ performance. In later session, recorded videos of SPs’ portraits were watched and rated by one medical educationist and one clinical pharmacist individually. The correlation coefficient between each SP who completed checklists and two experts’ assessments as a gold standard was evaluated.

To determine the reliability of SPs’ filling out the checklist under the same conditions, the test-retest approach was used. Initial filling out the checklists was done by SPs in the first encounter. Ten days later, SPs rated the pharmacy students’ performance for a second session. Ten days interval was considered in this study based on the results of studies which have shown there were no statistically significant differences in the test-retest reliability for 2 days or 2 weeks.[26] Furthermore, the second session with the same student was held considering previous scenario. The inter-rater reliability was evaluated by determining the correlation between scores.[27]

The inter-rater reliability was assessed by intraclass correlation coefficient and kappa coefficient. The intra-rater reliability of the SPs’ performance to complete the checklist was assessed by means of a Student's t-test analysis. Cronbach's alpha was calculated to assess internal consistency of checklist. The Pearson's correlation was computed to determine criterion validity.

RESULTS

The mean age of the SPs was 37 years old and all of them were women. The pharmacy students’ age range was 21–25 years. They were enrolled in their third or fourth year and 60% were women. The correlation between mean scores of raters and mean scores of SPs was 0.91 and 0.85, respectively. The Pearson correlation between mean scores of raters with SPs was 0.75. The checklists’ reliability, Cronbach's alpha, was calculated to be 0.72.

The results of inter-rater reliability using intra-class coefficient are presented in Table 1. The measured weighted Cohen's Kappa for the ratings of by each SP and the gold standard was between 0.53 and 0.57, indicating a moderate agreement [Table 2]. The inter-rater reliability kappa coefficient between raters was 0.75 (P = 0.01). When initial scores were compared to scores of retest session, there were no significant differences [Table 3]. The results of the check list showing the performance of pharmacy students’ communication skills and practice evaluated by simulated patient is presented in Table 4.

Table 1

The correlation coefficient between the mean scores of simulated patients and rater calculated by intraclass coefficient

Table 2

The correlation coefficient between the ability of simulated patients and the rater calculated by kappa

Table 3

Intra-rater reliability of the checklist completed by the simulated patients in test-retest approach

Table 4

Performance of pharmacy students’ communication skills evaluated by simulated patients

DISCUSSION

Prior studies have noted that the interaction between pharmacist and patient is remarkably dependent on the pharmacist’ ability of effective communication and counseling skills.[28] Therefore, improving pharmacist’ communication and counseling skills are considered by most organizations such as the Accreditation Council for Pharmacy Education.[3] As mentioned in the literature review, along with educating the communication and counseling skills in the pharmacy curriculum, there is a need to underpin communication and counseling assessment procedures.[1] The use of SP as a rigorous and robust method of measuring pharmacists’ competencies has received much attention in recent years.[29]

The quality of analysis should be assessed based on a valid interface between SPs and the pharmacy students. Hence, to achieve a suitable SP-based assessment, it is necessary to validate SP's performance. The validation of assessment depends on the raters’ accuracy; for this reason, the process and content of SPs’ training must be suitable to guarantee the quality of their performance.

The purpose of this investigation was to describe the procedures and results of training and validation of SPs in pharmacy communication and counseling skills at TUMS.

While previous studies showed that less systematic attention has been paid to validation process of SPs,[30] we have done several steps for it. In the present study, the SP's standardization process is to evaluate the consistency of the SPs’ performances, including the validity and reliability of their performance, and the checklist.

To check the validity, we obtained consensus among five experts about the key aspects that should be included in assessment tool. This issue was also emphasized by Swanson and van der Vleuten that standardized patient checklist errors can affect the student scores.[31] In this study, we reviewed, compiled, and standardized the checklist which was devoted much time and effort by the experts. In addition, our results indicated that the checklist had high internal consistency and reliability.

In this study, the inter-rater reliability of the SPs’ ability was tested by correlating between the SPs’ completing checklists and experts’ judgments which considered as a gold standard and was confirmed by a high association between them. In addition, our results showed a high correlation between the mean scores of examiners and SPs. This finding is in agreement with Zanten's findings which showed a strong correlation between SPs’ and raters’ scores.[32] In a similar study of general practitioners regarding the management of depression, Shirazi et al. demonstrated the strong coefficients.[2533] In contrast, McLaughlin et al. suggested that SP ratings are acceptable to students, but there was less agreement between SP-based assessment scores and experts’ scores.[3435] They concluded that experts, not SPs, are the only qualified persons who could judge the students’ performance. Nonetheless, the methods of SPs’ training, their standardization process and doing the research in a testing environment can affect the results of this study.[36]

Our findings indicated that the use of SP instead of experts is a good way for assessing communication skills of pharmacy. However, the generalizability of the results may be questionable because of using the small sample of students. Nevertheless, based on our findings, holding the training sessions for SPs, delivering feedback from their trainers, and developing a guideline for the assessment would be useful to increase the inter-rater reliability between SPs ratings.

The reliability of SPs’ ability to fill out the checklist under the same experimental conditions was determined by a paired t-test analysis. Our results demonstrated that there was no statistically significant difference between the scores of test-retest scores.

In line with our findings, another study has also presented well to very good SPs’ accuracy in completing checklist.[37] To increase the reliability of the SPs’ ability to fill out the checklist in this study, we trained SPs before they started to rate the pharmacy students and developed a guideline for the assessment.

Training and validation of SPs are fairly time-consuming, however, it is necessary to follow these steps to ensure the quality of the assessments. In addition, the number of SPs in our study was limited which might have affected the results, however, having a restricted budget was the important reason. In addition, all the SPs were women, however, we do not believe that the results will be changed dramatically if we recruited male SP.

Policy makers should note that when using SPs to assess communication and counseling skills, additional support to teach and validate their performance may be necessary. More studies may show the effects of using SPs in evaluating pharmacy students’ at large scale. The long-term effects of training in SPs’ performance in clinical settings are needed to augment this and other practice implications.

This study outlines how to train and validate the standardized patients to assess communication skills and counseling in pharmacy. The findings of this study indicated that a trained SP can act as a consistent rater for assessing pharmacists’ performances. It is necessary that policy makers consider SP as an important method for assessment of pharmacists in high stake exams to improve patient safety.

AUTHORS’ CONTRIBUTION

Mahboobeh Khabaz Mafinejad participated in the study design, conducted the analysis, and drafted and revised the manuscripts. Mansoor Rastegarpanah as a supervisor participated in the study design, supervised the data gathering process, and revised the manuscripts. Fereshteh Moosavi developed the study design and conducted the data gathering. Mandana Shirazi assessed the quality assurance of SPs and revised the analysis and manuscript.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.