Evaluating the impact of a pre-rotation workshop on student preparation for clinical advanced pharmacy practice experiences.

OBJECTIVES: This pilot study was designed to evaluate the impact of a pre-rotation workshop (PRW) on pharmacy students' clinical skills and preparation for clinical Advanced Pharmacy Practice Experiences (APPE) involving direct patient care.
METHODS: Randomized controlled trial of an educational intervention with Institutional Review Board approval. PRW activities designed to simulate rotation activities around five competencies, patient charts, medication histories, SOAP notes, patient presentations, and professionalism. Endpoints were evaluated using clinical rotation preceptors' evaluation of performance and students' performance on objective structured clinical exams (OSCE).
RESULTS: Eight fourth-year students and eight GPA matched controls (20% of the total class) were selected to voluntarily participate. The PRW demonstrated a positive impact on students' clinical skills and preparation for rotations by improving OSCE performance. However, no significant differences were found between groups when comparing preceptor evaluations of skills on rotations. These results are limited by the small sample size, potential OSCE "test-wiseness" effects, lack of OSCE evaluator blinding to study groups, potential case specificity effects due to the limited number of cases used on the OSCE and possible lack of sensitivity of the rotation evaluation tool to capture true differences among the experimental and control group participants.
CONCLUSION: The PRW was successful at advancing students' clinical skills and preparation for rotations and may be considered as a tool to help bridge didactic to clinical experiences in the Pharm.D. curriculum.

RCT Entities:   Population Interventions Outcomes

INTRODUCTION

The Advanced Pharmacy Practice Experiences (APPE) comprise at least twenty-five percent of the Doctor of Pharmacy (Pharm.D.) curriculum and can be considered the capstone component that assures students have achieved the professional competencies necessary to practice pharmacy.1 Furthermore, it is stated that these experiences must integrate, apply, reinforce, and advance the knowledge, skills, attitudes, and values developed through the other components of the curriculum. These experiences are required to provide direct interaction with patients and collaboration with other health care professionals and to include experiences in both acute care general medicine and ambulatory care settings. These clinical APPEs are designed to allow students to refine their professional competencies by allowing them to broaden their experience in patient care responsibilities, including conducting patient interviews and education, communicating and documenting recommendations for medication therapy, and participating in interdisciplinary team rounds. As such, colleges must teach clinical skills throughout the curriculum in order to bridge the traditional didactic and clinical portions and to ensure students enter the APPE year with baseline knowledge, skills, attitudes, and values required to perform the final year of patient care responsibilities. Students without those baseline professional competencies would not only perform very poorly on these rotations but also interrupt the delivery of patient care in these patient care settings.

Preparedness for the APPEs was identified to be a concern at our institution among clinical faculty preceptors through professional development discussions. Some described as much as one week of a four-week rotation was spent reviewing the expectations and skills needed to successfully complete the rotation, and all believed this absorbed valuable time away from applying those skills in the patient care setting.2 These discussions delineated five areas of professional competencies of which students were expected to enter rotations with baseline knowledge: patient charts, medication histories, SOAP notes, patient presentations, and professionalism. Several methods could be considered to interface clinical skills with the didactic coursework, including the structure and delivery of the Introductory Pharmacy Practice Experiences (IPPE), the utilization of portfolios, clinical coursework, pre-rotation training, as well as many other educational tools.1,2,3,4,5,6

The current pilot study was designed to evaluate the impact of a pre-rotation workshop (PRW) on fourth year students’ ability to demonstrate preparedness for clinical APPEs and to better apply those skills to patient care activities while on the clinical rotations. The results of this study should indicate if a PRW prepares students for the requirements of a clinical APPE and if it could be considered as an option to help bridge didactic to clinical experiences.

The study objective was to evaluate the impact of a PRW on four elements: students’ clinical skills; students’ preparation for clinical APPEs; student performance on clinical APPEs; and students’ retention of clinical skills.

METHODS

Participants

The University of Oklahoma (OU) College of Pharmacy is a 4 year (P1-P4) doctor of pharmacy program that has two campuses with approximately 80 students at the Oklahoma City campus (OKC) and 60 students at the Tulsa campus. At the time of this study, no fourth year students were enrolled on the Tulsa campus, therefore only OKC fourth year students were available. Students participate in introductory pharmacy practice experiences (IPPE) during the first through 3 years of the curriculum. During the fourth year, students complete 9 months of advanced pharmacy practice experience (APPE) which consist of spending one month at nine different sites.

The study was aimed at fourth year (P4) students preparing to enter APPE’s and was designed as a randomized controlled trial using the PRW as an experimental educational intervention. The study was approved by the University of Oklahoma Institutional Review Board, and informed consent was obtained from all study participants. All students in the fourth year of the entry-level Doctor of Pharmacy professional degree program (n=82) were eligible for the study. Ten percent of the class (N=8) was randomly selected using a table of random numbers and only those randomly selected students were invited to voluntarily participate in the experimental group of the study. The GPAs of the randomly selected students were then reviewed and categorized into low (2.0-2.5), moderate (2.5-3.0) and high (3.0 to 3.5) GPAs. The entire class was then organized according to these GPA guidelines and ten percent of the class (N=8) were randomly selected to voluntarily participate in the study to serve as a GPA matched control for the experimental group. Student participation was anonymous for all students and faculty outside of the experimental group and study investigators.

Intervention

The pre-rotation workshop was developed and administered by five clinical faculty members and one instructional design and assessment faculty member. The workshop was delivered as three days of eight-hour sessions led by the study investigators with a student to faculty ratio of three to one. The workshop topics were developed around five core areas that were determined to be assumed competencies at the start of clinical rotations: patient charts, medication histories, subjective objective assessment plan (SOAP) notes, patient presentations, and professionalism. The workshop activities were designed to simulate rotation activities with brief discussions with faculty (all topics), role-play scenarios in patient care activities (medication histories), and self-directed learning assignments (patient charts, SOAP notes, patient presentations). The simulated patient charts contained samples of a fictitious patient’s health related material (history and physical report, progress notes, physician’s orders, medication administration records, laboratory results, vital signs, and microbiology reports). All activities were assessed by the study investigators using standardized rubrics created and tested internally, and participants were given verbal and written feedback on all topics and assignments.

Endpoints

There were two primary endpoints in this study: the impact of the PRW on students’ clinical skills and on students’ preparation for rotations.

The impact of the PRW on students’ clinical skills was determined by comparing the experimental group’s performance on an objective structured clinical exam (OSCE) prior to and following the PRW. The OSCE was a standardized scenario reflecting a commonly encountered problem on a clinical rotation. This scenario focused more on professional skills emphasized in the workshop than on specialized knowledge and was chosen as the assessment tool due to its simulation of a clinical rotation interventions and integration of all clinical skills included in the PRW.7,8,9 Two OSCE cases were used in this study: case 1 was used in the experimental group prior to PRW and case 2 was used in the experimental and control groups prior to and following the year of APPE rotations. The use of two different OSCE cases (cases 1 and 2) to evaluate the impact of the PRW on the experimental group’s clinical skills (a primary endpoint) was to avoid the possibility of immediate recall of case specifics over the three days of PRW.

The other primary endpoint, impact of the PRW on students’ preparation for rotations, was determined by comparing the experimental and control groups using two endpoints: First, performance on an OSCE (case 2) at the start of the APPE rotations year was compared between the study and control groups. Second, the two groups were compared using a standardized evaluation of students’ rotation performance completed by faculty preceptors during the first week of a clinical APPE (involving direct patient care responsibilities). Every attempt was made to blind faculty preceptors to PRW participation. A small portion of faculty preceptors (3 out of 22 total) included study investigators; therefore it was impossible to blind every preceptor. However, student assignments to clinical rotations were completely random. All participants were evaluated using a standardized form, which assessed the five core areas of competency (patient charts, medication histories, SOAP notes, patient presentations, and professionalism) with a rating scale of 1 (low) to 7 (high).

Secondary endpoints included the impact of the PRW on APPE performance and on retention of clinical skills. Impact of the PRW on rotation performance was evaluated by comparing experimental to control groups using the previously described standardized evaluation of students’ rotation performance completed by faculty preceptors upon completion of a clinical APPE. Retention of clinical skills was evaluated by comparing the experimental and control groups’ performance on an OSCE at the end of fourth year APPE rotations. For this endpoint, the second OSCE case (case 2) was administered at both the start and completion of the year of APPE rotations in both groups. The possibility of students recalling specifics of case 2 as a confounding variable was not considered to be an issue because of the time that elapsed for this endpoint. The use of the same case was thought to help minimize differences in student specific knowledge as well.

Data Analysis

Performance on the OSCE was graded by the study investigators using a grading rubric specific to the two cases (cases 1 and 2) used in the study. Results were then interpreted to be passing (grade ≥70%) or failing (grade <70%). The pass rate of the PRW group was compared using the paired samples t-test. When the PRW group pass rate was compared to the control group, an independent samples t-test was used.

Rotation performance was evaluated using the standardized evaluation form completed by rotation preceptors. The responses were evaluated by the study investigators after completion of study follow- up and were categorized as having an either acceptable or unacceptable level of skills for rotations. An acceptable level of skill was defined as an average score of five to seven (overall and within each of the five categories), and an unacceptable level was defined as an average score of one to four. Percentages of students in each category were compared using the independent samples t test.

RESULTS

A total of sixteen students were recruited from the fourth year of the entry-level Doctor of Pharmacy professional degree program: Eight students (13% of the class) in the experimental group and eight students in the control group. Experimental and control group participants were categorized into the low GPA (Experimental=N=1 and control=N=1), moderate GPA (N=3 and 3), and the high GPA (N=4 and 4). All participants in the experimental group completed the PRW, including the OSCE cases both before (case 1) and after the PRW (case 2). All participants in both the experimental and control groups were administered the OSCE case (case 2) both before and after the year of APPE rotations. Twenty-two full-time college of pharmacy faculty completed a total of 47 preceptor evaluations of rotation preparation (completed at the beginning and end of the rotation). All participants were evaluated on at least two clinical APPEs (range 2-5 evaluations conducted per participant, average number of clinical rotations evaluated =2.94).

The difference in the experimental group’s performance on an OSCE prior to (38% pass rate) and following (88% pass rate) the PRW was statistically significant (p<0.05) measured by the paired samples t-test. In addition, there was a statistically significant difference between the experimental group’s performance on an OSCE prior to start of the APPEs (88% pass rate) compared to the control group’s performance (0% pass rate) (p=0.001) measured by the independent samples t-test. However, there was no statistically significant difference in performance on an OSCE after the completion of the APPEs between the experimental (75% pass rate) and control (50% pass rate) groups (p=0.252) measured by the independent samples t-test. No statistically significant difference was found between the experimental and control group’s demonstration of clinical skills during the first week or at the completion of the clinical APPEs (see table 1) as evaluated by the rotation preceptors.

Table 1

Clinical Rotations Evaluations.

	PRW Group (N=8) Mean† (SD)	Control Group (N=8) Mean† (SD)
Beginning of Rotation Month Evaluation Results
Average – 1st clinical rotation	3.61 (0.94)*	4.10 (1.10)*
Average – All clinical rotations	4.38 (1.14)*	4.47 (1.10)*
End of Rotation Month Evaluation Results
Average – 1st clinical rotation	5.54 (0.78)*	5.06 (0.90)*
Average – All clinical rotations	5.73 (0.69)*	5.54 (0.88)*

Based on a 7-point Likert scale ranging from 1 = low to 7 = high

no statistically significant differences between PRW and Control groups

DISCUSSION

The PRW was designed to be an introductory APPE for the transition from didactic coursework to clinical APPEs at our institution. The statistically significant difference in pre and post PRW OSCE results reveal that the PRW was demonstrated to be an effective method to improve students’ clinical skills, evidenced by the improvement in OSCE performance in the experimental group in this study. This finding would be expected because the workshop provided participants with intense, prolonged practice with direct feedback from clinical faculty.

The first important finding from this study is the impact of the PRW on students’ preparedness for clinical APPEs, as this was the underlying motivating issue at our institution. The results from this study suggest that the majority of students in this study were not prepared for clinical APPEs, evidenced by the student’s low pass rates (38% of the experimental group and 0% of the control group) on their first OSCE in this study taken prior to the start of the APPE year. The second important finding is that the study results suggest that the PRW as a three-day introduction rotation was effective at preparing the experimental group of students for the clinical skill requirements for the APPEs, evidenced by 88% of PRW students passing the OSCE.

While this result indicates the need for more experiences to help bridge didactic to clinical experiences throughout the curriculum, the result is limited by the small experimental group sample size. The use of only two OSCE cases also limits these study results. Although the two cases addressed general acute care issues and clinical skills with a similar level of complexity that all fourth year students must possess for APPE and should not have afforded an advantage to the experimental group, OSCEs were not commonly used as an evaluation tool with this class of students during the time of the study in any phase of the curriculum. Since none of the study participants had experience with OSCE exams prior to the study, the experimental group’s experience in taking the preworkshop exam might (by itself) have given that group an advantage (“test-wiseness”) over the control group when both groups took the case 2 OSCE exam since the experimental group may have had less apprehension and more familiarity with exam procedures as described by Mahamed, Gregory and Austin previously.10

The third limit to this finding is that the OSCE examiners were not blind with respect to study group and the OSCE examiners consisted of the faculty who provided the pre-rotation workshop. Furthermore, the OSCE’s were graded live by the evaluators instead of videotaping, therefore limiting scoring by other examiners. Although the examiners were trained with the rubric during sample OSCEs and demonstrated similarity in scoring, no formal assessment of inter-rater reliability was performed and the exam process was not validated.

The lack of a difference on the preceptor’s evaluation of student performance during the first week and at the rotation completion on clinical APPEs was discouraging; however this finding may point out several important limitations of this study. First, the use of a small, randomly selected sample size introduced a wide variability in assigned clinical rotations available to solicit these evaluations. This meant that students were not all entering clinical APPEs at the same time in the year (first clinical rotation ranged from June to April of that academic year) and that multiple faculty preceptors were completing these evaluations (a total of 22 preceptors completed the included 47 evaluations).

Furthermore, since the entire class did not participate in the workshop, many faculty preceptors may have continued reviewing baseline expectations and skills at the beginning of the month and conducting rotation activities to develop baseline competencies instead of refining higherlevel clinical skills. Limitations of the evaluation tool itself, such as the sensitivity to capture differences in and development of clinical skills and the interrater variability, is another potential limitation of this study.

APPE performance was not determined to be different between the groups in this study, as all students appeared to achieve rotation expectations by the end of the rotation. This may also be because specific APPE activities were not designed based on entering levels of performance. It is possible that the PRW group was able to spend more time in patient care activities, but this pilot study was not designed to gather this information. This pilot study was unable to determine whether the PRW had an impact on retention of clinical skills at the end of the APPE year when comparing clinical skills between students who completed the PRW with the control group. This could be due to the relatively low sample size or it could suggest that despite the level of clinical skills at the start of the APPE year, students were achieving the expected level of competency by the end of the APPE year through their clinical rotation experiences.

Overall, it does appear that providing pharmacy students with these additional opportunities for intense interaction with direct feedback from clinical faculty was beneficial to the participants. Each of the study investigators received numerous positive responses from the participants both at the completion of the PRW (less anxiety about rotations and faculty preceptors) and during the APPE year (improved patient care and APPE performance). For example, one student reported an instance of being proactive in taking a complete medication history on the first day of a clinical rotation. Future research should consider evaluating student perceptions and experiences in their APPE following a PRW. One outstanding question is to determine the optimal timing to introduce these and similar opportunities within a curriculum to further develop baseline professional competencies. Based on the results of this study, the incorporation of an introductory rotation is worthwhile in that it can improve students’ baseline knowledge, skills, attitudes, and values required to perform the patient care responsibilities at the start of the APPE year.

CONCLUSION

The pre-rotation workshop appears to be worthwhile in improving student’s preparation for clinical Advanced Pharmacy Practice Experiences, although it is unclear how effectively it will advance clinical skills overall. Additional studies will help to determine the role of this and other educational methods to help bridge didactic experiences in the Doctor of Pharmacy curriculum.