Clinical Advanced Pharmacy Practice Experience Rotations During COVID-19: Evaluation of a Transition to Virtual Learning.

BACKGROUND: All Advanced Pharmacy Practice Experience (APPE) pharmacy rotations at a large academic medical center were converted to virtual experiences during the beginning of the coronavirus disease 2019 (COVID-19) pandemic.
OBJECTIVE: This study aimed to describe information obtained through pre- and post-rotation surveys, implemented to improve experiences for future students who may be required to complete virtual APPE pharmacy rotations.
METHODS: A single-center, descriptive study was conducted at a 1382-bed academic medical center. A pre- and post-rotation survey was sent to 32 students, and a post-rotation survey was sent to 38 preceptors via email to assess newly implemented virtual rotations.
RESULTS: Students' response rate for pre- and post-rotation surveys was 59% and 41%, respectively, and the preceptors' response rate for the post-rotation survey was 37%. A statistically significant improvement in videoconferencing abilities after the rotation was found for students but no differences in other skills were noted. In the post-rotation survey, students rated all of the following areas as being "effective": rotation as a whole, virtual topic and patient discussions; but were "neutral" regarding the utility of the introductory training guide. In the post-rotation survey, preceptors rated all of the following areas as being "effective": rotation as a whole, virtual topic and patient discussions.
CONCLUSION: Abrupt shifts to virtual pharmacy clinical rotations due to COVID-19 have led to many challenges. Both students and preceptors felt that virtual rotations were an effective alternative to in-person experiences; however, further studies are warranted to evaluate actual performance compared to perceived effectiveness.

Introduction

Coronavirus disease 2019 (COVID-19) is a respiratory illness caused by a novel coronavirus (SARS-CoV-2) which placed a significant strain on the healthcare system.[1] This strain reduced access to personal protective equipment and raised concern for exposure to patients, providers, and students. Didactic and clinical education of medical, pharmacy, nursing, and all allied health professional students were severely impacted. Colleges across the nation closed their physical doors and moved all classes online, which led to significant challenges for those providing healthcare education. Professors who were still providing didactic education to students in the classroom had to quickly ramp up virtual education by pre-recording lectures, assigning homework, altering project work, and learning how to have beneficial discussions through virtual meeting platforms.

During this time, many students were in the practical or clinical portion of their healthcare education. The first change most institutions made when COVID-19 became present in their community was to start limiting visitors and all non-essential personnel from the hospital. This included preventing all healthcare learners from attending their clinical rotations and participating in direct patient care. Many of these students were about to begin the last rotation of their academic career, a required rotation many of them needed to graduate with their healthcare degree. It can be estimated that in pharmacy education alone this impacted a quarter of the 60 594 pharmacy students enrolled.[2] All healthcare education accreditation organizations charged colleges to evaluate each student’s specific requirements for graduation and to only have those students who required clerkship or clinical hours to continue clinical rotations.[3]

To maintain the safety of all students and preceptors, while ensuring that pharmacy students could graduate on time, the institution decided to allow those students who required clinical hours to graduate to maintain their rotation experience. Due to the requirement of social distancing and the fact that all learners were removed from the hospital on March 17th, 2020, all clinical experiences for the foreseeable future became “virtual” or “remote” rotations. Given the lack of data available to help direct these experiences and that there was no end in sight at that time, students were surveyed before and after each virtual rotation and preceptors were surveyed after each virtual rotation to gain information so that experience and learning opportunities could be improved for future pharmacy students. The purpose of this quality initiative was to describe information obtained through pre- and post-rotation surveys, with the intention to obtain feedback to improve the experience for students who may be required to complete virtual Advanced Pharmacy Practice Experience (APPE) pharmacy rotations in the future.

Methods

During the months of April and May 2020, all Advanced Pharmacy Practice Experience clinical pharmacy rotations at this practice site were virtual. The April students were on their last APPE rotation, and the May students were on their first APPE rotation. On average, this practice site (1382-bed academic medical center) takes approximately 15–20 students on rotation per month. A wide variety of rotations are available for students including the following practice areas: academia, acute care surgery, ambulatory care, cardiology, community retail pharmacy, critical care, emergency medicine, drug information, hematology, infectious disease, internal medicine, medication safety, nuclear medicine, oncology, pain and palliative care, pharmacy administration, psychiatric care, and solid organ transplant. Currently, this practice site only takes students from one College of Pharmacy which is a state school enrolling approximately 130 students per academic year and is one of seven health science colleges on campus. Prior to April 2020, the institution had never offered virtual rotations; therefore, there was a learning curve for preceptors and learners alike. To prepare for the month, all students were given remote access to the electronic medical records (EMRs) so they could review patients from home. An introductory training guide and a “Virtual Rotation Student Checklist” (Figure 1) was created to teach the student how to access the EMR remotely, how to utilize the various virtual meeting platforms available at the institution, and offer advice on how to be successful during the virtual clinical rotation. Tips on how to maximize the virtual meeting platforms, rules and regulations around sharing patient information on these platforms, maintaining Health Insurance Portability and Accountability Act compliance, and basic expectations were shared with the preceptors. Outside of the completely remote nature of the rotation, the expectations and day-to-day patient care that the student provided were no different than if they were on-site with the exception that they were unable to participate in patient care rounds. Preceptors were encouraged to schedule meeting times with the students every day to have in-depth discussions on the patients they were providing care for. As an attempt to improve efficiency across the health system, a team teaching calendar was developed for topic discussions. Each preceptor was encouraged to schedule their virtual topic discussions on the team teaching calendar and to share every topic discussion with their students, despite the area of focus for the rotation.

Figure 1.

Virtual rotation student checklist.

In an effort to assess the quick flip to virtual APPE rotations and to obtain feedback as a quality initiative from students and preceptors, a pre- and post-rotation Qualtrics® survey was developed and sent out to students and a post-rotation survey was sent out to preceptors. The pre-rotation survey was sent out via email to all students on rotation at the medical center one week prior to the rotation start date, with a reminder 2 days prior to the start date. The pre-rotation survey closed the day the rotation started. The post-rotation survey was sent out via email to both the students mentioned above and all preceptors who had those students on rotation on the last day of the rotation; a reminder was sent 1 week later, and the survey closed 2 weeks after the last day of the rotation. The surveys were voluntary, and there were no incentives for participation. In the pre-rotation survey, students were asked to describe their previous rotation experiences, their practice setting post-graduation if known, videoconferencing abilities, and how they rated themselves in various areas (i.e., written communication skills, verbal communication skills, public speaking, interpersonal skills, and follow through with assignments). Additionally, specific questions were added to the post-rotation survey to evaluate how effective they rated the following: the rotation as a whole for their learning, virtual topic discussions, virtual patient discussions, and the introductory training guide. All preceptors received a post-rotation survey, and were asked to describe their practice setting and questions to evaluate how effective they rated the following: the rotation as a whole for them to teach, virtual topic discussions, and virtual patient discussions. Additionally, both students and preceptors had an option to leave open-ended comments at the end of the post-rotation survey.

The results were anonymously compiled and aggregated data were assessed. When the data were pulled, each respondent was assigned a number. Those respondents that completed both the pre- and post-survey were included in the analysis to compare their skills in the various areas mentioned above. Three Likert scales were utilized dependent upon the question: Likert scale 1 was a 7-point scale to assess ability (0—far below average, 1—moderately below average, 2—slightly below average, 3—average, 4—slightly above average, 5—moderately above average, and 6—far above average), and Likert scale 2 was a 5-point scale on effectiveness (0—very ineffective, 1—ineffective, 2—neutral, 3—effective, and 4—very effective).

The primary outcome of this quality initiative was to describe information obtained through pre- and post-rotation surveys, with the intention to obtain feedback to improve the experience for future students who may be required to complete virtual APPE pharmacy rotations. Continuous data are reported as median (25–75% interquartile range (IQR)) or a percentage for descriptive data, as appropriate. Before and after Likert scale responses were analyzed using the related-samples, Wilcoxon signed rank tests and data are presented as median (interquartile range (IQR)).[4] All analysis was performed using IBM Corp. Released 2020. IBM SPSS Statistics for Windows, Version 27.0. Armonk, NY: IBM Corp.

Results

Students

An email to complete a pre- and post-rotation survey was sent out to a total of 32 unique students. The response rate for completion of these surveys was 59% (n = 19) and 41% (n = 13), respectively. Out of the 19 students who responded to the pre-rotation survey, 72.2% previously had an acute care rotation, 13.6% had an academia rotation, 50% had an ambulatory care rotation, 68.2% had a community/retail rotation, and 54.5% had an “other” type of rotation which was noted to be a home infusion, industry, managed care, drug information, and/or administration rotation. The practice setting that students would be entering upon graduation was the following: postgraduate year-one pharmacy residency (47%, n = 9), undetermined (26%, n = 5), acute care (11%, n = 2), community/retail (5.3%, n = 1), industry (5.3%, n = 1), and health-system pharmacy administration (5.3%, n = 1). Ten students completed both the pre- and post-rotation survey assessing how they rated the various skills including written communication, verbal communication, public speaking, interpersonal skills, follow through on assignments, and videoconferencing abilities. No differences were found in median response rates of skills assessed except in videoconferencing abilities where a statistically significant improvement was observed after the rotation (3 (IQR, 1–4.25) vs 5 (IQR, 3–5.25); P = .049) (Table 1). The increase in median score is representative of a response of “average” pre-rotation to “moderately above average” post-rotation based on Likert scale 1.

Table 1.

Student Self Rated Ability Assessment, n = 10.

Area assessed	Pre-virtual rotation	Post-virtual rotation	P value
Written communication skills	3 (1–5.25)	5 (3.75–5.25)	.072
Verbal communication skills	3.5 (2.5–4.5)	4 (3–5)	.491
Public speaking	3 (2.75–3.25)	4 (3–4)	.096
Interpersonal skills	4 (1–5.25)	5 (4–5.25)	.121
Follow through with assignments	5.5 (2.5–6)	5 (4–6)	.279
Videoconferencing abilities	3 (1–4.25)	5 (3–5.25)	.049

Data are presented as median (interquartile range).

Likert scale to assess ability: 0—far below average, 1—moderately below average, 2—slightly below average, 3—average, 4—slightly above average, 5—moderately above average, and 6—far above average.

Students’ responses on the post-rotation survey to the questions on the effectiveness of the rotation, topic and patient discussions, and introductory training guide can be found in Table 2. There were a total of 13 responses, and the median response to all question was “effective” on Likert scale 2, with an exception that they were “neutral” with the introductory training guide being helpful to prepare them for the virtual rotation. There were no constructive comments provided by the students in the open-ended comment box at the end of the survey as the majority of the comment boxes were blank or the comments were simply stating that utilization of a web camera helped simulate in-person interactions.

Table 2.

Student Evaluation of Virtual Rotation, n = 13.

Question	Very effective, n (%)	Effective, n (%)	Neutral, n (%)	Ineffective, n (%)	Very ineffective, n (%)
How effective would you rate the rotation as a whole for your learning?	6 (46%)	6 (46%)	0	1 (8%)	0
How effective were virtual topic discussions?	7 (54%)	4 (31%)	2 (15%)	0	0
How effective were virtual patient discussions?	6 (46%)	5 (38%)	1 (8%)	1 (8%)	0
How effective was the introductory training guide at preparing you for this virtual rotation?	0	3 (23%)	9 (69%)	1 (8%)	0

Preceptors

An email to complete post-rotation survey was sent out to a total of 38 preceptors and 14 responded, which was a response rate of 37%. Of those who responded (n = 14), a majority (71%) practiced in the acute care setting, 14% community/retail setting, 7% in ambulatory care, and 7% in a medication assistance program. Preceptors’ responses to the questions on the effectiveness of the rotation, topic, and patient discussions can be found in Table 3. The median response for all questions was “effective” on Likert scale 2. There were no constructive comments provided by the preceptors in the open-ended comment box at the end of the survey as the majority of the comment boxes were blank or the comments were simply stating that virtual rotations were an acceptable short-term solution and that utilization of videoconferencing with a web camera helped simulate in-person interactions.

Table 3.

Preceptor Evaluation of Virtual Rotation, n = 14.

Question	Very effective, n (%)	Effective, n (%)	Neutral,n (%)	Ineffective, n (%)	Very ineffective, n (%)
How effective would you rate the rotation as a whole for you to teach?	1 (7%)	8 (57%)	4 (29%)	1 (7%)	0
How effective were virtual topic discussions?	1 (7%)	10 (72%)	3 (21%)	0	0
How effective were virtual patient discussions?	3 (21%)	5 (36%)	5 (36%)	1 (7%)	0

Discussion

With the abrupt removal of learners from the clinical setting as the COVID-19 pandemic progressed, preceptors and students had to quickly adapt to virtual patient care, teaching, precepting, and learning. Concerns were anticipated from preceptors and students on the ability to conduct an effective clinical APPE rotation through videoconferencing and this quality initiative sought to assess perceptions and gather feedback for improvement as this uncharted territory was explored. This report suggests that virtual pharmacy clinical rotation were perceived, by both students and preceptors, to be an effective learning platform. Students and preceptors alike felt that both patient discussions and topic discussions were effective. One factor thought to have had a strong impact on the perceived effectiveness was that students were given remote access to the EMR. Without such access, real-time patient review would not have been possible and, thus, make a patient care rotation nearly impossible.

To our knowledge, this is the first report of a virtual clinical APPE rotation where students were given access to the EMR and were expected to review and work up patients from home. A recent publication by Johnston and colleagues discuss how their critical care faculty quickly created a virtual APPE rotation during the COVID-19 pandemic.[5] The rotation they initiated included topic discussions, journal clubs, case presentations, online learning modules, and formal writing assignments. The students only had access to an educational EMR where they worked through practice or pre-built patient cases. During the virtual rotations carried out in our report, as mentioned above, students had access to the actual EMR and reviewed patients admitted to the preceptors’ service daily. The day-to-day patient care that the student provided was no different than if they were on-site with the exception that they had limited interactions with the interdisciplinary team, and they were unable to participate in patient care rounds.

The results of this survey demonstrated significant improvement in videoconferencing abilities for APPE students. Utilization of videoconferencing allowed for frequent face-to-face interactions and screen sharing to give synchronous access to the EMR and facilitate patient care. From the comments that were provided in the post-rotation survey, both students and preceptors felt that utilization of videoconferencing helped simulate in-person interactions and positively impacted learning. The majority of preceptors met with students via videoconferencing twice daily, typically in the morning for patient discussions and in the afternoon for topic discussions. In addition to videoconferencing, preceptors and students communicated via secure messaging in the EMR for patient-related items and instant messaging for non-patient-related items. Students were also able to communicate with the interdisciplinary team via the secure messaging in the EMR facilitating some degree of interaction with other healthcare providers.

Currently, there is limited literature on the use of videoconferencing for synchronous clinical rotations. Given the students’ perceived improvement in videoconferencing skills during the virtual APPE rotation and feeling that the virtual rotation was effective for their learning, this may be the first report to support videoconferencing with off-site EMR access as a method to offer clinical rotations. This finding matches those highlighted by Eiland, which demonstrated that students who participated in an off-site academic APPE felt that communicating primarily through videoconferencing did not impede learning.[6] The COVID-19 pandemic has pushed students and preceptors to think outside of the box and try methods of teaching and learning that had previously not been attempted. The ability to offer virtual rotations could open up opportunities for students to gain experiences that they would have previously not had due to location, timing, cost, or other travel-related restrictions. Additionally, the use of videoconferencing in pharmacy education could also help develop preceptor communities, both locally and nationally, as more information on the direction of the pandemic and how virtual rotations could mold the APPE clinical rotation experiences moving forward.

Conducting a virtual clinical APPE did have several limitations. Students noted that a virtual rotation took away from some of their learning experiences, especially their ability to interact with other members of the healthcare team. The primary driver for this concern was the lack of interdisciplinary interaction with the patient care team and inability to be physically on patient care rounds. Prior to the start of the rotation, preceptors attempted numerous methods to be virtually present on rounds with the team (e.g., virtual teleconferencing or phone call), but were deemed not to be feasible given the current rounding structure. Issues that arose primarily were due to communication difficulties. For example, it was difficult to hear team members during patient presentations, to interject with pharmacy interventions, and to understand when a question was being posed to the pharmacist when only present via virtual teleconference. Depending on how patient rounds are conducted, a possible method to circumvent these last 2 issues would be to have a dedicated pharmacy time out during each patient or to implement “sit down” rounds in a room with videoconferencing. Having students reach out to patient care team members through the “Secure Chat” feature in the EMR and attaching the preceptor was a useful method to bridge this educational gap. This allowed for appropriate oversight from the preceptor while the student was able to learn necessary skills in alternative methods of communication to recommend therapeutic modifications. These limitations surrounding interprofessional skills are consistent with another hospital’s experience when conducting virtual inpatient APPE rotations utilizing de-identified or made-up patient cases during the COVID-19 pandemic.[7] If APPE rotations were required to be virtual again or if this opportunity was offered for another reason (extended student leave for health reasons or distant virtual clinical rotation), the problem of solving the communication difficulties for rounding to ensure students had the ability to engage with the interprofessional team in a more real-time manner would be paramount.

Another limitation identified by the students was the need for improved communication and planning. Students’ median response to the effectiveness of the introductory training guide was “neutral.” When planning the virtual rotations, the introductory training guide and “Virtual Rotation Student Checklist” (Figure 1) was created in an effort to help prepare students up front and in hopes that the rotation can be started in a positive direction. Additionally, the guidance was for each preceptor to establish additional expectations with their student and preceptors were encouraged to place their planned topic discussion on a shared teaching calendar on the Microsoft Outlook® platform. The preceptor was then responsible for sharing all available topics with their student. After receiving feedback from the students, it was recommended that preceptors create a detailed calendar prior to the start of rotation to set consistent times and methods for patient care discussions. In the future, it would be beneficial to make the real-time shared teaching calendar for topic discussions more readily available to students on a more accessible platform. This would help if lecturers needed to move the date or time of the topic discussion due to patient care responsibilities. When the lecturer was not the student’s primary preceptor, this caused confusion for the student and could often result in missing the topic discussion. To assist with some of these issues, it is very important that the shared teaching calendar include specific details for topic discussions including but not limited to the date, time, virtual location, lecturer name, and contact information. Overall, preceptors appreciated the shared teaching calendar. This provided a division of the workload and a sense of community, similar to that demonstrated by Ackman and Romanick when evaluating virtual communities and networks for pharmacy preceptor development.[8]

This quality improvement survey project was conducted to evaluate the urgent need to send both students and preceptors off-site during the height of the first phase of the pandemic. Given this information, the results of this study should be interpreted with caution due to the limitations of the survey design and quick rollout. First, this was a small snapshot in time (April and May 2020) when students were first required to be off-site. The goal was to capture a limited amount of information to improve survey compliance and to capture specific information to ensure that virtual rotations were an effective venue for students to learn in; however, this was all self-reported information, and clinical performance was not assessed. This report included a small number of preceptors and students surveyed at one academic medical center. It is possible that the perceived effectiveness would be reduced if multiple centers were involved due to a lack of cohesive approach to training and systems utilized. It must also be recognized that there was a low response rate from preceptors. This was a time of high stress and obligation for all preceptors and clinical staff as many working remotely, creating COVID-19 treatment guidelines, and trying to maintain their own safety in such difficult times. It is believed that the low response rate from the preceptor group was multifactorial. While grades were provided to students for their performance on the virtual rotation, the students’ actual grades were not obtained and thus were not compared to their perceived effectiveness of the rotation. If given optimal time to design a similar study, actual grades students were awarded would have been compared to their perceived effectiveness and to grades awarded for a similar rotation that was not virtual. Additionally, students could have also been surveyed again after entering their job or residency position to evaluate how prepared they felt. This type of information would give a better understanding of the actual effectiveness of a virtual clinical rotation.

Finally, it is important to note that when virtual rotations were initiated, the Accreditation Council for Pharmacy Education (ACPE) had not announced a stance on how virtual rotation would impact the students’ ability to collect the experiential hours needed to graduate. After implementation of virtual rotations, in August 2020, ACPE provided communication on recommendations for participation in experiential education activities during a national, regional, or local crisis. In this communication, they state that students may make up hours for the experiential education either online or through a virtual experience.[9] The ACPE’s stance along with reports like this one will continue to mold the future of APPE rotations for pharmacy students moving forward.

Conclusion

The COVID-19 pandemic disrupted the education of students at all levels of learning across the nation. For pharmacy students, the abrupt change from the usual in-person to virtual clinical APPE rotation format appeared to be an effective format for their learning; however, further studies are warranted to evaluate the actual performance compared to the perceived effectiveness.