Comparison of student pharmacists' performance on in-person vs. virtual OSCEs in a pre-APPE capstone course.

INTRODUCTION: Objective structured clinical examinations (OSCEs,) provided in-person or virtually, assess student pharmacist readiness for advanced pharmacy practice experiences (APPEs). During the COVID-19 pandemic in 2020, it was necessary for many educators to design and implement virtual OSCEs (vOSCEs). Impact on student performance utilizing in-person vs. vOSCE has not been well evaluated. The objective of this study was to determine if a difference existed in student performance when comparing in-person vs. vOSCE in a third year (P3) pharmacy pre-APPE capstone course.
METHODS: In winter 2019, four in-person OSCE stations were designed and implemented in a pre-APPE P3 capstone course. In winter 2021, the same four stations were transitioned into vOSCE stations. Assessment (summative) data from similar student cohorts from OSCE 2019 were compared vOSCE 2021 stations using Mann-Whitney U test.
RESULTS: There was no meaningful difference found when comparing student performance on in-person OSCE vs. vOSCE. There was no significant difference for the number of students offered remediation. For most stations, performance improved from formative to summative assessments.
CONCLUSIONS: Providing vOSCEs to assess students' skills in a pre-APPE P3 capstone course is a reasonable alternative to in-person OSCEs.

Introduction

Capstone courses in pharmacy curricula provide the opportunity for students to apply didactic knowledge in practice-based scenarios. The objective structured clinical examination (OSCE) is a skills-based assessment component of pharmacy education that is commonly found in capstone courses to assess readiness for advanced pharmacy practice experiences (APPEs).1, 2, 3 Our third year (P3) pre-APPE clinical capstone course has three main purposes: (1) to be a culmination point of the knowledge and skills students have acquired in the didactic portion of the pharmacy curriculum, (2) to provide review and assessments to better prepare our students prior to beginning their APPEs, and (3) to act as a critical assessment point to meet changing educational standards. A significant amount (30%) of the clinical capstone course grade is related to OSCE student performance, which highlights the importance of this skills-based assessment prior to fourth year APPEs.

In the past two years, the concept of virtual OSCEs (vOSCEs) has become more prominent in the literature evaluating educational methods in health profession students. For example, Blythe et al demonstrated the feasibility of vOSCEs in a small number of final year medical students. They found that virtual communication-based assessments were more feasible in comparison to virtual clinical examination assessments. Although technical problems were not encountered, the authors expressed that virtual assessment of their entire class of students would not be feasible due to limited staffing and quality control issues. Other articles demonstrated similar feasibility and limitations in dental education and medical specialty board exams. , While there are several studies across healthcare professions related to the development, implementation, and perceptions of virtual skills-based assessments, , , 7, 8, 9, 10, 11 literature is limited related to the evaluation of students' performance on virtual assessments. In pharmacy literature, VanLangen et al reflected upon their experience of in-person OSCEs vs. vOSCEs. While their primary study objective was to evaluate student and faculty perceptions of in-person vs. vOSCEs, a secondary objective was to determine if there was a categorical difference in performance between those that scored <80% vs. ≥ 80%. They found no significant differences in performance for two of three communication stations offered to third year students. Limitations in their study included a lack of consistency in patient types used in OSCE stations (in-person used standardized patients [SPs]; vOSCE used faculty), small station number, and structural differences in stations between years. Hsia et al addressed the implementation of a two-station vOSCE in a patient care skills course assessing communication of first year pharmacy students. The primary focus of the study was survey evaluation of SPs' and students' perceptions, but also included an outcome evaluating pass rate on the in-person 2019 OSCE vs. 2020 vOSCE. No difference was seen in the overall pass rates for the in-person vs. virtual OSCEs. Limitations cited were similar to those of VanLangen et al.

When the COVID-19 pandemic hit in March 2020, schools of pharmacy across the world were forced to develop remote teaching strategies and faced several challenges to do so emergently. This also required many pharmacy educators to design vOSCEs to assess skills that were previously assessed in a face-to-face setting. Similar to other programs, our P3 clinical capstone course needed quick transformation to 100% remote online learning. Given that it is an eight-week course offered the second half of the winter semester, the class had been in session for two weeks when the switch to remote online learning was mandated by the university. Consequently, an emergency action plan was developed to deliver the remainder of the course online through the end of April 2020. As the pandemic persisted into 2021, university policy required that remote online learning continue. The winter semester 2021 course planning commenced and included the necessary transformation of in-person OSCEs to vOSCE format. An unanswered question remained if the switch in testing environment and process impacted student performance. Thus, the objective of our study was to retrospectively evaluate if a difference existed between in-person OSCE student performance and vOSCE performance in a P3 pharmacy pre-APPE clinical capstone course.

Methods

OSCE case and station development

In 2019, four in-person OSCE stations were offered, including two incorporating live SP interactions and two written electronic assessments. As shown in Table 1 , SP stations included (1) collection of medication history and (2) over-the-counter medication counseling utilizing the SCHOLAR-MAC method. Written assessments included (3) renal dosing assessment for a hospitalized patient and (4) prescription verification with answers being entered into the university's electronic learning management system (LMS). The capstone course director and pharmacy program summative evaluation committee, comprised of representatives from experiential education and pharmacy practice faculty, developed the station concepts based on a survey of entrustable professional activities. Course faculty wrote the patient cases to be reflective of a selected number of disease states and focused on necessary skills needed to prepare for APPEs.

Table 1

Description of objective structured clinical examination stations.

OSCE station	Description of activity	Rubric / question allotment (% of station)
#1 Medication History	Accurately collect a medication history from a standardized patient and utilize appropriate patient interviewing and communication skills	• Communication elements (45%)a • Technical elements (45%)b • Global assessment (10%)c
#2 OTC Counseling	Apply the SCHOLAR-MAC method to collect information, assess the SP's self-care related issue, and provide an appropriate recommendation for self-care treatment or referral	• SCHOLAR-MAC interview (45%) • Patient impression, OTC selection and counseling (45%) • Global assessment (10%)c
#3 Renal Dosing in Hospitalized Patient	Evaluate a written patient case to determine factors and medications that contribute to AKI, calculate serum creatinine clearance, and perform appropriate renal dose adjustments to develop a care plan	• Calculation of creatinine clearance (26%) • Identification of patient contributors to AKI (16%) • Review of all medications to determine status of discontinue, modify, or continue (32%) • Description of dosing modifications (10%) • Monitoring (16%)
#4 Prescription Verification	Evaluate new and refill prescription orders based on patient-specific data provided. Three prescriptions with matching prescription history provided.	• Decision to accept or reject prescription as written and entered into pharmacy system (50%) • Description of decision making process (50%)
#5 DRP identification and Solutiond	Identify potential DRPs based on the medication history elicited from the SP in station #1 and provide solution(s) for the identified problems.	• Review of all medications to assign a potential DRP status (50%) • Describe three most important DRPs and solution (50%)
#6 Vaccine Needs Assessmentd	Evaluate a written patient case to perform a vaccine needs assessment and provide appropriate recommendations.	• Develop a vaccination plan for the next 12 months (63%) • Develop a vaccination plan for specified future date (i.e. 30 years in future) (37%)

AKI = acute kidney injury; DRP = drug related problem; OSCE = objective structured clinical examination; OTC = over-the-counter; SP = standardized patient.

Elements included introduction, relationship building, acknowledging and responding to emotions, rapport, question style, verbal/non-verbal expression, health literacy, confidence, professionalism and organization.

Elements included specific questions surrounding all medication-related products, allergies, how/why the patient is using the medication, problem inquiry, procurement and storage of medications, potential substances of abuse, and vaccination status.

Global assessment question was “I would seek help/advice from this pharmacist in the future.”

Stations #5 and #6 were introduced in 2021 and were not assessed in the 2019 in-person OSCE.

Assessment and remediation

All stations were graded via standardized rubrics developed for the course. Each rubric had a list of standardized questions and points assigned to them based on the case type. Details regarding rubric and question breakdown are shown in Table 1. Students were required to achieve a minimum score of 70% on each station. Stations using SPs had three-part checklists as described. Any student that received <70% or a “definitely not” on the global assessment as a response to the question “I would seek help/advice from this pharmacist in the future” did not pass. In these situations, a coordinator reviewed the video to determine accuracy and offered a one-time remediation opportunity.

Standardization and station flow

The group of SPs used were professional actors from the medical school's clinical exam center. The center works with course faculty to develop a training guide for each unique case to standardize instruction. All stations were piloted utilizing fourth-year pharmacy students on APPEs and SPs in person prior to the day(s) of the OSCE. Stations were scheduled in 20-min intervals at the clinical skills center; for SP-based stations, 10 min were allotted for interaction with the SP and 10 min for documenting the encounter. The full 20 min were permitted for written electronic stations. SP and faculty observer training were conducted in person about one week prior to event days. First time faculty observers were trained using the case information, observed a case, and then practiced assessing the case using a norming process. During this observation, a checklist is completed together to ensure graders understand the rubric and nuances of it. During the course, students had two scheduled days of OSCE assessments, one as a formative OSCE offered within the first three weeks of the course where they received feedback and a final, summative at the end of the course. The formative OSCE was strategically placed after instruction was provided regarding skills and station expectations. After the formative OSCE, students were given individual written reports on their performance on the SP-based, live stations and written stations were graded in the LMS or Examsoft, version Legacy (ExamSoft); however, only the summative OSCE grades were incorporated into their course grades. The same checklists were used on the formative and summative stations.

vOSCE development

Due to university policy, 100% of the P3 clinical capstone course was run virtually in 2021. For the 2021 transformation of the OSCEs to vOSCEs, course faculty attempted to mirror the previous in-person experience but had to adjust as described below to accommodate the online platform. vOSCE scheduling was different than in-person OSCE to allow for potential system issues and accommodate the SPs. Two days were scheduled for the formative vOSCE, each with three stations at 30-min scheduling intervals. Within those time intervals, SP-based communication stations provided up to 15 min for SP interaction and written stations provided up to 25 min. The five min increase in time made available to students per station interaction was provided solely to account for potential challenges in interacting online with SPs, connection issues, system loading delays, and uploading. A virtual platform was used for SP interactive stations, and a combination of the LMS or ExamSoft was used for written stations. To enhance exam security and reduce potential for student communication, all written stations were administered at the same time remotely. In addition, one of the long-term plans for the course was to add two new written stations, (5) drug-related problem (DRP) identification/resolution and (6) vaccine needs assessment. Course rubrics used for grading stations were the same as in 2019 except for the new rubrics developed for the two newly added stations.

In the “new” vOSCEs, each station blueprint and associated assessment type did not change from the 2019 OSCE administration. However, every year the disease states or drugs of focus are modified within each OSCE station to avoid duplication of assessments and enhance academic integrity. New cases are developed for the summative OSCE every year. Like the OSCEs, vOSCE content was piloted with SPs and APPE students to collect feedback and make modifications if needed. Faculty observers were utilized in the same fashion as with the OSCEs except that they observed virtually during the session on Microsoft Teams (Microsoft Corp.) and not using remote video like the in-person OSCEs.

Evaluation

A retrospective evaluation was conducted of the cohorts of comparable OSCE student performances of winter 2019 (in-person) and winter 2021 (vOSCE). The primary outcome was to determine differences in summative performance on the four comparable OSCE stations (OSCE 2019 vs. vOSCE 2021). Secondary outcomes evaluated included the following: (1) performance on the formative and summative OSCE stations within years and (2) number of students offered OSCE remediation.

Course level data were provided from the course coordinator and obtained from the LMS. Performance data were only included in the analysis of stations if students had taken the OSCEs or vOSCEs in the intended format and “on time”; those stations not meeting these parameters were excluded from evaluation. Once this exclusion occurred, data was de-identified for analysis. This project was submitted to the university's institutional review board and deemed non-human participant research (HPR determination no. 2020159).

All quantitative data was evaluated using SPSS for Windows, version 27 (IBM Corp.). Descriptive statistics were evaluated for all stations. The Mann-Whitney U test was utilized to compare cohorts and OSCE vs. vOSCE performance. The Wilcoxon Signed-Rank test was utilized to compare formative and summative scores within cohorts. Individual OSCE and vOSCE scores were reported as the actual station scores as continuous data with medians and interquartile ranges. Two-tailed Fisher exact probability tests were used to compare the number of students offered opportunities for station remediation. A P value below a significance threshold (α) of .05 was considered significant.

Results

Results from the in-person OSCE and vOSCEs are reported in Table 2 . The original class sizes each year were n = 97 (2019 OSCEs) and n = 96 (2021 vOSCEs). With the exception of the medication history station, there were no differences in standardized assessment scores for individual stations offered as in-person OSCEs compared to vOSCEs. There was significant improvement in standardized scores between formative and summative evaluations for each of the OSCE assessments (4/4 stations; P < .001) and on several vOSCE assessments (4/6 stations; all P < .001 except for medication history [P = .33] and prescription verification [P = .14]). In total, 10/93 students required remediation of the 2019 OSCE summative assessment while 13/96 students required remediation of the 2021 vOSCE as shown in Table 3 . There were no statistically significant differences between the cohorts in need for remediation overall or by specific station.

Table 2

Student scores for OSCE and vOSCE assessments.

Station(points in station)	2019 formative OSCE(n = 93)	2019 summative OSCE(n = 93)	2021 formative vOSCE(n = 96)	2021 summative vOSCE(n = 96)	P value for summative OSCE vs. vOSCE
	Median (Interquartile Range)
#1 Medication History (100)	88.2 (81.5–94.6)	98.1 (95.2–100)a	96.9 (91.9–99.0)b	96.0 (93.0–98.5)b	.002
#2 OTC Counseling (100)	71.4 (61.9–78.6)	84.5 (76.5–90.0)a	75.5 (64.6–79.9)b	85.0 (78.5–90.5)a, b	.30
#3 Renal Dosing in Hospitalized Patient (19)	15.0 (13.7–16.3)c	18.0 (17.1–19.0)a, c	15.0 (14.1–16.0)	18.8 (17.5–19.0)a	.23
#4 Prescription Verification (6)	5.0 (4.0–5.5)	5.7 (5.2–6.0)a	5.5 (5.5–5.5) b	5.7 (5.3–6.0)b	.12
#5 DRP Identification and Solution (12)	Not offered in 2019	Not offered in 2019	9.5 (8.0–10.5)	11.5 (11.0–12.0)a	Not applicable
#6 Vaccine Needs Assessment (24)	Not offered in 2019	Not offered in 2019	18.0 (18.0–21.0)b	21.0 (21.0–24.0)a, b	Not applicable

DRP = drug related problem; OSCE = objective structured clinical examination; OTC = over-the-counter; v = video.

P < .001 using Wilcoxon Signed-Rank test within same OSCE type/year.

Ninety-five examinees' data included in analysis; one excluded due to absence of student on test day resulting in make-up exam.

Nintey-one examinees data included; two excluded due to computer difficulties resulting in paper exam.

Table 3

Students offered remediation by station.

Station Type	Students requiring remediation in 2019 OSCE (n = 97), n (%)	Students requiring remediation in 2021 vOSCE (n = 96), n (%)
#1 Medication History	0 (0)	1 (1)
#2 OTC Counseling	3 (3.1)	6 (6.3)
#3 Renal Dosing in Hospitalized Patient	7 (7.2)	4 (4.2)
#4 Community Medication Order	0 (0)	2 (2.1)
#5 DRP Identification and Solution	Not applicable	0 (0)
#6 Vaccine Needs Assessment	Not applicable	7 (7.3)

DRP = drug related problem; OSCE = objective structured clinical examination; OTC = over-the-counter; v = video.

Discussion

Our study demonstrated a minimal difference in P3 student assessment scores on formative and summative in-person OSCE vs. vOSCE in a clinical capstone pre-APPE course. These results help provide insight into the impact of a virtual learning environment for a skills-based assessment in a pharmacy curriculum and are the first report of a formative and summative vOSCE used in a targeted P3 high stakes, pre-APPE assessment contained in a capstone course. For optimal comparison, no changes were made to the original OSCE blueprints or processes except for additional transition time added to the vOSCE stations. The cohort of trainees evaluated in this study was large compared to other existing literature in high stakes situations. , Lastly, our study incorporated the use of live SPs in both in-person and vOSCEs; this is novel in the pharmacy literature to date.

Like VanLangen et al., we had mixed results related to P3 student scoring on select stations. In VanLangen's P3 cohort, there was no difference seen in OSCE vs. vOSCE groups on students scoring ≥80% on their one verbal station of identifying and overcoming barriers with a patient and one written SOAP note, but a difference existed on their second verbal station regarding a healthcare provider interaction. Similar to their study, a difference was seen between summative scores of OSCE vs. vOSCE on one of our communication stations (medication history). This difference, although statistically significant (median [IQR] 98.1 [95.2–100] vs. 96.0 [93.0–98.5], P = .002), was not a meaningful difference in student performance on the station. No difference was seen in our over-the-counter medication counseling station. Lastly, one significantly different aspect in our communication stations was that we used SPs in both OSCE and vOSCE; VanLangen did not and utilized faculty as both the evaluator and the simulated patient. In our study, one SP communicated with a student while either a faculty member or SP observed dependent on the station type (see Table 1).

Hsia et al recently conducted a similar study that compared first-year student performance between formative in-person OSCE and summative vOSCE for two stations. This study was mainly focused on student perceptions and themes through qualitative analysis, yet their brief quantitative analysis also showed no significant difference in pass rates among students, similar to ours. Some considerations for why there was minimal difference in student performance in our study between OSCE and vOSCE could be contributed to the consistency in the OSCE transition process with case information, detailed instructions provided prior to the assessments, the use of live SPs both in-person and virtually, and consistent faculty scoring. Although our students were naïve to the vOSCE learning experience, they likely developed more comfort with online learning and assessments due to the university requiring 100% remote learning beginning in 2020. The additional time that was given for the vOSCE may have also contributed to better performance on some of the stations due to reduced time management stress and enhanced transition between stations. In contrast to the results of the Hsia group, we did see significant improvement from formative to summative assessment within cohorts. The exceptions to this occurred during the 2021 vOSCE medication history station in which an appreciable ceiling effect was observed during the formative assessment.

A recent systematic scoping review of 11 studies assessing vOSCE performance assessment found that despite significant heterogeneity in vOSCE design (even within specific health professions), most assessments were valid and reliable for skill evaluation purposes. One study compared medical students' performance on vOSCEs within a pediatric clerkship in comparison to the previous three years of live OSCEs and found no differences in mean scores or failure rates. However, it is not confirmed whether the same OSCE assessment was utilized over the four years. Despite these challenges, several have reported that both health profession students and faculty are generally accepting of vOSCEs when necessary, including pharmacy students.10, 11, 12, 13, 14 , Interestingly, despite some studies finding that students encounter less stress while undergoing vOSCEs vs. OSCEs, one study found that students would prefer live OSCEs where possible. , However, another stated that while students saw value in the vOSCE process, they felt it allowed for adequate demonstration of verbal or counseling skills but not for building patient rapport or non-verbal communication.

With COVID-19 accelerating the development of virtual options in pharmacy education and patient care, utilizing vOSCEs may contribute to student pharmacists' skill development in interacting with patients and healthcare providers in remote settings. Telepharmacy has a long track record of augmenting patient care but it has been challenging to effectively teach in the traditional doctor of pharmacy curriculum. Prior to the declaration of a pandemic, an international group of pharmacy educators had already identified both digital health and telemedicine as core professional skills students will need to develop in order to be at the forefront of pharmacy practice. Given the rapid transition many pharmacy programs made to virtual or remote learning experiences, the need for student pharmacists to demonstrate competence in communicating with patients using telemedicine has increased. Providing opportunities for learning and assessment in this realm using communication-based vOSCEs may provide confidence for students and faculty. The ability of vOSCEs and other web- or phone-based patient care simulations to develop and/or improve specific skills in telehealth and/or telepharmacy is a significant gap in the literature needing to be filled, regardless of pandemics keeping people at home.

Limitations of our study included the inability to test-retest to determine reliability of each case given the short nature of the course. The two student cohorts may have been under varying levels of personal stress due to pandemic-related restrictions. Lastly, for the SP-based stations, interrater reliability was unable to be assessed.

Performing this evaluation has provided benefit to our program to determine if there was a difference between types of offering OSCEs vs. vOSCEs for pre-APPE assessment. From a student assessment standpoint, the grade impact remained neutral with the change, and it confirmed there was definitive benefit from offering a formative followed by the summative evaluation for most stations. It also caused us to put a greater emphasis on affording more practice to students in over-the-counter medication counseling given the spread between individual formative and summative performances in both types (10%–13% improvement) whereas high performance on others like medication history didn't need as much attention. In regards to technology use, no major differences were seen in regards to number of student issues experienced. Very few technical issues were experienced on the vOSCE communication stations (two experience: one related to initial loading of the virtual platform which was quickly resolved, the other individual switched to a different virtual platform) and no technical issues were noted on written stations. Lastly, a financial evaluation was completed finding the vOSCE cost more than the OSCE; however, this was primarily due to increased cost per hour of SPs and extra scheduling burden online. Due to stakeholder request and finances, the OSCEs have been moved to in-person again in 2022. However, knowing that a successful vOSCE process is possible, a potential opportunity exists for us to utilize such an assessment for telehealth related topics in the future as well as looking for opportunities to reduce the cost burden to the program. A future consideration for vOSCE cost reduction is sharing a time management web-based system with other programs within the university as described by Shaban et al.

Many questions remain unanswered regarding student performance on OSCE vs. vOSCE assessment. In future research, comparison of assessment types utilizing validated tools should be explored. Further research is needed to determine if a difference exists between scoring of live-observations vs. recorded video observations and of faculty vs. SP-based evaluations. To enhance comparability of results, cohorts of students could be matched based on academic ability. Lastly, potential cost analyses could be completed to determine “hidden costs” of each type of evaluation and the potential impact on the institution and the student over the course of a curriculum. These areas of future inquiry would benefit both health professions programs and students.

Conclusions

This was the first study to demonstrate no significant overall difference in student performance on in-person OSCE vs. vOSCEs in a P3 capstone course focused on pre-APPE student assessment incorporating stations utilizing live SPs. An improvement in performance was seen between formative and summative assessments for both OSCE and vOSCE. Based on these findings, pharmacy programs should continue to explore the potential advantages and disadvantages of different OSCE formats to meet their curricular needs.