Evolving to Objective Structured Clinical Exams (OSCE): Transitional experience in an undergraduate pharmacy program in Kuwait.

BACKGROUND: Objective Structured Clinical Exams (OSCEs) can assess professional competencies in a structured manner and facilitate objective evaluation of clinical performance. With limited data from the Eastern Mediterranean region, this study aims to describe the development, implementation, and evaluation of OSCEs for final year pharmacy students in Kuwait. The study also aims to compare students' performance in two academic years (2015-2016 and 2016-2017).
METHODS: The design, implementation, and evaluation of the competency-based OSCE followed a 3-phase systematic evidence-based approach. The development phase involved establishing an OSCE working group to develop a blueprint and scoring rubrics and to organise assessors and standardised patient/physician training. The implementation phase involved conducting formative and summative OSCEs. The evaluation phase involved undertaking student and staff perception surveys.
RESULTS: The overall students' OSCE scores for the academic years 2015-2016 and 2016-2017 were (median (interquartile range)) (71.6%, 32.2) and (60.0% (30.7)) and respectively (p < 0.0001). The average students' performance score was high in stations covering 'patient consultation and diagnosis' competency (71.4% (95% CI: 66.7-73.3)) and lower in stations covering 'monitoring of medicine therapy' competency (50.0% (95% CI: 33.3-66.7)). Students perceived stations covering 'monitoring medicines therapy' and 'assessment of medicine' as difficult. However, staff perceived stations related to 'patient consultation and diagnosis' competency as the easiest. Students reported that the OSCE was a positive experience as it provided them an opportunity to practice real life scenarios in a safe learning environment.
CONCLUSION: The OSCE helped to identify the level of competency of students prior to graduation and areas to improve in the curriculum.

Introduction

In the past decade, medical education (including pharmacy) has evolved to a more competency-based education to decrease the gap between education and practice. According to the Lancet commission, current professional education is moving at a slower pace than practice advancement and this has led to a gap between the graduates’ competencies and population needs (Frenk et al., 2010). The Faculty of Pharmacy (FOP), in Kuwait, understood the need to implement a competency based approach towards pharmacy education and to make necessary changes in the curriculum to close this gap.

There are several ways of defining competency based education (CBE) and it has multiple interpretations across the academic curricula (Gervais, 2016). CBE can be defined “as an outcome‐based approach to education that incorporates modes of instructional delivery and assessment efforts designed to evaluate mastery of learning by students through their demonstration of the knowledge, attitudes, values, skills, and behaviours required for the degree sought.” (Gervais, 2016) The “assessment efforts” require a framework, such as the Miller’s pyramid (Miller, 1990), to serve as a foundation for the application of an assessment tool to assess the clinical competencies of a pharmacy student. The Miller’s pyramid depicts the learner’s need and expectations as four levels, starting from the bottom to the top as; “knows”, “knows how”, “shows how” and “does”. The pyramid helps to differentiate knowledge acquired at the bottom levels and the competencies to be demonstrated or achieved at the top levels. It emphasizes the inevitable requirement to assess whether students have achieved the necessary competencies that pharmacy educators expect them to be delivered in the society after graduation.

There is a considerable increase in the need for pharmacists to evolve and expand their services beyond dispensing of medications. Pharmacists are expected to provide more complex and diverse patient care services such as prevention and management of chronic diseases, clinical pharmacy services, provision of anticoagulation clinics, immunization and other public health services. To provide these services, pharmacy students have to develop problem-solving skills, provide evidence-based decisions, have adequate communications skills to be an effective member of the healthcare team, develop the ability to review literature for drug information, and resolve drug therapy problems (Salinitri et al., 2012). Pharmacy educators require valid and reliable tools to assess these competencies.

With this evolution in education and service requirements, traditional methods of assessments such as multiple-choice questions, written answer examinations, and essay writing assignments have proven to be insufficient for educators to assess the competence of pharmacy students in real-life practice (Austin et al., 2003). The traditional methods are capable of assessing the lower two levels of the Miller’s pyramid of “knows” and “knows how” but are not structured and objective to assess clinical competencies of the higher levels. Traditional methods encourage assessors to employ subjective criteria for grading answers, and restrict the span of knowledge to course book content, which limits a student’s potential for development of professional competencies. This is a problem encountered with other healthcare professionals as well (Eldarir et al., 2010).

One method to address the challenges and drawbacks of traditional assessment methods is the use of Objective Structured Clinical Exams (OSCEs), which can assess professional competencies in a structured manner, and facilitate the objective evaluation of clinical performance at the ‘shows how’ level on the Miller’s Pyramid. The OSCE originated in medical education in the 1970s and has become a global gold standard to assess clinical skills, knowledge, and behaviors. Consequently, its use in other healthcare professions, including pharmacy, continues to grow (Hasan, 2016, Kristina and Wijoyo, 2018, Lacy et al., 2019, Lynga et al., 2019, Wardman et al., 2017). A literature review explored the characteristics of 14 publications of OSCE implemented for pharmacy education, and concluded that OSCE has received significant approval by pharmacy faculties and students all over the world as a reliable and valid tool for competency assessment (Kristina and Wijoyo, 2018).

The OSCE uses a series of stations which simulate various ‘real life’ clinical problems for students to be assessed on, and apply their skills. These stations can be manned or interactive where a patient or clinician actor interacts with the student while being evaluated by a faculty staff. The OSCE also provides a safe learning environment with no concerns of adverse outcomes as with a real patient. The feedback provided by the educator after the OSCE, enhances the students’ self-learning and improvement of competencies and acquire confidence for their future practice of pharmacy. Finally, the students’ performance in the OSCE raises the opportunity for educators to understand the changes required in the curriculum to improve learning and teaching (Ho and Lin, 2016).

In the Eastern Mediterranean region, few studies were published focusing on the application of OSCE in pharmacy education (Hadi et al., 2018, Sobh et al., 2017, Wilby et al., 2016). One study in Qatar explored the stakeholders’ perceptions of the value of OSCE and found that OSCE was of benefit to the students and generally accepted (Sobh et al., 2017). Another study in Qatar also described the development of OSCE for pharmacy students as part of a collaboration with the Canadian Council for Accreditation of Pharmacy Programs (CCAPP) and concluded that OSCE provided significant positive educational outcomes compared to traditional methods of assessment (Wilby et al., 2016).

The FOP at Kuwait University was established in 1997 and offers a 5-year bachelor’s degree in pharmacy (BPharm). In 2016–2017, a 2-year Add-on PharmD program was started, and in November 2020–2021, an Entry-to-practice PharmD program was implemented to gradually replace the BPharm as the main undergraduate pharmacy program at Kuwait University. Our faculty recognized the urgent need for a valid and reliable assessment tool to provide competent pharmacists to the society. In 2014, the Faculty of Pharmacy (FOP) at Kuwait University (KU), introduced OSCEs as a form of assessment for the undergraduate pharmacy students’ competency in their final year of the BPharm program in the ‘Hospital Professional Experience' course. This course consists of placements at sites in either hospital pharmacies or on the medical wards of hospitals. The general aim of this course is to develop specific skills in the areas of clinical pharmacy, problem-solving, medication history taking, medications dispensing, patient counselling, and therapeutic planning. The aim of introducing the OSCE to the final year assessment was for it to act as a capstone exam and to provide evidence of each students’ competence and readiness to enter the real-life practice. The OSCE constitutes 45% of the total course grade, and the remaining is allocated for hospital and pharmacy placement experience which includes workbooks and preceptor evaluation. A student is required to pass the OSCE in addition to the placement experience, which contributes to the total course score. The approved pass score for the whole course is 60%. The student passes the course if the total score is equal to or greater than the pass score.

The learning style has an influence on an individual’s approach towards a method of assessment and teaching. Educators have to be aware of this factor as it helps in choosing the appropriate strategies for learning. A cross-cultural survey explored the learning styles and multiple intelligences of Kuwaiti and Taiwanese students taking the English course as first language. This study revealed that Kuwaiti students are extroverted, interpersonal learners with visual intelligences and hands-on tactile skills (Wu and Alrabah, 2009). Pharmacy students of Kuwait may have an assimilating learning style enabling them to learn in a logical, student‐centred background or an accommodating learning style with a teacher‐centred approach. Finding the influence of the learning style or culture of Kuwaiti pharmacy students on OSCE performance is beyond the scope of this study. Over the years, our experienced educators had considered the learning styles of our students and provided the students with a competency based curriculum that caters to the current expectations of a professional pharmacist.

Currently, there are no published studies from Kuwait with regard to the use of OSCE in pharmacy education. The required local, contextual pharmacy services, education, and the OSCE settings implemented in this study sets it apart from other publications on OSCE from other regions of the world. With the evolution of our pharmacy program in Kuwait, it is important to reflect on past experiences in assessment methods that would prepare the future generation of pharmacists to be accountable and responsible additions to the workforce. This study aims to report our experiences in transitioning from a sole traditional method of assessment to an additional, objective method by describing the development, implementation, and evaluation of the OSCE during the academic years of 2015–2016 and 2017–2018 of BPharm pharmacy students in the FOP in Kuwait. The study also aims to compare two cohorts of the final year pharmacy students’ performance and their perceptions of the OSCE. We hope this study will encourage our colleagues in the Middle East, and other parts of the world to overcome the skepticism of OSCE and promote the use of OSCE as an assessment tool to enhance the learning experience.

Methods

The design, implementation, and evaluation of the OSCE followed a systematic and evidence-based approach (Austin et al., 2003, Harden et al., 1975, Shirwaikar, 2015). This study was approved by the Ethics Committee of the Health Sciences Centre, Kuwait University (Reference number: VDR/EC/2268).

Development phase

This phase started with the establishment of an OSCE working group to assess the feasibility of implementing a summative OSCE as a new competency-based assessment for the final year students. The working group included academic and academic-support staff from the pharmacy practice department. This group reviewed the relevant literature and examined the prevalent design, implementation, assessment, and grading system applied for final year pharmacy students (Austin et al., 2003, Corbo et al., 2006, Rutter, 2001, Shirwaikar, 2015, Urteaga et al., 2015, Wilby et al., 2016, Awaisu et al., 2007, Awaisu et al., 2010).

A blueprint was developed that outlined different OSCE stations and case description for each station (Table 1). As no local competency framework existed at that time for undergraduate pharmacy education, the FIP Global Competency Framework (GbCF) was adopted for the blueprint (International Pharmaceutical Federation, 2012). The GbCF is a mapping tool that has a foundation in the outcomes of initial education and training. The competencies included in the blueprint were ‘assessment of medicines’, ‘dispensing’, ‘monitor medicines therapy’, ‘patient consultation and diagnosis’, and ‘communication’. Clinical academic staff at the FoP were recruited to prepare and submit case scenarios that cover the selected competencies. The complexity of the cases and problems ranged from simple to complex, with intermediate complexity in between. All cases were developed using the English language. The cases were reviewed by the OSCE working group to ensure that the cases were realistic, relevant, and valid for a fifth-year pharmacy student. The OSCE working group members, all of whom are pharmacists, had good practice experience with expertise in the subject matter. Contents of each station were also reviewed by an external expert reviewer (who is a pharmacy practice staff with clinical experience). Formative OSCEs (see implementation phase below) were also used as a face validity check (Alkhateeb et al., 2019). Following the review, the working group determined the resources and logistic support required for setting up each station. The resources, design of assessment rubrics, and staff training were identical for both academic years. The difference was in the content of the case scenarios.

Table 1

Blueprint for OSCEs conducted in the academic years 2015–2016 and 2016–2017 and overall performance percentage for each competency with confidence interval (CI).

		Competency – Academic Year 2015–2016
No.	Station*	Assessment of medicines	Dispensing	Monitor medicines therapy	Patient consultation and diagnosis	Communication
1	Calc		SS			NA
2	CV1	SS
3	CV2	CC
4	DM			CS
5	Joint Dis1				CS
6	Joint Dis2		SS
7	Resp		SS
8	RoS1				SC
9	RoS2				SS
10	Womens’ health		SS
Students’ average score		62.8% (95%CI: 56.8–68.8)	69.9% (95%CI: 64.3–71.4)	50.0% (95%CI: 33.3–66.7)	71.4% (95%CI: 66.7–73.3)	100% (95%CI: 100–100)
		Competency – Academic Year 2016–2017
No.	Station**	Assessment of medicines	Dispensing	Monitor medicines therapy	Patient consultation and diagnosis	Communication
1	Calc		SS			NA
2	CV1			CC
3	CV2	CC
4	DM		SS
5	Joint Dis				CS
6	Prescrip		SS			NA
7	Resp		SS
8	RoS1				SS
9	RoS2				SS
10	Womens’ health			SS
Students’ average score		49.2% (95%CI: 42.5–55.9)	60.0% (95% CI: 55.0–62.5)	30.0% (95%CI: 14.3–40.0)	55.3% (95%CI: 52.3–58.3)	87.5% (95%CI; 83.3–90)

Complexity Levels: SS = Simple patient, simple problem; SC = Simple patient, complex problem; CS = Complex patient, simple problem; CC = Complex patient, complex problem NA = Not Assessed.

Calc = pharmaceutical calculation, CV1 = atrial fibrillation, CV2 = hypertension, DM = Diabetes, Joint Dis1 = gout, Joint Dis2 = osteoarthritis, Resp = Respiratory disease (asthma), ROS1 = responding to symptoms (sleep disorder), ROS2 = responding to symptoms (obesity), women’s health = patient education on.

Cal = Calculation (dose calculation and quantity), CV1 = angina, CV2 = anticoagulants, DM = Diabetes, Joint Dis = gout, Prescrip = Prescription (label and check RX accuracy), Rep = Respiratory disease (asthma), ROS1 = responding to symptoms (cough), ROS2 = responding to symptoms (acne), Women’s health = patient counselling on contraception pills.

For each station, a scoring rubric was developed. The scoring rubrics contained an analytical checklist and a global rating score. The checklist provided an objective and standardized assessment of clinical skills (accounted for 75% of the total station grade), and communication skills (accounted for 25% of the total station grade). The list included the tasks expected to be performed by the student and the level of competencies based on the student’s completion of the task. It was divided into three categories of competencies level: (1) fully competent: if the student completes the task, (2) partially competent: if the task is not entirely completed, and (3) not competent: if the task is not performed at all. In addition to assessing performance on rating scales, an assessor can record narrative comments about the strengths and weaknesses of a student’s performance. These comments can justify the assessor’s judgment as well as provide specific and timely feedback to the students for improvement.

The instructions to students and standardised patients/physicians were developed based on the tasks assigned at individual stations.

Training of assessors and standardized patients and physicians (SPs) for their respective roles in their assigned stations was organised before the OSCE with a thorough explanation about the purpose of the OSCE. This prepared them to provide consistent responses in their assigned stations. The training also included an explanation of the competencies to be assessed by using the analytical checklists. As all assessors and SPs were teaching staff at FoP (part time of full time) consent was not required. The OSCE working group also developed the ‘Pharmacy OSCE guide’ to outline the purpose, the need, and the required steps for effective implementation of OSCE in the faculty.

Implementation phase

A formative OSCE was conducted before the summative OSCE. This was done to ensure validity and reliability of the OSCE. The formative exam introduces the students to OSCE and reduces the stress they would have on the day of the summative exam. The formative-OSCE design involved a blueprint development that served as a guideline for the development and face-validation of the summative OSCE (Alkhateeb et al., 2019). This also helped the OSCE working group to identify any potential problems that might occur.

In both academic years, the summative OSCE included 10 stations (1 non-interactive and 9 interactive stations in 2015–2016 and 2 non-interactive and 8 interactive stations in 2016–2017). Each station covered one of the following four competencies: ‘assessment of medicines’, ‘dispensing’, ‘monitor medicines therapy’, ‘patient consultation and diagnosis’. Communication, as a competency, was covered in all interactive stations. Each station lasted for nine minutes. In each academic year, academic and administrative staff as well as pharmacists from the MoH were recruited to run the OSCEs. The assessors were academic staff at FoP. Standardized patients/physicians were either practicing pharmacists from the MoH who worked part time at FoP as Teaching Assistants or teaching staff from at FoP. Recruiting of standardized patients/physicians was based on their expertise and their availability on the day/time of the OSCE and pre-exam staff training. Faculty’s administrative staff were present to ensure a smooth run of the OSCE with adequate control over time and movement of the participants.

In both academic years, students were divided into 4 groups. Every two groups of students were assigned to two identical, but separate, circuit of stations that were run simultaneously. It was repeated for the other 2 groups on the same day.

Evaluation phase

After each formative OSCE, the OSCE working group organised feedback sessions with the assessor and standardized patients/physicians responsible for each station and the students. The feedback was intended to debrief the students on their performance. The performance in each station was reviewed with the whole class. After each summative OSCE, individual feedback with a student who was borderline or who had performed poorly was provided. The feedback session was part of using the OSCE as a diagnostic tool to guide student learning (Corbo et al., 2006).

A students’ perception survey was administered immediately after the summative OSCEs in the academic years of 2015–2016 and 2016–2017. The survey was adapted from Awaisu et al. (2007) and comprised of 18 items that assessed students’ perception on the following domains: OSCE setting, self-assessment of performance, perception of reliability and validity of OSCE, as well as the level of difficulty and time adequacy of each station. Students responded to the first three domains using a 5-point Likert-type scale from 1 to 5, with 5 representing the strongest agreement. Students also reported the level of difficulty and time adequacy of each station using a three-point scale (Difficult, Intermediate, Easy) for the level of difficulty and (Long, Adequate, Short) for the time adequacy. Open-ended questions were also included in the evaluation survey to better understand the student’s experiences. To ensure the face and content validity of the questionnaire, the OSCE working group reviewed the instrument and modifications were made to better fit the study context (Awaisu et al., 2007). The students were asked to fill the questionnaire voluntarily. No personal details (e.g., name or student number) were disclosed to ensure confidentiality.

Staff perception of the OSCE experience was also collected. A short feedback form was provided for each assessor and standardized patients/physicians at every station. They were asked to evaluate the task difficulty level (Difficult, Intermediate, Easy) and time adequacy (Long, Adequate, Short) for their station in addition to a comment section to add their suggestions and any observations.

Data analysis

Data were analysed using Excel and Graphpad Prism version (8). The normality test was tested by D'Agostino & Pearson test. Individual students’ OSCE scores out of 10 were non-normally distributed and were presented as medians and interquartile range (IQR). The median of OSCE scores within the academic year was compared using Kruskal-Wallias test for the years 2015–2016 and 2016–2017. The overall OSCE score was correlated with total course grade using Pearson correlation for normally distributed data and Spearman correlation for non-normally distributed data. Percentage total OSCE score out of 100 was calculated and presented as median (95% confidence interval (CI)) for each academic year. Mann-Whitney test was used to compare the median percentages total OSCE score between the academic years 2015–2016 and 2016–2017.

Clinical skill competencies were analysed separately from communication skills and presented as percentage competency scores. The percentage competency score was calculated as a continuous variable for each competency by dividing individual student’s scores by the total score (7.5 for clinical skill and 2.5 for communication skill) and multiplying by 100. The percentage competency score was then presented as mean (SD) for normally distributed and median (IQR) for non-normally distributed data. The percentage competency score was divided into three groups, not competent (score < 60%), partially competent (score = 60–80%), and fully competent (score ≥ 80%). Chi-square test was used to compare frequencies in each competency between the academic years 2015–2016 and 2016–2017. The significance value was set at 0.05.

Data from the student and staff perception surveys were analysed using frequency and percentages. Answers to open-ended questions were thematically analysed to reflect the positive and negative aspects of students’ experiences with the OSCE.

Results

Summative OSCEs were conducted for a total of 84 students for the two academic years of 2015–2016 (n = 44), and 2016–2017 (n = 40). The majority of the students were female (n = 70, 83.3%). Table 1 shows the OSCE blueprint that was developed for summative OSCEs in both academic years as well as the distribution of competencies assessed and the levels of complexities across the different stations.

Table 2 presents students’ level of performance for the total OSCE score and for each competency for both academic years. Overall (two academic years combined) the mean percentage OSCE score was 65.1% (9.45). Almost one third of student were not competent (scored < 60%) (n = 26; 31.0%; 95% CI 22.1–41.5), two-thirds were partially competent (scored 60–80%) (n = 51; 60.7%; 95% CI: 50.0–70.5%), and less than one tenth were fully competent (scored > 80%) (n = 7 (8.30%; 95%CI: 4.09–16.2)).

Table 2

Students’ level of performance for the total OSCE score and for each competency.

Performance level	Academic Year 2015–2016N (%)	Academic Year 2016–2017N (%)	TotalN (%)	P value
Assessment of medicines (n* = 88 vs 40)
Not competent	36 (40.9)	27 (67.5)	63 (49.2)	0.0115
Partially competent	26 (29.5)	9 (22.5%)	35 (27.3)
Fully competent	26 (29.5)	4 (10.0%)	30 (23.4)
Dispensing (n* = 176 vs 160)
Not competent	52 (29.5)	78 (48.8)	130 (38.7)	< 0.0001
Partially competent	78 (44.3)	69 (43.1)	147 (43.8)
Fully competent	46 (26.1)	13 (8.12)	59 (17.6)
Monitor medicines therapy (n* = 44 vs 80)
Not competent	27 (61.4)	64 (80.0)	91 (73.4)	0.0211
Partially competent	6 (13.6)	10 (12.5)	16 (12.9)
Fully competent	11 (25.0)	6 (7.50)	17 (13.7)
Patient consultation and diagnosis (n* = 132 vs120)
Not competent	45 (34.1)	68 (56.7)	113 (44.8)	< 0.0001
Partially competent	39 (29.5)	42 (35.0)	81 (32.1)
Fully competent	48 (36.4)	10 (8.33)	58 (23.0)
Communication (n* = 396 vs 320)
Not competent	12 (3.03)	25 (7.81)	37 (5.20)	< 0.0001
Partially competent	49 (12.4)	73 (22.8)	122 (17.0)
Fully competent	335 (84.6)	222 (69.4)	557 (77.8)
Percentage total OSCE performance
Not competent	6 (13.6)	20 (50.0)	26 (31.0)	0.0003
Partially competent	31 (70.5)	20 (50.0)	51 (60.7)
Fully competent	7 (15.9)	0 (0.00)	7 (8.30)

n = number of students in all stations covering X competency or frequency of competency assessed.

The overall median percentage OSCE scores for the academic years 2015–2016 and 2016–2017 were (71.6%, 32.2) and (60.0% (30.7)) respectively (p < 0.0001). Overall students’ mean percentage score in ‘assessment of medicine’ competency showed that students were not competent (58.5% (27.0)), where 49.2% (n = 63; 95%CI: 40.7–57.8) were not competent, 27.3% (n = 35; 95% CI: 20.4–35.6) were partially competent, and 23.4% (n = 30; 95% CI: 16.9–31.5) were fully competent. Overall students’ median percentage score in dispensing showed partially competent (64.3% (100)), where 38.7% (n = 130; 95%CI: 33.6–43.9) not competent, 43.8% (n = 147; 95% CI: 38.5–49.1) partially competent, and 17.6% (n = 59; 95% CI: 13.9–21.9) fully competent. Overall students’ median percentage of ‘monitor of medicine therapy’ competency score was not competent (37.9% (100)), where 73.4% (n = 91; 95%CI: 64.9–80.4) not competent, 12.9% (n = 16; 95% CI: 8.10–19.9) partially competent, and 13.7% (n = 17; 95% CI:8.74–20.9) fully competent. Overall students’ median percentage of ‘patient consultation’ competency score was partially competent (63.3% (86.7)), where 44.8% (n = 113; 95%CI: 38.8–51.0) not competent, 32.1% (n = 81; 95% CI: 26.7–38.1) partially competent, and 23.0% (n = 58; 95% CI: 18.2–28.6) fully competent. Overall students’ median percentage of ‘communications’ competency score showed that students were fully competent (100% (83.3)), where 5.20% (n = 37; 95%CI: 3.77–7.04) not competent, 17% (n = 122; 95% CI: 14.5–19.9) partially competent; 77.8% (n = 557; 95% CI: 74.6–80.7) above average.

Median students’ performance score was higher in stations covering ‘patient consultation and diagnosis’ competency (71.4% (78.6)) and lower in stations covering ‘monitoring of medicine therapy’ competency (50.0% (100)) in the year 2015–2016. On the other hand, in the year 2016–2017, students performed well in stations covering ‘dispensing’ (median: 60.0% (100)) and were the lowest in stations assessing 'monitoring of medicines therapy' (median: 30.0% (100)). Regarding communication skills, students’ overall performance in communication was moderate to high (87.5–100%) in both years.

Table 3 presents the median (IQR) for each station score for both academic years, Individual students’ score was significantly different across all stations (P < 0.0001) for the year 2015–2016 and 2016–2017.

Table 3

Median (IQR) for each station score in both academic years.

No	Station	Clinical skill scores 7.5 Median (IQR)	Communication 2.5 Median (IQR)	Total station scores 10 Median (IQR)	Station	Clinical skill scores 7.5 Median (IQR)	Communication 2.5 Median (IQR	Total station scores 10 Median (IQR)
	Academic Year 2015–2016				Academic Year 2016–2017
1	Calc	4.56 (2.06)	NA	6.08 (2.75)	Calc	4.13 (1.78)	NA	5.50 (2.38)
2	CV1	5.09 (4.15)	2.5 (0.83)	7.75 (4.25)	CV1	3.75 (2.00)	2.5 (0.44)	6.25 (2.50)
3	CV2	4.82 (3.08)	2.08 (0.83)	6.88 (2.81)	CV2	3.50 (1.50)	2.00 (0.50)	5.50 (2.13)
4	DM	3.75 (4.38)	2.25 (0.25)	6.00 (4.56)	DM	4.57 (1.69)	2.5 (0.31)	6.75 (1.94)
5	Joint Dis1	6.5 (1.50)	2.5 (0.19)	9.00 (1.75)	Joint Dis	4.75 (1.94)	2.00 (0.75)	7.00 (1.75)
6	Joint Dis2	5.36 (1.20)	2.5 (0.00)	8.00 (1.00)	Prescrip	5.63 (1.88)	NA	7.50 (2.50)
7	Resp	4.02 (2.28)	2.08 (0.83)	6.63 (2.44)	Resp	5.16 (1.41)	2.5 (0.31)	7.50 (1.50)
8	RoS1	5.36 (2.95)	2.5 (0.42)	8.00 (2.00)	RoS1	3.50 (1.19)	2.25 (0.5)	5.88 (1.50)
9	RoS2	4.29 (2.14)	2.08 (0.42)	6.50 (2.38)	RoS2	4.22 (2.03)	1.87 (0.63)	6.23 (1.84)
10	Womens’ health	5.89 (1.87)	2.5 (0.42)	8.50 (1.94)	Womens’ health	1.07 (1.47)	2.18 (0.42)	3.00 (1.38)

Fig. 1 shows the correlation between students’ OSCE scores against their total course grades for both academic years. The correlation was strong between OSCE score and total course grades in the year 2015–2016 with Pearson correlation coefficient r = 0.90, p < 0.0001, which indicated that scoring high in OSCE resulted in high course grades. However, the correlation was reasonable (moderate correlation) between the variables in the year 2016–2017 with a similar pattern (Spearman correlation coefficient r = 0.68, p < 0.0001). These results indicated that a high OSCE score may lead to a high total course grade.

Fig. 1

Scatter plot of overall OSCE student performance and their final course grade.

Student perception

For both academic years, all students completed the evaluation survey (Table 4). In the academic year 2015–2016, 43% of students (n = 19) agreed that the tasks covered in OSCEs stations reflected those covered in the curriculum compared to only 25% (n = 10) for the year 2016–2017. Moreover, almost half of the students (n = 24, 55%) in 2015–2016 and (n = 22, 55%) in the year 2016–2017 agreed that having ‘patient’ actors was realistic.

Table 4

Students’ responses to the OSCE evaluation questionnaire.

No.	Statement	2015–2016 (N = 44)			2016–2017 (N = 40)
		AgreeN (%)	NeutralN (%)	DisagreeN (%)	AgreeN (%)	NeutralN (%)	DisagreeN (%)
Assessment of OSCE setting
1	The OSCE stations were well signposted	24 (54.5)	12 (27.3)	8 (18.2)	21 (52.5)	7 (17.5)	12 (30)
2	The stations were well resourced for each activity	22 (50)	16 (36.4)	6 (13.6)	22 (55)	8 (20)	8 (20)
3	The actors (Patient/Physician) were believable and realistic	24 (54.5)	13 (29.5)	7 (15.9)	22 (55)	8 (20)	10 (25)
4	Having staff of faculty of pharmacy as actors was nerving	28 (63.6)	9 (20.5)	7 (15.9)	22 (55)	5 (12.5)	12 (30)
5	Having non-staff pharmacists as actors was nerving	8 (18.2)	12 (27.3)	24 (54.5)	4 (10)	7 (17.5)	27 (67.5)
Assessment of the Quality of Performance in the OSCE
6	I was fully aware of the nature of the exam	21 (47.7)	13 (29.5)	10 (22.7)	18 (45)	8 (20)	13 (32.5)
7	The tasks reflected those taught	19 (43.2)	16 (36.4)	9 (20.5)	10 (25)	16 (40)	14 (35)
8	The time at each station was adequate	21 (47.7)	15 (34.1)	8 (18.2)	10 (25)	9 (22.5)	20 (50)
9	The setting and context at each station was realistic with current practice.	22 (50)	15 (34.1)	7 (15.9)	18 (45)	7 (17.5)	15 (37.5)
10	The Instructions to perform each activity were clear	24 (54.5)	14 (31.8)	6 (13.6)	21 (52.5)	5 (12.5)	14 (35)
11	The tasks asked to perform were fair	19 (43.2)	12 (27.3)	13 (29.5)	11 (27.5)	8 (20)	21 (52.5)
12	The sequence of stations was logical and appropriate	19 (43.2)	14 (31.8)	11 (25)	17 (42.5)	10 (25)	13 (32.5)
13	The OSCE provided opportunities to learn real life scenarios	20 (45.5)	14 (31.8)	10 (22.7)	16 (40)	9 (22.5)	15 (37.5)
Perception of the Validity and Reliability of an OSCE
14	Passing or failing an OSCE provides a true measure of clinical skills in pharmacy practice	11 (25)	9 (20.5)	24 (54.5)	7 (19.4)	6 (15)	26 (65)
15	OSCE was a practical and useful experience	15 (34.1)	16 (36.4)	13 (29.5)	16 (44.4)	8 (20)	15 (37.5)
16	Personality, ethnicity and gender will not affect OSCE scores	17 (38.6)	20 (45.5)	7 (15.9)	25 (69.4)	7 (17.5)	7 (17.5)
17	Different patient actors will not affect student performance/ scores	21 (47.7)	8 (18.2)	15 (34.1)	16 (44.4)	7 (17.5)	15 (37.5)
18	Different assessors will not affect student performance/scores	16 (36.4)	13 (29.5)	15 (34.1)	14 (38.9)	7 (17.5)	18 (45)

Fig. 2 and 3 show students’ perception of the level of difficulty and time adequacy for the overall competency assessed at each station for both academic years. In 2015–2016 and 2016–2017, students rated stations covering ‘monitoring medicines therapy’ and ‘assessment of medicine’ as difficult (n = 24, 55.8% and n = 47, 54.7%; n = 51, 76.1%; n = 13, 41.9% respectively). In 2015–2016 more than half of the students (n = 293, 68.1%) reported that the allocated time for stations were adequate, compared to (n = 172) 52.1% of students in 2016–2017.

Fig. 2

The overall students’ perception of level of difficulty (left) and of time adequacy (right) of task as per each clinical competency for the academic year (a) 2015–2016 and (b) 2016–2017.

Fig. 3

Students’ perception of level of difficulty (left) and of time adequacy (right) of task in each station for the academic year (a) 2015–2016 and (b) 2016–2017.

Students reported that the OSCE was a positive experience as it provided them an opportunity to practice real-life scenarios in a safe learning environment. On the other hand, students reported that the OSCE was a “very nerve-wracking” and “stressful experience” as it may negatively affect their final grade. They suggested having more formative practical sessions for OSCEs early in the curriculum to get familiar with the process.

Staff perception

The number of staff who were assessors and standardized actors were 34 in the academic year 2015–2016, and 32 in 2016–2017. For each assigned station, they rated their perception of task difficulty and time adequacy.

Overall, the majority of staff rated the level of difficulty for the OSCE in 2015–2016 and 2016–2017 as intermediate (50.0% (n = 17) and 48.4% (n = 15), respectively). Moreover, the adequacy of time for the overall OSCE was found to be mainly “adequate” in both 2015–2016 and 2016–2017 (with 76.5% (n = 26) and 93.6% (n = 29), respectively).

With regards to the competencies assessed, half the staff (n = 17) rated ‘assessment of medicines’ competency in 2015–2016 as ‘intermediate’, while in 2016–2017 50% (n = 16) of the staff rated it as ‘difficult’ and the other 50% (n = 16) as ‘intermediate’ (P = 0.157). Tasks related to ‘dispensing’ were rated in both years as ‘intermediate/easy’ (with 60% (n = 20) rating it as intermediates in 2015–2016 and 50% (n = 16) as intermediate in 2016–2017). The tasks related to ‘monitoring of medicines therapy’ were in general rated as easy in 2015–2016 (75%, n = 25) and intermediate in difficulty (75%, n = 24) in 2016–2017 (P = 0.03). Similarly, tasks related to ‘patient consultation and diagnosis’ were rated as either easy/intermediate in difficulty by the majority of staff in 2015–2016, but mostly easy in 2016–2017 (72.3%, n = 23).

A few staff members of 2015–2016 commented on the OSCE experience. Generally, they had a positive perception and described it as a “nice experience”. One assessor reported lack of time in responding to symptoms station. The assessor suggested that more prompts are required from the actor with additional questions such as “what shall I do now” or “shall I buy your recommended medicine now?”, as most students might be “nervous” and forget to provide certain recommendations. It was also pointed out that students did not use their time appropriately as they “kept on reading references”, resulting in an incomplete task.

Discussion

With limited data from the Eastern Mediterranean Region, this study, to our knowledge, is the first that describes an evidence-based approach to the development, implementation, and evaluation of pharmacy OSCE in undergraduate education. One strength of the present study is the competency-based analysis (rather than topic-based analysis) of the OSCE stations. Findings from this study could enlighten educators to consider using OSCEs as assessment tools with special considerations related to the development of a competency-based blueprint, and scoring rubrics that emphasize the competency needed to be mastered in order to accomplish the tasks in each station. The researchers acknowledge that, although OSCEs have been named as the gold standard of clinical assessment (Onwudiegwu, 2018), limitations to their use still exist. This includes the compartmentalization of the clinical patient in each station, which defers it from mimicking real life scenarios (where patients are complete and not segmented). Moreover, shorter stations are not able to assess in-depth history taking or longer communication scenarios (Shirwaikar, 2015).

Although published research on implementation of OSCEs does claim that OSCEs require time, effort and money, our research showed that the design, implementation, and evaluation of the OSCE for final-year BPharm students in Kuwait was technically feasible (Onwudiegwu, 2018). The OSCE exam provided a broad assessment of competency, to ensure final year pharmacy students meet an acceptable level of competency before graduation.

The approach

Blueprinting is an important step as it helps in developing OSCE stations with simulated tasks that are relevant to practice and considers all the competencies to be assessed which are crucial to establishing the validity of OSCE (Shirwaikar, 2015). Having a blueprint clarifying the competencies needed to be assessed provided a clear road map for developing case scenarios and appropriate assessors training. The involvement of staff and experts in blueprinting also enhances the validity of the exam (Hijazi and Downing, 2008). In the present study, formative OSCEs were found to help anticipate problems related to staff training and organisation. Formative OSCEs served as a teaching and learning tool for better implementation of the summative OSCE. Moreover, students’ and staff evaluation utilised in this study provided insights on the perceived level of difficulty and time adequacy of each OSCE station.

Overall performance

The results from the present study indicate that students performed best in ‘communication’, ‘dispensing’ and ‘patient consultation’, and least in ‘assessment of medicine’ and ‘monitor medicine therapy ’competencies’ which are expected competencies from a graduated pharmacist with BPharm degree. This was consistent with other studies that showed students performed best on stations which depend highly on communication skills and least on calculation and problem identification and resolution type stations (Corbo et al., 2006, Awaisu et al., 2010).

Performance comparison between two academic years

There was a clear difference in performance between the two academic years across all stations. This illustrates the impact of different students’ abilities and maybe teaching across different courses. These factors are difficult to assess. However, stations that assessed ‘monitor of medicine therapy’ competency, consistently had the lowest performance level in both academic years. This was clear from the students’ perception survey, where ‘monitor of medicine therapy’ was rated as difficult stations in both academic years. These results indicated that the current curriculum prepares pharmacy graduated to do traditional pharmacy roles rather than clinical roles which are needed for expanding the future pharmacy profession. These findings have highlighted possible areas of deficits in students’ knowledge and skills, and/or deficiencies in clinical training (Awaisu et al., 2010). The current results provide insight on strength and area to improve in the current pharmacy program and facilitate future curriculum development to focus on the clinical competencies expected from pharmacy graduates.

Correlation between OSCE scores and total course grades

Several studies showed that OSCE score could be a predictor of students’ final marks (Corbo et al., 2006, Rutter, 2001). This is consistent with the results of our study over the two academic years that were analysed which showed that students’ OSCE scores correlated well with their total scores for the course.

Results from the students’ perception questionnaire showed that the majority of students found the time allocated to OSCE stations adequate. This was found consistent with other published research (Branch, 2014). However, their perception of the level of station difficulties varies. Students rated stations related to ‘dispensing’ and ‘patient consultation and diagnosis’ as least difficult. This may indicate that students’ communication and patient-care skills were sufficiently developed, and therefore, the confidence in the student–patient interactive stations were not affected by the nerves and stressful exam experience. This was supported by findings from other studies in the literature (Branch, 2014, Aranda et al., 2019). As in the present study, previous research reported that students generally say that the OSCE provides useful practical learning experience with an opportunity to encounter real-life scenarios (Awaisu et al., 2007, Corbo et al., 2006, Branch, 2014). In the present study, students reported that the OSCE was a very stressful assessment method. A negative effect of OSCE-related anxiety on student performance was also reported in a cross-sectional survey of fifth-year Doctor of Pharmacy (PharmD) students in Saudi Arabia (Hadi et al., 2018). The findings of the present study are in agreement with other reports that elucidated that many students felt that the OSCE was an extremely anxiety-producing examination and should be introduced earlier in the curriculum (Awaisu et al., 2010, Awaisu et al., 2007).

The perception of staff varied as they were not aware of the tasks in all the stations. Their reported perceptions are related to their assigned station. The study is not meant to provide a meaningful comparison between student and staff perception. However, staff perception is relevant as they contribute to positive student outcomes (Regan et al., 2014). For both academic years, the difficulty was reported as intermediate by half of the staff, and a majority of staff considered time was adequate. This is encouraging as their positive perceptions and comments, is very important to ensure credibility and acceptance of OSCE by the faculty. There are insufficient well-designed studies that explore staff perception for pharmacy OSCE. Medicine faculty have reported staff perception and depict a positive perception for the implementation of OSCE for their students (Majumder et al., 2019). According to a survey in Egypt, more than 80% of the staff agreed that the time allocated for the medicine OSCE was sufficient (Gouda et al., 2019).

Among the competencies assessed, patient consultation and diagnosis was rated as the easiest. This is identical to the students’ perception. This similarity adds evidence to the fact that our students are adequately competent to provide patient counseling and consultation. The staff of the academic year 2015–2016 considered assessment of medicines as intermediate, whereas half of the staff of the academic year 2016–2017 reported this competency as difficult. This brings out the need to focus on the development of this competency through sufficient changes in the education curriculum. There is scope, in the future, for the investigation of the effect of staff perception on OSCE assessment, and student outcomes in our faculty. There was a significant difference in rating of the monitoring of medicines competency. We speculate that this difference could be attributed to the case design and different staff (assessors and actors) background.

Despite raising a concern about time by an assessor for one station, students perceived that time was adequately provided. Moreover, the time in the blueprint was set based on the experience with the formative OSCE with the same batch of students in every academic year. The need for further questions to help the student depends on the complexity of the case. Too many prompts and increased time will reduce complexity. Also, it will render the case inefficient to detect the competency level of the student. It is reported that changes, such as increased time, will not affect a student’s performance. (Stowe and Gardner, 2005)

Recommendations

The OSCE working group had generated a set of recommendations to ensure that students benefit the most from their OSCE experience and are better prepared for practice. These recommendations included: i) students need competency training during earlier years; ii) ‘Mini OSCEs’ to be introduced in 2nd, 3rd, and 4th year practical labs; iii) more staff needed for the committee (academic and support staff); iv) and a final OSCE to be held as an exit exam at the end of the academic year to ensure that graduates are prepared for practice. In the current era of pandemics, such as Covid-19, online OSCEs could be at the forefront for competency assessment (Major et al., 2020).

Limitations

This study has some limitations. First, two parallel OSCEs were conducted and students’ performance was assessed by trained examiners (faculty members) using standardized checklists in each of the OSCE stations in order to achieve high inter-rater reliability. However, no data were collected to support the sufficiency of this in ensuring the validity and reliability of the examination and to determine whether high inter-rater reliability of scores was achieved. Future research could aim at testing OSCE’s using procedures such as internal structure evaluation (ie, inter-rater reliability assessment). Second, activities were not videotaped, this was mainly due to logistic reasons. The OSCE ran in different rooms and videotaping stations was not possible with the available resources at the time. Videotaping OSCE stations would have helped with providing constructive feedback to students as well as staff involved in different stations, and this could be implemented in future work. Third, students and staff perceptions were collected using questionnaires that mainly generated quantitative data. Future studies could involve other methodologies such as focus groups with students and staff to provide in depth-understanding of their experiences. Finally, the order of stations as well as the precision of simulated patients and their effect on students’ performance were not assessed in this study, this could be an area for future work.

Conclusion

Our experience with OSCE has fulfilled the faculty’s need for a reliable and valid competency assessment tool. However, OSCE requires significant time, effort, staff, and monetary resources. The OSCE helped to identify the competency level of our students prior to graduation and the areas that need to be improved in the curriculum. There is a growing need for undergraduate pharmacy assessment methods that are competency-based to prepare competent pharmacists for future practice.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.