Development and Conduct of Tele-Objective Structured Clinical Examination (Tele-OSCE) to Assess Clinical Pharmacy Competencies During COVID-19 Emergency Remote Teaching.

Restrictions in face-to-face contact during the COVID-19 pandemic have necessitated the conversion to online teaching-learning activities. To assess relevant competencies of a Master's in Clinical Pharmacy student cohort, an online tele-objective structured clinical examination (OSCE) was developed and conducted using Microsoft Teams®. Afterward, a survey was conducted to determine the acceptance of tele-OSCE by students and faculty members. Students' performance was also compared to a previous cohort that underwent face-to-face OSCE. The majority of students generally agreed that tele-OSCE was operationally easy to undertake (94.0%) and did not deter their exam performance. The majority of faculty members also generally agreed that the online platform did not deter the assessment of students' performance and a minority disagreed on the ease of assessing counselling tasks or calculation work (13.0%). There was no statistically significant difference in students' overall scores between the tele-OSCE and the face-to-face cohort (p > 0.05). A thoughtfully planned tele-OSCE is a feasible method of examination that allows acceptable assessment of attained clinical competencies when social distancing measures are mandated. Stakeholders should look towards fortifying IT and online platform access to support optimal emergency remote teaching.

Background

Clinical pharmacy services, which traditionally involve the provision of pharmaceutical care to hospitalised inpatients as a part of integrated patient-centred care, have now matured across all healthcare settings, including ambulatory and transitional care [1, 2]. A clinical pharmacist optimises drug therapy by ensuring rational and safe use of medicines [3]. Clinical pharmacy is a specialised field attained and polished during practice at the workplace. With the evolution of the pharmacy profession together with advances in the healthcare system and patient care, pharmacists are increasingly required to provide clinical pharmacy services such as medication review, medication reconciliation, drug therapy monitoring, drug use optimisation, and more recently, vaccination [4, 5].

To meet the standards of providing higher demands of patient management, practising pharmacists looking to polish their clinical pharmacy skills pursue relevant postgraduate studies such as Master’s or postgraduate diploma in clinical pharmacy in the UK, Australia, and Malaysia, or postgraduate residency programs in the USA and Singapore [6, 7]. Courses in the programme usually combine didactic teaching of pharmacotherapeutics with experiential learning at hospitals to instil knowledge and skills for professional practice in line with the FIP Nanjing Statements on Pharmacy and Pharmaceutical Sciences Education [8].

The Master’s in Clinical Pharmacy programme at our institution is a postgraduate degree in clinical pharmacy which aims to produce skilled clinical pharmacists who can lead and work with multiple healthcare team members and patients. Periodically, students’ clinical pharmacy competencies were assessed by objective structured clinical examination (OSCE). OSCE provides assessments of knowledge, skills, and behaviour in the context of simulated clinical encounters.

In March 2020, the World Health Organisation declared the coronavirus disease 2019 (COVID-19) outbreak as a pandemic affecting more than 100 countries, including Malaysia [9]. Soon after, the Malaysian government announced a Movement Control Order which required all institutions and business premises to operate remotely and residents to be confined to their own home, leaving only the essential services to operate [10]. This emergency thus interrupted academic sessions at the university. All planned face-to-face teaching and learning (TL) activities were abruptly changed to online activities which students undertake remotely from home. The all-inclusive use of available technologies during this situation was unprecedented to both academicians and students.

While OSCE traditionally has been conducted through face-to-face encounters involving simulated actors, assessors, and students in a simulated clinical setting, the Movement Control Order sanctioned social distancing and prohibited interactions requiring close proximity. Considering that any delay to the academic semester will incur a financial impact on the students and sponsorship bodies, proceeding with assessments online was deemed necessary. Therefore, this study aimed to describe the development and conduct of a tele-OSCE that can enable the assessment of the Master’s in Clinical Pharmacy students’ learning during emergency remote learning. In gauging whether the tele-OSCE could be implemented continually in the future, students’, assessors’, and simulated actors’ acceptance of the conducted tele-OSCE was also examined. Finally, this study also compared the scores of students who underwent the tele-OSCE with a previous cohort of students who underwent face-to-face OSCE.

Methods

Tele-OSCE Development

A tele-OSCE was designed and implemented by involved faculty academicians for semester 2 2019/2020 of the Master’s in Clinical Pharmacy program in response to the call for restriction of face-to-face TL activities as an effort to curb the COVID-19 pandemic.

The University subscribed to two main learning management systems to support TL activities, Moodle® and Microsoft Office 365®. Microsoft Office 365® was selected to conduct tele-OSCE because it features the Microsoft Teams® application that supports virtual videoconferencing and real-time sharing of documents. Microsoft Teams® can be downloaded and accessed using all smart devices, including tablets and smartphones. Several roles and tasks were established to ensure the smooth conduct of the tele-OSCE (Table 1).

Table 1

Roles and responsibilities of individuals involved during Master’s in Clinical Pharmacy tele-OSCE

Roles	Tasks
Tele-OSCE manager	Ensuring smooth flow of students at dedicated stations and troubleshooting of technical issues
Question provider	Preparing tele-OSCE question
Assessor	Assessing students during delivery of tele-OSCEs task
Simulated actor	Simulated actor for required roles during tele-OSCE simulated clinical encounter

Content experts on OSCE and clinical pharmacy led the discussion and workshops on tele-OSCE development. OSCE tasks were aimed to assess the psychomotor and affective domains of learning in outcome-based education, which was possible to be assessed using tele-OSCE [11]. The teaching team decided to exclude the assessment of the cognitive/knowledge domain of learning in this tele-OSCE because it was extensively tested in other forms of assessment such as quizzes and the final written examination. OSCE tasks were designed to assess understanding and skills on the following topics taught in the second semester of study: critical care, musculoskeletal and rheumatology, oncology, and special populations. The setting and the tasks required for completion at each OSCE station were decided based on the complexity of the tasks: whether it could be successfully attempted within 15 min and whether it can be tested and assessed effectively via tele-OSCE.

The focus of this tele-OSCE was clinical rationalisation and problem solving, communication skills, and professionalism which were in accordance with the course learning outcomes. An overview of the tele-OSCE tasks to be undertaken at each of the six stations is tabulated in Table 2.

Table 2

Description of OSCE tasks at each station and corresponding domain of learning

Station number	1	2	3	4	5	6
Course learning outcomes	To demonstrate competency in interpreting patient’s clinical and laboratory data and recommend appropriate management in critical care therapeutics	To explain the pathophysiology of common musculoskeletal diseases and their management principles	To demonstrate competency in interpreting patient’s clinical and laboratory data and recommend appropriate management in oncology therapeutics	To explain the pathophysiology of common cancers and their management principles	To demonstrate competency in interpreting patient’s clinical and laboratory data and recommend appropriate management in special populations	To explain the pathophysiology of common conditions affecting special populations and their management principles
Domain of learning	P	A	P	A	P	A
OSCE tasks and setting	Responding to nurse’s inquiries on inotrope dilution	Counselling on malfunctioning teriparatide injection device	Responding to physician’s inquiries on chemotherapy dosing calculations and monitoring parameters	Counselling on conception concerns in a patient undergoing chemotherapy	Responding to physician’s inquiries on aspirin therapy in paediatric Kawasaki disease	Counselling on anxiety and insomnia in an elderly

P, psychomotor; A, affective

For each planned tele-OSCE station, question providers with adequate background in the area were tasked to create questions, scripts for simulated actors, and marking rubric for assessors in word processor format. The questions and supporting documents were subjected to rigorous online vetting processes by the faculty academic at the department level and subsequently at a higher-level vetting. Afterward, the final version of the answer schemes was converted into online assessment forms using the Microsoft Forms® software for assessors’ use during the actual examination. The tele-OSCE manager conducted a final vetting of the transcribed online rubric with the Microsoft Word document to identify and rectify errors during transcription.

Two duplicates of tele-OSCE stations were created to maximise time and efficiency of tele-OSCE conduction. The decision on number of duplicates was based on both the number of students and the number of academic faculty with adequate background to perform the assessment of learning in clinical pharmacy. For a total number of 17 Master’s in Clinical Pharmacy students, 6 assessors who were clinical pharmacy lecturers and 6 simulated actors per OSCE station were appointed.

Two mock sessions of tele-OSCE were trialled prior to OSCE day. The first session was conducted to test the operational flow for each of the four roles involved in each tele-OSCE station: tele-OSCE manager, assessors, simulated actors, and students. The second session was conducted to assure the quality of online communication between assessors, simulated actors, and students, and finalising the tele-OSCE flow. Written feedbacks were informally obtained through emails and iterations to the final tele-OSCE operational flow were made. A briefing session with assessors and standardised actors was also conducted to ensure standardised assessments by the assessors and standardised delivery of performance by the standardised actors during the tele-OSCE to minimise assessment bias. Before the actual tele-OSCE, all involved faculty members and student candidates signed a declaratory form to preserve academic integrity as a deterrent for cheating.

Tele-OSCE Implementation

On tele-OSCE day, the OSCE manager, assessors, simulated actors, and students logged online into Microsoft Teams® and entered a virtual room created and titled “Master’s in Clinical Pharmacy Tele-OSCE” 30 min ahead of OSCE start time. All individuals are required to use their University Microsoft Outlook account for this procedure to ease communication and online tracking which was harder to achieve when using an external email (guest) account. Upon instruction, the OSCE manager, assessors, and simulated actors adjourned into an online conference call at designated “stations” which were virtual breakout rooms created on Microsoft Teams®. As scheduled, students received an invitation on Microsoft Teams® to enter the station at predetermined times. Students successfully entering into the OSCE station marked the start of the 15 min allocated for each OSCE task.

The first 2 min required the student to verbally affirm viewing of the tele-OSCE question on the shared screen and for students to prepare necessary information to complete the task. After 2 min have elapsed, the assessor announces the start of interaction with the simulated actor. An indicator was given at minute 8 via the “raise hand” function and an alarm sound. At 12 min, the assessor instructed the student to complete their final sentence and end the task. Verbal formative feedback on the student’s performance of the tasks immediately ensued prior to adjourning. The student was removed from the call at the end of 15 min and the cycle was repeated with the next students according to rotation.

Feedback and Performance Analyses

At the end of the session, online links to two sets of surveys were uploaded to the Microsoft Teams®. Students were asked to complete the survey on experience, ease of use, and acceptability of the tele-OSCE. The academic faculty and simulated actors were asked to complete the survey on ease of participation in the tele-OSCE, and ease in assessing or responding to the students. The survey employed a 5-point Likert scale (1 = strongly agree, 2 = agree, 3 = neutral, 4 = disagree, 5 = strongly disagree). The settings on the online survey form were adjusted to disallow submission if the survey was incompletely filled. Students’ performance during the tele-OSCE was taken as the scores obtained from each station and converted into percentages (%). The mean scores were calculated for each station and compared with scores attained during face-to-face OSCE in the previous semester. An independent t-test was conducted using IBM Statistical Package for the Social Sciences (SPSS) version 24.0 after normality testing revealed a normal data distribution.

This study was a part of the end of semester routine course evaluation undertaken to enhance teaching–learning activities and was exempt from requiring approval from the institutional research ethics committee. Prior to attempting the online survey, respondents were made aware that completion of the survey constitutes giving informed consent on the use of information and future intended publication. All data were treated confidentiality and anonymously.

Results

The tele-OSCE survey received responses from 17 students (Table 3) which amounted to a 94.4% response rate and 16 assessors and simulated actors which amounted to an 80.0% response rate (Table 4). The majority of students agreed that tele-OSCE conduction using Microsoft Teams® was easy to undergo and that they were able to communicate and perform the task without difficulties. One or two students experienced difficulty during tele-OSCE which arose from internet connectivity and malfunctioning audio during the conference call. The technical issue was solved by changing from a laptop to a smart device phone. The majority of students also agreed that the video function of Microsoft Teams® permitted eye contact with simulated actors and enabled projection of empathy in response to critical situations, which was important and valuable. The students also agreed that tele-OSCE is an important assessment to be carried out during crises and must be incorporated into the curriculum. The majority of students agreed that they were equally prepared to undertake tele-OSCE and face-to-face OSCE.

Table 3

Students’ response to tele-OSCE (N = 17)

Domain	Statements	SA (%)	A (%)	N (%)	D (%)	SD (%)
Ease of operation	I found the Microsoft Teams meeting easy to use	41	53	0	6	0
	I have a good understanding and skills on using Microsoft Teams prior to Tele-OSCE	35	47	0	12	6
	I need more technical training on using Microsoft Teams prior to Tele-OSCE	0	29	0	47	24
Technical difficulties	My Tele-OSCE interaction with the Microsoft Team software was clear and understandable	41	41	0	12	6
	I did not experience technical difficulties during Tele-OSCE	29	47	0	18	6
	I did experience technical difficulties during Tele-OSCE but it was solved easily	18	18	47	12	6
Ease of performance	I found the Microsoft Teams videoconferencing software made it easy for me to make eye contact with patient/simulated actor during Tele-OSCE	29	53	0	18	0
	I found the Microsoft Team videoconferencing software made it easy for me to demonstrate empathy during Tele-OSCE	18	41	0	41	0
	I feel equally prepared to undergo Tele-OSCE and OSCE in person	47	35	0	18	0
General attitudes towards tele-OSCE	I think Tele-OSCE is a good alternative to face-to-face OSCE during crises/pandemic	53	35	0	12	0
	I think Tele-OSCE must be incorporated into curriculum to prepare future use of telemedicine in health care sectors	35	53	0	12	0

SA, strongly agree; A, agree; N, neutral; D, disagree; SD, strongly disagree

Table 4

Assessors/actors’ responses to tele-OSCE (N = 16)

Domain	Statement	SA (%)	A (%)	N (%)	D (%)	SD (%)
Ease of operation	I found the Microsoft Teams meeting easy to use	50	50	0	0	0
	My interaction with the Microsoft Teams software was clear and understandable	38	62	0	0	0
Technical difficulties	I did not experience technical difficulties during Tele-OSCE	38	25	0	38	0
	I did experience technical difficulties during Tele-OSCE but it was solved easily	19	31	44	6	0
Ease of assessment	I found the Microsoft Teams videoconferencing software made it easy for me to make eye contact with students	13	69	0	13	6
	I found the Microsoft Teams videoconferencing software made it easy for me to feel empathy projected by students	6	63	0	31	0
	I found the Microsoft Teams videoconferencing software made it easy for me to understand patient education/counselling/instruction by students	0	88	0	13	0
	I found the Microsoft Teams videoconferencing software made it easy for me to get answers requiring calculation or dosage regimen	6	75	0	13	6
General attitudes towards tele-OSCE	I think Tele-OSCE is a good alternative to face-to-face OSCE during crises/pandemic	63	38	0	0	0
	I think Tele-OSCE must be incorporated into curriculum to prepare future use of telemedicine in health care sectors	44	56	0	0	0

SA, strongly agree; A, agree; N, neutral; D, disagree; SD, strongly disagree

Similarly, assessors and simulated actors found tele-OSCE using Microsoft Teams® easy to use and did not experience any major technical difficulties. Both simulated actors and assessors agreed that responses requiring counselling, calculation, and dosage regimen are clear through virtual videoconferencing. However, a few assessors and simulated actors disagreed on the ease of assessing students’ performance on counselling and calculation work. This may be reflected in responses from students (Table 3), where almost one-third of students felt they need more training on using Microsoft Teams® prior to tele-OSCE for best performance.

The scores obtained from each of the six OSCE stations contributed to a portion (10%) of the final grade to the following courses: P5242, P5252, and P5262. The average performance between the student cohort who underwent face-to-face OSCE and the current cohort who underwent tele-OSCE for the same courses was similar, with no statistically significant differences (Table 5).

Table 5

Comparison of students’ performance between face-to-face OSCE (2018/2019) and tele-OSCE (2019/2020)

Courses	P5242		P5252		P5262
Cohort	2018/2019	2019/2020	2018/2019	2019/2020	2018/2019	2019/2020
Maximum score	8.51	8.33	7.28	9.00	7.69	8.88
Minimum score	4.87	2.65	4.25	2.75	3.12	4.02
Mean score	6.22	6.40	5.86	6.79	5.24	6.42
Standard deviation	1.12	1.43	0.94	1.63	1.72	1.70
p-valuea	0.739		0.176		0.155

aIndependent t-test; P5242, pharmacotherapeutics IV (critical care); P5252, pharmacotherapeutics V (oncology); P5262, pharmacotherapeutics VI (special populations)

Discussion

The use of telemedicine platforms to conduct assessments of competencies such as OSCE has been in place for schools that offer distance learning [12]. In the advent of the COVID-19 pandemic, tele-OSCE was a logical strategy for remote conduct of assessment [13, 14].

Graduates must be able to achieve satisfactory levels of competencies in order to deliver clinical pharmacy care in healthcare settings. This is assured by wide-ranging domains of assessments rather than a singular cognitive domain assessed during the final written examinations. OSCE presented a platform for the assessment of higher-level competencies on psychomotor skills, attitudes, and professionalism [15].

Switching to emergency remote learning requires the use of resources at hand. Microsoft Teams® is a cloud-based application that allows collaborative activities within organisation such as group conversations, videoconferencing, and document sharing [16]. Since the start of off-campus learning in March 2020, both faculty and students had been using the platform to engage in TL activities. The tele-OSCE involved assessors, examinees, and simulated actors to interact together on an online platform from different remote locations. This design is preferred over utilising virtual simulated environment and patient avatars because of the nature of the clinical pharmacy program. Graduates must convey a high degree of professionalism in delivering patient care in which assessment of professionalism cannot be automated.

In this exercise, the tele-OSCE was systematically developed until its execution. Its success was attributed to proper planning, clear assignment of roles and responsibilities, and having a dedicated OSCE manager. A comprehensive guide for online OSCE conduction that was recently published outlined roles and responsibilities of various members that are pertinent to the successful conduction of an online OSCE [17]. The ability to source simulated actors, having standardised rubric of assessments, and incorporation of individual feedbacks within 15 min of allocated time for each student were key strengths of our tele-OSCE.

There is an overall general acceptance of tele-OSCE among the students and faculty members. The majority of students had experienced ease in operating the Microsoft Teams® and did not face difficulties in conveying information and response needed in completing the OSCE tasks. One to two students who did not have a good experience during OSCE were attributed to poor network connectivity despite not reporting a problem during the tele-OSCE trial. Indeed, the challenges of online learning and online assessment in this region lie with internet connectivity rather than technical barriers or such as IT illiteracy or lack of access to computer devices [18, 19].

The majority of faculty members generally expressed high agreement with operating convenience and assessment convenience during tele-OSCE using the Microsoft Teams® platform and Microsoft Forms® online rubric. They also generally agreed that the tele-OSCE did not deter assessment of skills-based competencies such as empathic communications, responding to inquiries that require dosage calculations or adjustments, and patient counselling.

Advantages from the online tele-OSCE were reduction in paper trails, less error in marks transcription, and ease in data archiving and data analyses. Similar advantages of online OSCEs had also been reported by other authors [20, 21]. The scoring rubric used by assessors during the tele-OSCE was downloaded as Excel form where scores were subsequently automatically calculated after the application of a predetermined formula. This enabled the course coordinator to quickly identify students’ performance for each tele-OSCE station.

Students’ performance between cohorts that underwent face-to-face OSCE compared to cohorts that underwent tele-OSCE was comparable. This indicates that tele-OSCE did not present any significant limitations to students’ ability to perform required tasks adequately. The tasks designed for tele-OSCE were similar in focus and nature as previous tasks assessed during face-to-face OSCE. However, due to the remote nature of tele-OSCE, the proper assessment of psychomotor skills during the tele-OSCE was limited to calculations on dosing or dilution because of logistical challenges in providing medical devices such as inhalers or injection pens to each student residing at different locations.

As the pandemic is currently ongoing into its second year, competency-based exams will continue to be conducted online. Thus, it becomes necessary for stakeholders to invest or improve existing IT platforms and IT support to enable fair, valid, and reliable assessment of students. This is important especially for healthcare students who require a certain achieved level of competencies in order to provide a safe practice. At the national level, good telecommunications reach, internet speed, and coverage are required for effective and equitable online learning for all. Therefore, upgrading of fibre optics network infrastructure is a worthwhile agenda to remain ready should future needs for emergency remote learning arise again [22].

This tele-OSCE was our online intervention to ensure our students able to graduate with similar capacity and quality of competencies required for the Master’s in Clinical Pharmacy programme despite the COVID-19 pandemic. The success was reflected by full commitment showed by all developing members from the department, extensive vetting process and adequate workshops, piloting the tele-OSCE, and similar students’ performances between both face-to-face and tele-OSCEs. In addition, we organised 2 pilot tele-OSCEs to gain more rich information directly from the students, assessors, and simulated actors. However, our postgraduate clinical pharmacy cohort has been traditionally small, and may limit generalisability to larger groups. At present time, this model of tele-OSCE had been adopted for competency assessment of other pharmacy programs offered in our faculty. Internal course evaluations also revealed a general trend of acceptance and minimal technological barriers encountered. Further directions for the tele-OSCE include training on online communication so that professionalism can be effectively maintained while delivering clinical pharmacy services to patients and health professional colleagues. Additionally, an online proctoring system will be considered to strengthen the integrity of the tele-OSCE.

Conclusions

An online tele-OSCE is a feasible method for assessing higher-level skills-based clinical pharmacy competencies when face-to-face OSCE is not possible. Future activities should incorporate telehealth practices in clinical pharmacy teachings including its development, training, and implementation. The tele-OSCE model of assessment is important when the country is preparing to embrace the 5th Industrial Revolution and is even more relevant in the present pandemic when proven to complement the present clinical pharmacy assessments.