Delivering a modified continuous objective structured clinical examination for ophthalmology residents through a hybrid online method.

Since coronavirus disease 2019 was declared a global pandemic by the World Health Organization, it has become a challenging situation to continue medical education, including in Indonesia. The situation prohibited face-to-face (direct) educational activities in clinical settings, therefore also postponing examinations involving especially procedural skills. Adaptations were urgently needed to maintain the delivery of high-stake examinations to sustain the number of ophthalmology graduates and the continuation of eye health service. Objective structured clinical examination (OSCE) has been one of our widely used method to assess clinical competencies for ophthalmology residents, and is the one method that involves gatherings, close contact of examiners, examinees and patients, therefore the most difficult to adjust. Pandemic challenges brought technical changes in our delivering the OSCE to online, maximizing digital platforms of meetings, while still concerned to guarding the safety of candidates, patients and staffs. OSCE scenarios were also made as timely efficient as possible by changing continuous station models to a cascade one. The purpose of this article is to document our experience in conducting a feasible and reproducible OSCE in this pandemic era filled with limitations.

Introduction

The ophthalmology residency program in Indonesia refers to the principles and guidelines of the International Council of Ophthalmology. The main purpose of our program is to establish qualified ophthalmologists in academic and professional scopes to serve the community. Therefore, residents in training are required to pass a national high-stake board examination arranged by the Indonesian College of Ophthalmology (KIKMI) in order to become board-certified ophthalmologists. KIKMI has adapted the use of high-stake examinations to assess knowledge and clinical skill competencies of ophthalmology residents in their final year, and its outcome as a tool to assess program quality and accountability of each residency training center [1].

Indonesia has 12 residency training centers widely distributed throughout the nation, on different islands and provinces. Three means of examinations that are adapted by KIKMI in the national board exam are written tests, portfolio or interview, and objective structured clinical examination (OSCE). These examinations, further referred to as the national examination, is arranged by a national committee consisting of faculty members from accredited institutions across Indonesia. National examinations are regularly held 3 times a year, in April, August, and December, and traditionally takes place in one national center requiring all participants (examiners, examinees, committees) to travel to the appointed location. This centralized method has become impossible due to the limitation of travelling during the pandemic era. Before the coronavirus disease 2019 (COVID-19) pandemic, OSCE was conducted with more than 10 stations comprising of history taking, physical examination, diagnostic and clinical reasoning, performing surgical procedures, and doctor-patient communication, each for 15 minutes per station. A minimum of two break stations were usually added to the total circuit. Each station begins with different case scenarios, equipped with different tools according to the task given, and all clinical cases and tasks are made based on blueprints established by the committee months before examination. Scoring was usually done with paper-based rubrics consisting of clinical points to be assessed (score 0–3), and Global Rating Scale (GRS).

However, since the World Health Organization declared COVID-19 as a pandemic in March 2020, the whole continuance of medical education processes has become immensely challenging since face-to-face sessions and direct educational activities in clinical settings were restricted. Examinations were also postponed throughout the country causing final year residents to face the possibility of delay in completing their graduation. Ensuring a sufficient workforce in the pandemic era became an issue since delays in qualification completion would result in shortage of at least 100 ophthalmologists per year in providing health services across Indonesia. Therefore, every institution massively shifted from direct teaching and assessment to modified online adjustments, while several adaptations in a short timeframe were needed specifically to carry out end exams, increasing the challenge.

Among the modes of examinations, OSCE in our experience was the most difficult to implement due to its flow complexity, the necessity for direct interaction, and need for a well-thought scenario to create a reliable setting for assessing comprehensive clinical competencies. This article is intended to document our experience in preparing and conducting an online national examination for clinical skill assessments in a pandemic era for ophthalmology residents in Indonesia, using a continuous station model while prioritizing the safety of all participants affiliated in the examination.

Methods

1. Planning

In response to pandemic, the Indonesian government has limited traveling and prohibited large gatherings. Therefore, conducting a centralized national examination became impossible since the 12 training centers are distributed on different islands/provinces as aforementioned. This shifted our national examination to be decentralized to each center, monitored through an online video conference platform (Zoom; Zoom Video Communications Inc., San Jose, USA). It was also decided to reduce the examination from 3 times into twice a year (June and December 2020) considering the challenge of modifying from offline to online with such limited time and resources. Time needed to prepare and modify national examination was approximately 2 months, but it took another extra month to prepare for OSCE.

The process started from literature searching on studies conducting online examination models, followed by a series of online brainstorming sessions to create a design that can ensure the validity and reliability of process, and also determined which clinical skills were to be assessed in OSCE. These skills would have to conform both to the standard of competence blueprint which referred to the KIKMI curriculum, and also the online feasibility.

To minimize the risk of infection and ensure the safety of participants, several key principles were adapted following recommendations from previous studies [2,3]. First, avoid group gatherings as much as possible. Briefing and meeting regarding exam preparations were conducted through online video-call conferences. In designing OSCE stations, the number of participants had to be limited in each participating institution, including the committee and standardized patients. Mannequins or simulators for assessing technical skills were used in place of standardized patients whenever possible. Second, safety protocols were implemented at all costs, e.g., hand sanitizers were provided in every room, participants were encouraged to use proper personal protective equipment (N-95 masks and gloves), and physical distancing were implemented throughout the process. Each training center should conduct the examination in clinical skills or practice lab centers instead of hospitals. Before starting the exam, it was imperative to perform a quick screening to all participants through brief history-taking and measuring body temperature; while during examination, all participants were reminded to wash their hands before and after performing each station to minimize contact exposure from equipment used in the examination room. Third, several studies also advised to minimize the number of stations in one circuit and also to use a cascade method instead of circuit, to reduce contact between examinees [2,3]. Finally, we implemented digital-based scoring method instead of the usual paper-based.

2. Station and room design

The pre-pandemic OSCE that consisted of 12 stations in circuit method with dissociated clinical scenarios and tasks were adjusted to a cascade method that reduced the time needed for mobilization between rooms/station (Fig. 1). The number of stations itself were reduced to five without breaks in-between, and the clinical scenarios were connected between the now virtual “stations”. Given 10 minutes for each station, each examinee will finish in 50 minutes (maximum 60 minutes). Standardized patients were limited to only one station to minimize human resources, while history-taking or doctor-patient communication skills were simulated by the committee through Zoom.

Fig. 1.

The Pre-pandemic OSCE Design and the New Arrangement

The pre-pandemic objective structured clinical examination (OSCE) (top picture) design needed 12 examination rooms for 12 active stations and 2 break stations, with participants moving from station to station (2 minutes for moving in-between); hence an extra of 24 minutes for mobilization only. The new arrangement (bottom picture) where stations are relatively virtual only required 1 examination room with the participant moving only between tables of equipment within the room. Examination room is only limited to examinee, local committee with mobile phone acting as mobile camera, and one standardized patient for ultasonography station.

Institutions were to prepare an examination room and another two isolation rooms for pre- and pos-texamination, with seating arrangements distanced 1 meter apart. One examination room was to accommodate a maximum of four examinees. These rooms were equipped with multiple stationery cameras to monitor the room. The examination room had additional mobile audiovisual devices and computer for communication and displaying questions. Each camera was linked to individualized Zoom accounts according to its function, and all were supervised by the national committee.

The breakout room feature from Zoom was a very useful tool for conducting this parallel design of exam; it allowed us to create separate rooms within one Zoom meeting, therefore several examination rooms could exist simultaneously whilst easily being monitored by the national committee. Each breakout room was dedicated for one examination room from participating institution. It had been arranged that the examiners and examinee in one breakout room must not come from the same institutions to maintain objectivity.

3. Briefing

Before informing the modified design of examination to participating institutions, we performed a pilot simulation to evaluate the likelihood of station implementations, and identify areas of improvement regarding the method or tasks given. The whole examination process used the break-out room features from Zoom, where each breakout room consists of all cameras in the examination room from one institution and one central person in charge (PIC) to guide the simulation (Fig. 2). Therefore, we required all institutions to maximally accommodate a reliable internet connection to prevent interruptions during the examination; however, unstable internet connection was unfortunately the most inevitable factor.

Fig. 2.

Schematic Picture of Zoom Breakout Room in Each Institution

Both local person in charge (PIC) and examinee use smartphone to ensure mobility inside the room without losing audio, and ensuring flexibility on taking close-up shoots of the examinee’s performance when needed by the examiners. All laptops have to be provided with webcams. Hence all gadgets are linked to zoom and the whole process of examination is recorded and documented.

After refining, all participating institutions were informed on logistics obligated to be provided in the examination room. Multiple stationery cameras were intended for monitoring and should reveal the whole room from different angles. We also required two smartphones that will serve as mobile cameras and audios to function in (1) shooting close-ups of resident’s performance and as (2) microphones. All cameras mentioned were individually connected via Zoom to the national committee. Lastly, each institution was to assign at least two local staffs to aid in the process, further referred to as local PICs, who played significant roles in ensuring a seamless flow locally and to communicate with the appointed representative from the national committee (referred to as central PICs) need there were queries the examination. Written guidelines were generated for both PICs to elaborate their roles.

Days before the examination, examiners, examinees, and PICs were briefed separately, regarding the examination flow, group arrangement of examinees and examiners, and exam schedule. There was also one local PIC assigned to supervise and regulate the examination flow in pre-examination isolation rooms.

4. Rehearsal

One day prior the examination date, all centers partook in a rehearsal, simulating the real setting, without the examiners and examinees to maintain confidentiality. During rehearsal, central PICs ensured several important points: first, all tools and equipment in the examination room accords to the list and work properly; second, camera connectivity to Zoom were confirmed and able to reveal the entire room, and audios through mobile phone’s microphones were clear. Mobile cameras and audios were addressed by two separate mobile phones and were rehearsed by local PICs to follow and show close-up shots of examinee.

Central PICs must also ensure the locations of isolation and examination rooms were distant enough to prevent contact or communication between examinees. Unstable internet connection had to be corrected to prevent delays during examination day.

5. Scoring

Participating examiners were appointed by KIKMI and are faculty members in ophthalmology residency training institutions throughout Indonesia, who have enrolled and passed national examination examiners workshops held by KIKMI. For this online adaptation, we recruited examiners who have already experienced in assessing during the pre-pandemic era, and were familiar with the scoring methods. All examiners had been contacted individually via e-mail to obtain consent and a signed integration pact. Previous paper-based assessment rubrics for OSCE were converted into digital scoring systems using Google forms. Each rubric had three assessment points and were scored from 0 to 3, plus a GRS consisted of “fail”, “borderline”, “pass”, and “superior”. To prevent scoring breach, the links to these digital forms were only distributed 1–2 minutes before the beginning of each station on examination day, one form for each station.

6. Evaluation of the feedback on pilot examination

Proper evaluation is needed in order to establish more efficient examinations in the future. We composed anonymous feedback questionnaires for all examination participants including examinees, examiners, and head of ophthalmology resident programs from each institution (Appendices 1–3). These questionnaires were distributed shortly after the examination to reduce recall bias.

Results

1. Delivering the examination

There were 12 institutions participating in our online OSCE, 70 examinees and 36 examiners (18 pairs). Depending on the number of examiners, in our setting the examinees were divided into a group of (maximum) four persons for each pair of examiners. Since one examination room can only accommodate a maximum of four examinees, any center who had more than four participating residents were requested to prepare more rooms by-fold.

Approximately 1 hour before examination and after entering breakout rooms, the central PIC shortly briefed the examiners on tasks and what to assess in each station, conforming perception, and how to score using the digital scoring system. Each examiner will produce individual scorings.

Mostly, participating institutions used teaching room or clinical skills room for the examination room and library or discussion room for isolation room. For administrative purposes, the national committee will list examinee’s attendance in pre-exam isolation rooms by conforming ID cards to examination cards, while strictly reminding again on pandemic health protocols, and avoiding interactions between examinees throughout the examination.

2. Continuous station

Considering the large number of participants and to reduce the duration of participants being in the physical examination room, the examination time needs to be very efficient. Therefore, instead of the previously unassociated OSCE case contents with one scenario for one station, the case itself has been modified into one overall case but with different tasks for each station that runs in accord to the patient’s progression of disease. Creating a continuous linked one-patient scenario. The scenario and tasks were compiled into one PowerPoint presentation (Microsoft Corp., Redmond, USA) and shown slide-by-slide operated by the central PIC from national committee. All tasks had been adjusted to assess both knowledge and dexterity of the examinee. An example of the continuous scenario is as seen in Fig. 3.

Fig. 3.

An Example of Sequential Objective Structured Clinical Examination Station Slides Containing Tasks and Instructions

After confirming the identity of the candidate, the case scenario begins from (A) and finishes at (H), at a maximum of 60 minutes/examinee. (A, B) shows an introduction clinical scenario of a patient with corneolimbal scleral rupture. (C-F) With the clinical scenario dynamically changing, examinees are assessed on procedural skills either on an animal eye or standardized patient while simultaneously describing the steps one-by-one, and/or assessed on interpretation, diagnostic or management skills (i.e. antibiotic regiment decisions along with the intravitreal injection procedure). Animal eyes (wound structure) are always prepared fresh by local person in charge (PIC) who have been briefed through standardized briefed instructional videos. (G) Lastly, as the clinical scenario progressively worsens, the skill on patient education and delivering bad news is assessed with the central PIC as the standardized patient.

After completing all OSCE stations, the examinees left the examination room to proceed to the post-examination isolation room. Examiners were required to submit a completely filled digital assessment form, which will be cross-checked by the central PIC, before allowed to proceed to the next examinee. At the end, all assessment forms from all examiners were compiled into one Excel sheet (Microsoft Corp.); data will be analyzed using a borderline regression method to set the passing grade for each station.

3. Feedback on exam implementation

Most examiners, examinees, and institutions have reported that the briefing instructions were very informative and felt the mechanism of modified examination was appropriate during pandemic era. All examiners also felt that central PICs were very accommodative and thorough in aiding the examination flow. No one was having a difficulty in using digitally converted scoring system via Google form (Google LLC, Mountain View, USA). Some examiners had difficulty in assessing due to poor image quality and recording technique by local PIC. The implementation of modified OSCE was found to be successful by the residents as well. The technical implementation and difficulty were found to be at a good level. With this continuous manner, residents have reported that the examination flow was easier to follow. Some technical issues in diagnostic tools (e.g., broken ultrasonography [USG] machine, blurred microscope lens) and time management were considered as source of stress. There was also internet problem which caused interruption in video interpretation. Overall, residents were grateful to take part in the examination. They also appreciated our effort to conduct modified OSCE in the exceptional circumstances during pandemic era.

Discussion

In this study, our online method was shown to be comparable to the pre-pandemic offline method in terms of exam’s objectives (procedural skills and knowledge), passing results, and scoring aspects, except for technical preparation and the need for painstaking supervision to each participating institution. Ten out of 70 examinees were declared “failed” to pass the exam. This number showed no significant difference between pre and pandemic OSCE, where the number of passing examinees were around 80%–90%. In this study which involves a high-stake examination, validity and reliability are maintained since the content of exam still assesses procedural skill exams representing the curriculum and adequate number of stations in one circuit, unchanged regulated scoring rubrics, also the use of trained examiners [4]. It is also more cost-effective since the expense for travel to a centralized exam location is not needed, therefore favoring the travel limitation, and technically feasible to be conduct.

As we wanted to keep aspects from conventional OSCE mechanism, a modified face-to-face combined with virtual format was our best option. It was quite a challenge to modify the examination mechanism in short time with a lot to consider and there were limited studies around online clinical assessment for high-stake examination. However, it was recognized as effective in several studies. For example, Blythe et al. [5] held virtual OSCE in which nine medical students participated and there were no significant concerns regarding exam delivery and metrics. Hytönen et al. [6] also successfully implemented modified online OSCE in four dental institutes of Finland, and had positive feedbacks from participating examinees. Despite the similarities, those studies did not assess clinical procedural skills that might be challenging to execute virtually.

We used the blueprint for conventional OSCE to develop the stations and scenario, this ensured comparable examination content. Continuous scenario was attributed to disease progression that involves multiple fields of ophthalmology; in this case refractive surgery, external eye disease, and vitreoretinal diseases. To limit the number of human resources, we only used one standardized patient for USG station and central PIC were assigned to perform roleplay at doctor-patient communication skills assessment. Because the examination was held in individual institution, we realized that there was different technical preparation in each institution. They might have varying tools (e.g., surgical microscope), examination room setting, information technology (IT) staffs’ capability, and internet stability. Even though we had informed the recommended internet bandwidth in our briefings, we still cannot predict the stability in each region. Institutions located in Java Island almost had no connection problems during examination. But those located outside Java had multiple episodes of unstable connection. Previous studies stated the same difficulty. In the implementation of online OSCE for Harvard School of Dental Medicine, poor internet connection was also an issue. This resulted in low quality of images and sound shared online also created a disruption in accessing digital scoring platform [7]. Another study in Indonesia also shared similar problem regarding internet connection while conducting internet-based multiple mini-interviews in medical school, which resulted in delayed in the entire process [8]. Aside from technical difficulty and internet disruptions, we also had difficulty in recruiting examiners who are relatively familiar with the use of video conference technology.

Despite of these challenges, we believe that modified OSCE was the best way to perform high-stake examinations during pandemic era. It also required less budget due to huge cost reduction for transport, accommodation, and massive human resources in traditional setting. We realized this modified online examination has several limitations. Compared to traditional OSCE setting, we cannot fully observe and assess the resident’s performance. Although we tried our best to maintain secrecy regarding the stations’ objective, some things could go beyond our control and thus compromising fairness and objectivity.

To conclude, we have described the experience of implementing modified continuous online OSCE examination in ophthalmology residency setting in Indonesia. It is feasible to complement the traditional OSCE with modified online version. And it’s also possible to convert the usual circuit method into continuous method. However, it should be meticulously prepared to predict potential drawbacks and the execution should be well-organized with proper internet connection. Further discussion and evaluation to overcome connection problems should be done. Proper IT team in each participating institution should also be considered. Conducting the online OSCE ensure the continuation of ophthalmology residency program to get the residents to obtain their degree as an ophthalmologist, and thus, these residents were an addition to the workforce needed during pandemic. Despite several challenges and limitations, the modified continuous OSCE model was created and implemented successfully while preserving its validity and reliability. The result was also supported by participant’s positive feedbacks. This will encourage implementation of online national examination in the future.

Objective structured clinical examination (OSCE) station feedback

Station 1

	Strongly disagree	Disagree	Agree	Strongly agree
The clinical scenario was easily understood.	○	○	○	○
The task was easily understood.	○	○	○	○
The time given for the station was enough.	○	○	○	○

Were there any difficulties during station 1? Please write:        

Station 2

	Strongly disagree	Disagree	Agree	Strongly agree
The clinical scenario was easily understood.	○	○	○	○
The task was easily understood.	○	○	○	○
The time given for the station was enough.	○	○	○	○
The shared video was clear.	○	○	○	○

Were there any difficulties during station 2? Please write:        

Station 3

	Strongly disagree	Disagree	Agree	Strongly agree
The clinical scenario was easily understood.	○	○	○	○
The task was easily understood.	○	○	○	○
The time given for the station was enough.	○	○	○	○

Were there any difficulties during station 3? Please write:        

Station 4

	Strongly disagree	Disagree	Agree	Strongly agree
The clinical scenario was easily understood.	○	○	○	○
The task was easily understood.	○	○	○	○
The time given for the station was enough.	○	○	○	○
The shared video was clear.	○	○	○	○

Were there any difficulties during station 4? Please write:        

Station 5

	Strongly disagree	Disagree	Agree	Strongly agree
The clinical scenario was easily understood.	○	○	○	○
The task was easily understood.	○	○	○	○
The time given for the station was enough.	○	○	○	○

Were there any difficulties during station 5? Please write:        

Overall examination feedback

	Strongly disagree	Disagree	Agree	Strongly agree
The examiners were communicative and gave clear instructions.	○	○	○	○
The committees were helpful throughout the exam.	○	○	○	○
Pre-exam briefing was clearly understood and informative.	○	○	○	○
Overall, the examination was well-conducted.	○	○	○	○

If you have another feedback regarding the modified online OSCE, please write:        

Thank you.

	Strongly disagree	Disagree	Agree	Strongly agree
Pre-exam briefing was clearly understood and informative.	○	○	○	○
Exam preparation instructions was clearly understood.	○	○	○	○
Communication between national and local committee was appropriate in aiding the exam.	○	○	○	○
Overall, the examination was well-conducted.	○	○	○	○

Were there any difficulties during examination? Please write:          

If you have another feedback regarding this modified online exam, please write:      

Thank you.

	Strongly disagree	Disagree	Agree	Strongly agree
Pre-exam briefing was clearly understood and informative.	○	○	○	○
Communication between national and local committee was appropriate in aiding the exam.	○	○	○	○
Digital scoring system was convenient to use.	○	○	○	○
Overall, the examination was well-conducted.	○	○	○	○

Were there any difficulties during examination? Please write:          

If you have another feedback regarding this modified online exam, please write:      

Thank you.