Experience and Challenges of Objective Structured Clinical Examination (OSCE): Perspective of Students and Examiners in a Clinical Department of Ethiopian University.

BACKGROUND: Invented nearly half a century ago, Objective Structured Clinical Examination (OSCE) is overwhelmingly accepted clinical skills assessment tool and has been used worldwide for evaluating and teaching learners' competences in health care disciplines. Regardless of factors affecting the attributes, OSCE is considered as reliable and powerful tool with certain validity evidences. In spite of its advantages and various promotion efforts, the progress of OSCE implementation in Ethiopian public universities has not been satisfactory. Therefore, the objective of this study was to explore the experience and challenges of OSCE implementation from the perspective of clinical year-II medical students and their examiners in Ob-Gyn Department of Jimma University.
METHODS: Forty-nine students and seven examiners voluntarily participated in Ob-Gyn Department where OSCE has been used as one of summative assessment methods. Qualitative study design using structured open-ended questionnaire as a tool and descriptive phenomenology as underpinning method were employed. Collaizzi's descriptive analysis was used as phenomenological analysis approach. RESULT: Poor organization, inadequate student preparation time, and inadequate number and duration of stations were thematically emerged as umbrellas of factors negatively affecting OSCE implementation. Satisfaction with OSCE was the only theme with findings that encourage OSCE implementation.
CONCLUSION: There should be team approach, shared responsibility and proper planning among faculty to minimize hindering factors of OSCE implementation. Besides faculty development on OSCE, the department should improve skill lab utilization arranging schedule for both students and faculty members to increase guided students' exposure to simulation-based learning and ultimately enhance OSCE implementation.

Introduction

The conventional clinical skills assessment methods such as long case exam, short case exam, viva, work place subjective judgment by seniors, etc. were noted to have several shortcomings and necessitated invention of innovative approach like Objective Structured Clinical Examination (OSCE). In modern context, OSCE can be defined as one of competency-based skill assessment tools in which fairly sampled clinical tasks or skills across learned outcome represented in several stations that all examinees are observed against standardized scoring rubrics in simulated environment. Nearly half a century back, when Harden invented the first OSCE with a ‘circuit' consisting of 16 stations of five minutes each. His primary aim was to standardize the exam, avoid subjectivity related to examiner, minimize variables that may affect students' exam performance and add objectivity to the assessment of clinical skills (1). Moreover, both educators and learners had not been satisfied with traditional assessment methods used before the introduction of OSCE (2). Since its inception, OSCE has been overwhelmingly accepted as clinical skills assessment tool and used worldwide for teaching and evaluating learners' competences particularly in healthcare disciplines (3). The impact of this innovative format is not limited only to assessment, but also can be used for learning purpose when it is accompanied with proper feedback. There is also an attempt to develop learning tool named teaching OSCE (4) making modifications on the existing assessment oriented OSCE. Moreover, in some clinical skills, OSCE assessment improved learning outcome compared to some other skills teaching methods (5), and the acquired skills last longer (6). This may prove a notion, which is originally based on biomedical science education evidence (7), that 'assessment drives learning’ in clinical education.

Based on existing evidences, generally OSCE is valid, highly reliable, objective and a powerful tool than other traditional assessment formats such as short or long case examinations, viva, etc (8,9,10). Faculty perception of OSCE as better assessment tool than its traditional assessment counterpart (11) reveals its growing acceptance. There is a report that students may perceive that OSCE decreases exam related stress (12). Regardless of all these plus attributes, implementation of OSCE has never been free of challenges anywhere. Harden and his colleagues, as early as 1970s, for instance, noted drawbacks such as increased preparation time and separately viewing knowledge and skills (1). Scholars in handful of published evidences argue that the attributes of OSCE such as short time spent on each station, predetermined non-flexible checklists, its focus on “snapshot” aspect of patient-provider encounter, and unsimulatability of some skills may compromise its use for assessment (13). Problem of OSCE implementation is worse in schools of nonwestern countries and even worst in some other developing countries particularly due to its late inception and, consequently, lesser experience. Studies show that factors such as lack of skill lab, excess cost, shortage of trained faculty, unfair selection of tasks across the competency, increased stress, inconsistent tools, lack of standardized patients, absence of assessor training, assessors' intimidation, time shortage on stations, and difficulty in standard setting have affected implementation of OSCE mainly in schools of developing countries (12,14,15,16,17). Contentiously, some students may rate OSCE as more stressful and mentally demanding than traditional assessment formats (18). In summary, factors that may affect OSCE implementation are summarized using the conceptual framework below (Figure 1).

Figure 1

Conceptual framework of perceived challenges and factors negatively (dark) and positively (light) affecting OSCE implementation from perspectives of students and examiners as reported in literature

OSCE has relatively younger history in Ethiopia which traces back to not more than just two decades (14). Since its inception, much effort has been put by institutions and stakeholders to ensure OSCE implementation. However, in spite of the efforts and the recent governmental need to make it one of licensure exams formats in healthcare disciplines, its development is not up to the expectation nationally. Challenges related to lack of theoretical and practical skills of running OSCE along with resource related constraints were often observed after each OSCE session held in the universities. Although there is handful of quantitative survey reports of students' perception and attitude that are in favor of OSCE use in the country (12,14,15), there is no study conducted to explore experience, perceived challenges and other factors that limited much needed implementation of OSCE in Ethiopian universities and colleges. Moreover, there is unimproved utilization of OSCE during student assessment in Jimma University contrarily to the promotion efforts. Therefore, aimed at answering the research question “, this study explores experience and challenges of OSCE implementation from the perspective of medical students and their examiners in Obstetrics and Gynecology Department of the Institute of Health, Jimma University.

Methods

The study was conducted in Jimma University Institute of Health on students and examiners who participated in OSCE exam at the Department of Obstetrics and Gynecology (OB-GYN). Ethical clearance was obtained from the Institutional Review board (IRB), and all participants gave their consent for participation in the study. Participation was based on their will. The students were informed that the study would not have any effect on their performance evaluation and the confidentiality of information they gave would be kept throughout the study process. OSCE has been one of the four assessment methods- i.e. progressive assessment, written exam, long case, and OSCE- used for summative evaluation at the end of OB-GYN clinical attachment period of fifth year or clinical year two (C-II) students. The OSCE consisted of seven stations. The detailed overview of paradigmatic assumptions and research philosophies underpinning our methods was published elsewhere in an extended form of methodology part of this study (19). Interpritivist/constructivist tradition framed the methodological approaches resulting in phenomenological qualitative design. The purpose of phenomenological research is to understand the essence of social phenomena from the perspective of those who perceived it (20). Therefore, this approach enables us to understand the nature and meaning of students' and examiners' OSCE experience and perceived challenges in the department. Among the commonest phenomenological study variations (21), as it is recommended in areas where there is little or no previous research evidences exist (22), descriptive (Husserlian) phenomenology suits to our study.

Sample of 49 C-II medical students and seven examiners from the Department of Ob-Gyn were interviewed using self-report written questionnaire. At the end of OSCE, data was collected using self-administered or self-report open-ended structured interview as a tool adapted from a similar previous qualitative study (23) and modified in a way it fits to our phenomenological enquiry (Table 1). To give phenomenological sense to the adapted questionnaire, participants had been verbally informed in advance to express their feeling about OSCE experience and perceived challenges while answering the questions. Although there are several other data collection methods used in descriptive phenomenology (22), Marshal suggests possibility of employing open-ended structured interview using questionnaire to explore topics ranging from cultural differences, first hand encounters, and the perceptions, meanings, and interpretations of participants (24). To minimize “memory decay, alterations or participant response errors” related to self-report interview in phenomenological study (25), the questionnaire was distributed immediately after participants finished their OSCE exam. Analysis was carried out using Collaizzi's descriptive phenomenological analysis approach. Previous study by Obizoba, aimed to explore the strategies for mitigating the challenges of OSCE in baccalaureate nursing education program, was used as a benchmark for data analysis (26). Agreeing with Georgi's critics towards the seventh step of Collaizzi's method (27), we finalized analysis at sixth step as shown below (Figure 2). Regarding the reflexivity, the primary investigator of this study had experience as a coordinator of clinical skills lab and aware of some of the challenges faculty members and students had been facing. Such experience consciously was ignored throughout the study and served as just a base for understanding problem and developing research question. Moreover, the trustworthiness of the study was tried to maintain with bracketing, internal auditing, larger data, and ‘reader resonation’ (19).

Table 1

Data collection tool

No.	Questions
1	Did you have OSCE experience before? If yes for Q no.1, how many times did you examined (with) OSCE?
2	What do you think about the structure of OSCE?
3	Was the number of OSCE stations adequate? (if Yes/No, Explain your reason.
4	Was the time allocated in each station fair/ time management adequate for OSCE stations? (If Yes/No, Explain your reason).
5	Did the OSCE stations sufficiently cover the major areas of your course or attachment (Ob-Gyn)? (Explain your reason)
6	What do you feel if there was written stations in addition to skill/procedure stations?
7	Was the OSCE format easy to follow? (think of instructions, scenarios, materials, examiners, etc and explain).
8	What did you like about this OSCE exam?
9	What did you not like about this OSCE exam?
10	How the OSCE can be improved? Any recommendation?
11	Generally (not specific to this OSCE), what is challenging or difficult for you when you prepare for OSCE?

Figure 2

Modified Colliazi's seven steps descriptive analysis

Results

A total of 76 students took the OSCE exam, and only 49 students were willing to complete the open questions making the response rate 64.5%. However, all of the seven examiners completed the questioner. Regarding previous OSCE experience, only 10 out of 49 students reported that they took the exam for the first time, but the rest reported they had taken the exam one to four times previously. All of the examiners reported that they had participated in OSCE two to six times. Asked their feeling about inclusion of written stations along with the procedural ones, the majority of the students and examiners were against the idea despite the fact that students complained about lack of awareness about OSCE principles. Regarding the overall difficulty of the OSCE exam, the majority of the participants leveled it as medium. The perceived factors affecting OSCE exam were categorized under four themes as presented below. Across the themes, ‘S’ and ‘E’ along with code numbers in parenthesis stand for student and examiner participants, respectively. The themes, expected to encompasses the factors under investigation, were termed in self-explanatory way for the ease of understanding as follows.

Satisfaction with the OSCE: The satisfaction level of participants about the structure of the OSCE format based on their responses varies from “very nice” to “not well-structured”. The common terms and phrases used by participants to express their satisfaction with the OSCE include happy with it, I like it, excited about, assesses well, best, good, etc. The majority of the respondents indicated that they were satisfied with the assessment type.

“I like OSCE because it correlated [with] the major area of our field [of attachment]”[S19]

“… it is best strategy for developing [and assessing] technical skills…it should be strengthened.”[S32]

“…it's good and increase our confidence to do procedures on patients…”[S11].

“…the OSCE format was easy to follow, I was feeling excited about each procedure.”[S20]

These notions are examples of students' satisfaction and their perception of OSCE not only as assessment tool, but also as a tool for enhancing their learning of clinical skills. Additionally, OSCE was also perceived as a method that comprehensively assesses their clinical competence. In this regard, as an example, one student responded:

“…I think it is well organized, and assesses us [our competence] very well.”

All of the examiners were happy with the exam format and the way it was designed. Nevertheless, poor coordination and shortage of time affected students' satisfaction about the assessment format to some extent.

“…it is good; it has good structure but lacks full coordination.” “…time should be given adequately [with this structure].”[S40]

Poor organization: Poor organization implies problems related to OSCE processes such as design, setting up, preparations and running of OSCE. Overall organization of the OSCE was rated as good by both students and examiners team. However, students responded that planning and organization of OSCE should be improved. Some of the problems perceived a problem of organizing OSCE, beside failure to allocate proportional time for each tasks include unclear instruction and distracting or “shouting” examiners. Students also perceived that orally narrating case scenario by examiners caused some bias and confusion before the start of the task and they preferred written case scenario to oral narration for better clarity.

“…as a whole it was good, but some of instructions were not clear and it is better to make it clear…some of the examiners were shouting.“[S27]”

“…it causes unclarity, it is good if the cases are written rather than [verbally] told by someone [examiner].”[S36]

In order to adapt “unfairly” shorter duration of time allocated to stations and perform the task properly, students suggested breaks in-between which necessitates rest stations so that they make themselves psychologically ready.

“…no. [I don't think the time management was adequate and fair for the OSCE stations]. If the time allocated is very short it is better to have break”[S19]

Some of the students also did feel that there should be orientation about the basic principles of OSCE or provided with prior practice sessions of mock OSCE before actual exam in order to familiarize themselves to the assessment method.

“…most of us do not know how and what to prepare…we don't know resource to read about preparation [for OSCE]” [S35]

“…it is better if we practiced it before.” [S43]

Insufficient number and duration of stations: Both students and examiners criticized the sampled competencies in terms of the number of stations. It was challenging for the faculty to have fair representation of competency in OSCE stations. One of the examiners, for example, mentioned that it would be difficult even to imagine representation of competence of Obstetrics and Gynecology for C-II students only with four procedural skill stations such as IUCD insertion, MVA, AMTSL and neonatal resuscitation included in that OSCE.

“…the topics covered are very important in Obstetrics and Gynecology and they are very common procedures…but [only] limited number of stations [were sampled].”[S14]

Almost all of the examiners and the supervisor agreed that the number of sampled competencies or skills were important, but they were too few to assess students against what they learned during their clinical attachment. The reasons for planning lesser number of skills were shortage of examiners and lack of all necessary models and materials in simulation center.

The time allocated for each station was also inadequate and challenging for students. Some of the examiners also did agree on this regard. They feel that in spite of fewer stations, time allocated for each station was not enough. However, the majority of the finding in this theme was from students and most of them reported that the time spent on each station was not sufficient to perform the given task completely.

“…no, [time was not enough for stations]. I think it doesn't give us time to think and gather our thoughts…no, it's not fair. Even we didn't have time to think.”[S16]

“…it is better to have more number of stations with adequate time.” [S27]

Overall, regarding the number and duration of station, responses show that study participants from students' group emphasized the shortage of time allocated for station whereas examiners perceived unrepresentativeness of sampled stations as challenges.

Deficient preparation for the OSCE: Although the examiners were not concerned about it as such, students complained about inadequate time given for preparation before OSCE examination in two ways. Firstly, they had little or no previous exposure to simulated environment and they did not have experience of mock OSCE which they preferred to have.

“…it was challenging because we only practice when exam approaches and we do not internalize it” [S01] “it is better if enough [practice] time is given to practice before exam.”[S07]

Secondly, the schedules were too tight that they even were rushed to take the OSCE exam on the same day they took other formats of exams like viva. They often mentioned that it was difficult for them to cope with such academic pressure or load.

“…difficulty I have is timing. In the morning [we took] internal oral examination, OSCE in the afternoon. Not much time to revise. ”[S03]

“…our long exam was in the morning so that [we had] no time for preparation.”[S02] “…It is good not making the long exam and OSCE on the same day”[S06]

The majority of the students agreed that these practices inhibited them from properly completing the given task and affected their performance on the OSCE examination.

On the other hand, a handful of students, mainly those with more previous OSCE experience, stated that they are comfortable with making themselves ready for OSCE than other assessment types if practice time is properly arranged in advance.

“Practice days or schedules should be increased. No difficulty during preparation, it feels better to prepare for OSCE than oral.” [S04]

From examiners side, the challenge of planning and preparation for OSCE was mainly perceived as related to logistics and examiners' capacity building.

“…I feel it is better if supervisors or examiners are trained before the exam. Limited number of residents or man power [examiners] and instruments [simulators and other lab equipment] are challenging” [E06]

Discussion

Satisfaction of students and faculty towards OSCE format was the major promising finding presented in this study. Medical students' and examiners' satisfaction with the OSCE format is consistent with several evidences in the literature (11, 28). In a study conducted in Nigeria, a country with comparable experience with Ethiopia, Ameh et al confirmed that students were satisfied with OSCE and preferred it over other assessment methods (29). The main reason for their preference was that OSCE truly measures their clinical skills and has more validity and reliability. Therefore, students' satisfaction and motivation may not be a negative factor at all anywhere; instead, it has positive impacts. In spite of the satisfaction which encourages the implementation of OSCE in the department, the challenges commonly discussed and presented as the major findings were poor organization, insufficient number and duration of stations and deficient preparation for OSCE.

Poor organization is a finding related to failure of proper planning for the OSCE exam. During planning phase, OSCE team should arrange meeting and training sessions before the exam day to prepare exam blue print, design stations, write scenarios, orient examiners, prepare the rooms and equipment, and address other management issues. Both students and examiners complained of problems directly or indirectly related to poor organization. Intimidating or interfering examiners, unclear instructions, unfair station time allocation and absence of necessary materials were some of the problems from reports. To combat poor organization and related problems, Kamran et al suggested responsibility sharing among the team, preparing exam schedules, developing rules and regulations, preparing exam blueprint, deciding station numbers and exam duration, developing intended OSCE stations, preparing marking tools, and finally piloting developed circuits before actual OSCE exam (30). Insufficient number of station is related to unfair representation of competencies. Fairly sampling adequate number of OSCE stations from competencies in line with the course objective has been challenging whenever the format is planned. It needs training and orientation for examiners or exam developers, if any. Whereas the number of stations in our study was only seven, the recommended number varies from 12 to 20 with reliability score of greater than 0.6 (31). However, probably as a result of resource related constraints, fewer number of OSCE stations is not unprecedented as far as the national experience is concerned. The first OSCE, which was used as an exit exam of surgical residents in the top Ethiopian university, for instance, consisted of just seven stations (14). Inadequate time for each task in a station was also a finding under the same theme along with less number of stations. In this regard, a study conducted by Emadzadeh et al also reported shortage of time in each station as the main challenge while running OSCE exam. In the same study, the authors correlated stress, due to insufficient time for the tasks, to poor result in the exam (32). However, increasing time for a station may make students finish the exam tiresomely late or may force the team to reduce the number of stations to minimize the total time. According to the finding, shorter time was roughly associated with mental stress. In contrast to the finding in this study, students who participated in the study conducted in Ethiopia by Shitu et al also reported that the time allocated for each station is adequate and overall OSCE has less stress than other assessment methods (13). Whereas the duration of a task or skill depends on its complexity and available resource, the number of stations or tasks represented depends on the objectives of the course. To avoid these challenges, the exam team should rely on the course objectives and the exam blueprint to set individual stations' duration, and ultimately that of the overall exam. Several students complained lack of preparation time for OSCE. They sat for another clinical exam format earlier on the same day before the OSCE. In principle, mixing assessment tools—traditional and innovative approaches— is highly recommended. However, taking these exams on the same day may add unnecessary cognitive load on students which eventually affects their performance. However, there is no evidence regarding how long students should prepare before OSCE and the decisions are commonly made based on the curriculum and the policy of the schools.

Theories were sought to frame the discussion of our result; yet, we could not find a single best purely educational theory particularly devised for assessment in medical education. Derived from other fields, theories of psychometrics and theories of development of medical expertise (TDME) are the most common theories on which clinical assessments are based (33); but, we chose TDME to frame the overall discussion of OSCE as an assessment method in relation to our finding. The TDME gives more emphasis to importance of the processes, i.e. what and how to assess with OSCE or any other assessment before interpretation of data. The focus of TDME is different from Psychometric theories (PT) that mainly rely on interpretation of students' performance data or result to determine the validity and reliability of the test. Analyzing students' data based on PT is beyond the scope of our study. Derived from the original theory of expertise development with classical model of distinguishing amateur and grand master in chess game (34), TDME assumes that there exist unique features that distinguish learners on the spectrum of novice to expert (35). According to the theory, medical students gain “rich, elaborated causal networks” elucidating the etiology and outcome of the disease in relation to underlying biological and pathophysiological presentations on the course of their training. An experienced physician with expert knowledge (“illness scripts”) or any other learner approaching to the same level with repeated exposure and practice operates on unique knowledge structure than novices while diagnosing and managing patient that indicates the learner's performance and competence in real and simulated environments, respectively. After repeated exposure and application, “illness scripts” get encapsulated and become easily retrievable and stored in the memory in the form of a model of high-level concepts. In summary, I directly quote Schmidt and Boshuizen stating, finally knowledge structures acquired during the different phases of development, pathophysiological networks, encapsulated structures, illness scripts, and episodic traces of instantiated scripts, do not decay; neither do they become inert, nor inaccessible (35). This also applies to students performing tasks during OSCE. In our finding, it was noted that lack of prior exposure to practice sessions or mock OSCE affected the performance of those who took the exam for the first time. Students with repeated previous exposure, on the other hand, found the exam easier than other assessment methods. These findings imply that repeated exposure to simulation and mock OSCE with feedback improves not only their confidence and performance on the test, but also the efficiency— i.e. takes shorter time on stations— while completing the tasks. The drawback of OSCE based on TDME, if employed as the sole assessment tool, would be domain specificity and idiosyncrasy of the expertise (33). Although it is not a problem in OB-GYN Department for now as the students took at least viva and written exam in addition to OSCE, domain specificity deals with the principle that performance of a student on one station or task is not good predictor of his or her performance on other task or station (e.g. station not included in sampled stations). In our study, students and examiners perceived that sampled stations were not adequate, and that it might be due to the very reason that they felt performance could have been better on other clinical tasks that had not been included. In order to overcome such attribute of OSCE and use the strength of other clinical assessment methods, assessment “toolbox” approach—i.e. mixing various assessment methods to measure students' competence—is recommended. Idiosyncrasy of a learner means students on different level of skill development continuum, especially on the higher end of expertise, have different ways of doing tasks—they just focus on the final output than the prescriptive basic steps. The implication of this notion to our OSCE in the department is that we cannot judge the competence of students based on few steps they missed on the checklist of the station as long as they completed the tasks properly. Hence, removing unnecessary details on checklists and focusing on fewer essential steps towards final output of the task may necessitate.

As strength of the study, use of self-administered questionnaire for the structured interview minimized interviewer bias. Participants were interviewed immediately after OSCE exam experience, and in so doing, self-report related “memory decay, alterations or participant response errors” were minimized. As limitations, the data was collected solely with self-administered questionnaire without triangulation with other methods. It could have been better if in-depth interview was conducted and triangulated with FDG, or vice versa. Moreover, the students' response rate (64.5%) was too low for qualitative survey, but the sample size (forty-nine) was more than enough for qualitative study that it did not affect the analysis of the result. The main reason for nonparticipation was postexam fatigue. Self-administered report questionnaire used in our study per se has drawback that are expected in self-reported data (19).

In conclusion, OSCE is overwhelmingly advocated at the ministry level to the extent of preparing licensure exam which consists of OSCE and MCQ questions in accredited institutions across the country. For this very reason, clinical departments wish to implement and run quality OSCE. Both students and examiners were happy about and preferred the format over other clinical assessment methods. Poor organization, inadequate stations, insufficient time allocated for a station and inadequate inter-exams time gap were factors affecting the much needed implementation of OSCE in Jimma University OB-Gyn Department. These factors are related to concepts of TDME in some way and could have been addressed during the process of OSCE. The department should form exam team and share responsibility of organizing OSCE so that problems related to poor planning are solved. There should also be separate schedules for OSCE and other assessment methods so as to avoid unnecessary stress and burnout among students. Standard steps of OSCE preparation should be followed, and at least 12 stations should be sampled based on the exam blueprint to improve validity and reliability. We also suggest inclusion of at least two rest stations in the circuit. In addition to training examiners, students should be oriented about OSCE exam and practice mock OSCE before actual exam to make them familiar with the assessment method. Eventually, proposed factors in this study should be tested with robust quantitative design so as to detect their association with pedagogical and administrative outcomes.