Challenges of OSCE national board exam in Iran from participants' perspective.

BACKGROUND: The national board exam for residents in Iran is held in two parts: Multiple-choice and Objective Structured Clinical Examination (OSCE). The OSCE is a suitable method for evaluation of residents' clinical qualifications. However, it requires experienced human resources, accurate planning, facilities and reliable evaluation tools.
OBJECTIVE: To determine the challenges of the OSCE National Board Exam in Iran.
METHODS: This cross-sectional study was conducted on all the final-year pediatrics and gynecology residents of Mashhad University of Medical Science, who participated in the board exam in September 2014. A questionnaire was designed to evaluate the residents' opinion on challenges of the OSCE. Data was analyzed with SPSS16. We used U Mann-Whitney test independent t-test, and Pearson correlation coefficient.
RESULTS: Fourteen pediatrics and eleven gynecology residents participated. In the gynecology group, there was no significant statistical correlation between the individual marks and questionnaire scores. However, in the pediatrics group, there was a significant correlation (p=0.046, r=-0.763). Based on pediatrics residents' perspective, the main challenge of the OSCE part of the exam was the imbalance and disproportion between the allowed time and the task load in each exam stage. In other words, they believed that the tasks could not be fulfilled in the given time. In the gynecology group, the main challenge reported was the delay in announcing the exam results. In the pediatrics group, the main complaint was the disproportion of the allowed time and the task load in the exam stages.
CONCLUSION: Some of the challenges of the board exam were associated with the examiners and the exam environment, and some of them were related to the home university where the candidates had studied. To solve the problems, both aspects should be considered.

1. Introduction

Clinical skills evaluation is one of the crucial parts of medical education to assess students’ qualifications (1). The Objective Structured Clinical Examination (OSCE) is one the most known methods for this purpose. This exam has been analyzed in different studies and it has been reported that OSCE can be a suitable assessment tool to evaluate students’ knowledge and emotional and psychomotor skills (1). OSCE is usually used in periodical evaluations of both medical students and residents along with the written exams (2). OSCE is a golden standard for evaluation based on performance to assess the clinical skills in specialized studies. It is a multi-stage with limited time exam where the candidates do common tasks such as interview, examination, clinical actions, recovery and consultation like a real scenario. (3). As discussed above, OSCE is a suitable method to evaluate students’ clinical skills. However, the exam organization is difficult and needs accurate planning, skilled human resources, appropriate environment and reliable and precise evaluation equipment (4). The national board exam in Iran is annually held in September for the residents who passed the pre-board exam. It is composed of a written part, and the candidates who succeed the written exam attend the second part, the OSCE, which is held some days later. For residents, it is naturally a stressful exam. For instance, one of their main concerns is the inconsistency and mismatch of the studies and offered courses in home universities, and the exam contents. The OSCE is one of the most influencing exams in the academic life of the residents as success in this exam may provide better professional opportunities, higher income and even higher academic ranks and chance of becoming a faculty member. However, there has been no comprehensive study on the board exam, especially the OSCE part, or from the perspective of the participant residents. The goal of this study was to determine the challenges of the OSCE National Board Exam in Iran.

2. Material and Methods

2.1. Research design and participants

This was an analytic cross-sectional study that was conducted on all the final-year pediatrics and gynecology residents, who participated in the board exam in September 2014 from Mashhad University of Medical Science. The exclusion criteria was the lack of satisfaction in participating in the study.

2.2. Sampling

The sampling was a stratified sampling where all 27 clinical studies groups at the faculty of medicine of Mashhad University of Medical Sciences were classified into two groups of surgical and non-surgical. Then, based on a simple random method, the pediatrics group composed of 14 residents (non-surgical), and gynecology group composed of 11 residents (surgical) were included in the study. After asking for their consent and assuring all 25 residents that their information would remain confidential, the questionnaire was completed in a telephone interview conducted by a skilled survey expert.

2.3. Instrument

A questionnaire was designed by the research team which included demographic questions such as sex, field of study and score in the board exam (first part) and 15 questions regarding the opinion of the participants about the 2014 OSCE (second part). These 15 questions had been designed based on Likert scaling (totally agree, agree, no opinion, disagree, totally disagree). The overall score of the questionnaire, the sum of all 15 questions, was in the range of 15 to 75 where the minimum score observed was 40 and the maximum was 68. The questions are provided here:

2.3.1. Question 1: “The date and time of the exam were convenient.”

The reason that this question has been included in the questionnaire was that the OSCE exam is held only once a year, in the capital (Tehran), on a weekday and in early morning hours. In a metropolis, such as Tehran, early morning is rush hour which creates transportation difficulties for the candidates to arrive at the exam venue. It would be recommended to have the exam at least twice a year (e.g., September and March), in the middle hours of the day and in different regions of the country.

2.3.2. Question 2: “The equipment provided for the OSCE exam was appropriate.”

In OSCE, equipment such as moulage, computer and medical instruments, is needed. Any interruption or failure in those instruments and equipment affects the exam progress and will be stressful for the participants. It is thus needed to employ reliable and verified equipment and to check it during the exam. Gynecology residents gave a higher average score to this question. Only three pediatrics residents disagreed or totally disagreed, which implies that employed instruments were in an acceptable condition.

2.3.3. Question 3: “The number of questions was appropriate”

Naturally the number of exam stages and the duration of each stage should not exceed the physical and mental capacities of the students. One of the most common types of OSCE is composed of 20 stages (considering the limited time of the exam, more stages implies a higher exam reliability (6)). Gynecology residents’ average score to this question was higher than the other group. In total, five residents disagreed the number of OSCE questions.

2.3.4. Question 4: “The time assigned to each exam stage was appropriate.”

The standard time for each stage is decided independently and it depends on the stage tasks. In different exams, durations of 5 to 15 minutes have been reported. However, most of the stages’ duration is set to 5 minutes. Naturally, the considered duration for each stage should be long enough for the work load of the stage, so the students will not feel under pressure (unless time is also an evaluation parameter). The students should be assigned at least 30 seconds to move to the next stage (6). As discussed, pediatrics residents gave the lowest score to this question, and four gynecology residents were not satisfied either with the considered time limits. We believe this question and related problem needs more investigation.

2.3.5. Question 5: “OSCE questions of the national board exam were similar to the pre-board OSCE exams held in your home university.”

The national board exam is annually held in September for students who have successfully passed the pre-board exams; therefore, similar questions and structure will help students to prepare themselves with lower stress. This was the only question whose average score was significantly different between two groups (p =0.014). Gynecology residents gave the highest grade to this question. It is therefore recommended that the questions in the pre-board exams are also provided based on standards, from the recommended references and curriculum. Five pediatrics residents disagreed or totally disagreed with this question.

2.3.6. Question 6: “There were similarities between the national OSCE exam questions and the internally held OSCE exams in different groups during residency studies”

During residency studies, some exams are held internally in the OSCE format. However, as the exams are supervised and the questions are designed by a limited group of professors, the board exam standards are not necessarily respected, while having similar structures and contents as the national board exam can prepare the students for the main exam and evaluate their qualifications more appropriately. Only two gynecology residents disagree with this question, which implies that both groups had experienced similar exams internally.

2.3.7. Question 7: “OSCE exam questions were compatible with the courses offered during the residency studies.”

The efforts have been on having questions in the exams which are compatible with the recommended references and curriculum. Professors also try to cover the same references in their lectures. The pediatrics group gave the highest score to this question and all residents either totally agreed or agreed with the question statement. In the gynecology group, except one person, all others agreed or totally agreed with the statement. We can interpret from the results that the students were satisfied with the teaching methods and offered lectures, and that the professors were knowledgeable and skillful.

2.3.8. Question 8: “The OSCE exam of 2014 could appropriately asses your clinical skills.”

In a study on the OSCE exam potentials to evaluate the pediatrics residents’ qualifications, it was concluded that OSCE is a valid and reliable method for assessment of clinical skills of pediatrics residents (7). When OSCE is used along with other evaluations methods, it can very well represent the progress of a resident. However, holding an OSCE exam is costly and needs accurate planning, therefore, it should be more investigated and discussed whether, in comparison to other formative assessment skills, OSCE is more advantageous or not. In the conclusion of the article, the authors mention that even though OSCE is an appropriate formative assessment method, the approaches to improve its performance should be investigated (7).

2.3.10. Question 10: “The results of the 2014 OSCE were announced on-time.”

The national board exam is one of the most important exams in the academic life of the residents because all future opportunities, from working as a faculty member to becoming a specialist physician in a small city, depend on this exam. The results (pass or fail) are announced one week later via internet and naturally that one week would be very stressful for the candidates. Furthermore, even though the exam questionnaires are coded, some boardman professors may have access to the results in advance and inform their own students which is unfair since some students come from universities which do not have a boardman on the executive committee. Female candidates naturally bear more stress, which is also reflected in the low score they gave to Question 9. We believe advancing the results announcement, similar to the written exam results which are available the day after, may play a major role in reducing the stress of the candidates.

2.3.11. Question 11: “The exam results were predictable and had been expected.”

Usually the candidates know their level of preparation for the exam and anticipate their score in advance based on the amount of studies and their scores in previous exams. However, there may be parameters such as the exam venue, exam organization, mental and physical state of the candidate and type of questions that affect the results. Sometimes the student believes that he answered the questions very well, but the result is different. To check the questions and answers will be easy in the written exam, as the exam questions are multi-choice, but will be difficult in the OSCE part as the questions should be answered in a descriptive manner, and after the exam, no solution manual is published. It is thus recommended to have a solution manual available after the exam with clear instructions. In the pediatrics group, 10 students and in the gynecology group, 7 students agreed that their score had been expected.

2.3.12. Question 12: “Having a board man professor in the home university academic group is effective.”

Every year, a group of faculty members from different universities are invited to participate in the executive committee of the board exam, and design questions. However, there could be universities with no member on the OSCE committee. Even though it is expected that having a boardman positively affects the exam experience for the students from the same university, in our study, 7 pediatrics and 10 gynecology residents believed that having a boardman from the home university had no advantage.

2.3.13. Question 13: “OSEC exam stages and your field of study clinical needs matched.”

OSCE is based on the golden standard, and provides similar clinical scenarios for assessment of several candidates (3). Exam questions should naturally represent the most important clinical topics of a field of study. Having questions on rare scenarios is not common. From the pediatrics group, 10 residents and from the gynecology group, 6 residents agreed that the questions were matched with the important clinical needs of their field of study.

2.3.14. Question 14: “Questions asked in exam stages were clear and valid (based on the given references).”

Ambiguous questions confuse the candidates and affect their performance. All pediatrics residents and 10 gynecology residents believed that the questions were clear and valid.

2.3.15. Question 15: “Exam grading and corrections were fair and appropriate.”

Based on direct and indirect observations and using checklists, the student is evaluated in OSCE based on standard grading systems. Therefore, it is more efficient and less error-prone than traditional methods. However, as a subjective grading method, giving a score would still be challenging. Totally, 20 residents, 10 from each group, believed that the grading system had been appropriate.

2.4. Validity and reliability

The validity of the questionnaire was confirmed by experts, professors who were in the evaluation committee of the exam, and previous-year participant residents. For content validity, relevance, simplicity and clarity of each item were measured, as well as being essential. Face validity was checked, whether the tool was designed to be suitable for evaluating the target or not. Also for this reason, difficulty, and the disproportion and ambiguity of items was reviewed and modified. The reliability of the questionnaire was determined using Cronbach’s alpha in a group of 19 residents at the beginning of the research work, equal to 0.789.

2.5. Statistical analysis

Data was analyzed using SPSS 16 (SPSS Inc., Chicago, Illinois, USA). We used U Mann–Whitney test., independent t-test and in this study, to examine the relationship between the individual scores and the questionnaire scores in each group: the Pearson correlation coefficient was used, we consider p<0.05 to be meaningful.

3. Results

Among the 25 final-year residents who participated in the national board and OSCE exam, 14 candidates (56%) were from the pediatrics group and 11 candidates (44%) were from the gynecology group. All gynecology and 8 pediatrics residents were female. Regarding the written exam, passing it was the prerequisite of advancing to the OSCE, in pediatrics group, the minimum, maximum, mean and standard deviations were respectively equal to 110, 132, 121.86, and 7.819. In the gynecology group, the minimum was equal to 105, the maximum 134, the mean 116.57 and the standard deviation was equal to 12.191. The overall score of the questionnaire, the sum of all 15 questions, was in the range of 15 to 75 where the minimum score observed was 40 and the maximum was 68 with a mean score equal to 55±7.22. We separately analyzed the score for two groups of pediatrics and gynecology, the minimum, maximum, mean and standard deviation were respectively equal to 40, 66, 55.5 and 7.429 in pediatrics group. In the gynecology group, we had the minimum, maximum, mean and standard deviation respectively equal to 41, 68, 54.36 and 7.24. Comparing the overall score in the two groups using independent t-test and p=0.705 revealed that they did not have any significant difference. Table 1 provides the mean score for each of 15 questions for both groups. Each question score was between 1 and 5. For the pediatrics group, overall, the score range of each question was between 14 and 70, and for the gynecology group, was 11 to 55. We found that only questions 7 and 15 had a non-normal distribution, so to compare the two groups, the Mann-Whitney U test was used. For all other questions with a normal distribution, independent t-test was used. We did not observe any significant difference between the mean score of the same question in the two groups except question 5 where the difference of the mean score was significant and much higher in the gynecology group (p=0.014). We also analyzed the correlation between the questionnaire score as a dependent variable and sex, field of study and written exam score as independent variables. Using linear regression, the model was not found to be meaningful and the p value of ANOVA test was found equal to 0.722. To investigate the relation between the written exam score and the questionnaire score in the two groups, as both had normal distribution, we used Pearson correlation test. The results indicated no significant correlation in the gynecology group (r=0.587 and p =0.166) and an inverse significant correlation in the pediatrics group (r=−0.763, p=0.046).

Table 1

Comparison of mean score for each question of the questionnaire for two groups of pediatrics and gynecology

Question	Group	Mean	STD
1: The date and time of the exam were convenient	Pediatrics	4.14	0.86	0.214
	Gynecology	3.64	1.12
2: The equipment provided for the OSCE exam was appropriate	Pediatrics	3.64	1.22	0.233
	Gynecology	4.09	0.54
3: The number of questions was appropriate	Pediatrics	3.58	1.10	0.750
	Gynecology	4	1.09
4: The time assigned to each exam stage was appropriate	Pediatrics	3	1.18	0.223
	Gynecology	3.63	1.36
5: OSCE questions of the national board exam were similar to the pre-board OSCE exams held in your home university	Pediatrics	3.29	1.27	0.014
	Gynecology	4.36	0.5
6: There were similarities between the national OSCE exam questions and the internally held OSCE exams in different groups during residency studies	Pediatrics	3.86	0.66	0.718
	Gynecology	3.72	1.10
7: OSCE exam questions were compatible with the courses offered during the residency studies.	Pediatrics	4.28	0.47	0.5011
	Gynecology	4	0.77
8: OSCE exam of 2014 could appropriately asses your clinical skills	Pediatrics	3.5	1.16	0.635
	Gynecology	3.27	1.19
9: The results of the 2014 OSCE were announced in an appropriate manner	Pediatrics	3.5	1.16	0.484
	Gynecology	2.9	1.70
10: The results of the 2014 OSCE were announced on-time	Pediatrics	3.36	1.50	0.101
	Gynecology	2.36	1.36
11: The exam results were predictable and had been expected	Pediatrics	3.93	0.92	0.196
	Gynecology	3.36	1.21
12: Having a boardman professor in the home university academic group is effective	Pediatrics	3.28	1.64
	Gynecology	3.72	1.10
13: OSEC exam stages and your field of clinical study needs matched	Pediatrics	3.86	1.03	0.430
	Gynecology	3.54	1.21
14: Questions asked in exam stages were clear and valid (based on the given references)	Pediatrics	4	0.55	0.795
	Gynecology	3.9	1.136
15: Exam grading and corrections were fair and appropriate	Pediatrics	4.14	1.03	0.3731
15-Exam grading and corrections were fair and appropriate	Gynecology	3.81	0.98

Mann–Whitney U test and Mean Rank comparison

4. Discussion

In the pediatrics group, the main complaint was the disproportion and imbalance of the allowed time and the task load in the exam stages, which needs follow-up and intervention. In the gynecology group, the main complaint was the delay of announcing the exam results. The minimum score to pass the written part of the exam was 105. The mean score was higher in the pediatrics group, but the difference was not statistically significant (p =0.353), so both groups were almost in the same level. Regarding the OSCE part of the exam, all pediatrics residents, but only 9 out of 11 of gynecology residents passed the exam. Two gynecology residents did not answer the question regarding success in the OSCE exam probably because they did not pass it. The overall score of the questionnaire, calculated separately for each group, did not have any significant statistical difference between the two groups (p =0.705). Regarding each question separately, the pediatrics residents gave the highest mean score to question 7 of the questionnaire equal to 60. This implies that the OSCE exam questions matched the courses and topics visited during the residency studies, and curriculum and references recommended for the exam. The minimum mean score was given to question 4 equal to 42 (“the time assigned to each exam stage was appropriate”). In other words, they believed that the time assigned to some exam stages was short and did not match the task load of the stage. Naturally, the level of stress was higher due to the tight time limit, which affected the exam results. In the gynecology group, question 5 had the highest mean score implying that there were similarities between the OSCE and the pre-board OSCE exams. It indicates that the executive committee of the pre-board exam at Mashhad University of Medical Science were familiar with the type of questions in the main board exam, so participating in the pre-board exam had been a good preparation for the residents before the main exam which also positively affected the main board exam results. The gynecology residents gave the minimum mean score to question 9 equal to 32 (“The results of the 2014 OSCE were announced on-time”). The results of the OSCE part are announced online via internet some days after the exam while the results of the written part are announced one or at most, two days after the exam. To advance the announcement of the OSCE partly decreases the participants’ stress because they will have the chance to ask for a review. When the announcement is delayed, many of the participants would have already left for their home cities (the exam is held in the capital), so they would not have the opportunity to ask for an exam review. If the OSCE exam is held in a non-standard way, it may result in students’ inconvenience and increased stress, so that the students gradually find the exam an inappropriate evaluation method. Therefore, considering the international success of OSCE, we should look for strategies which improve the quality of the exam organization and management (5). Based on the regression model that we developed, variables such as sex, field of study and the score of the OSCE and written exams, did not affect the score given to the questionnaire by the residents. In the gynecology group, no significant correlation was observed between the score given to the questionnaire and the exam score. However, in the pediatrics group, there was a meaningful inverse correlation meaning that whoever had a lower score in the exam gave a higher score to the questionnaire. It thus implies that they answered the questions honestly and did not try to justify their low score in the exam. One of the limitations of this research work was studying only two academic groups (pediatrics and gynecology). Obtaining access to the graduates of other groups was difficult, so as a future work, we suggest that a similar study should be conducted in other groups of Mashhad University of Medical Sciences and in other universities and different years. After comparison and analysis, the results can be sent to all educational groups to be considered in future strategies.

5. Conclusions

Some of the challenges of the board exam were associated with the examiners and the exam environment, and some of them were related to the home university where the candidates had studied. To solve the problems, both aspects should be considered. In the pediatrics group, the main complaint was the disproportion of the allowed time and the task load in the exam stages, which needs follow-up and intervention. In the gynecology group, the main complaint was the delay of announcing the exam results while they are available just a few hours after the exam. This delay not only increases the stress; it lowers the chance of candidates coming from other cities, and leave the capital after the exam, to ask for a review. The results of this study can be considered for future strategies and plans regarding the OSCE exam structure and contents.