Investigating the Attitude of Graduate Psychiatrists towards Objective Structured Clinical Examination (OSCE) and Conventional Clinical Interview Examination.

OBJECTIVE: In the present study, we investigated the attitude of psychiatrists who graduated in 2002-2009 towards Objective Structured Clinical Examination (OSCE) and conventional clinical interview examination (Individual Patient Assessment).
METHOD: We studied 134 psychiatrists graduated; half of whom were examined with conventional clinical interview and the others with OSCE. A questionnaire was prepared by a specialist workgroup to assess the participants' attitude towards the exams. The questionnaire was initially examined in a pilot study. The findings of the questionnaire were used to assess the graduates' attitude towards each examination, as well as to compare the examinations.
RESULTS: The OSCE group indicated a significantly more positive attitude compared to the conventional group (p = 0.03). Furthermore, the OSCE group believed the role of theoretical knowledge (p = 0.01) and pre-test practice (p = 0.03) to be significantly greater for success compared to the other group. The structure of OSCE was reported to be superior to conventional examination in terms of fairness and homogeneity (p = 0.004). First participation in exam (p = 0.04) and ultimate success in the exam (p = 0.009) were predictors of graduates' attitude.
CONCLUSION: Based on examinees 'attitudes, OSCE may be a more appropriate choice for graduation examinations of psychiatry compared to the conventional clinical interview examination.

Graduation examinations in residency programs have always been a problematic issue. This may be due to the extensive curricula, limited learning time and the heterogeneity of education in different universities. These examinations have certain shortcomings, most conspicuously personal attitudes resulting in unbalanced questions, circumstantial judgment of the abilities and examinees’ limitations for presenting their capabilities; this is particularly true of psychiatry. The conventional method for oral examination is based on interviewing a patient in a limited time in the presence of examiners (Individual Patient Assessment). Due to the presence of a real patient and the delicacies of psychiatric interview (which is greatly influenced by the examinee’s mental and psychological conditions), the reliability of the conventional method may be questioned. The objective structured clinical examination (OSCE) may have certain advantages in creating homogeneity and expanding the spectrum of evaluation.

Harden was the first to describe the objective structured clinical examination in 1979 (Harden et al., 1979), stating that the time has come to evaluate the effects of this type of education and examination on capabilities of physicians in interaction with patients, families and colleagues. Recently, researchers have confirmed that OSCE improves the interpersonal communication skills of geriatrics with patients and their families (O’Sullivan et al., 2008). A review on articles published from 1986 to 2006 indicated that standard patients and simulation constitute helpful methods for training and evaluating medical students and residents of psychiatry (McNaughton et al., 2008). Similarly, another study reported that OSCE enhances the range of diagnoses made by medical students (Blaskiewicz et al., 2004). Finally, some studies confirm the high construct validity of this method, and emphasize using checklists for evaluation, diversity of stations as well as using standard patients and essay stations (Walters et al., 2005; Park et al., 2004).

Therefore, in this study, we recruited a number of exam participants during the last few years, some evaluated with conventional method and others with OSCE, to assess and compare their attitudes towards these two methods of examination using a questionnaire. The attitude consists of the overall attitude as well as attitude towards each element of the exam. Moreover, we assessed the role of age, sex, frequency of participation in the exam, score on written exam, university and the final outcome of the exam as probable factors affecting the attitude of the psychiatrists. It seems that there are few studies available on the examinees’ attitude, and it also seems that this item has an important role in designing exams in the future. Therefore, conducting similar studies are of importance. We hope this study will contribute to the positive modification and improvement of graduation examinations for residency in psychiatry.

Material and Methods

This is a descriptive-analytic study. We initially reviewed the literature and references and brought out the items; then designed a preliminary questionnaire holding the appropriate variables and sent it to the faculty members of the universities in the field of psychiatry. A modified version of the questionnaire was prepared in focus groups holding 15 faculty members of university of Social Welfare and Rehabilitation. In fact, they finally opted for the 11 questions in the focus group. Sampling from the population was randomized, and we subsequently recruited 134 psychiatrists as our sample. By considering 1/2 ratio from the population ((N = 400), α=0/05, d=0/02), the sample size stimulated 67 for each group. We chose the participants through the distribution of the questionnaire and call for participation in the annual conference of psychiatrists and the site of Iranian Psychiatric Association. Half of these participants were graduated in 2002-2005 and taken the conventional oral board examination (Individual Patient Assessment), while the other half, were graduated in 2006-2009, and had taken OSCE for their board examination. This number was based on an estimate of about 400 participants who have taken the test during the mentioned eight years. All these individuals had received their graduation certificate and participated in the oral exam after passing the written exam. The board exam in Iran is done in two stages (written and oral). First, one has to pass the written exam to be qualified to participate in the oral exam. Prior to the final study, we conducted a pilot study on 14 individuals – 7psychiatrists who graduated in 2002-2005 with the conventional exam and 7 who graduated in 2006-2009 with OSCE. The participants answered the questions and were asked to answer them again after two weeks. The reliability of the questionnaire was thus found to be 99%. After preparing the questionnaire, we used factor analysis to assess validity. The questionnaire had a main factor called attitude, so constant validity was also approved. Considering questions show intent of the researchers to assess so the questionnaire has content validity. After preparing the questionnaire, the characteristics of every question was discussed in the focus group, and finally the remained items were approved by all members. (Iterate validity).

Subsequently, the questions were submitted in person during the national psychiatrist’s convention or via email to all graduates of the mentioned years; and 134 psychiatrists completed the questionnaires (equally distributed between the two groups and collected in a similar manner). We developed two questionnaires, each pertaining to one group of the study. Each questionnaire consisted of 11 statements, and the participants expressed their opinion about them through a grading system of 0 to 4 (strongly disagree, disagree, neither agree nor disagree, agree, strongly agree). The statements were similar for both questionnaires, but the phrasing was selected in such a way as to reflect the respective examination in question. Each questionnaire also inquired about age, sex, frequency of participation in the exam, score on written exam, university of education and the final outcome. We used U Mann-Whitney test and regression analysis for data analysis. Because we used ranks instead of real grades, and different participants achieved different scores in separate situations, we used the non-parametric Mann-Whitney test to compare the means.

Prior to the test, we used Kolmogrov-Smirnoff test to verify the normal distribution; and the findings indicated that the variance of the scores was not significantly different between the two groups. Thus, we used the Mann-Whitney test to compare the ranks. We used a regression model to determine the impact of each variable on prediction of the attitude towards the exams.

Result

Out of 134 participants, 127 answered the question regarding the frequency of their participation in the exam, with 105 (82.68%) taking the test once, 17 (13.38%) taking it twice, 3 (2.36%) taking it three times, and 2 (1.58%) taking it more than three times. The mean age of the participants was 38.89 years (SD=4.29).

Of the participants, 83 (62.9%) were men and 49 (37.1%) were women.

The mean score on the written exam was 115.31 of 150 (SD = 8.1) for our participants, with the OSCE group achieving a mean score of 115.16 of 150 (SD = 7.67), and the conventional group achieving 115.45 (SD = 8.58). Out of 134 participants, 129 answered the questions regarding the final outcome of the exam, with 90 (69.8%) passing the exam and 39 (30.2%) failing it.

In the OSCE group, 48 participants (73.8%) passed the exam and 17 (26.2%) failed it, whereas in the conventional group, 42 participants (65.6%) passed and 22 (34.4%) failed the exam. There was a significant difference between the two groups in passing the exam (P = 0/01).

As for the university of education, 127 participants had answered the question, with 104 (81.89%) having studied in a first degree university and 23 (18.11%) in a second degree university. The grading of universities in Iran is announced by the Ministry of Health and Medical Education. The first degree universities consisted of medical universities of Tehran, Iran, Shahid Beheshti, Mashhad, Shiraz, Tabriz, Ahvaz and Isfahan, while the second degree universities included medical universities of Gilan, Mazandaran, Kerman and University of Social Welfare and Rehabilitation Sciences. Being graduated from a first degree university may indicate a higher quality of education, and often the majority of the examiners are chosen from the first degree universities.

No significant difference was observed between the two groups in demographic characteristics.

The overall attitude of the graduates was reflected in the question 11 of the questionnaires, with the results shown in Table 1.

Table 1

Elements and overall attitudes of participants towards Objective Structured Clinical Examination (OSCE) and conventional clinical examination

Elements of Participants’ Attitudes towards their respective exam	Groups	Strongly disagree		disagree		Neither agree nor disagree		agree		Strongly agree
		n	%	n	%	n	%	n	%	n	%
1) Patient ’s cooperation (skills of the simulated patients) affected my success.	Conventional	5	7.5	13	19.4	21	31.3	22	32.8	6	9
	OSCE	3	4.5	4	6	26	38.8	22	32.8	6	9
2) The selected disorders (subjects in each station) were appropriate for evaluating my abilities.	Conventional	5	7.5	9	13.4	25	37.3	23	34.3	5	7.5
	OSCE	0	0	9	13.4	28	41.8	26	38.8	3	4.5
3) The structure of the exam eliminated the personal attitudes of examiners.	Conventional	7	10.4	24	35.8	21	31.3	13	19.4	2	3
	OSCE	8	11.9	12	17.9	25	37.3	18	26.9	3	4.5
4) The duration of the exam (for each OSCE station) was sufficiently long.	Conventional	2	3	15	22.4	22	32.8	22	32.8	5	7.5
	OSCE	1	1.5	15	22.4	16	23.9	30	44.8	4	6
5) Prior knowledge of the exam affected my success.	Conventional	4	6	13	19.4	20	29.9	25	37.3	5	7.5
	OSCE	5	7.5	8	11.9	11	16.4	35	52.2	6	9
6) My preparations during the days leading to the exam affected my success more than the abilities I had acquired during my residency program.	Conventional	8	11.9	22	32.8	15	22.4	19	28.4	3	4.5
	OSCE	4	6	11	16.4	25	37.3	22	32.8	4	6
7) My theoretical knowledge affected my success on oral exam.	Conventional	4	6	15	22.4	31.3	31.3	22	32.8	5	7.5
	OSCE	2	3	3	4.5	23	34.3	32	47.8	6	9
8) The structure of the exam was capable of assessing my abilities to establish rapport with patients.	Conventional	5	7.5	16	23.9	18	26.9	21	31.3	7	10.4
	OSCE	6	9	14	20.9	13	19.4	24	35.8	7	10.4
9) The final outcome of the exam was the same as what I expected at the end of the exam.	Conventional	8	11.9	16	23.9	17	25.4	22	32.8	4	6
	OSCE	9	13.4	11	16.4	15	22.4	25	37.3	6	9
10) The structure of the exam was served to maintain homogeneity and justice for all participants.	Conventional	12	17.9	26	38.8	19	28.4	9	13.4	1	1.5
	OSCE	10	14.9	11	16.4	20	29.9	22	32.8	3	4.5
11) Overall, this examination is appropriate as a graduation exam.	Conventional	7	10.4	26	38.8	16	23.9	14	20.9	4	6
	OSCE	9	13.4	9	13.4	18	26.9	22	32.8	7	10.4

As Table 1 demonstrates, the majority of the participants did not have a good attitude towards the conventional clinical exam, as 26.9% chose the “agree or strongly agree” and 49.2% chose the “disagree or strongly disagree” options.

On the other hand, 43.2% of the participants evaluated OSCE as agree or strongly agree, while 26.8% considered it disagree or strongly disagree, and 26.9% had a neuter attitude towards it.

Table 1 presents the participants’ attitudes towards each element of the questionnaire.

Table 2 compares the overall attitude, as well as the attitude towards each element between OSCE and conventional clinical examination.

Table 2

Comparing the participants’ attitudes towards Objective Structured Clinical Examination (OSCE) and conventional clinical examination in overall attitude and separate elements

Elements	Groups	Average of ranks	Mann-Whitney	P
1) Patient ’s cooperation (skills of the simulated patient) affected my success.	Conventional	60.99	1808.50	0.23
	OSCE	98.35
2) The selected disorder (subjects in each station) was appropriate for evaluating my abilities.	Conventional	65.16	2088	0.56
	OSCE	68.86
3) The structure of the exam eliminated the personal attitudes of examiners.	Conventional	62.02	1877.50	0.12
	OSCE	72.05
4) The duration of the exam (for each OSCE station) was sufficiently long.	Conventional	63.96	2010.50	0.42
	OSCE	69.04
5) Prior knowledge of the exam affected my success.	Conventional	61.53	1844.50	0.11
	OSCE	71.62
6) My preparations during the days leading to the exam affected my success more than the abilities I had acquired during my residency program.	Conventional	60.40	1769	0.03
	OSCE	73.70
7) My theoretical knowledge affected my success on oral exam.	Conventional	59.16	1686	0.01
	OSCE	74.95
8) The structure of the exam was capable of assessing my abilities to establish rapport with patients.	Conventional	64.87	2068.50	0.72
	OSCE	67.18
9) The final outcome of the exam was the same as what I expected at the end of the exam.	Conventional	64.34	2032.50	0.40
	OSCE	69.70
10) The structure of the exam was served to maintain homogeneity and justice for all participants.	Conventional	57.65	1584.50	0.004
	OSCE	76.49
11) Overall, this examination is appropriate as a graduation exam.	Conventional	59.65	1718.50	0.03
	OSCE	73.56

As the table depicts, OSCE is significantly superior compared to the conventional clinical examination in the mean ranks of overall attitude towards exams (p = 0.03) in the role of preparation during the days leading to the exam (p = 0.03) in the impact of theoretical knowledge (p = 0.01) and in the role of the exam structure in maintaining homogeneity and justice.

In the elements of patient cooperation (skills of the simulated patient), the selected subject (patient), exam structure, duration of the exam, prior knowledge of the exam, capability of the exam to assess patient-therapist rapport and predictability of the exam outcome, OSCE achieved higher mean ranks compared to the conventional exam. The difference, however, was not significant.

Considering the F value of ANOVA (F = 2.58) which is significant at a confidence interval of α = 0.02,(Table 3), it may be stated that the variable attitude can be deduced from age, sex, frequency of participation, passing or failing the exam, university of education and score of the written exam. Thus, a better understanding of the impacts of these independent variables requires careful examination of regression coefficients, presented in Table 4.

Table 3

Results of ANOVA

Source of changes	Sum of squares	Mean of squares	Degree of freedom	F	P
Regression	18.37	3.06	56	2.58	0.02
Remaining	98.52	1.18	83
Total	116.9		89

Table 4

Stepwise Regression coefficients Models of predicting variables

Predicting Variables	Standard Coefficients (D)	T	P
Age	0.810	0.91	-
Sex	0.004	0.04	-
Frequency of Participation	0.22	2.08	0.04
Success in Exam	0.31	2.69	0.009
University of Education	0.04	0.38	-
Score of Written Exam	0.112	0.07	-

As the table indicates, the variables “frequency of participation” and “success in exam” is able to predict the attitude: i.e., these variables may determine the attitude of the participant towards the examination.

The predicting power for the variable “success in exam” is greater than that of the variable “frequency of participation”. Comparing the means reveals that those participants who pass the exam and those who take it for the first time have a more positive attitude towards it. Age, sex, score of written exam and university of education did not influence the participants’ attitude towards the exam.

Discussion

Previous studies have suggested the need for using OSCE in evaluating physician’s abilities on graduation (Harden et al., 1979), and its positive impact on psychiatric residents has been demonstrated (O’Sullivan et al., 2008).

Perhaps no previous study compared OSCE with the older methods of examination in the field of psychiatry; and our findings indicate that test-takers who are evaluated using OSCE tend to have a more positive attitude towards the test and believe the impact of theoretical knowledge and preparation for the exam to be greater compared to the other group. Of course, this finding does not necessarily indicate that success in OSCE has a direct connection with the achieved abilities during the whole period of the residency. Moreover, they believe OSCE to be superior to the conventional examination (Individual Patient Assessment) in terms of maintaining justice and homogeneity. Probably these items were effective in the attitude of examinees, so OSCE may be preferred considering this aspect.

A study reported the residents’ feedback to be positive in terms of OSCE’s impartiality and appropriateness (Sauer et al., 2005). Also, another study demonstrated that the OSCE scores are homogenous for different schools throughout one nation (Hickling et al., 2005). In addition, researchers indicated that OSCE eliminates the confounding impact of difference in essay abilities of psychiatric residents (Park et al., 2005).

On the other hand, our study indicates that the final outcome and the frequency of participation in the exam influence the attitude towards it; thus, it appears that in the future studies, participants should be inquired about the examination immediately after taking it and before the results are announced. Furthermore, participants should be categorized based on the frequency of their participation in the exam, and each category must be analyzed separately.

One limitation of our study is the difference in performing the abilities of simulated patients over the duration of our study. We recommend that the future studies use a group of trained simulated patients over different years without changing them. Different performances can cause injustice for the examinees. Another limitation is the issue of the residents’ feedback regarding the simulated patients. Also, the effect of time on candidate’s opinion is an important factor which seems to be different between the conventional method (IPA) and OSCE.

Another limitation is related to the “Cohort effect” which means that investigating the attitude sometimes cannot show reality because it is possible that some cases have more positive attitude towards the exam and this is against the randomized sampling.

A study conducted at Rouzbeh Psychiatric Hospital in Tehran emphasized the need for the confirmation of the trained standardized patients by residents (Sadeghi et al., 2007). Moreover, another study reported that OSCE is a reliable and cost-effective method requiring little resources; however, it needs evaluation and feedback systems to improve its capabilities (Chandra et al., (2009). On the other hand, one study dealt with the score assigned to the resident by the standardized patient (Whelan et al., 2009).

In general, based on the examinees’ attitudes, it may be concluded that OSCE is an appropriate option for the oral graduation exam as confirmed by previous studies (McNaughton et al., 2008; Walters et al., 2005;Park et al., 2004).This is the first study to compare the attitude of the examinees’ towards OSCE and conventional interview examination. We recommend future studies to use larger populations and more varied questions. In addition, further studies are required to combine both methods and assess the results.