Evaluation of moral competency using standardized patient: presenting an experience.

In this study we discuss our experience of including an ethics objective structured clinical examination (OSCE) station in endocrinology board exam. One OSCE station on truth telling was developed and a standardized patient was trained for role playing in this station. Based on a pilot study, the evaluation checklist got modified. Then the finalized station added into the OSCE phase of endocrinology board exam. Based on this experience, adding ethics station in board exams is practical and reasonable. Since OSCE method could evaluate students' ethical decision making and communication skill it could be used in combination with other kinds of evaluation in assessing ethics competency of graduates. Using this method could push the ethics learning approach toward more practical and skill based ones.

Introduction

One of the important challenges in medical ethics education is the evaluation of trainees. Evaluation of students’ competencies is one of the essential tools for assessing success of education programs in accomplishment of its objectives. On the other hand, evaluation is a major determinant of students’ learning patterns. In other words what is not evaluated will not be considered important (1, 2). Evaluation method should be compatible with educational objectives. Some believe that ethical evaluation of medical students should emphasize on ethical knowledge and moral reasoning while some others think that in addition to these objectives, compassion, respect and altruism are essential moral competences which need to be evaluated (3). It means that although moral reasoning is an important goal, this competency does not necessarily end to ethical and professional practice of students.

The other challenge in moral evaluation is that how it is possible to evaluate students` attitude and values and ethical practice in a valid and reliable way (4). In published literatures there are so many papers introducing and discussing different tools for evaluation of ethical attitude, reasoning and behavior (2, 5–7). Defining issue test (DIT), sociomoral reflection measure (SRM), and moral judgment interview (MJI) have been developed for evaluation of moral reasoning, not being enough specific they are used generally for research purpose though (8). In practice, for evaluation of students’ medical ethics competency, methods such as multiple choice questions, short essay, portfolio, objective structured clinical examination (OSCE), faculty evaluation and 360 degree evaluation are being used (2, 7). Although there is no consensus among ethicists on the best evaluation method, they all agree that different methods should be used (7). One method which is able to evaluate trainee’s competencies in practice is OSCE. This method not only evaluates trainee’s ethics knowledge but also assesses their moral reasoning and ethical behavior. Nowadays increasing number of medical ethics OSCE stations are including in comprehensive exams in different educational phases in medical schools around the world (2, 9–11).

In Iran, there is a consensus about the necessity of evaluation of graduate’s ethical competency among medical education leaders and they have mandated adding some medical ethics multiple choice questions (MCQ) to pre-internship comprehensive exam, residency entrance exam, annual residency and board exam. Since multiple choice question method alone is not enough for moral competency evaluation, we decided to assess practicality of evaluation of ethical competency by designing and conducting an OSCE station for medical ethics. Success of this method could convince educational leaders to use other methods for evaluation of graduate’s ethical competency. In this paper we present our experience of including one medical ethics station in seven endocrinology OSCE stations of endocrinology board exam.

Method

We used bioethics OSCE experience of Toronto center for bioethics to design one station on truth telling. The scenario is about an inoperable pancreatic cancer patient whose wife asks physician not to tell the diagnosis to the patient. A standardized patient played the role of the patent’s wife in this station and talked to examine physicians. We used a ten item checklist to rate physicians’ interaction in this station. We also used two global rating questions using Likert scale to score their ethical decision making and communication skill.

This station was pilot tested in a medical ethics education workshop in May 2009 which the standardized patient was trained for. In pilot conduct of this station some ethics teachers and bioethics PhD students had participated. One of workshop attendees participated as the examinee and other six participants rated his performance. All scores were very close to each other, indeed 5 out of six participants gave the same mark to the examinee and the reliability of station was 0.83. Participants also evaluated the face and content validity of the checklist. Based on their suggestion one item added to the checklist hence the station and its checklist got standardized.

The finalized station included in the OSCE part of the endocrinology board exam in September 2009. Nineteen physicians took this exam. Time for each physician’s interview with simulated patient in ethics station was 5 minutes. One rater (first author) was present in the station to fill checklists and rate examinees.

Results

Most participants had good to excellent communication skill and only one participant did not communicate with the standardized patient and easily accepted her request (table 1). In their ethical decision making, %26 (5) of participants had poor performance and only 3 of them had good or acceptable ethical encounter.

Table 1

Frequency (percent) of ethical performance and communication skill score of endocrinology board examinee.

	poor	fair	good	Very good	excellent
Attention to ethical issues	5 (26.3)	11 (57.9)	3 (15.8)	0 (0.0)	0 (0.0)
Communication skill	1 (5.3)	0 (0.0)	6 (3.16)	5 (26.3)	7 (36.8)

Most participants asked for the patient’s wife reason for her request (15 people) and almost all of them (18 people) mentioned that patient had right to know about his condition. The most prevalent mentioned justification for patient’s right was the need to plan for the rest of life (12 people). Two people told that patient might ask about his condition and one participant gave several different reasons for refusing her request such as she is not telling lie to patient because it is unprofessional, finally patient would find out they were hiding something form him, also if he heards about his condition he would lose his trust on physicians and would be affected by a worse emotional stress. Nine participants refused to withhold the truth from patient, however no one acknowledged the patient’s right not to know. All of them believed that they had to give information to the patient in any case.

Mean number of mentioned items by participants was 3.53 out of 11(1–7). There was a significant correlation between participant’s score of communication skill and score of their ethical decision making (r=0.48, P=0.03).

Although the mean score of board OSCE for those with good ethical performance was higher, this correlation was not significant (table 2). A positive correlation has been observed between participants’ OSCE score and their communication skill score but it was not significant. Mean ethical performance score of female participants (2, 17) was significantly higher than male ones (1.43 out of 5) (P=0.01), however there was no significant difference between their communication skill.

Table 2

Mean score of board exam of participants with different level of ethical performance and communication skill.

		poor	fair	good	Very good	excellent	total
Pay attention to all ethical issues	Mean score of OSCE board exam	79.8	77.1	90.3	-	-	79.9
	number	5	11	3	0	0	19
Communication skill	Mean score of OSCE board exam	75	-	76	79.8	84	79.8
	number	1	0	6	6	7	19

Discussion

Small though, this experiment showed practicability of conducting and including an OSCE ethics exam in evaluation of trainees. Low score of participant’s ethical performance was due to lack of training in their educational course and we could not expect them to show good ethical competency just by reading ethics text books.

OSCE is a good method for integrated evaluation of trainees’ knowledge and skills in facing with ethical dilemmas. This method of evaluation could improve teaching and learning methods. Evaluation method is an important determinant for learner’s learning behaviors (5). If we just use multiple choice questions for evaluation of ethics competency, students learning will be limited to knowledge while using methods that could evaluate their moral judgment and behavior will encourage them to learn practical ethics competencies. Moreover standardized patient could be used as a teaching method. In teaching medical ethics besides reflection and discussion on ethical dilemmas, students need to practice their ethical skills and receive feedbacks on their practice (13).

We should notice that OSCE method alone is not enough for precise evaluation of ethical practice. Since the pattern of all ethics station could be recognized by students and they could get ready for showing good competency in those stations, being able to have ethical performance is not equal to having ethical practice (5). In other words, OSCE method is not able to evaluate students’ ethical attitude. Thus other evaluation methods such as 360 degree evaluation are needed to assess this dimension of educational objectives in medical ethics (2). Furthermore OSCE method is very costly and having enough number of ethics station will pose logistical difficulties. In a study conducted by Singer et al, 4–6 ethics station had internal consistency reliability (Cronbach’s α) of 0.28 to 0.46. They estimated that at least 41 ethical stations are needed in an OSCE test for achieving acceptable internal consistency of 0.8 (12).

Although inter-rater reliability of our station in its pilot phase was good, because of small sample size in the pilot study, we should have evaluated its reliability in the main study. Unfortunately due to logistical problems we were not able to have two raters, so we did not evaluate inter-rater reliability of the station.

The other important pitfall of our study was that one station has not content validity for assessing ethical performance. In other words, we could not be sure of ethical competency of physicians in encountering with different types of ethical problems in their professional life through watching their conduct in facing with just one ethical problem.

Conclusion

This study showed that it is practical to develop an ethics station in comprehensive medical exams in different phase of medical education course and this method could motivate medical students to learn ethical practice. Despite all straight points of this evaluation method, it should be noticed that conducting one or two OSCE station in graduate comprehensive exam is not enough for evaluation of their ethical practice and other complementary evaluation methods such as continuous peer and faculty evaluation of students during their educational course are necessary.