Combined student ratings and self-assessment provide useful feedback for clinical teachers.

Many evaluation instruments have been developed to provide feedback to physicians on their clinical teaching but written feedback alone is not always effective. We explored whether feedback effectiveness improved when teachers' self-assessment was added to written feedback based on student ratings. 37 physicians (10 residents, 27 attending physicians) from different specialties (Internal Medicine, Surgery, Obstetrics/Gynecology, Pediatrics, Neurology, Dermatology, Ophthalmology, ENT, and Psychiatry) were invited to fill out a self-assessment questionnaire on their teaching skills. Students completed an almost identical questionnaire to evaluate the same teachers based on their experiences during clerkships. After receiving written feedback incorporating their self-assessment and the student ratings, the teachers indicated their perceptions of the self-assessment exercise and the written feedback in a questionnaire (five-point Likert scale items) and next, in more detail, in semi-structured interviews with a purposive sample of 12 of the participating teachers. 25 physicians participated (67%). The results showed that self-assessment and student feedback were both perceived as useful (3.7, SD 1.0) but the latter was considered more effective. The physicians we interviewed considered the combination of self-assessment with student ratings more effective than either self-assessment or written feedback alone. Notably, discrepancies between student ratings and self-assessment were deemed a strong incentive for change. We conclude that self-assessment can be a useful tool to stimulate improvement of clinical teaching when it is combined with written feedback based on student ratings. Future research among larger groups is needed to confirm our findings and examine whether these combined tools actually lead to improved teaching.

Introduction

Clinical teachers make an indispensible contribution to the learning of medical students in the clinical workplace (Dolmans et al. 2002; Dornan 2006; Stern et al. 2000). This contribution is the more important since research and experience have shown that workplace learning is implicit, haphazard, and dictated by the demands of the workplace (Eraut 2004) rather than by students’ learning needs (Collins et al. 1989). Clinical teachers face the daunting task of meeting high performance standards of both service and teaching. Good teachers can optimize students’ learning experiences by providing support, direction and feedback (Kilminster and Jolly 2000), and they can be helped to do this effectively by feedback to gauge their performance and point out areas for improvement. Although various instruments, mostly written questionnaires, are available for evaluating the teaching skills of clinical teachers (Beckman et al. 2004), these instruments have limitations, especially regarding the uptake of feedback. Whether feedback is assimilated, accepted, and eventually applied for learning and improvement depends on various internal and external factors (Sargeant et al. 2008). The latter comprise the nature of feedback—written feedback must be specific, diagnostic and descriptive (Braskamp and Ory 1994)—the credibility of feedback, the professional culture of the organization, and facilitation of feedback. As for internal factors, self perception of performance and personal reflection are equally likely to hamper as stimulate the uptake of feedback. Moreover, research has demonstrated that written feedback alone is not sufficient to convince teachers to change their teaching (Beijaard and De Vries 1997). For teachers to effectively change their teaching behavior they need to be convinced that there is a real problem. Alternatively, change can be the upshot of extensive reflection, which can be triggered by new ideas about effective teaching (Hashweh 2003; Richardson and Placier 2001). Reflection, which has been shown to clarify a person’s perceptions (Eva and Regehr 2008), including those of teaching skills, can be stimulated by self-assessment. Knowing that feedback alone has limited power to foster improvement while reflection can be instrumental in this, we examined the effectiveness of feedback combined with self-assessment to enhance teaching quality.

In medicine, self-assessment is generally explored from the perspective of the accuracy of physician self-assessment, usually of clinical knowledge or diagnostic accuracy (Davis et al. 2006). Studies are of the “guess your grade” type in which physicians are asked to predict their score on an external examination (Colliver et al. 2005). In a teaching context a different perspective seems preferable, one in which self-assessment is interpreted as a learning experience that can be instrumental in stimulating workplace learning by identifying standards of excellence, deficits, and suggestions for improvement (Colliver et al. 2005; Ross and Bruce 2007). For the present study, we used the following definition of self-assessment (Eva and Regehr 2008, p. 15):

(…) a pedagogical process by which one takes personal responsibility for looking outward, seeking feedback and explicit information from external sources, then using these externally generated sources of assessment data to direct performance improvements.

In a review of studies on the effectiveness of self-assessment, Colthart et al. (2008) deplored the general neglect of the views of those who engage in self-assessment, calling for research into the acceptability of self-assessment to those who use it as a learning tool. This prompted us to seek medical teachers’ views on the potential of self-assessment to supplement and augment the effects of feedback from students in guiding and encouraging their efforts to optimize clinical teaching.

We investigated whether a structured combination of written feedback based on student ratings and teacher self-assessment could be an incentive for clinical teachers to adapt their teaching practice. The following research questions were addressed:

To what extent do clinical teachers perceive self-assessment as a useful method to obtain insight into their strengths and weaknesses as a clinical teacher and do they think it will help them to improve or change their teaching?

To what extent do clinical teachers perceive written feedback (based on student ratings) as useful for obtaining insight into their strengths and weaknesses as a clinical teacher and do they think it will help them to improve or change their teaching?

Do clinical teachers think that a combination of self-assessment and written feedback augments the effectiveness of feedback?

Method

Context

Between March 2007 and March 2008 we collected data in a non-academic teaching hospital in the Netherlands. First, all the students on clinical rotations in the hospital were asked to complete an evaluation form for each of the three clinical teachers who had been most involved with their learning during the last rotation they had completed. Next, we invited the clinical teachers who had been evaluated by at least six students to participate in our study. The minimum of six ratings was based on evidence that this number is required to obtain reliable student ratings for feedback (Dolmans and Ginns 2005).

Participants

37 physicians (10 residents and 27 attending physicians) from different specialties (Internal Medicine, Surgery, Obstetrics/Gynecology, Pediatrics, Neurology, Dermatology, Ophthalmology, ENT, and Psychiatry) were asked to participate. Participation was voluntary and the principal investigator had no professional or personal relation with any of the participants. All research aims and procedures were explained to the participants and data were handled confidentially.

Instruments and procedure

Self-assessment, written feedback, and evaluation

In May 2008, the 37 invited physicians received a validated self-assessment questionnaire consisting of 25 Likert-type items (1 = fully disagree; 5 = fully agree) and two open-ended questions inviting comments about their strengths and weaknesses as a teacher (Appendix Table 3) (Stalmeijer et al. 2008). The items asked about teaching-related issues, like providing feedback to students, and were identical to those of the student evaluation questionnaire only phrased from the teacher’s point of view (student version: ‘the clinical teacher stimulated me to ask questions in order to increase my understanding’, physician version: ‘I stimulate students to ask questions in order to increase their understanding’). Data on age, years of experience in medicine, years of experience as a clinical teacher, and attendance of courses in clinical teaching were also collected. The questionnaires and a letter explaining the goals and procedures of the study were mailed to the physicians; non-responders received an email reminder.

Table 3

The self-assessment instrument

The physicians who returned the completed self-assessment questionnaire received individual, written feedback (Appendix Table 4) comprising (1) student ratings: mean, SD, and number of students; (2) their self-assessment on each item, and (3) the overall score of their colleagues who had received six or more ratings. Attached to the feedback was a 10-item Likert type questionnaire asking the teachers to evaluate the process of self-assessment and written feedback (Table 2).

Table 4

Example of feedback provided to physicians

Table 2

Mean ratings and SDs on a Likert scale (1 = fully disagree; 5 = fully agree), for the perceived effects of self-assessment and feedback

	N	Mean	SD
1. The items of the self-assessment questionnaire are in alignment with my views on clinical teaching	24	3.7	0.8
2. Filling out the self-assessment questionnaire has given me a better idea of my performance as a clinical teacher	25	3.7	1.0
3. Filling out the self-assessment questionnaire has provided me with insights into how I can improve my performance as a clinical teacher	25	3.8	1.1
4. The feedback of the students closely resembled my self-assessment	25	3.6	1.0
5. The feedback from the students has provided me with insights into how I can improve my performance as a clinical teacher	25	3.8	1.1
6. Based on the feedback I am planning to change/improve my clinical teaching	25	3.7	1.0
7. I learned a lot by filling out the self-assessment questionnaire	24	3.6	1.1
8. I learned a lot from the student feedback	25	4.2	0.8
9. I learned a lot from students’ responses to the open-ended questions	25	4.0	1.1

Interviews

Based on the outcomes of the self-assessment, the student ratings, and the process evaluation questionnaire, 12 physicians were invited to explore, in a 40-min interview, the value of self-assessment and feedback as quality improvement tools for clinical teachers. The demographic information enabled purposive sampling of participants to ensure well-balanced representation of the population.

RES (an educationalist) conducted the semi-structured interviews (June 2008–October 2008), which were audio recorded. The interviews were guided by an interview protocol (Appendix Table 5) directed at the physicians’ general views of clinical teaching, factors influencing the effectiveness of self-assessment, the written feedback (Braskamp and Ory 1994; Coe 1998; Sargeant et al. 2008), and the value of self-assessment and feedback from students. Participants were assured of confidentiality and received the transcript of their interview by e-mail with the request to further clarify their comments and the option to withdraw statements if deemed desirable. They were also assured that no statements would be traceable to individuals and conclusions would apply to the population as a whole and not to individuals.

Table 5

Interview protocol

1. In what way do you think students can learn the most from you?/What do you have to offer to students?

Can you describe how you guide students?

Is there a discrepancy between how you would like to teach and how you are actually teaching?

2. Are the items of the self-assessment questionnaire in alignment with your views regarding clinical teaching during clerkships?

Which items are in good agreement with your approach to clinical teaching?

Which items are not in agreement with your approach to clinical teaching?

If items are not in alignment with your views: do you consider them useful? Why/why not?

3. Do you think it is helpful to complete a self-evaluation questionnaire?

Is self-assessment new to you? Have you self assessed in other situations? Did you think self-assessment was useful in those cases?

4. Has completing the self-assessment given you a clearer picture of how you function as a clinical teacher for students?

5. Has completing the self-assessment given you concrete suggestions for changing or improving your teaching of students?

6. Has completing the self-assessment led to concrete changes in the way you teach medical students/or do you intend to change the way you teach? If yes, how? If no, why not?

7. Do you think this combination of written feedback and additional data is useful for you?

If yes, why, if not, why not?

Which information do you find particularly useful?

(useful to receive information from students ← credibility)

(How did you experience the feedback (positive/negative ← acceptance))

8. Were there striking discrepancies between your self-assessment, student assessment, and your colleagues’ mean ratings?

If yes, which discrepancies were there? If no, what are your thoughts about that?

How has this affected your views regarding the teaching of medical students?

9. Has the feedback you received given you concrete ideas for changing or improving the way you teach medical students?

Since you received the feedback have you changed aspects of your teaching of medical students?

If yes, please give a concrete example? If not, why not?

Are there any concrete factors that are a barrier to changing your teaching? ← How might this be resolved?

10. How can this type of written feedback be improved?

Which additional information?

Which additional judges (besides students)?

Different presentation? Different form of communication?

11. In retrospect, which has been more useful for you: self-assessment, feedback, or the combination of the two? Why?

Analysis

Evaluation questionnaire of the self-assessment and the written feedback based on student ratings

Means and SDs were calculated for all questionnaire items.

The interviews were transcribed verbatim and submitted to the participants for approval. RES summarized the transcripts guided by the interview protocol. Next DHJMD and IHAPW each read half of the interviews, comparing them with the summaries and discussing points of disagreement to reach consensus. Finally, RES used ATLAS-ti (version 5.2) to analyze the data searching for recurring themes and concepts in relation to factors pertinent to the usefulness and effectiveness of the self-assessment and the written feedback.

Results

Self-assessment and feedback

Response

Of the 37 physicians invited to fill out the self-assessment questionnaire, 30 returned the questionnaire (response 81%) and received feedback. Of the respondents 27% were female and 70% attending physician. All major disciplines were represented (Internal Medicine, Surgery, Obstetrics/Gynecology, Pediatrics, Neurology, Dermatology, Ophthalmology, ENT, and Psychiatry). The median age was 45 years and the average experience in clinical teaching 5 years or more. Forty percent had never attended a course on clinical teaching, while 60% had attended one or more courses (see Table 1).

Table 1

Demographic information respondents (N = 30)

Sex	N (%)	Age	N (%)	Status	N (%)
Male	22 (73%)	<30	5 (17%)	Resident	9 (30%)
Female	8 (27%)	30–40	9 (30%)	Attending	21 (70%)
		40–50	6 (20%)
		>50	10 (33%)

Evaluation of self-assessment and feedback

Of the 30 physicians who received feedback, 25 returned the evaluation questionnaire (83%), i.e. 67% of the initial group. Of the five non-responders, one had switched hospitals. After inquiry, the four other physicians (attending physicians with multiple years of experience) admitted to have forgotten to fill out the evaluation form. Their student ratings did not differ significantly from the overall population.

Questionnaire

The physicians appreciated the process of self-assessment and written feedback (mean rating 3.7). Student ratings and their written comments on teachers’ general strengths and weaknesses were valued the most. Seventy-six percent of the respondents would like to receive similar feedback in the future (Table 2).

Interviews

All 12 invited physicians [three residents, nine attendings, three female, nine male, from nine disciplines (Table 1)] agreed to be interviewed. They received the transcript of their interviews and three made minor textual changes.

Self-assessment

The self-assessments gave rise to three interview themes: (1) self-assessment in relation to physicians’ views on effective clinical teaching; (2) the usefulness of the self-assessment exercise, and (3) the extent to which self-assessment promoted teachers’ insight and changes in teaching practice.

Self-assessment + views of clinical teaching

Nine of the participating physicians felt the items of the self-assessment instrument resonated with their personal views on clinical teaching: demonstrating skills and students shadowing them, discussing patients and asking questions, and creating opportunities for students to perform tasks under supervision. Nevertheless, several physicians thought the entire list could only be realized in an ideal situation with sufficient time and opportunity for clinical teaching. One physician referred to the instrument as a list of the basic skills any clinical teacher needs. All physicians thought the instrument offered a useful framework for self-assessment.

Usefulness of self-assessment

Several physicians felt the self-assessment exercise was useful and provided a framework and standards for good clinical teaching. It stimulated reflection and helped them to look at their teaching in a more structured way.

But it is definitely worthwhile to do it because (…) [for you think] hey, should I do something about that? (…) But that way just completing the list shows you where you might make improvements. (Resident 1)

No, this sort of gives more structure. So what you know about yourself is usually much more general not split into 25 different themes. And with some things you think, God, yes how is that really, so in that sense I think it is useful to look at it that way, definitely. (Attending Physician 6)

Other physicians disliked the subjectivity of self-assessment and preferred feedback from colleagues or students.

I think it is much more important what others think of me. Whether I think that I do enough in demonstrating things to students, I mean that is totally irrelevant. If the rest of the students or residents here were to say ‘You do not demonstrate enough’ then I would have to pay attention to it. (Attending Physician 8)

Did self-assessment foster insight and improvement?

The majority of the teachers thought the self-assessment merely confirmed their existing views and teaching activities and did not shed new light on their strengths and weaknesses. The more experienced clinical teachers in particular said they had learned nothing new. Although the exercise was said to have provided eye-openers and suggestions for improvement, only a few physicians reported actual changes in their teaching. The prevailing opinion was that nothing new could be added to their teaching repertoire and that time constraints prevented changes in teaching. Any reported changes were small and practical like stimulating students to formulate personal learning goals.

Written feedback based on student ratings

Several issues were raised concerning the feedback: (1) physicians’ stance toward student ratings, (2) the nature of the feedback, (3) whether the feedback provided new information, and (4) improvements resulting from the feedback.

Physicians’ stance toward student ratings

The written feedback (Appendix Table 3) was highly appreciated and quite a few physicians said they had been very curious about their personal ratings. Student feedback posed no problems. The physicians felt it kept them on their toes and saw it as a part of today’s world. Some teachers said that since they gave feedback to students, students should be entitled to give them feedback too.

(…) for I see students as clients too. And then you should also ask, well are you satisfied with how things are going? And how are we going to turn you into the best physicians … (Attending Physician 9)

The nature of the feedback

The physicians characterized the feedback as specific, informative and illuminating. They liked the additional written comments and some even said they would like face to face feedback from students but realized this was probably a bit much to ask given the inherent inequality of teacher–student relationships.

Feedback usefulness

Discrepancies between their self-assessment and student ratings on the item level were experienced as powerful triggers for reflection and action. Occasionally, physicians had been pleasantly surprised, but they all remarked on the confrontational nature of these data. All agreed that item-level discrepancies were highly informative and served as a focus and foundation for improvement. Reported student ratings of colleagues had little impact but did offer a point of reference and helped to define standards. Interest was stronger in ratings of colleagues in the same department and discipline than at the hospital level.

For you can see whether you are doing worse or better. Yes, on the one hand, interesting, of course it is about the students. But on the other hand I think, if you want more insight into your general functioning then it is a good thing to see; how would you assess yourself and what do your colleagues think, so where is the discrepancy. (Resident 3)

Improvements made based on the feedback

Steps taken by teachers in response to the feedback were: encouraging students to formulate learning goals, providing a more structured learning experience, and creating more opportunities for students to undertake activities under supervision. Interestingly, some physicians from specialties with shorter clerkships (ENT, Ophthalmology, and Dermatology) commented they had too little time with students for meaningful clinical teaching and it was just not feasible to undertake additional or new teaching activities.

Combining self-assessment and feedback

When asked what was more useful, self-assessment or feedback, all but one physician opted for feedback. However, they also acknowledged that self-assessment did impact on the evaluation process.

[The feedback] indicates the discrepancies just that little bit more clearly, so I think you learn more than when you just assess yourself (…) because it is specifically [in the feedback] that you can see the differences [with the self-assessment] and I think that tells you more. (Attending Physician 3)

Eleven physicians were in favor of annual or biannual feedback. As for the self-assessment they proposed a repeat every 4 or 5 years with the current measurement as baseline. It was suggested, as an improvement, to enable physicians to discuss their feedback with a supervisor or educationalist. However, asking a colleague for feedback on their clinical teaching was deemed neither desirable nor viable, because colleagues were not considered acceptable judges of their clinical teaching skills, especially since clinical teaching was typically viewed as a ‘solo endeavor’. A wish for additional information on clinical teaching, like specific tips and tricks for supervising students, was expressed by some physicians.

Discussion

There are numerous instruments for eliciting student ratings of clinical teaching as an incentive for improvement (Beckman et al. 2004). However, many factors interfere with the effectiveness of student feedback and much is left to serendipity (Coe 1998). Looking for ways to increase the impact of student feedback, we sought clinical teachers’ perceptions of the effectiveness of supplementing student feedback with teachers’ self-assessment.

Self-assessment

Teachers generally thought that self-assessment as a single tool had a limited impact as a stimulus for change. While the majority valued the self-assessment, only a minority were actually motivated to implement changes. The reported congruence of the self-assessment instrument with the participants’ views on teaching is significant, because a feedback instrument can only be effective when it reflects views not too far removed from those of the target group. The instrument also appeared to be effective in achieving one of the aims of self-assessment, i.e. to help define standards of excellence (Ross and Bruce 2007), with physicians saying that it afforded a more structured picture of their clinical teaching. While acknowledging the value of self-assessment many physicians saw no need for change. Particularly, the older physicians (≥50 years) commented on the instrument’s low news value, which is not surprising considering that physicians with many years of teaching experience are likely to have a firmly established sense of self-efficacy as a teacher and thus feel they have little to learn from self-assessment (Colthart et al. 2008). When interpreting these results, one of the questions becomes to what extent the physicians had already engaged in significant reflection on their teaching skills without the encouragement of a formal self-assessment rendering this intervention obsolete. The interview data showed that several physicians were already engaged in some sort reflection on their clinical teaching but that it was no more than a general idea of their function as clinical teacher. As one of the physicians formulated it ‘So what you know about yourself is usually much more general not split into 25 different themes’ meaning that the self-assessment stimulated deeper and more structured reflection about their clinical teaching skills. Additionally, to explain the lack of news value, it should be noted that the instrument was implemented top down and not to meet a specific need or interest expressed by clinical teachers. Meeting a real need is a strong determinant of success of self-assessment as a learning tool (Coe 1998; Sargeant et al. 2008).

Written feedback based on student ratings

The physicians took student feedback very seriously and qualified it as specific, informative and illuminating. This suggests that the physicians experienced the feedback as credible and were open to it, which is considered favorable for feedback effectiveness (Coe 1998).

Written feedback and self-assessment

Although the effect of self-assessment in itself was limited, the findings on self-assessment in combination with student ratings were more promising. As mentioned above, the physicians were open to student feedback and, even more significantly, experienced discrepancies between student feedback and their self-assessment as a strong incentive to contemplate change. That physicians take student feedback seriously is in line with a study on multi-source feedback by Sargeant et al., who found that feedback from patients, who judged their physicians based on concrete experience, was a stronger stimulus for physicians to consider modifying their practice than feedback from colleagues or allied health professionals. Perhaps clinical teachers view student feedback in a similar way, as is suggested by the teacher who said ‘I see students as clients too.’ Similarly, the physicians in Sargeant’s study and the clinical teachers in our study appear to share doubts regarding feedback from colleagues (Sargeant et al. 2007). There were indications that the potential effectiveness of self-assessment and feedback was hampered by limitations in the teaching environment. Time constraints during brief clerkships were perceived as a barrier to introducing new teaching practices and while physicians’ endorsed the intrinsic value of the teaching approach reflected in the self-assessment instrument they also commented that it was not feasible in practice. The latter remark also reflects on the instrument we used, confirming criticism that it is rather specific in nature (Stalmeijer et al. 2008) while the clinical teaching ‘style’ it advocates is not considered viable for shorter clerkships (Stalmeijer et al. 2009). Perhaps additional coaching may be helpful to physicians in implementing an adapted version of the instrument. The fact that many physicians were interested in additional information on clinical teaching and in being coached can be interpreted as a positive outcome of the self-assessment on feedback exercise and as a sign that it has augmented the physicians’ interest in improving their clinical teaching in an effective and professional manner.

Strengths and weaknesses

Feedback effectiveness is a complex issue involving numerous interacting elements and this study addressed only a selection of these elements. However many elements which are known to influence feedback effectiveness, like self-perceptions of performance or beliefs, are difficult to measure because they tap into beliefs and self-efficacy, which are outside the awareness of many people (Pajares 1992). Although the study was conducted within one teaching hospital and the number of participants was small, the physicians we interviewed were purposively sampled to provide an accurate representation of the population. It is a strength of this study that all the instruments were based on research and thus can provide strong guidance for further research (Bowden and Marton 2000). Furthermore, the mixed methods design afforded a deeper understanding of the research topic and the research questions.

Future research

The results of this study will have to be confirmed by studies among a larger and more varied population of physicians in different hospitals. Larger studies may also shed more light on the influence of self-assessment on feedback effectiveness. It also seems worthwhile to study long-term effects of self-assessment and feedback by (1) examining data on repeated feedback and self-assessment over several years and (2) observation studies to examine whether physicians actually incorporate feedback into their teaching practices. Additionally, research should focus on the deeper understanding of internal and external factors that influence feedback effectiveness. Finally, self-assessment is only one way to optimize feedback and we recommend that other methods be considered too, such as coaching, and extending the combination of self-assessment and student feedback into a multi-source feedback process.