Does the format residents use to give and receive feedback about teaching affect the usefulness of the feedback?

PURPOSE: An important element in each teaching workshop for resident doctors at the University of Saskatchewan is the microteaching sessions, including feedback. We set out to test our observations that one condition for organizing the feedback increased the quality of feedback. In one condition, residents provide and receive feedback in all areas listed on our feedback form; while in the other condition, they provide and receive feedback in some areas.
METHODS: Over 115 residents participated in the teaching workshop in the 2019-2020 academic year. Each resident experienced both conditions for giving and receiving feedback-about half with one condition first and the other half in the opposite order. We developed and tested a simple survey that asked about the usefulness of the feedback.
RESULTS: We used the Mann-Whitney U test for differences between some areas or all areas. We found a statistically significant difference with small to moderate effect sizes (Cohen's d) favouring the some areas condition.
CONCLUSION: Residents found the usefulness of feedback given or received using the feedback condition in some areas greater than all areas. We will now only use the some areas condition and recommend that other teaching workshops that use microteaching practice sessions consider using this condition.

Introduction

In this paper we report on our study exploring the usefulness of two different formats for observing practice teaching and giving feedback, with all other factors held stable. First, we will describe the context in which this study took place. Every academic year, the College of Medicine offers the Teaching Improvement Project Systems (TIPS) workshop to over one hundred residents at the University of Saskatchewan, Saskatoon, Canada. The purpose of the TIPS workshop is to enhance residents’ basic skills and expertise as teachers of medical students, fellow residents, other healthcare providers, patients, and families. These workshops have been quite successful.[1] All residents attend this two-day workshop, the vast majority of whom participate during their first year of residency. Typically, the two days of the workshop occur one week apart. Both days of the workshop include instruction about teaching and preparation for the practice teaching followed by a short “microteaching” session of approximately 5-10 minutes, during which residents have an opportunity to practice the skills learned earlier in the day with feedback from peers and the facilitator.

The TIPS courses teach the set-body-closure framework, popularly used among instructors to structure teaching sessions in learning environments.[2],[3] Set is the beginning where the resident instructor establishes the mood, motivation, and learning objectives of the session.[3] The Body is the bulk of the teaching session where residents are encouraged to manage an appropriate amount of content while keeping learners engaged with active learning methods.[3] The Closure summarizes key components & learning objectives while giving learners a sense of accomplishment in acquiring new knowledge or skills.[3]

Following this instruction and the preparation for the microteaching sessions, residents are placed in groups of three to five with a faculty or staff facilitator. Each resident teaches the others a short (5-10 minute) lesson (microteaching session). An essential part of each microteaching session is the debriefing that follows. These debriefing sessions are structured around an observation and feedback form divided into five areas: set, body, closure, verbal/non-verbal skills, and instructional media. During or after the micro-teaching session, residents make notes on the feedback form about their colleague’s teaching. They may then use their notes to give oral and written feedback to the resident who just taught. Sometimes the resident colleagues are directed to observe and comment on the feedback form on some areas of teaching (one or two components out of the five on the form) and sometimes on all areas of teaching (all five components on the form).

Initial observations by two of the authors (SP is the workshop leader and MD an occasional facilitator) seemed to indicate that some areas approach to feedback provided better feedback. This made us curious and motivated to test that hypothesis. First, we looked for existing research.

Literature on medical education feedback is broad, describing many new or reformed approaches to feedback. Common models and methods include the sandwich method,[4] ask-tell-ask,[5] Pendleton method,[6] the 5-step micro-skills model (or one-minute preceptor method),[7] the dialogic feedback model,[8] the teacher feedback observation scheme,[9] the conversational framework,[10] and the peer-review feedback method.[11] Other less common strategies include self-assessment, review visit, graded profile, overall summative judgment, subject review report, and the effect of reflection-in-action.[12]-[15] Most authors on feedback in clinical settings argue that feedback should be provided bidirectionally in a SMART fashion (specific, measurable, attainable, relevant and time-bound) to maximize learning goals.[1],[7],[8],[10],[11],[16]-[20] Some suggest that non-evaluative feedback based on directly observed conditions provides the best opportunity for the learner to improve.[17],[21] Despite this seeming abundance of advice for feedback in learning environments, we could not answer our curiosity-driven question. The primary aim of this study, therefore, was to investigate which of the two feedback conditions (using all areas compared with using some areas on the feedback form) provided a better structure for delivering useful feedback during the TIPS two-day microteaching sessions for residents.

Methods

Study participants

We invited all residents who participated in the two-day workshop session during the 2019-2020 academic year to participate in this study.

Study design

This study applied a cross-sectional design. If on the first day of the TIPS course, residents provided feedback on all areas on the TIPS feedback form (i.e., observing and commenting on all five areas of set, body, closure, verbal/non-verbal skills, and instructional media) then on Day 2 the same group provided feedback on some areas of the feedback form (i.e. observing and commenting on only one or two areas). If on the first day of the TIPS course, residents provided feedback on some areas, then on Day 2 they gave feedback on all areas. All residents had therefore given and received feedback for both all areas and some areas feedback conditions. To be clear, the resident teacher received feedback on all five areas of their teaching under each condition. The difference is that those providing feedback had different instructions; either they would give feedback on all of the areas or each would give feedback on one or two of the five areas. Even under some areas condition, the resident teacher received feedback on all five areas from their peers collectively, with each peer contributing feedback in one or two areas only.

We used a locally developed questionnaire to collect data on the quality of feedback given by peers for each day of the 2-day TIPS workshop. This questionnaire was different from the TIPS observation and feedback form on which residents and facilitators provided feedback. Our questionnaire asked the participants to rate the quality of feedback comparing the conditions (some or all areas) for feedback in two different but related situations: feedback participants gave to peers (other residents who taught in the microteaching sessions), and feedback provided by all participants to those who taught. See Table 1.

Table 1

Data collection protocol

Two-Day TIPS Course for Residents: Data Collection
Day 1		Day 2
Large group	Small group	Large group	Small group	Data collection
instruction and preparation for microteaching	microteaching then feedback from peers using either condition:all areas or some areas	instruction and preparation for microteaching	microteaching then feedback from peers using some areas if all areas Day 1or all areas if some areas Day 1	For each condition (all areas and some areas), residents rated the quality of the feedback 1) they gave to their peers and 2) provided to them and all other teachers

For each condition (some areas or all areas), the residents rated the feedback using a 5-point Likert scale gradient (1 = Not useful, 5 = Extremely useful). We also left open-ended spaces for comments by participants for each question asked. Since the qualities of effective feedback are usually discussed during the TIPS course,[1] we imagined that the residents would be able to use that knowledge to adequately provide feedback to their peers as well as evaluate the overall feedback received as teachers in their respective groups. We pilot tested the questionnaire for readability and overall quality with residents who participated in the TIPS workshop the preceding academic year (from July 2018 to February 2019). We then made minor revisions to the questionnaire based on residents’ feedback before embarking on our study in July 2019. For our study, we gave the questionnaire to all participating residents the end of Day 2 after they had completed microteaching sessions under both conditions. Approximately half of the residents used the all areas approach on Day 1, followed by some areas approach on Day 2, while the other half used some areas approach first (see Tables 1 and 2).On Day 2, we distributed our survey to only those residents who had started Day 1 in that particular TIPS course to avoid the situation where participants completing our questionnaire may not have experienced both conditions for giving and receiving feedback.

Table 2

Group assignments to feedback condition for Day 1 and Day 2

Feedback condition	Day 1	Day 2
all areas	66 (53.7%)	50 (43.5%)
some areas	57 (46.3%)	65 (56.5%)
Total	123 (100%)	115 (100%)

Data analyses

Data collected were collated and entered in Microsoft Office Excel[22] and later exported to the Statistical Package for the Social Sciences (SPSS) version 25 for analysis.[23] We re-coded and categorized rated scores from the 5-point Likert scale into three groups. Scores 1 or 2 were recoded as low useful, score 3 as moderate useful, and scores 4 or 5 recoded as high useful. Because our data were ordinal, we used the Mann-Whitney U non-parametric test to investigate for differences between and effect sizes (Cohen’s d) for feedback conditions (some areas or all areas). A p-value of < 0.05 was set for the level of significance for the statistical test.

Ethical considerations

The University of Saskatchewan Behavioural Research Ethics Board determined this study to be exempt per Article 2.5 of the Tri-Council Policy Statement (TCPS2).[24] This study commenced on July 14, 2019, when the TIPS workshop began for the academic year 2019-2020 and continued until January 24, 2020.

Results

While 123 residents attended the first day of the TIPS courses, 115 completed the questionnaire for us on Day 2. Of a potential participant pool of 123, we ended up with 115 valid questionnaires, a completion rate of over 93%. Table 2 displays the descriptive results of feedback conditions (some areas or all areas) for the participants for both Day 1 and Day 2.

The Mann-Whitney U test for differences between feedback conditions (some areas or all areas) showed statistical significance for responses to feedback received and given to peers (p < 0.0001). Effect sizes (Cohen’s d) were small with values, 0.37 and 0.26 for feedback to peers and the entire group’s feedback to all teachers, respectively (see Table 3).

Table 3

Test results for feedback usefulness between conditions

Survey question	Survey responses	Feedback Conditions Frequency (%)		Mann Whitney U	Cohen’s d
		Some areas	All areas
Your feedback to your peers	Low Useful	1(0.9)	12 (10.4)	4301.5*	0.37
	Moderate Useful	15 (13.0)	43 (37.4)
	High Useful	99 (86.1)	60 (52.2)
Entire group’s feedback to all teachers	Low Useful	3 (2.6)	10 (8.7)	5030.5*	0.26
	Moderate Useful	17 (14.8)	37 (32.2)
	High Useful	95 (82.6)	59.1)

p-value significant at 0.05

Discussion

The initial observations of two of the TIPS facilitators (MD, SP) indicated that the condition using some areas seemed to provide more specific, robust feedback compared to all areas. Our hypothesis, therefore, was that structuring the feedback using some areas would be rated higher for usefulness by the residents. Our results supported that residents found greater usefulness of feedback given or received for some areas compared to feedback given or received for all areas. Comments by residents given on the surveys indicated a preference for providing feedback using some areas as they were more ‘focussed,’ ‘concentrative,’ and ‘less distracting’ than employing an all areas approach. These comments also align with Renting et al.,[25] Jug et al.[17] and de la Cruz et al.,[18] on the implications of using succinct and focussed feedback approaches.

There are a few weaknesses of this study that must be considered. This study was completed with only one year of residents in the College of Medicine at the University of Saskatchewan, and, as such, it would be interesting to see if these results would be replicated in other schools with different groups of learners. The definition of useful was not explicitly stated on the survey form. One of the topics during the TIPS workshops is feedback, where the qualities of effective feedback–specific, helpful, focused, behavioural, constructive–are discussed. While we are confident that residents would use these qualities to determine the usefulness of the feedback within the microteaching sessions, we cannot be certain. Furthermore, in our survey form, we attempted to be clear that residents should focus on the quality of the feedback that they (and their peers) gave during the debriefing sessions. Still, there is a possibility that they responded with their preference of conditions instead. Finally, the design of our study did not allow us to analyze the experience of the receiver of the feedback to compare their experience under the two conditions.

Given the very high response rate in our study and the clear differences between the reported usefulness of feedback in the two conditions we tested, we will use the some areas condition and believe that other similar train-the-teacher workshops should also seriously consider using and experimenting with this condition. Our study did not investigate the reasons why some areas condition seemed to result in more useful feedback. We speculate that directing those who give feedback to focus on a specific area of skills allows, or even encourages, them to give more substantive and detailed feedback in that area. Conversely, instructing them to provide feedback on a wider array of skills might force more superficial and even less specific feedback. When the person who is observing can be more focused in gathering the information needed to deliver useful feedback, and there are several observers as there are in the TIPS course, the person receiving the feedback gets concentrated feedback on one or two areas (some areas) from each observer and collectively receives feedback on all areas from the many contributors taken together. Further research could attempt to replicate our findings or test the two conditions in other training programs in different settings, such as communication or leadership skills development. Additional studies may also want to explore why some areas approach to feedback was rated higher for usefulness.