Doing More With Written Feedback: Improving Learner Satisfaction and Reflection With the LEAF (Learner-Engaged Analysis of Feedback) Method.

PROBLEM: Written feedback is often overly positive, nonspecific, and difficult to interpret. Learner satisfaction with written feedback is low and obtaining written feedback that encourages self-reflection is challenging. Improving feedback quality is laborious and only modestly effective. APPROACH: The authors developed the LEAF (Learner-Engaged Analysis of Feedback) method to improve learner satisfaction with, and reflection on, existing written feedback. The method pairs a learner and coach to methodically identify themes in the learner's written feedback. Themes occurring more frequently or less frequently than typical offer areas for reflection, as they may identify learners' relative strengths or weaknesses. The method was introduced at the Massachusetts General Hospital in 2017 during program director (PD) meetings with anesthesiology residents. In 2018, resident satisfaction was measured (1 to 5 Likert-type questions, 1 = "not at all satisfied," 5 = "extremely satisfied") for 4 feedback sources, 2 related to the LEAF method (PD meetings, written feedback) and 2 unrelated (verbal feedback, mentor feedback). Residents' comments were qualitatively assessed to explore the impact on self-reflection. OUTCOMES: Residents who had participated in a LEAF session (n = 54), compared with those who had not (n = 11), reported higher satisfaction with written feedback (mean 3.1 versus 2.5, d = 0.53, P = .03) and PD meeting feedback (mean 3.8 versus 2.8, d = 0.80, P = .03). There were no significant differences between groups for satisfaction with feedback unrelated to the LEAF method. Qualitative analysis of comments suggested that residents found the method useful for providing holistic self-assessment, facilitating goal setting, uncovering blind spots, and improving feedback interpretation. NEXT STEPS: Next steps should include studies determining if the association between increased learner satisfaction with written feedback and the LEAF method is causal, and whether this feedback process changes learners' subsequent behaviors.

Problem

Written feedback for learners tends to be nonspecific, overly positive, and targeted at personal (nonbehavioral) traits. [1] Moreover, many who provide feedback avoid explicitly addressing improvement areas for fear of offending learners. [2] Even when constructive comments are provided, their meaning can be ambiguous or misinterpreted, and comments from different feedback givers may be contradictory. Uncertainty about how to synthesize feedback may leave learners poorly equipped to integrate it to inform a holistic self-assessment. [3] Despite recommendations for improving the quality of written feedback, substantial organized efforts have produced only modest effects. [4]

Effective use of written feedback is additionally challenging because it does not inherently promote partnership between feedback providers and learners in a context that fosters vulnerability, honesty, and receptivity. Accordingly, some learners reject negatively perceived comments by discrediting the source, blaming external factors, or otherwise discounting feedback validity. Unsurprisingly, medical trainees have reported substantial dissatisfaction with written feedback, [5] making it unlikely that they will reflect on feedback holistically or use it to identify their growth needs.

To overcome these problems, we developed the Learner-Engaged Analysis of Feedback (LEAF) method at the Massachusetts General Hospital to promote learners’ use of currently available written feedback. Its aims are to allow written feedback from multiple sources to be gathered, interpreted, and shared with learners meaningfully and to facilitate learners’ cognitive and affective processing to integrate and reflect on feedback. We hypothesized that the LEAF method would produce higher levels of trainee satisfaction with written feedback and promote self-reflection.

Approach

Many feedback givers hesitate to write feedback that may be perceived as negative or critical—the so-called mum effect. [6] For learners to correctly interpret feedback, they must understand how the written feedback they receive compares with typical comments; this can mitigate the mum effect by helping them appreciate what is written and what is not written. For example, a trainee with a pleasant personality but poor medical decision making may receive many positive comments about interpersonal skills but relatively few about clinical judgment. If the typical trainee receives numerous positive comments about clinical judgment, a relative absence of such comments (due to the mum effect) may identify an area for improvement for this learner. Conversely, the higher-than-normal frequency of positive comments about this learner’s positive personality may represent a relative strength.

With the LEAF method, a coach and learner identify themes in each written comment (e.g., pleasant personality, good clinical judgment, good technical skills) and track the number of positive and negative comments for each theme. They then compare the learner’s frequencies with those of a typical learner, so the learner can better integrate feedback and identify blind spots.

For these comparisons to be possible, it is essential to summarize typical feedback for the relevant cohort. To accomplish this, the LEAF method employs a stepwise approach mirroring the approach taken by qualitative researchers when identifying themes in narrative (textual) data [7] (Figure 1). Feedback across the cohort of learners is coded to determine the themes and typical frequencies for that cohort. This is done once, before any LEAF sessions, to generate a “typical” learner profile. Subsequently, each learner meets with a coach to code that learner’s feedback and compare their themes with the typical learner’s profile, paying particular attention to themes identified in substantially higher or lower frequency than is typical. Finally, the learner uses this to self-reflect and develop learning goals and action plans. Figure 2 outlines the LEAF process for a hypothetical learner. At our institution, LEAF sessions take 30–60 minutes.

Figure 1

The LEAF method as it parallels the steps of qualitative framework analysis. Abbreviations: LEAF, Learner-Engaged Analysis of Feedback; ACGME, Accreditation Council for Graduate Medical Education.

Figure 2

Using the “coding index” during a LEAF session. Before any LEAF sessions, a coding index is developed. A simplified portion of the coding index is shown (top) that describes the “Practice-based learning and improvement” domain. During a LEAF session (middle), the coach and learner code the learner’s written comments onto the coding index. Themes are highlighted to help summarize how the learner’s comments compare with the comments a typical learner gets. In the last step of the LEAF session (bottom), the learner and coach reflect on the themes to inform a more deliberate self-assessment. Abbreviation: LEAF, Learner-Engaged Analysis of Feedback.

More robust descriptions of each step of the LEAF method are detailed in Supplemental Digital Appendix 1, available at http://links.lww.com/ACADMED/B78.

Connections to constructivism and qualitative framework analysis

The LEAF method approach is informed by constructivist theory. The steps parallel the steps of framework analysis, a well-established qualitative research approach to understanding narrative comments. [7]

Constructivism.

Constructivism is a theoretical concept describing the active processes of learning and making meaning. It proposes that learners take in new information, link it to existing knowledge structures, and construct new and increasingly complex levels of understanding. [8] Constructivism highlights the shared, social aspects of learning as well as the importance of active learning in constructing meaning. In the LEAF method, both the learner and coach examine the narrative comments and co-construct a profile to assist the learner in reflecting on performance. The informational style and active learning features of this process embrace a constructivist approach, allowing for greater impact on learning and subsequent behavior change.

Qualitative framework analysis.

The steps in the framework analysis are outlined in Figure 1. Framework analysis is a powerful qualitative approach to meaningfully synthesizing feedback that offers the following advantages [7]:

Enabling in-depth exploration and understanding of text-based data

Establishing a process wherein multiple investigators (e.g., learner, coach) can develop a shared understanding of themes present in the data

Synthesizing data from multiple perspectives (e.g., feedback from multiple feedback providers)

Systematically guiding investigators to identify their own assumptions and biases related to data interpretation (e.g., defensiveness)

Identifying gaps or expected content areas that may be missing from the data

We developed and began using the LEAF method with anesthesiology residents during their semiannual, one-on-one program director (PD) meetings in 2017. At the time, the program had 74 residents. Written feedback was requested from faculty every week they worked with a resident. PD meetings were scheduled every 6 months, and all written faculty feedback from the preceding 6-month period formed the basis of each LEAF session. With this approach, most residents received comments from between 10 and 25 unique faculty members (see Supplemental Digital Appendix 1, at http://links.lww.com/ACADMED/B78). The PD served as the coach for all LEAF sessions.

Analysis

To assess the LEAF method’s impact on learners, we included relevant questions in our anonymized 2018 annual program survey. At that point, some but not all residents had participated in a LEAF session, which allowed for comparison between groups. Residents who did not have a LEAF session had access to all their written comments, but their most recent PD meeting was before incorporation of the LEAF method. In these non-LEAF sessions, residents were asked to reflect on themes from their written comments they had identified on their own. There were no other known systematic differences between groups.

On the annual program survey, residents rated their satisfaction with 4 different areas of feedback on 1 to 5 Likert-type scales (1 = “not at all satisfied” to 5 = “extremely satisfied”). Two were related to the LEAF method: PD meeting feedback (in which LEAF sessions took place) and written feedback from faculty (the data source used during the LEAF sessions). The other 2 did not relate to the LEAF method: verbal feedback from faculty and feedback from faculty mentors (who have access to written feedback but were not instructed to make specific use of it). Because this study was not randomized, these were included to help determine if differences between groups were related to some general score inflation, instead of the intervention. To minimize response bias, we made no mention of the LEAF method before or within this section of the annual program survey. In a subsequent section, residents were asked to provide free-text comments about the LEAF method.

To assess the impact of the LEAF method on satisfaction, we compared group means on each question using independent t tests. Effect sizes are represented using Cohen’s d (with d = 0.2, 0.5, and 0.8 interpreted as small, medium, and large effects, respectively). [9] To assess the impact on self-reflection, 2 authors (D.S.-K., K.B.) used framework analysis [7] to identify themes in the open-ended survey items.

This study met institutional criteria as a clinical quality improvement project and was exempted from ethical/human subjects review.

Outcomes

Sixty-five of 74 residents responded to the anonymized annual program survey (88% response rate). Of the 65 residents, 54 had participated in a PD meeting with a LEAF session and 11 had a PD meeting and access to their feedback without a LEAF session (comparator group). Trainees who had a LEAF session were more satisfied with written feedback from faculty (mean 3.1 [SD 1.2] versus 2.5 [SD 0.7], d = 0.53, t statistic = 2.34, P = .03) and with PD meeting feedback (mean 3.8 [SD 1.3] versus 2.8 [SD 1.2], d = 0.80, t statistic = 2.35, P = .03) than trainees in the comparator group (Figure 3). Satisfaction with verbal feedback from faculty and satisfaction with faculty mentor feedback (not related to the LEAF method) were not significantly different between groups (mean 3.1 versus 2.7, P = .41 and mean 3.4 versus 3.4, P = .92, respectively).

Figure 3

Learner satisfaction (mean +/− interquartile ranges) for 4 different types of feedback as reported on the program’s annual survey in 2018. Learner satisfaction was measured with 1 to 5 Likert-type questions, 1 = “not at all satisfied” to 5 = “extremely satisfied.” Effect sizes are Cohen’s d (difference in means divided by pooled standard deviation) with small, medium, and large effects taken to be 0.2, 0.5, and 0.8, respectively. Learner satisfaction with both feedback types related to the LEAF method (written feedback and PD meeting feedback) was statistically significantly higher in the group that had a LEAF session than in the group that did not have a LEAF session. Abbreviations: LEAF, Learner-Engaged Analysis of Feedback; PD, program director; NS, nonsignificant.

Forty-one residents wrote comments about the LEAF method in the survey, which are summarized in Supplemental Digital Appendix 2, at http://links.lww.com/ACADMED/B78. Thirty-four comments included positive elements, and 8 included negative or neutral elements (1 resident comment included both positive and neutral elements). Four themes were identified in the positive comments. Residents found the LEAF method useful for getting a “big picture” of their themes (n = 20), setting goals (n = 8), uncovering blind spots (n = 4), and interpreting comments (n = 3). Two themes were identified in the negative or neutral comments. Specifically, residents questioned the validity of the process (n = 7) or commented that it took substantial time (n = 4).

Next Steps

Using the LEAF method was associated with an increase in trainee satisfaction with written feedback. Resident comments suggest that the method can inform holistic understanding of feedback and self-reflection. Importantly, the LEAF method uses written feedback that currently exists and does not require faculty members to change what they write.

Improving the quality of written feedback appears to be a Sisyphean task. [4] The LEAF method takes a different approach by extracting more meaning from feedback that is already available. Furthermore, it encourages self-reflection and may improve a learner’s ability to interpret feedback. Thus, the LEAF method may represent an effective supplemental approach to other feedback efforts. After developing the coding index, the ongoing cost at our institution is approximately 1 hour per learner per year for LEAF sessions.

Despite these promising results and the robust framework upon which the LEAF method is designed, there are limitations to this work. First, the groups were not randomly assigned, so we cannot claim causality. That said, if there were some systematic difference between groups, we would expect a general score inflation, but there were no significant differences in satisfaction with feedback measures unrelated to the LEAF sessions (verbal feedback or faculty mentor feedback). This suggests that the observed differences were related to the LEAF method. Second, though our response rate was high (88%), the anonymized nature of the survey means that we cannot claim that nonresponders were the same as responders. Third, a single coach performed all LEAF sessions. Whether the benefits would extend to other coaches is unknown. Fourth, one of the premises of the LEAF method (that substantially fewer positive comments about a given attribute may suggest an area for improvement) is conceptually sound, but it has not been empirically validated. Though standards for use of qualitative data for evaluating learner performance have been proposed, [10] the LEAF method is not intended to be evaluative. It is intended to inform and encourage learners’ self-reflection. Last, although increasing learner satisfaction with feedback is important given how pervasive low satisfaction with feedback is, future studies should describe outcomes that are arguably more important. Some of these were suggested in our qualitative analysis, such as the ability to interpret feedback, the formation of developmental goals, and behavior change.

Even with these limitations, we found statistically significant differences with our small sample, and we received strong positive narrative feedback and useful critiques to guide ongoing improvement. The LEAF method appears to augment the interpretation and use of written feedback by applying the systematic and rigorous principles of qualitative analysis and grounding it in the well-accepted and tested conceptual model of constructivism. Next steps should include determining whether the association is causal and whether the LEAF method changes learners’ subsequent behaviors.

Acknowledgments:

The authors would like to acknowledge faculty members and residents at the Massachusetts General Hospital for their participation in providing written feedback and for participating in LEAF sessions.