A Pilot Study of a Direct Teaching Observation Tool for Residents.

It is important to empower physicians to teach effectively so they are prepared to provide high quality education to medical trainees. The residency experience is recognized as an opportunity to help future faculty enhance their teaching skills, but there are few formalized tools available to facilitate this. We piloted a new observation tool for faculty to use when observing residents teach. The goal of our study was to determine if our observation tool is a useful and well accepted mode to facilitate feedback from faculty to residents about their teaching. A small number of senior residents were observed using the tool without prior knowledge of the tool components. Our early findings suggest that the tool is well-accepted by faculty and residents and may be useful as a catalyst in discussing a teaching framework. More data are being collected to determine if the tool is useful in improving resident teaching skills over time.

Introduction

Since the late 1990s, residents-as-teacher curricula have become increasingly popular as physicians-in-training have vocalized the desire to receive explicit education on teaching skills [1-4]. In response, a handbook on this topic was published by the American Academy of Pediatrics in 2002 [5]. While most curricula described in the literature include formal didactic sessions, few include validated methods of assessment that can be easily reproduced at other institutions [6]. One important method of assessment is direct observation of teaching skills with feedback. There is moderate evidence suggesting that the observed structured teaching encounter is a reliable and valid format of teaching assessment, although few rating instruments have been adequately studied [7].

In the field of general education, teaching observation tools have been well established [8]. However, within the medical education domain, teaching observation tools are less ubiquitous. Direct observation tools published in the medical education literature include those that evaluate clinical teaching at the bedside or in the ambulatory setting [9,10], teaching as part of a formal Observed Structured Clinical Encounter (OSCE) [11], as well as peer observation of teaching [12]. One such tool was created at Cincinnati Children’s Hospital and designed for faculty observation of brief didactic resident teaching [13]. Their tool evaluates nine teaching skills in three domains (preparation, teaching, and reflection), where each skill is rated on a scale from 1 to 3 for a maximum score of 27 points. This tool was evaluated as part of a resident-as-teacher curriculum during a controlled, prospective pre-/post-educational study directed towards first-year pediatric residents teaching clerkship students on their inpatient hospital rotation. Their curriculum, in conjunction with this tool, showed a significant pre-post difference in one of the key domains of teaching competency with a notable trend for overall competency [4].

We will be describing an early-phase pilot study of a new direct observation tool for faculty to use while observing brief resident teaching sessions presented to learners on the inpatient wards. We feel that our teaching tool is more comprehensive and adaptable to an online platform than those which currently exist.

Methods

Research Goal

The goal of our study was to determine if our observation tool is a useful and well-accepted mode to facilitate feedback from faculty to residents about their teaching.

Research Design

A qualitative study design was implemented to analyze the data collected from direct observation using a teaching observation tool.

Sample

Prior to the study, an email was sent to 52 senior residents in the pediatric residency program at Yale New Haven Hospital offering the opportunity to be observed by a faculty member using a new teaching observation tool. The email described our goal of facilitating feedback between faculty and attending physicians about effective teaching strategies. Four residents responded and gave verbal assent to participate in the study after reviewing a consent form assuring that participation in this pilot study was a voluntary experience. It would neither negatively impact their relationship with the pediatric faculty nor performance evaluation if they chose to participate or not to participate. They were informed that their results would be de-identified prior to data analysis. Three pediatric hospital medicine faculty volunteered as observers after the pilot project description was presented at a section meeting. Faculty received a verbal overview about how to use the tool. Yale Institutional Review Board exemption was sought and granted. This project was approved by the pediatric residency program leadership.

Data Collection

Our 30-item direct teaching observation tool was created from review of the literature by our colleague Dr. Janet Hafler, the Associate Dean for Educational Scholarship at Yale University School of Medicine [14-16]. The tool encompasses six domains of teaching behaviors including establishing the learning environment, promoting learner engagement, managing the learning session, content, verbal/non-verbal interaction, and use of media (see Figure 1). The tool was modified from its original form including a revision in the language and format enhancing clarity and evaluator accessibility.

Figure 1

Observation of teaching form.

A total of four teaching encounters were observed in our cross-sectional pilot study using the same observation tool for each encounter. The teaching was conducted by four different pediatric senior residents and observed by three different pediatric hospital medicine attending physicians. The teaching topics were chosen by the resident teachers and included hyperbilirubinemia, asthma, intravenous fluids, and the newborn exam. Each resident was directly observed by a faculty member during a 15-minute educational talk given to interns and/or medical students in a workroom on an inpatient unit. Faculty observers used a paper version of the tool to record observations of teaching behaviors during the session. Behaviors were recorded as present or not present with qualitative descriptive comments about the behavior. Up to three teaching strengths and areas for improvement were noted by the observer. The observation tool was not shared with the resident teachers until after the observation when it was viewed jointly between the faculty and resident during a 5-minute post-session feedback encounter.

Results

For each of the four teaching encounters, the faculty completed an assessment of the resident’s teaching skills using the observation tool. Upon analysis of the 30 items, 25 items were deemed applicable to all teaching sessions with five items considered applicable only in certain situations (i.e. adapting to unexpected situations, respecting those with divergent opinions, and use of media). Three residents performed 24 out of the 25 applicable items and one resident performed 23 out of the 25 applicable items. Behaviors not performed by all the residents included self-introduction, evidence of preparation, communication of session goals, summarization of key take-away points, and establishment of learner’s level of knowledge.

Faculty comments about resident teaching strengths included “open to questions,” “created a safe learning environment,” “organized drawings on the whiteboard,” and “determined learner’s level of knowledge with questions at the beginning of the session.”

Comments regarding areas for improvement included “emphasize why a teaching topic is important,” “ensure a learning environment where distractions are minimized,” and “end session with 2-3 take-away points.”

Resident feedback about the encounter was positive. One resident remarked that she was interested to learn about the components of the tool as reinforcement of the behaviors she was already doing. Two residents expressed that they had never received formal feedback on their teaching skills before participation in the pilot study and they said that they appreciated the faculty taking the time to provide thoughtful, practical input. One resident felt that the experience opened the door to ask questions about controversial teaching methods, “like cold-calling.” Faculty observers noted that the tool was a useful way to provide organized, constructive feedback, and stimulate discussion about effective teaching strategies.

Discussion

This was a pilot study intended to introduce a new teaching observation tool for faculty members to use while observing brief resident didactic teaching sessions on an inpatient unit. We do not have enough data to comment on generalizability and are working to observe additional teaching sessions. We are in the process of validating the tool using expert reviewers locally and nationally. To improve the consistency between observations, we created a formal instructional video for faculty on how to use the tool.

All resident teachers in our study performed very well on the observations. Future research will incorporate a second teaching session using the observation tool, focusing on how they incorporated the feedback they received from the prior observation. In addition, conducting interviews of both the faculty and resident participants would enable us to collect qualitative and robust data about the impact of this tool.

Key components of successful lectures and presentations have been identified in the literature, including engagement of learners, providing clarity and identifying key points [17]. When evaluating resident teachers using our tool, it may be worthwhile to weigh these items more heavily as to emphasize these important elements that create a high-quality didactic experience for learners.

One of the primary drivers behind our study was the request from our residents for more instruction and feedback on their teaching. Our study shows that residents may be more competent teachers than they realize – and thus our tool may be useful as a way to introduce a formal teaching framework to identify, reinforce, and organize existing skills. However, more observations are needed to determine an accurate baseline level of resident teaching competency.

Our teaching observation tool appears to be a well-accepted, useful mode of feedback facilitation between faculty and residents about effective teaching methods. As medical education is shifting toward more competency-based measurement of outcomes [18], this tool may be a useful component not only for formative feedback but for summative assessment of resident teaching skills prior to graduation. It could also be used to establish teaching competency for medical faculty at academic institutions or for faculty peer observation of teaching [19-20]. We have developed an electronic version of the tool which has made it easier to distribute and access for impromptu teaching sessions using a smart phone or tablet. A saved electronic record of teaching performance as part of a teaching portfolio allows for enhanced reflection by the resident teachers as well as documentation of skill development. In addition, we will continue to work on educator development for the faculty observers.